Log API Traffic

API7 Enterprise Edition supports collecting route access information and recording it as logs, such as host, client IP, and request timestamp. This key information will be of great help in troubleshooting related problems. A flexible plugin extension system and a number of existing plugins for logging are provided. For example:

• Push to HTTP/TCP/UDP Logging Server
• SkyWalking
• Kafka
• RocketMQ
• Clickhouse
• Syslog
• Aliyun SLS
• Google Cloud Logging Service
• Splunk HTTP Event Collector (HEC)
• Specific file on the disk.
• Elasticsearch
• TencentCloud CLS
• Grafana Loki

Prerequisite(s)

1. Obtain a User Account with Super Admin or API Provider Role.
2. Complete Add Service from API Definition.
3. Get the host of the your own ClickHouse database.
4. Install Docker.

Configure ClickHouse

Start a ClickHouse instance named quickstart-clickhouse-server with a default database quickstart_db, a default user quickstart-user and password quickstart-pass in Docker:

docker network create apisix-quickstart-net

docker run -d \
  --name quickstart-clickhouse-server \
  --network=apisix-quickstart-net \
  -e CLICKHOUSE_DB=quickstart_db \
  -e CLICKHOUSE_USER=quickstart-user \
  -e CLICKHOUSE_PASSWORD=quickstart-pass \
  -e CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT=1 \
  --ulimit nofile=262144:262144 \
  clickhouse/clickhouse-server

Connect to the ClickHouse instance using the command line tool clickhouse-client in Docker:

docker exec -it quickstart-clickhouse-server clickhouse-client

Create a table test in database quickstart_db with fields host, client_ip, route_id, @timestamp of String type, or adjust the command accordingly based on your needs:

CREATE TABLE quickstart_db.test (
  `host` String,
  `client_ip` String,
  `route_id` String,
  `@timestamp` String,
   PRIMARY KEY(`@timestamp`)
) ENGINE = MergeTree()

If successful, you should see Ok on the output.

Enter exit to exit from the command line interface in Docker.

Configure Logging for All Services

For optimal monitoring and tracking, Enabling the logging plugin as a global rule is strongly recommended to ensure that all services and routes are consistently tracked.

1. Select Plugins > Plugin Global Rules, then click Add Plugin Global Rules.
2. Choose clickhouse-logger as the plugin.
3. Apply the following configuration to the JSON Editor.

{
    "log_format": {
        "host": "$host",
        "@timestamp": "$time_iso8601",
        "client_ip": "$remote_addr"
    },
    "user": "quickstart-user",
    "password": "quickstart-pass",
    "database": "quickstart_db",
    "logtable": "test",
    "endpoint_addrs": ["http://quickstart-clickhouse-server:8123"]
}

4. Click Add.

Validate

Send a request to the route to generate an access log entry:

curl -i "http://127.0.0.1:9080/pet/1"

Connect to the ClickHouse instance using the command line tool clickhouse-client in Docker:

docker exec -it quickstart-clickhouse-server clickhouse-client

Query all records in table quickstart_db.test:

SELECT * from quickstart_db.test

You should see an access record similar to the following, which verifies clickhouse-logger plugin works as intended.

┌─host───────────┬─client_ip───┬─route_id─────────────────────────────┬─@timestamp────────────────┐
│ 127.0.0.1      │ 172.19.0.15 │ 6433300c-311b-4047-800e-579244d42aa7 │ 2023-09-01T09:24:16+00:00 │
└────────────────┴─────────────┴──────────────────────────────────────┴───────────────────────────┘