Log API Traffic

API7 Enterprise supports recording route access logs with information such as the host, client IP address, and request timestamp. These logs can help identify and troubleshoot issues.

API Enterprise integrates with many logging platforms through its plugin system. The following integrations are available out of the box:

• HTTP/TCP/UDP Logging Server
• SkyWalking
• Kafka
• RocketMQ
• ClickHouse
• Syslog
• Aliyun SLS
• Google Cloud Logging Service
• Splunk HTTP Event Collector (HEC)
• File logging
• Elasticsearch
• TencentCloud CLS
• Grafana Loki

This tutorial guides you through configuring the clickhouse-logger plugin to log API traffic to a ClickHouse database.

Prerequisites

1. Publish a service.
2. Get the host of your ClickHouse database.

Configure ClickHouse

1. Start a ClickHouse instance named quickstart-clickhouse-server with a default database quickstart_db, a default user quickstart-user, and password quickstart-pass in Docker:

   docker network create apisix-quickstart-net
   
   docker run -d \
     --name quickstart-clickhouse-server \
     --network=apisix-quickstart-net \
     -e CLICKHOUSE_DB=quickstart_db \
     -e CLICKHOUSE_USER=quickstart-user \
     -e CLICKHOUSE_PASSWORD=quickstart-pass \
     -e CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT=1 \
     --ulimit nofile=262144:262144 \
     clickhouse/clickhouse-server

2. Connect to the ClickHouse instance using the command line tool clickhouse-client in Docker:

   docker exec -it quickstart-clickhouse-server clickhouse-client

3. Create a table test in database quickstart_db with fields host, client_ip, route_id, and @timestamp of String type, or according to your needs:

   CREATE TABLE quickstart_db.test (
     `host` String,
     `client_ip` String,
     `route_id` String,
     `@timestamp` String,
     PRIMARY KEY(`@timestamp`)
   ) ENGINE = MergeTree()

   If successful, you should see Ok on the output.

4. Enter exit to exit from the Docker container.

Configure Logging for All Services

Enabling the logging plugins in a global rule is strongly recommended to ensure all traffic to all services and routes is consistently tracked.

Dashboard

1. Select Gateway Groups from the side navigation bar and then select the gateway group to enable logging.

2. Select Plugin Settings from the side navigation bar.

3. Select the Plugin Global Rules tab and then search for the clickhouse-logger plugin in the Plugins field.

4. Click the Plus icon (+).

5. In the dialog box that appeared, apply the following configuration to the JSON Editor:

   {
     "log_format": {
       "host": "$host",
       "@timestamp": "$time_iso8601",
       "client_ip": "$remote_addr"
     },
     "user": "quickstart-user",
     "password": "quickstart-pass",
     "database": "quickstart_db",
     "logtable": "test",
     "endpoint_addrs": ["http://quickstart-clickhouse-server:8123"]
   }

6. Click Enable.

ADC

To use ADC to enable logging, create the following configuration:

adc.yaml

global_rules:
  clickhouse-logger:
    _meta:
      disable: false
    database: quickstart_db
    endpoint_addrs:
      - http://quickstart-clickhouse-server:8123
    log_format:
      "@timestamp": $time_iso8601
      client_ip: $remote_addr
      host: $host
    logtable: test
    password: quickstart-pass
    user: quickstart-user

Synchronize the configuration to API7 Enterprise:

adc sync -f adc.yaml

note

ADC uses the configuration files as the single source of truth. Make sure to pass all configuration files to the adc sync command using the -f flag.

Validate

1. Send a request to the route to generate an access log entry:

   curl -i "http://127.0.0.1:9080/ip"  

2. Connect to the ClickHouse database using the command line tool clickhouse-client in Docker:

   docker exec -it quickstart-clickhouse-server clickhouse-client

3. Query all records in table quickstart_db.test:

   SELECT * from quickstart_db.test

   You should see an access log entry similar to the following:

   ┌─host───────────┬─client_ip───┬─route_id─────────────────────────────┬─@timestamp────────────────┐
   │ 127.0.0.1      │ 172.19.0.15 │ 6433300c-311b-4047-800e-579244d42aa7 │ 2023-09-01T09:24:16+00:00 │
   └────────────────┴─────────────┴──────────────────────────────────────┴───────────────────────────┘