OpenMetrics로 Trino 메트릭 수집

OpenMetrics로 Trino 메트릭 수집 (Trino metrics with OpenMetrics)

Trino는 오픈소스 시스템 모니터링·알림 도구 키트인 Prometheus에서 유래한 메트릭 표준 OpenMetrics를 지원해요.

출처: 문서

본문

메트릭은 자동으로 활성화되며 코디네이터의 /metrics 엔드포인트에서 사용할 수 있어요. 이 엔드포인트는 Web UI와 클라이언트 프로토콜과 동일하게, 설정된 인증으로 보호돼요.

예를 들어 localhost:8080에서 실행 중인 보안이 설정되지 않은 Trino 서버에서 사용자 이름 foo로 메트릭 데이터를 가져올 수 있어요:

curl -H X-Trino-User:foo localhost:8080/metrics

반환되는 메트릭은 다음과 같은 OpenMetrics 형식이에요:

# TYPE io_airlift_http_client_type_HttpClient_name_ForDiscoveryClient_CurrentResponseProcessTime_Min gauge
io_airlift_http_client_type_HttpClient_name_ForDiscoveryClient_CurrentResponseProcessTime_Min NaN
# TYPE io_airlift_http_client_type_HttpClient_name_ForDiscoveryClient_CurrentResponseProcessTime_P25 gauge
io_airlift_http_client_type_HttpClient_name_ForDiscoveryClient_CurrentResponseProcessTime_P25 NaN
# TYPE io_airlift_http_client_type_HttpClient_name_ForDiscoveryClient_CurrentResponseProcessTime_Total gauge
io_airlift_http_client_type_HttpClient_name_ForDiscoveryClient_CurrentResponseProcessTime_Total 0.0
# TYPE io_airlift_http_client_type_HttpClient_name_ForDiscoveryClient_CurrentResponseProcessTime_P90 gauge
io_airlift_http_client_type_HttpClient_name_ForDiscoveryClient_CurrentResponseProcessTime_P90 NaN

Prometheus로 수집하기

도커 네트워크를 만들고 Trino 컨테이너를 실행해 볼게요:

docker network create platform

docker run -d \
  --name=trino \
  --network=platform \
  --network-alias=trino \
  -p 8080:8080 \
  trinodb/trino:latest

prometheus.yml에 스크레이프 설정을 추가해요:

scrape_configs:
- job_name: trino
  basic_auth:
    username: trino-user
  static_configs:
    - targets:
      - trino:8080

Prometheus 컨테이너를 실행해요:

docker run -d \
  --name=prometheus \
  --network=platform \
  -p 9090:9090 \
  --mount type=bind,source=$PWD/prometheus.yml,target=/etc/prometheus/prometheus.yml \
  prom/prometheus

정리할 때는 컨테이너를 순서대로 멈추고 제거하면 돼요:

docker stop prometheus
docker stop trino

docker start trino
docker start prometheus

docker rm trino
docker rm prometheus
docker network rm platform

Kubernetes에서 수집하기

Kubernetes에서 실행 중이라면 Trino 파드에 주석(annotation)을 달아 스크레이프 대상을 표시할 수 있어요:

coordinator:
  annotations:
    prometheus.io/trino_scrape: "true"
worker:
  annotations:
    prometheus.io/trino_scrape: "true"

워커를 스크레이프하는 Prometheus 설정의 예시는 다음과 같아요:

    - job_name: trino-metrics-worker
      scrape_interval: 10s
      scrape_timeout: 10s
      kubernetes_sd_configs:
        - role: pod
      relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_trino_scrape]
        action: keep # scrape only pods with the trino scrape anotation
        regex: true
      - source_labels: [__meta_kubernetes_pod_container_name]
        action: keep # dont try to scrape non trino container
        regex: trino-worker
      - action: hashmod
        modulus: $(SHARDS)
        source_labels:
        - __address__
        target_label: __tmp_hash
      - action: keep
        regex: $(SHARD)
        source_labels:
        - __tmp_hash
      - source_labels: [__meta_kubernetes_pod_name]
        action: replace
        target_label: pod
      - source_labels: [__meta_kubernetes_pod_container_name]
        action: replace
        target_label: container
      metric_relabel_configs:
          - source_labels: [__name__]
            regex: ".+_FifteenMinute.+|.+_FiveMinute.+|.+IterativeOptimizer.+|.*io_airlift_http_client_type_HttpClient.+"
            action: drop # droping some highly granular metrics
          - source_labels: [__meta_kubernetes_pod_name]
            regex: ".+"
            target_label: pod
            action: replace
          - source_labels: [__meta_kubernetes_pod_container_name]
            regex: ".+"
            target_label: container
            action: replace

      scheme: http
      tls_config:
        insecure_skip_verify: true
      basic_auth:
        username: myuser # replace with a username that has system information permission
        # DO NOT ADD PASSWORD

코디네이터를 스크레이프할 때는 HTTPS 인그레스 주소로 대상 주소를 바꿔줄 수 있어요:

    - job_name: trino-metrics-coordinator
      scrape_interval: 10s
      scrape_timeout: 10s
      kubernetes_sd_configs:
        - role: pod
      relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_trino_scrape]
        action: keep # scrape only pods with the trino scrape anotation
        regex: true
      - source_labels: [__meta_kubernetes_pod_container_name]
        action: keep # dont try to scrape non trino container
        regex: trino-coordinator
      - action: hashmod
        modulus: $(SHARDS)
        source_labels:
        - __address__
        target_label: __tmp_hash
      - action: keep
        regex: $(SHARD)
        source_labels:
        - __tmp_hash
      - source_labels: [__meta_kubernetes_pod_name]
        action: replace
        target_label: pod
      - source_labels: [__meta_kubernetes_pod_container_name]
        action: replace
        target_label: container
      - action: replace  # override the address to the https ingress address
        target_label: __address__
        replacement: {{ .Values.trinourl }}
      metric_relabel_configs:
          - source_labels: [__name__]
            regex: ".+_FifteenMinute.+|.+_FiveMinute.+|.+IterativeOptimizer.+|.*io_airlift_http_client_type_HttpClient.+"
            action: drop # droping some highly granular metrics
          - source_labels: [__meta_kubernetes_pod_name]
            regex: ".+"
            target_label: pod
            action: replace
          - source_labels: [__meta_kubernetes_pod_container_name]
            regex: ".+"
            target_label: container
            action: replace

      scheme: https
      tls_config:
        insecure_skip_verify: true
      basic_auth:
        username: myuser # replace with a username that has system information permission
        password_file: /some/password/file

더 알아보기 (Learn more)

JMX 기반 모니터링이 궁금하다면 JMX로 모니터링 문서를 이어서 읽어 보세요.