VictoriaMetrics Stack 을 활용한 모니터링 구축

이전 포스팅에서 논의 했던 VictoriaMetrics Stack 을 활용한 모니터링 구축을 진행할 것이다.

 

쿠버네티스 환경이며, 모든 것은 쿠버네티스 자원을 통해 생성 될 것이다.

 

구축 순서는 아래와 같다.

 

1. NS 생성

2. VictoriaMetrics Operator 생성

3. VMSingle 생성(Metrics 저장소)

4. VictoriaLogs 생성(Log 저장소)

5. Jeager 생성(Trace 저장소(올인원))

6. VMAgent 생성(스크래핑 + 집계)

7. OTEL Collector(노드 로그 수집)

8. OTEL Collector(게이트웨이)

9. Grafana

 

우선 모든것이 설치될 네임스페이스 먼저 생성핸다.

kubectl create ns monitoring

 

그리고 repo 를 추가한다.

helm repo add vm https://victoriametrics.github.io/helm-charts
helm repo add jaegertracing https://jaegertracing.github.io/helm-charts
helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm repo add grafana https://grafana.github.io/helm-charts
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

 

그리고 VictoriaMetrics Operator values 파일을 작성한다.

# vm-operator-values.yaml
operator:
  # Prometheus CRD(ServiceMonitor 등) 자동 변환 활성화
  enable_converter_ownership: true
  # 감시할 네임스페이스 (비워두면 전체)
  watchNamespaces: []

resources:
  limits:
    cpu: 200m
    memory: 256Mi
  requests:
    cpu: 50m
    memory: 128Mi

helm install vm-operator vm/victoria-metrics-operator \
  -n monitoring \
  -f vm-operator-values.yaml

 

설치확인

kubectl get pods -n monitoring
kubectl get crds | grep victoriametrics

 

이어서 VMSingle 설치를 진행한다.

# vmsingle-values.yaml
server:
  # 데이터 보관 기간 (월 단위)
  retentionPeriod: 3

  persistentVolume:
    enabled: true
    size: 20Gi
    # storageClassName: "" # 필요시 지정

  resources:
    limits:
      cpu: 500m
      memory: 512Mi
    requests:
      cpu: 100m
      memory: 256Mi

  extraArgs:
    opentelemetry.usePrometheusNaming: "true"

helm install vmsingle vm/victoria-metrics-single \
  -n monitoring \
  -f vmsingle-values.yaml

kubectl get pods -n monitoring
kubectl get pvc -n monitoring

 

VictoriaLogs도 설치한다.

# vmlogs-values.yaml
server:
  # 데이터 보관 기간
  retentionPeriod: 30d

  # 스토리지
  persistentVolume:
    enabled: true
    size: 20Gi
    # storageClassName: "" # 필요시 지정
    
  resources:
    limits:
      cpu: 300m
      memory: 256Mi
    requests:
      cpu: 50m
      memory: 128Mi

helm install vmlogs vm/victoria-logs-single \
  -n monitoring \
  -f vmlogs-values.yaml

kubectl get pods -n monitoring
kubectl get pvc -n monitoring

 

다음으로 jeager를 설치한다.

# jaeger-values.yaml
allInOne:
  enabled: true

  resources:
    limits:
      cpu: 300m
      memory: 256Mi
    requests:
      cpu: 50m
      memory: 128Mi

  # Badger 영구 저장소 설정
  extraEnv:
    - name: BADGER_EPHEMERAL
      value: "false"
    - name: BADGER_DIRECTORY_VALUE
      value: "/badger/data"
    - name: BADGER_DIRECTORY_KEY
      value: "/badger/key"

  persistence:
    enabled: true
    size: 10Gi
    
storage:
  type: badger

collector:
  enabled: false
query:
  enabled: false
agent:
  enabled: false

helm install jaeger jaegertracing/jaeger \
  -n monitoring \
  -f jaeger-values.yaml

kubectl get pods -n monitoring

 

다음으로 vmagent를 생성한다.

helm install node-exporter prometheus-community/prometheus-node-exporter \
  -n monitoring

# vmagent-values.yaml

# VictoriaMetrics로 전송
remoteWrite:
  - url: http://vmsingle-victoria-metrics-single-server.monitoring.svc.cluster.local:8428/api/v1/write

resources:
  limits:
    cpu: 300m
    memory: 256Mi
  requests:
    cpu: 50m
    memory: 128Mi

# Kubernetes 인프라 스크래핑
config:
  global:
    scrape_interval: 30s

  scrape_configs:
    # 노드 메트릭 (kubelet)
    - job_name: kubernetes-nodes
      scheme: https
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        insecure_skip_verify: true
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      kubernetes_sd_configs:
        - role: node
      relabel_configs:
        - action: labelmap
          regex: __meta_kubernetes_node_label_(.+)

    # 파드 메트릭
    - job_name: kubernetes-pods
      kubernetes_sd_configs:
        - role: pod
      relabel_configs:
        - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
          action: keep
          regex: "true"
        - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
          action: replace
          target_label: __metrics_path__
          regex: (.+)
          
    - job_name: kubernetes-nodes-cadvisor
      scheme: https
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        insecure_skip_verify: true
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      kubernetes_sd_configs:
        - role: node
      relabel_configs:
        - action: labelmap
          regex: __meta_kubernetes_node_label_(.+)
        - target_label: __metrics_path__
          replacement: /metrics/cadvisor

    - job_name: node-exporter
      kubernetes_sd_configs:
        - role: endpoints
      relabel_configs:
        - source_labels: [__meta_kubernetes_service_label_app_kubernetes_io_name]
          regex: prometheus-node-exporter
          action: keep
        - source_labels: [__meta_kubernetes_endpoint_node_name]
          target_label: node

    # kube-state-metrics
    - job_name: kube-state-metrics
      static_configs:
        - targets:
          - kube-state-metrics.monitoring.svc.cluster.local:8080

helm install vmagent vm/victoria-metrics-agent \
  -n monitoring \
  -f vmagent-values.yaml

kubectl get pods -n monitoring

 

 

다음으로 OTEL Collector를 설치한다.

# otel-daemonset-values.yaml

image:
  repository: otel/opentelemetry-collector-contrib
 
mode: daemonset

config:
  receivers:
    filelog:
      include:
        - /var/log/pods/*/*/*.log
      include_file_path: true
      operators:
        - type: json_parser
          id: parser-docker
        - type: move
          from: attributes["log.file.path"]
          to: resource["log.file.path"]

  processors:
    batch: {}
    memory_limiter:
      limit_mib: 256
    k8sattributes:
      auth_type: serviceAccount
      extract:
        metadata:
          - k8s.pod.name
          - k8s.namespace.name
          - k8s.node.name
          - k8s.deployment.name

  exporters:
    otlp/gateway:
      endpoint: otel-gw-opentelemetry-collector.monitoring.svc.cluster.local:4317
      tls:
        insecure: true

  service:
    pipelines:
      logs:
        receivers: [filelog]
        processors: [memory_limiter, k8sattributes, batch]
        exporters: [otlp/gateway]

resources:
  limits:
    cpu: 200m
    memory: 256Mi
  requests:
    cpu: 50m
    memory: 128Mi

helm install otel-ds open-telemetry/opentelemetry-collector \
  -n monitoring \
  -f otel-daemonset-values.yaml

 

 

# otel-gateway-values.yaml

image:
  repository: otel/opentelemetry-collector-contrib

mode: deployment
replicaCount: 1

config:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 0.0.0.0:4317
        http:
          endpoint: 0.0.0.0:4318

  processors:
    batch: {}
    memory_limiter:
      limit_mib: 512

  exporters:
    otlphttp/vmagent:
      endpoint: http://vmagent-victoria-metrics-agent.monitoring.svc.cluster.local:8429/opentelemetry/api/v1/push
      tls:
        insecure: true

    # Logs → VictoriaLogs
    otlphttp/vmlogs:
      endpoint: http://vmlogs-victoria-logs-single-server.monitoring.svc.cluster.local:9428/insert/opentelemetry
      tls:
        insecure: true

    # Traces → Jaeger
    otlp/jaeger:
      endpoint: jaeger.monitoring.svc.cluster.local:4317
      tls:
        insecure: true

  service:
    pipelines:
      metrics:
        receivers: [otlp]
        processors: [memory_limiter, batch]
        exporters: [otlphttp/vmagent]
      logs:
        receivers: [otlp]
        processors: [memory_limiter, batch]
        exporters: [otlphttp/vmlogs]
      traces:
        receivers: [otlp]
        processors: [memory_limiter, batch]
        exporters: [otlp/jaeger]

resources:
  limits:
    cpu: 300m
    memory: 512Mi
  requests:
    cpu: 100m
    memory: 256Mi

helm install otel-gw open-telemetry/opentelemetry-collector \
  -n monitoring \
  -f otel-gateway-values.yaml

kubectl get pods -n monitoring
kubectl get daemonset -n monitoring

마지막으로 Grafana를 설치해준다.

UI 접근을 위해 어떠한 방식으로 할 것인지 결정해야한다.

Service타입을 LoadBalancer로 할 것인지, 별도의 ingress를 통해 접근할 것인지 등.

 

본 포스팅에서는 ingress로 접근하겠다.

VPN 환경이기에 별도의 annotaion을 통해 private으로 변경해줬다.

# grafana-values.yaml
adminPassword: "패스워드 입력"

persistence:
  enabled: true
  size: 5Gi

# VictoriaLogs 플러그인 설치
plugins:
  - victoriametrics-logs-datasource

service:
  type: NodePort

ingress:
  enabled: true
  annotations:
    alb.ingress.kubernetes.io/network-type: "private"
  ingressClassName: alb
  hosts:
    - grafana.yourdomain.com

# Datasource 자동 설정
datasources:
  datasources.yaml:
    apiVersion: 1
    datasources:
      # Metrics - VictoriaMetrics (Prometheus 호환)
      - name: VictoriaMetrics
        type: prometheus
        url: http://vmsingle-victoria-metrics-single-server.monitoring.svc.cluster.local:8428
        access: proxy
        isDefault: true

      # Logs - VictoriaLogs
      - name: VictoriaLogs
        type: victoriametrics-logs-datasource
        url: http://vmlogs-victoria-logs-single-server.monitoring.svc.cluster.local:9428
        access: proxy

      # Traces - Jaeger
      - name: Jaeger
        type: jaeger
        url: http://jaeger.monitoring.svc.cluster.local:16686
        access: proxy

resources:
  limits:
    cpu: 300m
    memory: 256Mi
  requests:
    cpu: 100m
    memory: 128Mi

helm install grafana grafana/grafana \
  -n monitoring \
  -f grafana-values.yaml

kubectl get pods -n monitoring
kubectl get service -n monitoring

 

 

LB가 생성됬다면, 위의 grafana yaml파일에서 hosts를 수정해서 upgrade 한다.

helm upgrade grafana grafana/grafana -n monitoring -f grafana-values.yaml

 

업그레이드 후, 
접근하여 admin/지정한 패스워드 입력시 로그인하면 접속이 되는데,

 

좌측 네비게이션에서 Connections -> Data sources 에 들어가면, Grafana yaml에서 설정했던 Datasource가 이미 들어가 있다.

 

이제 각 컴포넌트를 클릭해서 밑에 Test를 눌러 3개다 성공했다면 설치는 끝이다.