Kubernetes의 Log System 를 알아보자

kubectl logs <pod-name>을 했을 때는 어떤 파일을 보고 로그들을 출력해주는 걸까라는 의문이 들었다. 

 

그래서 https://kubernetes.io/docs/concepts/cluster-administration/logging/ 를 보고... 로컬환경에서 KIND를 통해서 찾아보았다. 

 

1. docker exec <KIND컨테이너명> 

2. cd /var/log 

root@local-cluster-control-plane:/var/log# ls
alternatives.log  containers  pods

이렇게 containers와 pods가 있다. 

root@local-cluster-control-plane:/var/log/containers# ls -l | grep was
lrwxrwxrwx 1 root root  93 Jun  1 06:07 was-5cb779664c-sk8hd_default_httpbin-1cdb3b07d574ed2603c2fc5e7d5b1dd95ffd949c678b2154996831826aceb039.log -> /var/log/pods/default_was-5cb779664c-sk8hd_a1743ef9-7506-4534-816e-b219b6d62042/httpbin/2.log
lrwxrwxrwx 1 root root  93 May 26 10:44 was-5cb779664c-sk8hd_default_httpbin-22f9f08ed76eb13e8ee1a2b25268cbee88ae43e17ae49666916826a3943257ab.log -> /var/log/pods/default_was-5cb779664c-sk8hd_a1743ef9-7506-4534-816e-b219b6d62042/httpbin/1.log

containers에 들어가서 ls -l로 확인해보면 컨테이너 로그파일들이 soft link가 걸려있고 /var/log/pods의 파드가 소프트링크의 목적지 인것을 알 수 있다. 

 

root@local-cluster-control-plane:/var/log/containers# cat was-5cb779664c-sk8hd_default_httpbin-1cdb3b07d574ed2603c2fc5e7d5b1dd95ffd949c678b2154996831826aceb039.log
2025-06-01T06:07:47.76411738Z stderr F [2025-06-01 06:07:47 +0000] [1] [INFO] Starting gunicorn 19.9.0
2025-06-01T06:07:47.766403922Z stderr F [2025-06-01 06:07:47 +0000] [1] [INFO] Listening at: http://0.0.0.0:80 (1)
2025-06-01T06:07:47.766413755Z stderr F [2025-06-01 06:07:47 +0000] [1] [INFO] Using worker: gevent
2025-06-01T06:07:47.769316089Z stderr F [2025-06-01 06:07:47 +0000] [27] [INFO] Booting worker with pid: 27

root@local-cluster-control-plane:/var/log/containers# cat /var/log/pods/default_was-5cb779664c-sk8hd_a1743ef9-7506-4534-816e-b219b6d62042/httpbin/2.log
2025-06-01T06:07:47.76411738Z stderr F [2025-06-01 06:07:47 +0000] [1] [INFO] Starting gunicorn 19.9.0
2025-06-01T06:07:47.766403922Z stderr F [2025-06-01 06:07:47 +0000] [1] [INFO] Listening at: http://0.0.0.0:80 (1)
2025-06-01T06:07:47.766413755Z stderr F [2025-06-01 06:07:47 +0000] [1] [INFO] Using worker: gevent
2025-06-01T06:07:47.769316089Z stderr F [2025-06-01 06:07:47 +0000] [27] [INFO] Booting worker with pid: 27

당연히도 link가 걸려있으니 내용은 동일하다 

그러면 이 로그 경로는 어디에서 지정할까? 

바로 Kubelet의 설정 파일인 config.yaml에서 설정한다. 

root@local-cluster-control-plane:/var/lib/kubelet# cat config.yaml
apiVersion: kubelet.config.k8s.io/v1beta1
authentication:
  anonymous:
    enabled: false
  webhook:
    cacheTTL: 0s
    enabled: true
  x509:
    clientCAFile: /etc/kubernetes/pki/ca.crt
authorization:
  mode: Webhook
  webhook:
    cacheAuthorizedTTL: 0s
    cacheUnauthorizedTTL: 0s
cgroupDriver: systemd
cgroupRoot: /kubelet
clusterDNS:
- 10.96.0.10
clusterDomain: cluster.local
containerRuntimeEndpoint: ""
cpuManagerReconcilePeriod: 0s
evictionHard:
  imagefs.available: 0%
  nodefs.available: 0%
  nodefs.inodesFree: 0%
evictionPressureTransitionPeriod: 0s
failSwapOn: false
fileCheckFrequency: 0s
healthzBindAddress: 127.0.0.1
healthzPort: 10248
httpCheckFrequency: 0s
imageGCHighThresholdPercent: 100
imageMaximumGCAge: 0s
imageMinimumGCAge: 0s
kind: KubeletConfiguration
logging:
  flushFrequency: 0
  options:
    json:
      infoBufferSize: "0"
    text:
      infoBufferSize: "0"
  verbosity: 0
memorySwap: {}
nodeStatusReportFrequency: 0s
nodeStatusUpdateFrequency: 0s
rotateCertificates: true
runtimeRequestTimeout: 0s
shutdownGracePeriod: 0s
shutdownGracePeriodCriticalPods: 0s
staticPodPath: /etc/kubernetes/manifests
streamingConnectionIdleTimeout: 0s
syncFrequency: 0s
volumeStatsAggPeriod: 0s

원래라면 podLogsDir: /var/log/pods 이렇게 나와야하지만 없기 때문에 기본값인 /var/log/pods로 설정된다. 

 

그래서 Elasticsearch의 Fleet Policy에 Pod의 로그를 수집하는 정책을 넣어준다면 

-> /var/log/pods/~~의 경로를 넣어주면 해당 파드의 로그를 수집할 수 있다