calico/node 설치하기
Calico the hard way 시리즈의 한 부분으로, calico/node를 DaemonSet으로 배포해 모든 클러스터 노드에서 Calico Open Source 에이전트가 실행되도록 만드는 방법을 다룹니다. Typha와의 TLS 인증서 발급부터 RBAC 구성, DaemonSet 설치까지 직접 따라 해 보게 됩니다.
출처: 문서
본문
calico/node는 세 개의 데몬을 실행합니다
- Felix, 노드별 Calico 데몬
- BIRD, BGP 프로토콜을 사용해 다른 노드에 라우팅 정보를 전파하는 데몬
- confd, Calico 데이터스토어의 구성 변경을 감시하고 BIRD의 구성 파일을 갱신하는 데몬
이 실습에서는 calico/node를 데몬셋으로 구성하고 설치합니다.
인증서 준비하기
calico/node가 Typha와 인증할 때 사용할 키와 인증서 서명 요청(CSR)을 만듭니다
openssl req -newkey rsa:4096 \
-keyout calico-node.key \
-nodes \
-out calico-node.csr \
-subj "/CN=calico-node"
이 인증서는 Common Name(CN)을 calico-node로 제시하며, 이것이 지난 실습에서 Typha가 받아들이도록 구성한 값입니다.
앞서 만든 CA로 Felix 인증서에 서명합니다
openssl x509 -req -in calico-node.csr \
-CA typhaca.crt \
-CAkey typhaca.key \
-CAcreateserial \
-out calico-node.crt \
-days 365
키와 인증서를 calico/node가 접근할 Secret에 저장합니다
kubectl create secret generic -n kube-system calico-node-certs --from-file=calico-node.key --from-file=calico-node.crt
calico/node가 TLS로 Typha에 접근할 때 사용할 키와 인증서를 검증합니다:
curl https://calico-typha:5473 -v --cacert typhaca.crt --resolve calico-typha:5473:$TYPHA_CLUSTERIP --cert calico-node.crt --key calico-node.key
결과
* Added calico-typha:5473:10.103.120.116 to DNS cache
* Hostname calico-typha was found in DNS cache
* Trying 10.103.120.116:5473...
* Connected to calico-typha (10.103.120.116) port 5473 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* CAfile: typhaca.crt
* CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.2 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, CERT verify (15):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: CN=calico-typha
* start date: Jan 27 19:35:44 2024 GMT
* expire date: Jan 26 19:35:44 2025 GMT
* common name: calico-typha (matched)
* issuer: CN=Calico Typha CA
* SSL certificate verify ok.
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> GET / HTTP/1.1
> Host: calico-typha:5473
> User-Agent: curl/7.81.0
> Accept: */*
>
* TLSv1.2 (IN), TLS header, Unknown (21):
* TLSv1.2 (IN), TLS alert, close notify (256):
* Empty reply from server
* Closing connection 0
* TLSv1.2 (OUT), TLS header, Unknown (21):
* TLSv1.2 (OUT), TLS alert, close notify (256):
curl: (52) Empty reply from server
이 결과는 calico/node 인증서와 키를 제시했을 때 Typha가 TLS 인증서를 제시하고 연결을 받아들였다는 것을 보여 줍니다.
RBAC 준비하기
calico/node가 실행될 ServiceAccount를 만듭니다
kubectl create serviceaccount -n kube-system calico-node
Calico 데이터스토어 객체를 읽고 수정할 권한을 가진 클러스터 롤을 준비합니다
kubectl apply -f - <<EOF
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: calico-node
rules:
# The CNI plugin needs to get pods, nodes, and namespaces.
- apiGroups: [""]
resources:
- pods
- nodes
- namespaces
verbs:
- get
# EndpointSlices are used for Service-based network policy rule
# enforcement.
- apiGroups: ["discovery.k8s.io"]
resources:
- endpointslices
verbs:
- watch
- list
- apiGroups: [""]
resources:
- endpoints
- services
verbs:
# Used to discover service IPs for advertisement.
- watch
- list
# Used to discover Typhas.
- get
# Pod CIDR auto-detection on kubeadm needs access to config maps.
- apiGroups: [""]
resources:
- configmaps
verbs:
- get
- apiGroups: [""]
resources:
- nodes/status
verbs:
# Needed for clearing NodeNetworkUnavailable flag.
- patch
# Calico stores some configuration information in node annotations.
- update
# Watch for changes to Kubernetes NetworkPolicies.
- apiGroups: ["networking.k8s.io"]
resources:
- networkpolicies
verbs:
- watch
- list
# Used by Calico for policy information.
- apiGroups: [""]
resources:
- pods
- namespaces
- serviceaccounts
verbs:
- list
- watch
# The CNI plugin patches pods/status.
- apiGroups: [""]
resources:
- pods/status
verbs:
- patch
# Used for creating service account tokens to be used by the CNI plugin
- apiGroups: [""]
resources:
- serviceaccounts/token
resourceNames:
- calico-node
verbs:
- create
# Calico monitors various CRDs for config.
- apiGroups: ["crd.projectcalico.org"]
resources:
- globalfelixconfigs
- felixconfigurations
- bgppeers
- globalbgpconfigs
- bgpconfigurations
- ippools
- ipamblocks
- globalnetworkpolicies
- globalnetworksets
- networkpolicies
- networksets
- clusterinformations
- hostendpoints
- blockaffinities
verbs:
- get
- list
- watch
# Calico must create and update some CRDs on startup.
- apiGroups: ["crd.projectcalico.org"]
resources:
- ippools
- felixconfigurations
- clusterinformations
verbs:
- create
- update
# Calico stores some configuration information on the node.
- apiGroups: [""]
resources:
- nodes
verbs:
- get
- list
- watch
# These permissions are required for Calico CNI to perform IPAM allocations.
- apiGroups: ["crd.projectcalico.org"]
resources:
- blockaffinities
- ipamblocks
- ipamhandles
verbs:
- get
- list
- create
- update
- delete
- apiGroups: ["crd.projectcalico.org"]
resources:
- ipamconfigs
verbs:
- get
# Block affinities must also be watchable by confd for route aggregation.
- apiGroups: ["crd.projectcalico.org"]
resources:
- blockaffinities
verbs:
- watch
EOF
클러스터 롤을 calico-node ServiceAccount에 바인딩합니다
kubectl create clusterrolebinding calico-node --clusterrole=calico-node --serviceaccount=kube-system:calico-node
데몬셋 설치하기
calico/node는 데몬셋으로 실행되어 클러스터의 모든 노드에 설치됩니다.
데몬셋을 만듭니다
kubectl apply -f - <<EOF
kind: DaemonSet
apiVersion: apps/v1
metadata:
name: calico-node
namespace: kube-system
labels:
k8s-app: calico-node
spec:
selector:
matchLabels:
k8s-app: calico-node
updateStrategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
template:
metadata:
labels:
k8s-app: calico-node
spec:
nodeSelector:
kubernetes.io/os: linux
hostNetwork: true
tolerations:
# Make sure calico-node gets scheduled on all nodes.
- effect: NoSchedule
operator: Exists
# Mark the pod as a critical add-on for rescheduling.
- key: CriticalAddonsOnly
operator: Exists
- effect: NoExecute
operator: Exists
serviceAccountName: calico-node
# Minimize downtime during a rolling upgrade or deletion; tell Kubernetes to do a "force
# deletion": https://kubernetes.io/docs/concepts/workloads/pods/pod/#termination-of-pods.
terminationGracePeriodSeconds: 0
priorityClassName: system-node-critical
containers:
# Runs calico-node container on each Kubernetes node. This
# container programs network policy and routes on each
# host.
- name: calico-node
image: calico/node:v3.20.0
env:
# Use Kubernetes API as the backing datastore.
- name: DATASTORE_TYPE
value: "kubernetes"
- name: FELIX_TYPHAK8SSERVICENAME
value: calico-typha
# Wait for the datastore.
- name: WAIT_FOR_DATASTORE
value: "true"
# Set based on the k8s node name.
- name: NODENAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
# Choose the backend to use.
- name: CALICO_NETWORKING_BACKEND
value: bird
# Cluster type to identify the deployment type
- name: CLUSTER_TYPE
value: "k8s,bgp"
# Auto-detect the BGP IP address.
- name: IP
value: "autodetect"
# Disable file logging so kubectl logs works.
- name: CALICO_DISABLE_FILE_LOGGING
value: "true"
# Set Felix endpoint to host default action to ACCEPT.
- name: FELIX_DEFAULTENDPOINTTOHOSTACTION
value: "ACCEPT"
# Disable IPv6 on Kubernetes.
- name: FELIX_IPV6SUPPORT
value: "false"
# Set Felix logging to "info"
- name: FELIX_LOGSEVERITYSCREEN
value: "info"
- name: FELIX_HEALTHENABLED
value: "true"
# Location of the CA bundle Felix uses to authenticate Typha; volume mount
- name: FELIX_TYPHACAFILE
value: /calico-typha-ca/typhaca.crt
# Common name on the Typha certificate; used to verify we are talking to an authentic typha
- name: FELIX_TYPHACN
value: calico-typha
# Location of the client certificate for connecting to Typha; volume mount
- name: FELIX_TYPHACERTFILE
value: /calico-node-certs/calico-node.crt
# Location of the client certificate key for connecting to Typha; volume mount
- name: FELIX_TYPHAKEYFILE
value: /calico-node-certs/calico-node.key
securityContext:
privileged: true
resources:
requests:
cpu: 250m
lifecycle:
preStop:
exec:
command:
- /bin/calico-node
- -shutdown
livenessProbe:
httpGet:
path: /liveness
port: 9099
host: localhost
periodSeconds: 10
initialDelaySeconds: 10
failureThreshold: 6
readinessProbe:
exec:
command:
- /bin/calico-node
- -bird-ready
- -felix-ready
periodSeconds: 10
volumeMounts:
- mountPath: /lib/modules
name: lib-modules
readOnly: true
- mountPath: /run/xtables.lock
name: xtables-lock
readOnly: false
- mountPath: /var/run/calico
name: var-run-calico
readOnly: false
- mountPath: /var/lib/calico
name: var-lib-calico
readOnly: false
- mountPath: /var/run/nodeagent
name: policysync
- mountPath: "/calico-typha-ca"
name: calico-typha-ca
readOnly: true
- mountPath: /calico-node-certs
name: calico-node-certs
readOnly: true
volumes:
# Used by calico-node.
- name: lib-modules
hostPath:
path: /lib/modules
- name: var-run-calico
hostPath:
path: /var/run/calico
- name: var-lib-calico
hostPath:
path: /var/lib/calico
- name: xtables-lock
hostPath:
path: /run/xtables.lock
type: FileOrCreate
# Used to create per-pod Unix Domain Sockets
- name: policysync
hostPath:
type: DirectoryOrCreate
path: /var/run/nodeagent
- name: calico-typha-ca
configMap:
name: calico-typha-ca
- name: calico-node-certs
secret:
secretName: calico-node-certs
EOF
클러스터의 각 노드에서 calico/node가 실행되고 있고, 몇 분 안에 Running 상태가 되는지 확인합니다.
kubectl get pod -l k8s-app=calico-node -n kube-system
결과
NAME READY STATUS RESTARTS AGE
calico-node-99ksc 1/1 Running 0 9m51s
calico-node-cbgxr 1/1 Running 0 9m21s
calico-node-j456w 1/1 Running 0 9m42s
calico-node-rflbk 1/1 Running 0 9m32s
calico-node-xlpkh 1/1 Running 0 9m12s
다음 단계
더 알아보기 (Learn more)
calico/node는 Felix, BIRD, confd 세 데몬이 한 pod에서 함께 동작하는 구조입니다. 인증서와 RBAC가 왜 필요한지 이해하고 나면 다음 단계인 BGP 피어링 구성이 훨씬 수월해집니다.