L7 회로 차단

L7 회로 차단 (Circuit Breaking)

Cilium Service Mesh는 CiliumClusterwideEnvoyConfig CRD를 정의하며, 이를 통해 사용자는 Cilium agent에 내장된 Envoy 컴포넌트의 구성을 설정할 수 있어요. 이 문서는 회로 차단(circuit breaking) 규칙을 구성하고 의도적으로 회로 차단기를 "트립"시켜 테스트하는 방법을 보여줘요.

출처: L7 Circuit Breaking

본문

Cilium Service Mesh는 CiliumClusterwideEnvoyConfig CRD를 정의하며, 이를 통해 사용자는 Cilium agent에 내장된 Envoy 컴포넌트의 구성을 설정할 수 있어요.

회로 차단은 탄력적인 마이크로서비스 애플리케이션을 만드는 데 중요한 패턴이에요. 회로 차단을 사용하면 실패, 지연 시간 급증, 네트워크 특성의 기타 바람직하지 않은 영향의 영향을 제한하는 애플리케이션을 작성할 수 있어요.

CiliumClusterwideEnvoyConfig로 회로 차단 규칙을 구성한 다음, 이 예제에서 의도적으로 회로 차단기를 "트립"시켜 구성을 테스트할 거예요.

테스트 애플리케이션 배포 (Deploy Test Applications)

$ kubectl apply -f https://raw.githubusercontent.com/cilium/cilium/1.20.2/examples/kubernetes/servicemesh/envoy/test-application-proxy-circuit-breaker.yaml

테스트 워크로드는 다음으로 구성돼요:

하나의 클라이언트 Deployment, fortio-deploy
하나의 Service, echo-service

이 Pod들에 대한 정보를 확인하세요:

$ kubectl get pods --show-labels -o wide
NAME                             READY   STATUS    RESTARTS   AGE     IP           NODE                       NOMINATED NODE   READINESS GATES   LABELS
echo-service-59557f5857-xh84s    2/2     Running   0          7m37s   10.0.0.125   cilium-control-plane   <none>           <none>            kind=echo,name=echo-service,other=echo,pod-template-hash=59557f5857
fortio-deploy-687945c6dc-6qnh4   1/1     Running   0          7m37s   10.0.0.109   cilium-control-plane   <none>           <none>            app=fortio,pod-template-hash=687945c6dc

Envoy 회로 차단기 구성 (Configuring Envoy Circuit Breaker)

CiliumClusterwideEnvoyConfig를 정의하는 envoy-circuit-breaker.yaml 파일을 적용하세요.

$ kubectl apply -f https://raw.githubusercontent.com/cilium/cilium/1.20.2/examples/kubernetes/servicemesh/envoy/envoy-circuit-breaker.yaml

Note 이 Envoy 리소스들은 K8s에서 전혀 검증되지 않으므로, Envoy 리소스의 모든 오류는 이 CRD를 관찰하는 Cilium Agent에서만 보여요. 즉 kubectl apply는 성공을 보고하지만, 노드 로컬 Envoy 인스턴스의 리소스 파싱 및/또는 설치가 실패했을 수 있어요. 현재 이것을 검증하는 유일한 방법은 Cilium Agent 로그에서 오류와 경고를 관찰하는 것이에요. 또한 Cilium Agent는 클러스터의 충돌하는 Envoy 리소스에 대해 경고 로그를 출력해요.

Note Cilium Ingress Controller는 필요할 때 내부적으로 Envoy 리소스를 구성해요. 충돌이 없는지 확인하려면 Envoy 리소스를 명시적으로 만들 때 Cilium Agent 로그를 확인하세요.

CiliumClusterwideEnvoyConfig가 올바르게 생성됐는지 확인하세요.

$ kubectl get ccec envoy-circuit-breaker -oyaml
apiVersion: cilium.io/v2
kind: CiliumClusterwideEnvoyConfig
...
resources:
- "@type": type.googleapis.com/envoy.config.cluster.v3.Cluster
  name: "default/echo-service"
  connect_timeout: 5s
  lb_policy: ROUND_ROBIN
  type: EDS
  circuit_breakers:
    thresholds:
    - priority: "DEFAULT"
      max_requests: 2
      max_pending_requests: 1
  outlier_detection:
    split_external_local_origin_errors: true
    consecutive_local_origin_failure: 2
services:
- name: echo-service
  namespace: default

CiliumClusterwideEnvoyConfig 설정에서 max_pending_requests: 1과 max_requests: 2를 지정했어요. 이 규칙들은 하나 이상의 연결과 요청을 동시에 초과하면, envoy가 추가 요청과 연결에 대해 회로를 열면서 일부 실패를 보게 된다는 것을 나타내요.

Envoy 회로 차단기 트립 (Tripping Envoy Circuit Breaker)

fortio용 Pod 이름으로 환경 변수를 만드세요:

$ export FORTIO_POD=$(kubectl get pods -l app=fortio -o 'jsonpath={.items[0].metadata.name}')

-c 2 플래그로 두 개의 동시 연결을 사용하고 -n 20 플래그로 20개 요청을 보내 Service를 호출하는 다음 명령을 사용하세요:

$ kubectl exec "$FORTIO_POD" -c fortio -- /usr/bin/fortio load -c 2 -qps 0 -n 20 http://echo-service:8080

출력:

$ kubectl exec "$FORTIO_POD" -c fortio -- /usr/bin/fortio load -c 2 -qps 0 -n 20 http://echo-service:8080
{"ts":1692767216.838976,"level":"info","file":"scli.go","line":107,"msg":"Starting Φορτίο 1.57.3 h1:kdPlBiws3cFsLcssZxCt2opFmHj14C3yPBokFhMWzmg= go1.20.6 amd64 linux"}
Fortio 1.57.3 running at 0 queries per second, 4->4 procs, for 20 calls: http://echo-service:8080
{"ts":1692767216.839520,"level":"info","file":"httprunner.go","line":100,"msg":"Starting http test","run":"0","url":"http://echo-service:8080","threads":"2","qps":"-1.0","warmup":"parallel","conn-reuse":""}
Starting at max qps with 2 thread(s) [gomax 4] for exactly 20 calls (10 per thread + 0)
{"ts":1692767216.842149,"level":"warn","file":"http_client.go","line":1104,"msg":"Non ok http code","code":"503","status":"HTTP/1.1 503","thread":"1","run":"0"}
{"ts":1692767216.854289,"level":"info","file":"periodic.go","line":832,"msg":"T001 ended after 13.462339ms : 10 calls. qps=742.8129688310479"}
{"ts":1692767216.854985,"level":"info","file":"periodic.go","line":832,"msg":"T000 ended after 14.158587ms : 10 calls. qps=706.2851681456631"}
Ended after 14.197088ms : 20 calls. qps=1408.7
{"ts":1692767216.855035,"level":"info","file":"periodic.go","line":564,"msg":"Run ended","run":"0","elapsed":"14.197088ms","calls":"20","qps":"1408.739595049351"}
Aggregated Function Time : count 20 avg 0.0013703978 +/- 0.000461 min 0.00092124 max 0.002696039 sum 0.027407957
# range, mid point, percentile, count
>= 0.00092124 <= 0.001 , 0.00096062 , 10.00, 2
> 0.001 <= 0.002 , 0.0015 , 90.00, 16
> 0.002 <= 0.00269604 , 0.00234802 , 100.00, 2
# target 50% 0.0015
# target 75% 0.0018125
# target 90% 0.002
# target 99% 0.00262644
# target 99.9% 0.00268908
Error cases : count 1 avg 0.00133143 +/- 0 min 0.00133143 max 0.00133143 sum 0.00133143
# range, mid point, percentile, count
>= 0.00133143 <= 0.00133143 , 0.00133143 , 100.00, 1
# target 50% 0.00133143
# target 75% 0.00133143
# target 90% 0.00133143
# target 99% 0.00133143
# target 99.9% 0.00133143
# Socket and IP used for each connection:
[0]   1 socket used, resolved to 10.96.182.43:8080, connection timing : count 1 avg 0.000426815 +/- 0 min 0.000426815 max 0.000426815 sum 0.000426815
[1]   2 socket used, resolved to 10.96.182.43:8080, connection timing : count 2 avg 0.0004071275 +/- 0.0001215 min 0.000285596 max 0.000528659 sum 0.000814255
Connection time histogram (s) : count 3 avg 0.00041369 +/- 9.966e-05 min 0.000285596 max 0.000528659 sum 0.00124107
# range, mid point, percentile, count
>= 0.000285596 <= 0.000528659 , 0.000407128 , 100.00, 3
# target 50% 0.000346362
# target 75% 0.00043751
# target 90% 0.0004922
# target 99% 0.000525013
# target 99.9% 0.000528294
Sockets used: 3 (for perfect keepalive, would be 2)
Uniform: false, Jitter: false, Catchup allowed: true
IP addresses distribution:
10.96.182.43:8080: 3
Code 200 : 19 (95.0 %)
Code 503 : 1 (5.0 %)
Response Header Sizes : count 20 avg 370.5 +/- 85 min 0 max 390 sum 7410
Response Body/Total Sizes : count 20 avg 2340.15 +/- 465.7 min 310 max 2447 sum 46803
All done 20 calls (plus 0 warmup) 1.370 ms avg, 1408.7 qps

위 출력에서 일부 요청의 응답 코드가 503인 것을 볼 수 있는데, 이는 회로 차단기가 발동했음을 나타내요.

동시 연결 수를 4로 올리세요.

출력:

$ kubectl exec "$FORTIO_POD" -c fortio -- /usr/bin/fortio load -c 4 -qps 0 -n 20 http://echo-service:8080
{"ts":1692767495.818546,"level":"info","file":"scli.go","line":107,"msg":"Starting Φορτίο 1.57.3 h1:kdPlBiws3cFsLcssZxCt2opFmHj14C3yPBokFhMWzmg= go1.20.6 amd64 linux"}
Fortio 1.57.3 running at 0 queries per second, 4->4 procs, for 20 calls: http://echo-service:8080
{"ts":1692767495.819105,"level":"info","file":"httprunner.go","line":100,"msg":"Starting http test","run":"0","url":"http://echo-service:8080","threads":"4","qps":"-1.0","warmup":"parallel","conn-reuse":""}
Starting at max qps with 4 thread(s) [gomax 4] for exactly 20 calls (5 per thread + 0)
{"ts":1692767495.822424,"level":"warn","file":"http_client.go","line":1104,"msg":"Non ok http code","code":"503","status":"HTTP/1.1 503","thread":"3","run":"0"}
{"ts":1692767495.822428,"level":"warn","file":"http_client.go","line":1104,"msg":"Non ok http code","code":"503","status":"HTTP/1.1 503","thread":"0","run":"0"}
{"ts":1692767495.822603,"level":"warn","file":"http_client.go","line":1104,"msg":"Non ok http code","code":"503","status":"HTTP/1.1 503","thread":"1","run":"0"}
{"ts":1692767495.823855,"level":"warn","file":"http_client.go","line":1104,"msg":"Non ok http code","code":"503","status":"HTTP/1.1 503","thread":"0","run":"0"}
{"ts":1692767495.825250,"level":"warn","file":"http_client.go","line":1104,"msg":"Non ok http code","code":"503","status":"HTTP/1.1 503","thread":"1","run":"0"}
{"ts":1692767495.825285,"level":"warn","file":"http_client.go","line":1104,"msg":"Non ok http code","code":"503","status":"HTTP/1.1 503","thread":"0","run":"0"}
{"ts":1692767495.827282,"level":"warn","file":"http_client.go","line":1104,"msg":"Non ok http code","code":"503","status":"HTTP/1.1 503","thread":"0","run":"0"}
{"ts":1692767495.827514,"level":"warn","file":"http_client.go","line":1104,"msg":"Non ok http code","code":"503","status":"HTTP/1.1 503","thread":"2","run":"0"}
{"ts":1692767495.829886,"level":"warn","file":"http_client.go","line":1104,"msg":"Non ok http code","code":"503","status":"HTTP/1.1 503","thread":"0","run":"0"}
{"ts":1692767495.830156,"level":"info","file":"periodic.go","line":832,"msg":"T000 ended after 9.136284ms : 5 calls. qps=547.268451812575"}
{"ts":1692767495.830326,"level":"warn","file":"http_client.go","line":1104,"msg":"Non ok http code","code":"503","status":"HTTP/1.1 503","thread":"2","run":"0"}
{"ts":1692767495.831175,"level":"warn","file":"http_client.go","line":1104,"msg":"Non ok http code","code":"503","status":"HTTP/1.1 503","thread":"3","run":"0"}
{"ts":1692767495.832826,"level":"warn","file":"http_client.go","line":1104,"msg":"Non ok http code","code":"503","status":"HTTP/1.1 503","thread":"3","run":"0"}
{"ts":1692767495.834028,"level":"warn","file":"http_client.go","line":1104,"msg":"Non ok http code","code":"503","status":"HTTP/1.1 503","thread":"3","run":"0"}
{"ts":1692767495.834116,"level":"info","file":"periodic.go","line":832,"msg":"T003 ended after 13.09904ms : 5 calls. qps=381.7073617608619"}
{"ts":1692767495.834865,"level":"info","file":"periodic.go","line":832,"msg":"T001 ended after 13.846811ms : 5 calls. qps=361.09397318992796"}
{"ts":1692767495.835370,"level":"info","file":"periodic.go","line":832,"msg":"T002 ended after 14.352324ms : 5 calls. qps=348.3756358900482"}
Ended after 14.386516ms : 20 calls. qps=1390.2
{"ts":1692767495.835489,"level":"info","file":"periodic.go","line":564,"msg":"Run ended","run":"0","elapsed":"14.386516ms","calls":"20","qps":"1390.1906479650806"}
Aggregated Function Time : count 20 avg 0.0024801033 +/- 0.001782 min 0.000721482 max 0.008055527 sum 0.049602066
# range, mid point, percentile, count
>= 0.000721482 <= 0.001 , 0.000860741 , 10.00, 2
> 0.001 <= 0.002 , 0.0015 , 45.00, 7
> 0.002 <= 0.003 , 0.0025 , 80.00, 7
> 0.003 <= 0.004 , 0.0035 , 85.00, 1
> 0.005 <= 0.006 , 0.0055 , 95.00, 2
> 0.008 <= 0.00805553 , 0.00802776 , 100.00, 1
# target 50% 0.00214286
# target 75% 0.00285714
# target 90% 0.0055
# target 99% 0.00804442
# target 99.9% 0.00805442
Error cases : count 13 avg 0.0016602806 +/- 0.0006006 min 0.000721482 max 0.00281812 sum 0.021583648
# range, mid point, percentile, count
>= 0.000721482 <= 0.001 , 0.000860741 , 15.38, 2
> 0.001 <= 0.002 , 0.0015 , 61.54, 6
> 0.002 <= 0.00281812 , 0.00240906 , 100.00, 5
# target 50% 0.00175
# target 75% 0.00228634
# target 90% 0.00260541
# target 99% 0.00279685
# target 99.9% 0.00281599
# Socket and IP used for each connection:
[0]   5 socket used, resolved to 10.96.182.43:8080, connection timing : count 5 avg 0.0003044688 +/- 0.0001472 min 0.000120654 max 0.00053878 sum 0.001522344
[1]   3 socket used, resolved to 10.96.182.43:8080, connection timing : count 3 avg 0.00041437933 +/- 9.571e-05 min 0.000330279 max 0.000548277 sum 0.001243138
[2]   3 socket used, resolved to 10.96.182.43:8080, connection timing : count 3 avg 0.00041114067 +/- 0.0001352 min 0.000306734 max 0.00060203 sum 0.001233422
[3]   4 socket used, resolved to 10.96.182.43:8080, connection timing : count 4 avg 0.00038631225 +/- 0.0002447 min 0.000175125 max 0.00080311 sum 0.001545249
Connection time histogram (s) : count 15 avg 0.0003696102 +/- 0.0001758 min 0.000120654 max 0.00080311 sum 0.005544153
# range, mid point, percentile, count
>= 0.000120654 <= 0.00080311 , 0.000461882 , 100.00, 15
# target 50% 0.000437509
# target 75% 0.000620309
# target 90% 0.00072999
# target 99% 0.000795798
# target 99.9% 0.000802379
Sockets used: 15 (for perfect keepalive, would be 4)
Uniform: false, Jitter: false, Catchup allowed: true
IP addresses distribution:
10.96.182.43:8080: 15
Code 200 : 7 (35.0 %)
Code 503 : 13 (65.0 %)
Response Header Sizes : count 20 avg 136.5 +/- 186 min 0 max 390 sum 2730
Response Body/Total Sizes : count 20 avg 1026.9 +/- 1042 min 241 max 2447 sum 20538
All done 20 calls (plus 0 warmup) 2.480 ms avg, 1390.2 qps

이제 예상되는 Circuit breaking 동작을 보기 시작할 수 있어요. 요청의 35%만 성공했고 나머지는 Circuit breaking에 걸렸어요.

Code 200 : 7 (35.0 %)
Code 503 : 13 (65.0 %)

정리 (Cleaning up)

규칙을 제거하세요.

$ kubectl delete -f https://raw.githubusercontent.com/cilium/cilium/1.20.2/examples/kubernetes/servicemesh/envoy/envoy-circuit-breaker.yaml

테스트 애플리케이션을 제거하세요.

$ kubectl delete -f https://raw.githubusercontent.com/cilium/cilium/1.20.2/examples/kubernetes/servicemesh/envoy/test-application-proxy-circuit-breaker.yaml