TroubleShooting
[Troubleshoot] Kubernetes 네임스페이스 Terminating 상태 지속
개요
- ingress-controller 설치 과정
Warning: Detected changes to resource ingress-nginx which is currently being deleted.
namespace/ingress-nginx unchanged
clusterrole.rbac.authorization.k8s.io/ingress-nginx unchanged
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx unchanged
ingressclass.networking.k8s.io/nginx unchanged
validatingwebhookconfiguration.admissionregistration.k8s.io/ingress-nginx-admission configured
clusterrole.rbac.authorization.k8s.io/ingress-nginx-admission unchanged
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx-admission unchanged
Error from server (Forbidden): error when creating "https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.1.1/deploy/static/provider/baremetal/deploy.yaml": serviceaccounts "ingress-nginx" is forbidden: unable to create new content in namespace ingress-nginx because it is being terminated
Error from server (Forbidden): error when creating "https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.1.1/deploy/static/provider/baremetal/deploy.yaml": configmaps "ingress-nginx-controller" is forbidden: unable to create new content in namespace ingress-nginx because it is being terminated
Error from server (Forbidden): error when creating "https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.1.1/deploy/static/provider/baremetal/deploy.yaml": roles.rbac.authorization.k8s.io "ingress-nginx" is forbidden: unable to create new content in namespace ingress-nginx because it is being terminated
Error from server (Forbidden): error when creating "https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.1.1/deploy/static/provider/baremetal/deploy.yaml": rolebindings.rbac.authorization.k8s.io "ingress-nginx" is forbidden: unable to create new content in namespace ingress-nginx because it is being terminated
Error from server (Forbidden): error when creating "https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.1.1/deploy/static/provider/baremetal/deploy.yaml": services "ingress-nginx-controller-admission" is forbidden: unable to create new content in namespace ingress-nginx because it is being terminated
Error from server (Forbidden): error when creating "https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.1.1/deploy/static/provider/baremetal/deploy.yaml": services "ingress-nginx-controller" is forbidden: unable to create new content in namespace ingress-nginx because it is being terminated
Error from server (Forbidden): error when creating "https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.1.1/deploy/static/provider/baremetal/deploy.yaml": deployments.apps "ingress-nginx-controller" is forbidden: unable to create new content in namespace ingress-nginx because it is being terminated
Error from server (Forbidden): error when creating "https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.1.1/deploy/static/provider/baremetal/deploy.yaml": serviceaccounts "ingress-nginx-admission" is forbidden: unable to create new content in namespace ingress-nginx because it is being terminated
Error from server (Forbidden): error when creating "https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.1.1/deploy/static/provider/baremetal/deploy.yaml": roles.rbac.authorization.k8s.io "ingress-nginx-admission" is forbidden: unable to create new content in namespace ingress-nginx because it is being terminated
Error from server (Forbidden): error when creating "https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.1.1/deploy/static/provider/baremetal/deploy.yaml": rolebindings.rbac.authorization.k8s.io "ingress-nginx-admission" is forbidden: unable to create new content in namespace ingress-nginx because it is being terminated
Error from server (Forbidden): error when creating "https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.1.1/deploy/static/provider/baremetal/deploy.yaml": jobs.batch "ingress-nginx-admission-create" is forbidden: unable to create new content in namespace ingress-nginx because it is being terminated
Error from server (Forbidden): error when creating "https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.1.1/deploy/static/provider/baremetal/deploy.yaml": jobs.batch "ingress-nginx-admission-patch" is forbidden: unable to create new content in namespace ingress-nginx because it is being terminated
원인분석
일전에 만들어둔 ingress-nginx 네임스페이스가 Terminating 상태에서 진행되지 않음
root@k8s-m:~# kubectl get namespaces
NAME STATUS AGE
default Active 112d
ingress-nginx Terminating 112d
kube-node-lease Active 112d
kube-public Active 112d
kube-system Active 112d
일전에 클러스터 업그레이드 후 scheduler와 controller-manager가 unhealthy 상태였기에 발생한 문제
root@k8s-m:~# kubectl get componentstatuses
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME STATUS MESSAGE ERROR
scheduler Unhealthy Get "[<http://127.0.0.1:10251/healthz>](<http://127.0.0.1:10251/healthz>)": dial tcp 127.0.0.1:10251: connect: connection refused
controller-manager Unhealthy Get "[<https://127.0.0.1:10257/healthz>](<https://127.0.0.1:10257/healthz>)": dial tcp 127.0.0.1:10257: connect: connection refused
etcd-0 Healthy {"health":"true","reason":""}
해결방안
- /etc/kubernetes/manifests 내 kube-controller & kube-scheduler yaml파일 수정
Step 1. kube-controller-manager.yaml 파일 수정
▶ 26번 line --port=0 주석처리
Step 2. kube-scheduler.yaml 파일 수정
▶ 19번 line --port=0 주석처리
Step 3. 정상 동작 확인
root@k8s-m:~# kubectl get componentstatuses
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME STATUS MESSAGE ERROR
scheduler Healthy ok
controller-manager Healthy ok
etcd-0 Healthy {"health":"true","reason":""}
이제 namespace가 정상적으로 Terminated 된 것을 확인할 수 있었다.
+) 내용 추가 계획
- scheduler & controller-manager 가 namespace에 미치는 영향도
- --port=0 주석처리 이유
댓글