What system:bootstrappers:kubeadm:default-node-token？
system:bootstrappers:kubeadm:default-node-token是个Group(常量值），谁在用？通过实验发现这个Group只有通过kubeadm init 安装后的集群才有，通过二进制安装的集群是没有。
通过kubeadm token list 和 kubectl get secrets 可以查看到bootstrap-token的auth-extra-groups为system:bootstrappers:kubeadm:default-node-token
kubeadm token list
TOKEN TTL EXPIRES USAGES DESCRIPTION EXTRA GROUPS
f723b1.e6cf92ad0c6cac77 <forever> <never> authentication,signing kubelet-bootstrap-token system:bootstrappers:kubeadm:default-node-token
kubectl get secrets -n kube-system bootstrap-token-f723b1 -o jsonpath='{.data.auth-extra-groups}' |base64 -d
system:bootstrappers:kubeadm:default-node-token
1）从rolebindings查看看
kubectl get rolebindings -A -o json |grep "system:bootstrappers:kubeadm:default-node-token" | wc -l
4
kubectl get rolebindings -A -o json | jq -r '.items[] | select(.subjects[0].kind=="Group") | select(.subjects[0].name=="system:bootstrappers:kubeadm:default-node-token")' | grep -A2 metadata | grep name |cut -d \" - -f4
kube-proxy
kubeadm:kubeadm-certs
kubeadm:nodes-kubeadm-config
只有三条？继续再改下
kubectl get rolebindings -A -o json | jq -r '.items[] | select(.subjects[0].kind=="Group") | select(.subjects[1].name=="system:bootstrappers:kubeadm:default-node-token")' | grep -A2 metadata | grep name |cut -d \" - -f4
kubeadm:kubelet-config
2）从clusterrolebindings查看看
kubectl get clusterrolebindings -o json |grep "system:bootstrappers:kubeadm:default-node-token" | wc -l
3
kubectl get clusterrolebindings -o json | jq -r '.items[] | select(.subjects[0].kind=="Group") | select(.subjects[0].name=="system:bootstrappers:kubeadm:default-node-token")' | grep -A2 metadata | grep name |cut -d \" - -f4
kubeadm:get-nodes
kubeadm:kubelet-bootstrap
kubeadm:node-autoapprove-bootstrap
export rolebinding clusterrolebinding role and clusterrole
PAGER="cat"
command -v "kubectl-neat" &>/dev/null && PAGER="kubectl neat"
file=/dev/stdout
(
echo "---"
kubectl get rolebinding kube-proxy -n kube-system -o yaml | $PAGER
echo "---"
kubectl get rolebinding kubeadm:kubeadm-certs -n kube-system -o yaml | $PAGER
echo "---"
kubectl get rolebinding kubeadm:nodes-kubeadm-config -n kube-system -o yaml | $PAGER
echo "---"
kubectl get rolebinding kubeadm:kubelet-config -n kube-system -o yaml | $PAGER
echo "---"
kubectl get clusterrolebinding kubeadm:get-nodes -o yaml | $PAGER
echo "---"
kubectl get clusterrolebinding kubeadm:kubelet-bootstrap -o yaml | $PAGER
echo "---"
kubectl get clusterrolebinding kubeadm:node-autoapprove-bootstrap -o yaml | $PAGER
echo "---"
kubectl get role kube-proxy -n kube-system -o yaml | $PAGER
echo "---"
kubectl get role kubeadm:kubeadm-certs -n kube-system -o yaml | $PAGER
echo "---"
kubectl get role kubeadm:nodes-kubeadm-config -n kube-system -o yaml | $PAGER
echo "---"
kubectl get role kubeadm:kubelet-config -n kube-system -o yaml | $PAGER
echo "---"
kubectl get clusterrole kubeadm:get-nodes -o yaml | $PAGER
echo "---"
kubectl get clusterrole system:node-bootstrapper -o yaml | $PAGER
echo "---"
kubectl get clusterrole system:certificates.k8s.io:certificatesigningrequests:nodeclient -o yaml | $PAGER
) > $file
导出结果
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: kube-proxy
namespace: kube-system
rules:
- apiGroups:
- ""
resourceNames:
- kube-proxy
resources:
- configmaps
verbs:
- get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: kubeadm:kubeadm-certs
namespace: kube-system
rules:
- apiGroups:
- ""
resourceNames:
- kubeadm-certs
resources:
- secrets
verbs:
- get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: kubeadm:nodes-kubeadm-config
namespace: kube-system
rules:
- apiGroups:
- ""
resourceNames:
- kubeadm-config
resources:
- configmaps
verbs:
- get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: kubeadm:kubelet-config
namespace: kube-system
rules:
- apiGroups:
- ""
resourceNames:
- kubelet-config
resources:
- configmaps
verbs:
- get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: kubeadm:get-nodes
rules:
- apiGroups:
- ""
resources:
- nodes
verbs:
- get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
annotations:
rbac.authorization.kubernetes.io/autoupdate: "true"
labels:
kubernetes.io/bootstrapping: rbac-defaults
name: system:node-bootstrapper
rules:
- apiGroups:
- certificates.k8s.io
resources:
- certificatesigningrequests
verbs:
- create
- get
- list
- watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
annotations:
rbac.authorization.kubernetes.io/autoupdate: "true"
labels:
kubernetes.io/bootstrapping: rbac-defaults
name: system:certificates.k8s.io:certificatesigningrequests:nodeclient
rules:
- apiGroups:
- certificates.k8s.io
resources:
- certificatesigningrequests/nodeclient
verbs:
- create
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: kube-proxy
namespace: kube-system
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: kube-proxy
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: Group
name: system:bootstrappers:kubeadm:default-node-token
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: kubeadm:kubeadm-certs
namespace: kube-system
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: kubeadm:kubeadm-certs
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: Group
name: system:bootstrappers:kubeadm:default-node-token
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: kubeadm:nodes-kubeadm-config
namespace: kube-system
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: kubeadm:nodes-kubeadm-config
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: Group
name: system:bootstrappers:kubeadm:default-node-token
- apiGroup: rbac.authorization.k8s.io
kind: Group
name: system:nodes
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: kubeadm:kubelet-config
namespace: kube-system
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: kubeadm:kubelet-config
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: Group
name: system:nodes
- apiGroup: rbac.authorization.k8s.io
kind: Group
name: system:bootstrappers:kubeadm:default-node-token
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: kubeadm:get-nodes
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: kubeadm:get-nodes
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: Group
name: system:bootstrappers:kubeadm:default-node-token
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: kubeadm:kubelet-bootstrap
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:node-bootstrapper
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: Group
name: system:bootstrappers:kubeadm:default-node-token
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: kubeadm:node-autoapprove-bootstrap
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:certificates.k8s.io:certificatesigningrequests:nodeclient
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: Group
name: system:bootstrappers:kubeadm:default-node-token
# Kubernetes <1.24
JWT_TOKEN_DEFAULT_DEFAULT=$(kubectl get secrets \
$(kubectl get serviceaccounts/default -o jsonpath='{.secrets[0].name}') \
-o jsonpath='{.data.token}' | base64 --decode)
# Kubernetes 1.24+
JWT_TOKEN_DEFAULT_DEFAULT=$(kubectl create token default)
export KUBE_API="https://192.168.122.200:8443"
curl $KUBE_API/apis/apps/v1/namespaces/default/deployments \
--cacert /etc/kubernetes/pki/ca.crt --header "Authorization: Bearer $JWT_TOKEN_DEFAULT_DEFAULT"
shell we need system:anonymous binding cluster-admin?
查询下谁(rolebindings或clusterrolebindings)绑定了system:anonymous这个User?
kubectl get rolebindings -A -o json | jq -r '.items[] | select(.subjects[0].kind=="User")' |grep -C20 system:anonymous
可以查询到在kube-public空间有个叫kubeadm:bootstrap-signer-clusterinfo的rolebinding
kubectl get role,rolebindings -n kube-public
NAME CREATED AT
role.rbac.authorization.k8s.io/kubeadm:bootstrap-signer-clusterinfo 2022-12-30T01:46:36Z
role.rbac.authorization.k8s.io/system:controller:bootstrap-signer 2022-12-30T01:46:33Z
NAME ROLE AGE
rolebinding.rbac.authorization.k8s.io/kubeadm:bootstrap-signer-clusterinfo Role/kubeadm:bootstrap-signer-clusterinfo 21d
rolebinding.rbac.authorization.k8s.io/system:controller:bootstrap-signer Role/system:controller:bootstrap-signer 21d
利用脚本将这些roles和rolebindings导出
PAGER="cat"
command -v "kubectl-neat" &>/dev/null && PAGER="kubectl neat"
file=/dev/stdout
(
echo "---"
kubectl get rolebinding kubeadm:bootstrap-signer-clusterinfo -n kube-public -o yaml | $PAGER
echo "---"
kubectl get rolebinding system:controller:bootstrap-signer -n kube-public -o yaml | $PAGER
echo "---"
kubectl get role kubeadm:bootstrap-signer-clusterinfo -n kube-public -o yaml | $PAGER
echo "---"
kubectl get role system:controller:bootstrap-signer -n kube-public -o yaml | $PAGER
)>$file
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: kubeadm:bootstrap-signer-clusterinfo
namespace: kube-public
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: kubeadm:bootstrap-signer-clusterinfo
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: User
name: system:anonymous
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
annotations:
rbac.authorization.kubernetes.io/autoupdate: "true"
labels:
kubernetes.io/bootstrapping: rbac-defaults
name: system:controller:bootstrap-signer
namespace: kube-public
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: system:controller:bootstrap-signer
subjects:
- kind: ServiceAccount
name: bootstrap-signer
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: kubeadm:bootstrap-signer-clusterinfo
namespace: kube-public
rules:
- apiGroups:
- ""
resourceNames:
- cluster-info
resources:
- configmaps
verbs:
- get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
annotations:
rbac.authorization.kubernetes.io/autoupdate: "true"
labels:
kubernetes.io/bootstrapping: rbac-defaults
name: system:controller:bootstrap-signer
namespace: kube-public
rules:
- apiGroups:
- ""
resources:
- configmaps
verbs:
- get
- list
- watch
- apiGroups:
- ""
resourceNames:
- cluster-info
resources:
- configmaps
verbs:
- update
- apiGroups:
- ""
- events.k8s.io
resources:
- events
verbs:
- create
- patch
- update
经过分析，kubeadm将system:anonymous用户赋予了kubeadm:bootstrap-signer-clusterinfo角色，
kubeadm安装的kubelet启动的token的auth-extra-groups必须为system:bootstrappers:kubeadm:default-node-token，如果kubelet bootstrap-token-secret中的auth-extra-groups不是system:bootstrappers:kubeadm:default-node-token则会报错
kubelet[11247]: E0118 15:40:29.064946 11247 kubelet_node_status.go:92] "Unable to register node with API server" err="nodes is forbidden: User \"system:anonymous\" cannot create resource \"nodes\" in API group \"\" at the cluster scope"
有一个解决办法是将system:anonymous用户授予最高权限则可通过，但此方法不是安全的。
以下脚本来自网络
kubectl create clusterrolebinding system:anonymous --clusterrole=cluster-admin --user=system:anonymous
# kubectl delete clusterrolebinding system:anonymous
kubectl auth can-i --list --as=system:anonymous -n default
Resources Non-Resource URLs Resource Names Verbs
*.* [] [] [*]
[*] [] [*]
[/healthz] [] [get]
[/livez] [] [get]
[/readyz] [] [get]
[/version/] [] [get]
[/version] [] [get]