K9S and Kubeadm
After playing K8S for couple of weeks and started to deploy K8S and debugging network issues
1 K9S
This is the LIFESAVOR of K8S management. and everything is so easy from now on.
wget https://github.com/derailed/k9s/releases/download/v0.32.5/k9s_linux_amd64.deb
sudo apt install ./k9s_linux_amd64.deb
2 Add a node to K9s
There is always a k8s cluster running on node A and I would like to add node B to the cluster.
# Cleanning up
sudo kubeadm reset
sudo systemctl stop kubelet docker # Stop related services
sudo rm -rf /etc/kubernetes /var/lib/etcd /var/lib/cni
# restart
sudo systemctl restart docker kubelet
# Running on node A
# This command would generate the join command for node B
kubeadm token create --print-join-command
# Run on node B
kubeadm join <node_A_IP>:6443 --token <token> --discovery-token-ca-cert-hash sha256:<hash>
3 Copy Kube Configs
Applying following scripts for every user who want to run kubectl.
mkdir -p /home/USERNAME/.kube
# use admin.conf as the config file
cp /etc/kubernetes/admin.conf /home/USERNAME/.kube/config
# change permission
chown USERNAME:USERNAME /home/USERNAME/.kube/config
4 Network Debugging
I found that Ray cluster worker failed to start and the issue is that I can’t ping the IP of a Pod, even on the same node as the Pod. (The node was the newly added from last step) The solutions worked was resinstall Calico
# Delete and reapply Calico manifest
kubectl delete -f https://docs.projectcalico.org/manifests/calico.yaml
kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml
# Restart Calico pods
kubectl -n kube-system rollout restart daemonset/calico-node
There are something else tried but may be helpful next time.
sysctl net.ipv4.ip_forward # Should return 1
echo 1 > /proc/sys/net/ipv4/ip_forward # If not 1, try this
# Check ip route
ip route get <POD_IP>
# check CNI
journalctl -u calico-node -n 100
# check network policies
kubectl get networkpolicies -A
# check which port is used
sudo netstat -tulnp | grep :10250
# t-TCP u-UDP l-Full n-numerical p-protocal