Skip to main content

Troubleshooting

Docker​

Does Docker run on Linux, macOS, and Windows?
You can run both Linux and Windows programs and executables in Docker containers. The Docker platform runs natively on Linux (on x86-64, ARM and many other CPU architectures) and on Windows (x86-64). Docker Inc. builds products that let you build and run containers on Linux, Windows and macOS.

Do I lose my data when the container exits?
Not at all! Any data that your application writes to disk gets preserved in its container until you explicitly delete the container. The file system for the container persists even after the container halts.

How far do Docker containers scale?
Some of the largest server farms in the world today are based on containers. Large web deployments like Google and Twitter, and platform providers such as Heroku run on container technology, at a scale of hundreds of thousands or even millions of containers.

How do I connect Docker containers?
Currently the recommended way to connect containers is via the Docker network feature.

What platforms does Docker run on?
Linux:

  • Any distribution running version 3.10+ of the Linux kernel
  • Specific instructions are available for most Linux distributions, including RHEL, Ubuntu, Oracle Linux, SuSE, and many others. Microsoft Windows:
  • Windows Server 2016
  • Windows 10 Cloud:
  • Amazon EC2
  • Google Compute Engine
  • Microsoft Azure
  • Rackspace
  • Oracle Cloud Infrastructure

How do I connect Docker containers?
Currently the recommended way to connect containers is via the Docker network feature. You can see details of how to work with Docker networks here.

Running out of Space​

CentOS consumes around 1.2GB

Docker Images consumes around 5,5GB space

  • kobil/kafkazookeeper 1.2.0 f73eb1b9f6ba 4 days ago 677MB
  • kobil/kafkabroker 1.2.0 0f33f138d8fa 4 days ago 677MB
  • kobil/kobil-resource-server 1.2.0-a ddd7b4dd5d64 4 days ago 735MB
  • kobil/kobil-pooler 1.2.0-a 55d8984661ce 4 days ago 712MB
  • kobil/kobil-rest-connector 1.2.0-a 93ba350abaea 4 days ago 676MB
  • kobil/kobiliam 1.2.0 fee5a1f9fb34 4 days ago 1.15GB
  • kobil/kobil-pooler 1.1.1 a44a04adb4ae 6 weeks ago 712MB
  • portainer/portainer latest a01958db7424 6 weeks ago 72.2MB

In case you are experiencing increment of space, please verify with command

$ df -h

Run also following command to see status of dockers components. Those commands list the

$ sudo docker image ls
$ sudo docker volume ls
$ sudo docker container ls

Clear up Docker Container, Volumes and Docker Platform​

Run following command to stop and delete all docker container(s)

$ sudo docker stop $(sudo docker ps -a -q)

Run following command to delete all docker volumes

$ sudo docker volume prune

Run following command to clean up docker system

$ sudo docker system prune -a

Get detailed Docker Container Stats​

$ docker stats

Health Page​

Call Health Page and see what functionality is not running.

Kubernetes Troubleshooting​

Not possible to join a v1.18 Node to a v1.17 cluster due to missing RBAC​

In v1.18 kubeadm added prevention for joining a Node in the cluster if a Node with the same name already exists. This required adding RBAC for the bootstrap-token user to be able to GET a Node object.

However this causes an issue where kubeadm join from v1.18 cannot join a cluster created by kubeadm v1.17.

To workaround the issue you have two options:

Execute kubeadm init phase bootstrap-token on a control-plane node using kubeadm v1.18. Note that this enables the rest of the bootstrap-token permissions as well.

or

Apply the following RBAC manually using kubectl apply -f ...

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: kubeadm:get-nodes
rules:
- apiGroups:
  - ""
  resources:
  - nodes
  verbs:
  - get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kubeadm:get-nodes
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kubeadm:get-nodes
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:bootstrappers:kubeadm:default-node-token

ebtables or some similar executable not found during installation​

If you see the following warnings while running kubeadm init

[preflight] WARNING: ebtables not found in system path
[preflight] WARNING: ethtool not found in system path

Then you may be missing ebtables, ethtool or a similar executable on your node. You can install them with the following commands:

  • For Ubuntu/Debian users, run apt install ebtables ethtool.
  • For CentOS/Fedora users, run yum install ebtables ethtool.

kubeadm blocks waiting for control plane during installation​

If you notice that kubeadm init hangs after printing out the following line:

[apiclient] Created API client, waiting for the control plane to become ready

This may be caused by a number of problems. The most common are:

  • network connection problems. Check that your machine has full network connectivity before continuing.
  • the default cgroup driver configuration for the kubelet differs from that used by Docker. Check the system log file (e.g. /var/log/message) or examine the output from journalctl -u kubelet. If you see something like the following:
error: failed to run Kubelet: failed to create kubelet:
misconfiguration: kubelet cgroup driver: "systemd" is different from docker cgroup driver: "cgroupfs"

There are two common ways to fix the cgroup driver problem:

  • Install Docker again following instructions here.

  • Change the kubelet config to match the Docker cgroup driver manually, you can refer to "Configure cgroup driver used by kubelet on Master Node" under Kubernetes Documentation.

If control plane Docker containers are crashlooping or hanging, you can check this by running docker ps and investigating each container by running docker logs.


kubeadm blocks when removing managed containers​

The following could happen if Docker halts and does not remove any Kubernetes-managed containers:

sudo kubeadm reset
[preflight] Running pre-flight checks
[reset] Stopping the kubelet service
[reset] Unmounting mounted directories in "/var/lib/kubelet"
[reset] Removing kubernetes-managed containers
(block)

A possible solution is to restart the Docker service and then re-run kubeadm reset:

sudo systemctl restart docker.service
sudo kubeadm reset

Inspecting the logs for docker may also be useful:

journalctl -ul docker

coredns (or kube-dns) is stuck in the Pending state​

This is expected and part of the design. kubeadm is network provider-agnostic, so the admin should install the pod network solution of choice. You have to install a Pod Network before CoreDNS may be deployed fully. Hence the Pending state before the network is set up.

HostPort services do not work​

The HostPort and HostIP functionality is available depending on your Pod Network provider. Please contact the author of the Pod Network solution to find out whether HostPort and HostIP functionality are available.

Calico, Canal, and Flannel CNI providers are verified to support HostPort.

If your network provider does not support the portmap CNI plugin, you may need to use the NodePort feature of services or use HostNetwork=true

TLS certificate errors​

The following error indicates a possible certificate mismatch.

#kubectl get pods
Unable to connect to the server: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")
  • Verify that the $HOME/.kube/config file contains a valid certificate, and regenerate a certificate if necessary. The certificates in a kubeconfig file are base64 encoded. The base64 --decode command can be used to decode the certificate and openssl x509 -text -noout can be used for viewing the certificate information.
  • Unset the KUBECONFIG environment variable using:
unset KUBECONFIG

Or set it to the default KUBECONFIG location:

export KUBECONFIG=/etc/kubernetes/admin.conf

Another workaround is to overwrite the existing kubeconfig for the “admin” user:

mv  $HOME/.kube $HOME/.kube.bak
mkdir $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config