Kubernetes: adventures and misadventures of patching (kubectl patch).

Kubernetes is a powerful container orchestration tool where many different objects are executed and at some point in time we will be interested in modifying.

For this, Kubernetes offers us an interesting mechanism: patch, we are going to explain how to patch and we will see that this tool is far from being enough tool as would be desirable.

Patching operations in Kubernetes

According to the Kubernetes documentation and the Kubernetes API rules, three types are defined (with –type):


This is the type of patch that Kubernetes uses by default and is a native type, defined in the SIG. It follows the structure of the original object but indicating the changes (by default join: merge, that’s why it is known as strategic merge patch) in a yaml file. For example: if we have the following service (in service.yml file):

apiVersion: v1
kind: Service
    app: traefik
  name: traefik
  namespace: kube-system
  externalTrafficPolicy: Cluster
  - name: http
    nodePort: 30200
    port: 80
    protocol: TCP
    targetPort: http
  - name: https
    nodePort: 31832
    port: 443
    protocol: TCP
    targetPort: https
    app: traefik
  type: LoadBalancer

We are going to use the command kubectl patch -f service.yml --type="strategic" -p "$(cat patch.yml)" --dry-run -o yaml to allow us to perform tests on objects without the danger of modifying its content in the Kubernetes cluster.

In this case, if we want this service to listen for an additional port, we will use the “merge” strategy and apply the following patch (patch.yml):

  - name: dashboard
    port: 8080
    protocol: TCP
    targetPort: dashboard

As we can see, the patch only follows the service object as far as we want to make the change (the “ports” array) and being a “strategic merge” type change it will be added to the list as seen in the command dump:

  externalTrafficPolicy: Cluster
  - name: dashboard
    port: 8080
    protocol: TCP
    targetPort: dashboard
  - name: http
    nodePort: 30200
    port: 80
    protocol: TCP
    targetPort: http

But if instead of “merge” we use “replace” what we do is eliminate all the content of the subtree where we are indicating the label “$patch: replace” and instead, directly put the content of the patch. For example to change the content of the array we use the file “patch.yml”:

  - $patch: replace
  - name: dashboard
    port: 8080
    protocol: TCP
    targetPort: dashboard

In this example, the entire contents of “ports:” are deleted and instead the object defined after the “$patch: replace” tag is left, although the order is not important, the tag can go back and has the same effect . The result of the above is:

  - name: dashboard
    port: 8080
    protocol: TCP
    targetPort: dashboard

Finally, “delete” indicated by “$patch: delete” deletes the content of the subtree, even if there are new content, it is not added.

  $patch: delete

The result will be the empty spec content:

apiVersion: v1
kind: Service
    app: traefik
  name: traefik
  namespace: kube-system
spec: {}


This type of patch is a radical change compared to “strategic” because it requires the use of JSON Merge Patch (RFC7386), it can be applied as yaml or json and is based on a procedure of changes application from a pair of target and content.

If the target exists and the value is not “null” it replaces it, if the value is “null” it eliminates the target. If the target does not exist, it creates it. The rest of the content is unchanged.

To clearly see the differences if we apply the following patch to the service.yml file

  - name: dashboard
    port: 8080
    protocol: TCP
    targetPort: dashboard

With kubectl patch -f service.yml --type="merge" -p "$(cat patch.yml)" --dry-run -o yaml we will get the same result as with the type “stategic replace”, since the pointed object (spec.ports) is replaced by the choosen value.

To delete a target (for example, remove the selector) we will usekubectl patch -f service.yml --type="merge" -p '{"spec": {"selector": null}}' --dry-run -o yaml

And to modify the selector we can also use a json as kubectl patch -f service.yml --type="merge" -p '{"spec": {"selector": "newtraefik"}}' --dry-run -o yaml


The two types of patches above are interesting for simple applications such as modifying a name, changing or deleting a tag or adding a check to a deployment without editing it.

But if we need to work with lists, we find them very limited and in addition many objects are not well prepared to be able to understand these patches (for example the Ingress object suffers from list problem) and instead of modifying the lists it always replaces.

That is why this third type of patch exists, more efficiently than the previous two, since it has more operations (“add”, “remove”, “replace”, “move”, “copy”, or “test”), and a more precise pointer to the object we want to modify (defined with JavaScript Object Notation (JSON) Pointer RFC6901).

  • “JSON Pointer

It is the string of objects separated by “/” targetting to the object we want to manipulate. The characters “/” and “~” must be encoded as “~0” and “~1”.

It is especially interesting for arrays, since it can refer to an element of an array indicating its position with a number or indicate a non-existent element with “-“. Unlike JSONPath it is not able to select an element of an Array using a search, so although this method is a better one it still has important limitations.

The patch will have the format:
‘{“op”: “OPERATION”, “PARAMETER1”, “VALUE1”, …}
it will usually have at least two components: operation “op” and path “path”. The following are the operations with this type of patch:

  • Add
    • Operation parameter: “add”
    • Path parameter: “path”
    • Parámetro valor: “value”

If the object targetted by route exists is replaced, if it does not exist it creates it and if it is an Array it creates it in the indicated position in the Array, and the element “-” is used to add new elements to an Array.

  • Remove
    • Operation parameter: “remove”
    • Path parameter : “path”

If the object pointed by the route exists, it is deleted, along with the corresponding branch.

  • Replace
    • Operation parameter: “replace”
    • Path parameter : “path”
    • Parámetro valor: “value”

If the object pointed by the route exists, replace it with the one indicated in value.

  • Move
    • Operation parameter: “move”
    • Source path parameter: “from”
    • Destination path parameter: “path”

If both routes exist, it will proceed to copy the object pointed by the origin route to the destination route and delete the one that exists in the origin route.

  • Copy
    • Operation parameter: “copy”
    • Source path parameter: “from”
    • Destination path parameter: “path”

If both routes exist, it will proceed to copy the object pointed by the origin route to the destination route.

  • Test
    • Operation parameter: “test”
    • Path parameter: “path”
    • Value parameter: “value”

Compare the objectpassed as value with the one that exists pointed by the path, if the path does not exist or the object is different it will return an error.

Applying a JSON patch to a Kubernetes Ingress object

To understand all better we will take as an example an Ingress element whose main function is to work as a reverse proxy from the outside towards the services that run on the platform.

Kubernetes: adventures and misadventures of patching (kubectl patch). 1
It is always interesting to publish web services to outside through an Ingress

In the graph we see an Ingress above all, functioning as a reverse proxy basically needs to know three parameters: the domain through which it is published, the route through which it is published and what service we publish.

Kubernetes: adventures and misadventures of patching (kubectl patch). 2

This Ingress will have a yaml like the following:

apiVersion: extensions/v1beta1
kind: Ingress
  name: traefik
  namespace: external-dns
  - host: powerdns-admin.disasterproject.com
      - backend:
          serviceName: powerdns-admin
          servicePort: 80
        path: /

As we can see the specification format of an Ingress includes at least two lists, one of services and another of service routes. Let’s see that these types of objects even with the “json” type patch are of complex manipulation.

We can use “add” to create new “hosts”, for example a new service for the powerdns-api:

kubectl -n external-dns patch ingress traefik --type "json" -p '[{"op":"add","path":"/spec/rules/-","value":{"host":"powerdns-api.disasterproject.com","http":{"paths":[{"backend":{"serviceName":"powerdns-api","servicePort":8081 },"path":"/"}]} } }]'

As we can see, we add a new host at the end of the rules with the result:

Kubernetes: adventures and misadventures of patching (kubectl patch). 3

Now we have two elements in the Array of rules and within the Array of backends, we have an important problem now because JSON Pointer is not able to search and you can only indicate an object in an Array by its order (0,1,2,3 ,etc). Now, there only a solution: to do a search with jsonpath and number the results, for example with:

$ kubectl -n external-dns get ingress/traefik -o jsonpath='{.spec.rules[*].host}'|tr ' ' '\n'|cat --number
     1  powerdns-admin.disasterproject.com
     2  powerdns-api.disasterproject.com

If we want to modify the path of powerdns-api “/” to “/api” we already have the order number for the JSONPointer (the order number in the array starts with 0, so we must subtract one):

$ kubectl -n external-dns patch ingress/traefik --type "json" -p '[{"op":"replace","path":"/spec/rules/1/http/paths/0/path","value":"/api" }]'

$ kubectl -n external-dns get ingress/traefik -o yaml
  - host: powerdns-api.disasterproject.com
      - backend:
          serviceName: powerdns-api
          servicePort: 8081
        path: /api

But what if I just want to modify the domain where the “/api” route is serving? JSON Pointer shows important limitations at this stage, since we have to look (surely with scripts) for each array the order it occupies in it. It clearly lacks the power of jsonpath that allows queries such as the following that will be returned to us by the backend of the powedns-api.disasterproject.com domain that has a “/api” path:

$ kubectl get -n external-dns ingress traefik -o jsonpath='{.spec.rules[?(@.host=="powerdns-api.disasterproject.com")].http.paths[?(@.path=="/api")]}'

map[backend:map[serviceName:powerdns-api servicePort:8081] path:/api]

In this jsonpath we see two searchs @.host==”powerdns-api.disasterproject.com” and @.path==”/api”, not available in any of the patching strategies. The most advanced JSON patch is the test operation, but it suffers from the problems of JSON Pointer:

$ kubectl -n external-dns patch ingress traefik --type="json" -p '[{"op": "test","path": "/metadata/name","value": "traefik2"}]' --dry-run -o yaml
error: Testing value /metadata/name failed

And therefore it is inefficient in scripts, leaving as the only jsonpath solution to list all the objects, filter the results, take the obtained index and perform the operation again for all arrays that the object we are going to patch has.


All tests have been performed in the test environment defined inhttps://www.disasterproject.com/index.php/2019/07/12/entorno-minimo-para-demos-de-kubernetes/ and in https://www.disasterproject.com/index.php/2019/07/18/kubernetes-mas-simple-k3s/

Docker: Reduce the size of a container

In container environments (Docker or Kubernetes) we need to deploy quickly, and the most important thing for this is their size. We must reduce them so that the download of them from the registry and their execution is slower the larger the container is and that minimally affects the complexity of the relations between services.

For a demonstration, we going to use a PowerDNS-based solution, I find that the original PowerDNS-Admin service container (https://github.com/ngoduykhanh/PowerDNS-Admin/blob/master/docker/Production/Dockerfile) has the following features:

  • The developer is very active and includes python code, javascript (nodejs) and css. The images in docker.hub are obsolete with respect code.
  • Production Dockerfile does not generate a valid image
  • It is based on Debian Slim, which although deletes a large number of files, is not sufficient small.

In Docker Hub there are images, but few are sufficient recent or do not use the original repository so the result comes from an old version of the code. For example, the most downloaded image (really/powerdns-admin) does not take into account the modifications of the last year and does not use yarn for the nodejs layer.

First step: Decide whether to create a new Docker image

Sometimes it is a matter of necessity or time, in this case it has been decided to create a new image taking into account the above. As minimum requirements we need a GitHub account (https://www.github.com), a Docker Hub account (https://hub.docker.com/) and basic knowledge of git as well as advanced knowledge of creating Dockerfile.

In this case, https://github.com/aescanero/docker-powerdns-admin-alpine and https://cloud.docker.com/repository/docker/aescanero/powerdns-admin is created and linked to automatically create the images when a Dockerfile is uploaded to GitHub.

Second step: Choose the base of a container

Using a base of very small size and oriented to the reduction of each of the components (executables, libraries, etc.) that are going to be used is the minimum requirement to reduce the size of a container, and the choice should always be to use Alpine (https://alpinelinux.org/).

In addition to the points already indicated, standard distributions (based on Debian or RedHat) require a large number of items for package management, base system, auxiliary libraries, etc. Alpine eliminates these dependencies and provides a simple package system that allows identifying which package groups have been installed together to be able to eliminate them together (very useful for developments as we will see later)

Using Alpine can reduce the size and deployment time of a container by up to 40%.

Another option of recent appearance to consider is Redhat UBI, especially interesting is ubi-minimal (https://www.redhat.com/en/blog/introducing-red-hat-universal-base-image).

Third step: Install minimum packages

One of Alpine’s strengths is a simple and efficient package system that in addition to reducing the size reduces the creation time of the container.

The packages that we need to install are going to be divided into two blocks, the block of packages that will be left in the container and those that will help us to develop the container:

FROM alpine:3.10
MAINTAINER "Alejandro Escanero Blanco <aescanero@disasterproject.com>"

RUN apk add --no-cache python3 curl py3-mysqlclient py3-openssl py3-cryptography py3-pyldap netcat-openbsd \
  py3-virtualenv mariadb-client xmlsec py3-lxml py3-setuptools
RUN apk add --no-cache --virtual build-dependencies libffi-dev libxslt-dev python3-dev musl-dev gcc yarn xmlsec-dev

The first line will tell Docker what is the basis to use (Alpine version 3.10), the second is information about who creates the Dockerfile and from the third we are creating the script that will generate the container.

The first RUN installs the basic python3 packages with SSL, Mysql and LDAP support with the “apk add” command and a very important feature of apk: “--no-cache” that avoids saving downloaded files as package or temporary indexes.

The second RUN line has another very useful feature of apk: “--virtual VIRTUAL_NAME_PACK“, the list of packages to be installed is assigned to a single name that will allow us to remove all the packages that are installed in this step. So this list will include all the packages that are used to create a development and then we will remove at the end.

Fourth step: Python

Python is a programming language with a huge number of modules, usually those that are required for an application must be in a file called requirements.txt and installed with a package system called pip (pip2 for Python2, pip3 for Python3).

We find another package system, which unlike the usual ones (apk, deb, rpm) will download the sources and if it requires binaries it will proceed to compile them. This forces us to have a development environment that in the previous step we have labeled in order to eliminate that environment.

One of the problems that we can find is one in which the pip package management system is really slow, but there is a package (case of py_lxml that is to be installed by pip3) or the case of some installation process from sources such as the case of python_xmlsec that we install quickly with a single line:

RUN mkdir /xmlsec && curl -sSL https://github.com/mehcode/python-xmlsec/archive/1.3.6.tar.gz | tar -xzC /xmlsec --strip 1 && cd /xmlsec && pip3 install --no-cache-dir .

Another process that accelerates the creation of a container is not to use git when this is possible, as in the previous case for xmlsec, in powerdnsadmin we will do the same, remove the packages already installed from requirements.txt and run the python package installer at same as in Alpine without saving cache of any kind with “--no-cache-dir

RUN mkdir -p /opt/pdnsadmin/ && cd /opt/pdnsadmin \
  && curl -sSL https://github.com/ngoduykhanh/PowerDNS-Admin/archive/master.tar.gz | tar -xzC /opt/pdnsadmin --strip 1 \
  && sed -i -e '/mysqlclient/d' -e '/pyOpenSSL/d' -e '/python-ldap/d' requirements.txt \
  && pip3 install --no-cache-dir -r requirements.txt

Finally, we must undestand that python applications usually use virtualenv, we must minimize its effect by indicating that you use the system packages and do not install in the local directory with “virtualenv --system-site-packages --no-setuptools --no-pip” The container is already a virtual environment, it is not necessary to add more layers.

Fifth step: Nodejs

The last step in development is the optimization of the deployment tasks of the javascript libraries and the style sheets, for this we must check if it uses npm or yarn for the installation of packages. The use of yarn is recommended because it is more efficient in downloading packages and the ability to eliminate duplicates within the tree of downloaded packages.

As the application to be installed uses “flask assets“, whose function is to unify the javascript libraries and the style sheets that the application will use, we have to check what elements of the node_modules folder will use javascript libraries and the style sheet, in the case of powerdns-admin are bootstrap, font-awesome, icheck, ionicons and multiselect, as well as some other files that we keep in a temporary folder and then return it to its original place. The remaining installed elements as well as the yarn cache are deleted.

Sixth step: other resources

Finally, it is necessary to check what happens when the container starts the application and see what applications can be replaced by elements already installed.

In this case the entry point of the container is an entrypoint.sh file that executes a some database queries before launching the service. The first query is to know if the port is open and does it with netcat, but the requirements of netcat are so meager that its replacement is meaningless.

However, if we have a candidate with mariadb-client, since it is only used to launch two simple SQL queries. Creating a python script that does exactly the same is easy and visibly reduces (about 20MB) the size of the container.


The difference between a container without optimization and another optimized can be really important affecting the creation, download and deployment times. As an example (using the test environment explained at Choose between Docker or Podman for test and development environments ) in the following video we see the creation of the container without optimization:

And here optimized:

As we can see there are big differences, the first is that the image not optimized is generated in 6:51 and the one optimized in 2:35. The second and most important is the size, the first is 1.11 GB and the optimized 321 MB both uncompressed. Server storage also appreciates it.

Open post
K3s: Simplify Kubernetes 4

K3s: Simplify Kubernetes

What is K3s?

K3s (https://k3s.io/) is a Kubernetes solution created by Rancher Labs (https://rancher.com/) that promises easy installation, few requirements and minimal memory usage.

For the approach of a Demo/Development environment this becomes a great improvement on what we have talked about previously at Kubernetes: Create a minimal environment for demos , where we can see that the creation of the Kubernetes environment is complex and requires too many resources even if Ansible is the one who performs the difficult work.

We will see if what is presented to us is true and if we can include the Metallb tools that will allow us to emulate the power of the Cloud providers balancers and K8dash environments that will allow us to track the infrastructure status.

K3s Download

We configure the virtual machines in the same way as for Kubernetes, with the installation of dependencies:

sudo apt-get install -y ebtables ethtool socat libseccomp2 conntrack ipvsadm
sudo yum install -y ebtables ethtool socat libseccomp conntrack-tools ipvsadm

We download the latest version of k3s from https://github.com/rancher/k3s/releases/latest/download/k3s and put it in /usr/bin with execution permissions. We must do it in all the nodes.

What is K3s?

K3s includes three “extra” services that will change the initial approach we use for Kubernetes, the first is Flannel, integrated into K3s will make the entire layer of internal network management of Kubernetes, although it is not as complete in features as Weave (for example multicast support) it complies with being compatible with Metallb. A very complete comparison of Kubernetes network providers can be seen at https://rancher.com/blog/2019/2019-03-21-comparing-kubernetes-cni-providers-flannel-calico-canal-and-weave/ .

The second service is Traefik that performs input functions from outside the Kubernetes cluster, it is a powerful reverse proxy/balancer with multiple features that will perform at the Network Layer 7, running behind Metallb that will perform the functions of network layer 3 as balancer.

The last “extra” service of K3s is servicelb, which allows us to have an application load balancer, the problem with this service is that it works in layer 3 and this is the same layer where metallb work, so we cannot install it.

K3s Master Install

On the first node (which will be the master) to be installed, we execute /usr/bin/k3s server --no-deploy servicelb --bind-address IP_MACHINE, if we want the execution to be carried out every time the machine starts we need to create a service file /etc/systemd/system/k3smaster.service


ExecStart=/usr/bin/k3s server --no-deploy servicelb --bind-address


And then, we execute

sudo systemctl enable k3smaster
sudo systemctl start k3smaster

To launch K3s and install the master node, at the end of the installation (about 20 seconds), we will save the contents of the file /var/lib/rancher/k3s/server/node-token since it is the activation code for the nodes (token), it is important to see that teh token is base64 and to use it you have to decode it.

Configure Metallb

Before preparing the nodes we proceed to install Metallb and the K8dash panel:

$ sudo mkdir ~/.kube
$ sudo cp -i /etc/rancher/k3s/k3s.yaml $HOME/.kube/config
$ sudo k3s kubectl apply -f "https://raw.githubusercontent.com/danderson/metallb/master/manifests/metallb.yaml"
$ sudo k3s kubectl apply -f "https://raw.githubusercontent.com/herbrandson/k8dash/master/kubernetes-k8dash.yaml"

We must note that to minimize space, kubectl is included within the k3s executable itself, which makes it ideal for environments with very little storage space.

To activate Metallb we create a configuration that will have the content:

apiVersion: v1
kind: ConfigMap
  namespace: metallb-system
  name: config
  config: |
    - name: my-ip-space
      protocol: layer2

That when applying with k3s, kubectl apply -f pool.yml will configure Metallb so that any service with loadBalancer obtain one IPs defined in the specified range, in our example the range corresponds to the network defined in inventory.k3s.yml and is between xxx240 to xxx254.

apiVersion: v1
kind: Service
  name: k8dashlb
  namespace: kube-system
  - name: http
    port: 80
    protocol: TCP
    targetPort: 4654
    k8s-app: k8dash
  type: LoadBalancer

To test this we load a service to access K8dash:

apiVersion: v1
kind: Service
  name: k8dashlb
  namespace: kube-system
  - name: http
    port: 80
    protocol: TCP
    targetPort: 4654
    k8s-app: k8dash
  type: LoadBalancer

We apply with k3s kubectl apply -f service.yml and we will see which IPs have the active services with:

$sudo k3s kubectl get services -A
NAMESPACE     NAME         TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)                      AGE
default       kubernetes   ClusterIP      <none>          443/TCP                      32m
kube-system   k8dash       ClusterIP    <none>          80/TCP                       31m
kube-system   k8dashlb     LoadBalancer   80:32324/TCP                 31m
kube-system   kube-dns     ClusterIP     <none>          53/UDP,53/TCP,9153/TCP       32m
kube-system   traefik      LoadBalancer   80:30668/TCP,443:31859/TCP   30m

We see on the one hand that the services marked as loadBalancer have correctly taken the IPs, both has beed configured, the Traefik service deployed by K3s and the K8dashlb.

Install K3s Nodes

In the rest of the nodes we execute /usr/bin/k3s agent --server https://IP_MASTER:6443 –token DECODED _BASE64_TOKEN, if we want the execution to be every time the machine starts we need to configure a service creating the file /etc/systemd/system/k3s.service


ExecStart=/usr/bin/k3s agent --server https://IP_MASTER:6443 --token TOKEN_DECODIFICADO_BASE64


And we run to configure the service

sudo systemctl enable k3s
sudo systemctl start k3s


If we access to the service IP for K8dashlb service obtained previously and with the token obtained with   k3s kubectl get secret `k3s kubectl get secret|grep ^k8dash|awk '{print $1}'` -o jsonpath="{.data.token}"|base64 -d (note: in Ansible deployment must be returned the IP and the token to access). We can check then the operations in the platform:
K3s: Simplify Kubernetes 5

Finally a small comparison between kubernetes and k3s in memory use, both fresh installed:

  • Kubernetes master: 574 MB
  • Kubernetes node: 262 MB
  • K3s master: 260 MB
  • K3s node: 137 MB
Open post
Kubernetes: Create a minimal environment for demos 6

Kubernetes: Create a minimal environment for demos

Every day more business environments are making a migration to Cloud or Kubernetes/Openshift and it is necessary to meet these requirements for demonstrations.

Kubernetes is not a friendly environment to carry it in a notebook with medium capacity (8GB to 16GB of RAM) and less with a demo that requires certain resources.

Deploy Kubernetes on kubeadm, containerd, metallb and weave

This case is based on the Kubeadm-based deployment (https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/) for Kubernetes deployment, using containerd (https://containerd.io/) as the container life cycle manager and to obtain a minimum network management we will use metallb (https://metallb.universe.tf/) that will allow us to emulate the power of the cloud providers balancers (as AWS Elastic Load Balancer) and Weave (https://www.weave.works/blog/weave-net-kubernetes-integration/) that allows us to manage container networks and integrate seamlessly with metallb.

Finally, taking advantage of the infrastructure, we deploy the real-time resource manager K8dash (https://github.com/herbrandson/k8dash) that will allow us to track the status of the infrastructure and the applications that we deploy in it.

Although the Ansible roles that we have used before (see https://github.com/aescanero/disasterproject) allow us to deploy the environment with ease and cleanliness, we will examine it to understand how the changes we will use in subsequent chapters (using k3s) have an important impact on the availability and performance of the deployed demo/development environment.

First step: Install Containerd

The first step in the installation is the dependencies that Kubernetes has and a very good reference about them is the documentation that Kelsey Hightower makes available to those who need to know Kubernetes thoroughly (https://github.com/kelseyhightower/kubernetes-the-hard-way), especially of all those who are interested in Kubernetes certifications such as CKA (https://www.cncf.io/certification/cka/).

Kubernetes: Create a minimal environment for demos 7

We start with a series of network packages

sudo apt-get install -y ebtables ethtool socat libseccomp2 conntrack ipvsadm
sudo yum install -y ebtables ethtool socat libseccomp conntrack-tools ipvsadm

We install the container life manager (a Containerd version that includes CRI and CNI) and take advantage of the packages that come with the Kubernetes network interfaces (CNI or Container Network Interface)

sudo sh -c "curl -LSs https://storage.googleapis.com/cri-containerd-release/cri-containerd-cni-1.2.7.linux-amd64.tar.gz |tar --no-overwrite-dir -C / -xz"

The package includes the service for systemd so it is enough to start the service:

sudo systemctl enable containerd
sudo systemctl start containerd

Second Step: kubeadm and kubelet

Now we download the executables of kubernetes, in the case of the first machine to configure it will be the “master” and we have to download the kubeadm binaries (the installer of kubernetes), kubelet (the agent that will connect with containerd on each machine. To know which one is the stable version of kubernetes we need to execute:

VERSION=`curl -sSL https://dl.k8s.io/release/stable.txt`

And we download the binaries (in the master all and in the nodes only kubelet is necessary)

sudo curl -Ol https://storage.googleapis.com/kubernetes-release/release/$VERSION/bin/linux/amd64/{"kubectl","kubelet","kubeadm"} -o /usr/bin/"#1"
sudo chmod u+x /usr/bin/{"kubectl","kubelet","kubeadm"}

We configure a service for kubelet on each machine by creating the file /etc/systemd/system/kubelet.service that will depend on whether the machine has the function of master or node. For the master the content is:

Description=kubelet: The Kubernetes Node Agent

# This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating the KUBELET_KUBEADM_ARGS variable dynamically
Environment="KUBELET_EXTRA_ARGS='--network-plugin=cni --container-runtime=remote --container-runtime-endpoint=unix:///run/containerd/containerd.sock --node-ip={{ vm_ip }}'"
Environment="KUBELET_KUBECONFIG_ARGS='--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf'"


For the other nodes:

Description=kubelet: The Kubernetes Node Agent

Environment="KUBELET_EXTRA_ARGS='--network-plugin=cni --container-runtime=remote --container-runtime-endpoint=unix:///run/containerd/containerd.sock --node-ip={{ vm_ip }}'"
Environment="KUBELET_KUBECONFIG_ARGS='--bootstrap-kubeconfig=/var/lib/kubelet/bootstrap-kubeconfig --kubeconfig=/var/lib/kubelet/kubeconfig'"


Once the configuration is loaded, we proceed to use kubeadm on the first node (master), indicating the ip to be published, the internal network of the pods (a /16, don’t forget that it is a demo/development environment so that shouldn’t be a problem.) We are not going to use kubeadm on the rest of the nodes, so we do not need to collect information about the execution of this command (the token showed in console isn’t neccesary).

$ sudo kubeadm init --apiserver-advertise-address {{ vm_ip }} --pod-network-cidr= --cri-socket /run/containerd/containerd.sock
Kubernetes: Create a minimal environment for demos 8

Third step: MetalLB (layer 4 load balancer) and Weave (network management)

Before preparing the nodes we proceed to load two elements, the first “metallb” will allow us to have “loadbalancer” services accessible in the same range as the virtual machines and the second “weave” is a network manager that will allow the communication between pods that run in different machines within the network that we have defined above. Both services are loaded with the following commands executed in master:

$ sudo mkdir ~/.kube
$ sudo cp -i /etc/kubernetes/admin.conf ~/.kube/config
$ sudo kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(sudo kubectl version | base64 | tr -d '\n')"
$ sudo kubectl apply -f "https://raw.githubusercontent.com/danderson/metallb/master/manifests/metallb.yaml"

Fourth step: Add nodes

To add the nodes we have to create a user, as we don’t have integration with ldap or active directory, we will use “system accounts” or sa. For this we’ll generate a “initnode” account to add the nodes with the following yaml:

apiVersion: v1
kind: ServiceAccount
  creationTimestamp: null
  name: initnode
  namespace: kube-system
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
  creationTimestamp: null
  name: initauth
  namespace: kube-system
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:node-bootstrapper
- kind: ServiceAccount
  name: initnode
  namespace: kube-system

Once the service account is generated, we execute kubectl get serviceaccount -n kube-system initnode -o=jsonpath="{.secrets[0].name}"   which will give us the name of the token and kubectl get secrets "TOKEN_NAME" -n kube-system -o=jsonpath="{.data.token}"|base64 -d will give us the value of the token that we have to save.

In addition to creating the user, you must create a startup configuration for the nodes, for this we execute the following commands:

$ sudo kubectl config set-cluster kubernetes --kubeconfig=~/bootstrap-kubeconfig --certificate-authority=/etc/kubernetes/pki/ca.crt --embed-certs --server
$ sudo kubectl config set-credentials initnode --kubeconfig=~/bootstrap-kubeconfig --token=VALOR_TOKEN
$ sudo kubectl config set-context initnode@kubernetes --cluster=kubernetes --user=initnode --kubeconfig=~/bootstrap-kubeconfig
$ sudo kubectl config use-context initnode@kubernetes --kubeconfig=~/bootstrap-kubeconfig

The generated file must be copied to all the nodes in/var/lib/kubelet/bootstrap-kubeconfig and start the kubelet service on each node that will be in charge of contacting request access to the master.

sudo systemctl enable kubelet
sudo systemctl start kubelet

Para autorizar el o los nodos en el master hemos de ir al mismo y consultar por las peticiones con el comando kubectl get certificatesigningrequests.certificates.k8s.io
, y autorizarlas con el comando kubectl certificate approve

To authorize the node(s) in the master, we must ask for requests in the master with the command kubectl get certificatesigningrequests.certificates.k8s.io
, and authorize them with the kubectl certificate approve command.

Once the nodes are authorized, we will have the ability to use this platform.

Choose between Docker or Podman for test and development environments

When we must choose between Docker or Podman?

A lot of times we find that there are very few resources and we need an environment to perform a complete product demonstration at customer.

In those cases we’ll need to simulate an environment in the simplest way possible and with minimal resources. For this we’ll adopt containers, but which is the best solution for those small environments?


Docker is the standard container environment, it is the most widespread and put together a set of powerful tools such as a client on the command line, an API server, a container lifecycle manager (containerd), and a container launcher (runc).

running docker with containerd

Install docker is easy, since docker supplies a script that execute the process of prepare and configure the necessary requirements and repositories and finally installs and configures docker leaving the service ready to use.


Podman is a container environment that does not use a service and therefore does not have an API server, requests are made only from the command line, which has advantages and disadvantages that we will explain at the article.

Install podman is easy in a Centos environment (yum install -y podman for Centos 7 and yum install -y container-tools for Centos 8) but you need some work in a Debian environment:

# sudo apt update && sudo apt install -y software-properties-common dirmngr
# sudo apt-key adv --keyserver ha.pool.sks-keyservers.net --recv-keys 0x018BA5AD9DF57A4448F0E6CF8BECF1637AD8C79D
# sudo sh -c "echo 'deb http://ppa.launchpad.net/projectatomic/ppa/ubuntu bionic main' > /etc/apt/sources.list.d/container.list"
# sudo apt update && sudo apt install -y podman skopeo buildah uidmap debootstrap

Deploy with Ansible

In our case we have used the Ansible roles developed at https://github.com/aescanero/disasterproject, to deploy two virtual machines, one with podman and the other with docker.

In the case of using a Debian based distribution we must to install Ansible:

$ sudo sh -c 'echo "deb http://ppa.launchpad.net/ansible/ansible/ubuntu trusty main" >>/etc/apt/sources.list'
$ sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 93C4A3FD7BB9C367
$ sudo apt-get update && sudo apt-get install -y ansible 

We proceed to download the environment and configure Ansible:

$ git clone https://github.com/aescanero/disasterproject
$ cd disasterproject/ansible
$ chmod 600 files/insecure_private_key

Edit the inventory.yml file which must have the following format:

          memory: MEMORY_IN_MB
          vcpus: vCPUS_FOR_VM
          vm_ip: "IP_VM_MACHINA_NAME1"
          linux_flavor: "debian|centos"
          container_engine: "docker|podman"
          memory: MEMORY_IN_MB
          vcpus: vCPUS_FOR_VM
          vm_ip: "IP_VM_MACHINA_NAME2"
          linux_flavor: "debian|centos"
          container_engine: "docker|podman"
        network_name: NETWORK_NAME
        network: "VM_NETWORK"

There are some global variables that hang from “vars:”, which are:

  • network_name: Descriptive name of the libvirt network that we will use and that will also be the name of the interface that will be configured on the KVM host and that will serve as the gateway of the virtual machines
  • network: the first three fields of the IPv4 address to conform a network with mask, virtual machines must have an IP of that range (minus .1 and .255)

The format of each machine is defined by the following attributes:

  • machine_name: Descriptive name of the virtual machine to be deployed, it will also be the hostname of the virtual machine.
  • memory: Virtual machine memory in MB
  • vcpus: Number of virtual CPUs in the virtual machine
  • vm_ip: IP of the virtual machine, must belong to the range defined in the general variable “network”
  • linux_flavor: It is the linux distribution of the virtual machine, three options are allowed: debian, centos and oracle (for oracle linux 8).
  • container_engine: (Optional) It is the container engine that can be deployed in the virtual machine, four options are allowed: docker, podman, kubernetes and k3s

We deploy the environment with:

$ ansible-playbook -i inventory.yml create.yml --ask-become-pass

How to access

As a result, we obtain two virtual machines that we can access via ssh with ssh -i files/insecure_private_key vagrant@VIRTUAL_MACHINE_IP, in each of them we proceed to launch a container, docker run -td nginx and podman run -dt nginx. Once the nginx containers are deployed, we analyze what services are executed in each case.

Results of the comparison

  • Docker, start 32 processes to launch nginx:
                     |                 `-10*[{containerd-shim}]
3194 root      20   0  546332  92220  39252 S  0.0  9.5   0:07.23 dockerd
 2314 root      20   0  463200  41696  25768 S  0.0  4.3   0:00.57 containerd
 3621 root      20   0   32648   5244   4564 S  0.0  0.5   0:00.08 nginx
 3603 root      20   0   11720   4612   3856 S  0.0  0.5   0:00.03 containerd-shim
 3500 root      20   0   20472   2692   1656 S  0.0  0.3   0:00.00 dhclient
 3479 root      20   0   20352   2676   1636 S  0.0  0.3   0:00.00 dhclient
 3656 systemd+  20   0   33100   2508   1468 S  0.0  0.3   0:00.00 nginx

Podman, start only a process to launch nginx:

 3471 root      20   0   32648   5180   4500 S  0.0  0.5   0:00.06 nginx
 3482 systemd+  20   0   33100   2420   1380 S  0.0  0.2   0:00.00 nginx
 3461 root      20   0   85800   1924   1768 S  0.0  0.2   0:00.00 conmon

About the size of the virtual machines there is only 100 MB of difference between the two:

1,5G docker.qcow2
1,4G podman.qcow2

In conclusion we have the following points:

Life cycle management, for example restart of containers that fail automatically, start containers automatically when the computer restarts, run checks on containers, start containers in a certain order, etc.It’s compatible with Docker at the CLI level, image and load from registry.
It depends on systemd to manage the life cycle management of containers.
It requires more resources, but in a demo / development environment with few containers it should not require significant resources.It left more resources available for containers, which is ideal for a virtual machine demo environment.
It occupies a smaller space and requires less dependencies.

Podman is a RedHat project that is gaining strength and many development services are considering adoption (for example the syslog-ng project: https://www.syslog-ng.com/community/b/blog/posts/replacing-docker-with-podman-in-the-syslog-ng-build-container)

More information on how to adapt Podman in Replacing Docker with Podman.

Linux virtual machine with KVM from the command line

If we want to raise virtual machines in a Linux environment that does not have a graphical environment, we can raise virtual machines using the command line with a XML template.

This article explains how the deployment performed with Ansible-libvirt at KVM, Ansible and how to deploy a test environment works internally

Install Qemu-KVM and Libvirt

Linux virtual machine with KVM from the command line 9

First: we must to install libvirt and Qemu-KVM. In Ubuntu / Debian is installed with:

$ sudo apt-get install -y libvirt-daemon-system python-libvirt python-lxml

And in CentOS / Redhat with:

$ sudo yum install -y libvirt-daemon-kvm python-lxml

To launch the service we must do: $ sudo systemctl enable libvirtd && sudo systemctl start libvirtd

Configure a network template

Libvirt provides us with a powerful tool for managing virtual machines called ‘virsh’, which we must use to be able to manage KVM virtual machines from the command line.

For a virtual machine we mainly need three elements, the first is a network configuration that provides IP to virtual machines via DHCP. To do this libvirt needs XML template like the next template (which we will designate “net.xml”):

  <forward mode='nat'>
      <port start='1' end='65535'/>
  <bridge name='BRIDGE_NAME' stp='on' delay='0'/>
  <ip address='IP_HOST' netmask='NETWORK_MASK'>
      <range start='BEGIN_DHCP_RANGE' end='END_DHCP_RANGE'/>

Whose main elements are:

  • NETWORK_NAME: Descriptive name that we are going to use to designate the network, for example, “test_net” or “production_net”.
  • BRIDGE_NAME: Each network creates an interface on the host server that will serve as gateway of the input/output packets of that previous network to the outside. Here we assign a descriptive name that let as identify the interface.
  • IP_HOST: The IP that such interface will have on the host server and that will be the gateway of the virtual machines.
  • NETWORK_MASK: Depends on the network, usually for testing must be use a class C (
  • BEGIN_DHCP_RANGE: To assign IPs to virtual machines using libvirt, there are an internal DHCP server (dnsmasq based), here we define the first IP of the range that we can serve to virtual machines.
  • END_DHCP_RANGE: And here we define the last IP that virtual machines can obtain.

Preparing the operating system image

The second element is the virtual machine image, this image can be created or downloaded, the second is recommended to reduce the deployment time. An image source for virtual machines with KVM / libvirt is Vagrant (https://app.vagrantup.com/boxes/search?provider=libvirt), to obtain an image of the virtual machine that interests us, we must download from https://app.vagrantup.com/APP_NAME/boxes/APP_TAG/versions/APP_VERSION/providers/libvirt.box where APP_NAME is the name of the application we want to download (e.g. debian), APP_TAG is the distribution of this application (e.g. stretch64) and finally APP_VERSION is the version of the application (e.g. 9.9.0). Even so, there are specific image repositories such as CentOs that are at http://cloud.centos.org/centos/7/vagrant/x86_64/images.

Once the libvirt.box file is obtained, it must be decompressed with tar -zcf libvirt.box, which generates three files, one of which is the image of the virtual machine (box.img) that we will rename to NAME_DESCRIPTIVE.qcow2 and copy it to the standard folder for libvirt virtual machine images (/var/lib/libvirt/images) and we will give the libvirt user permissions so that he can manage that image (chown libvirt /var/lib/libvirt/images/NAME_DESCRIPTIVE.qcow2)

It is important to know that if we want to access the virtual machine we will need a password that we can download with wget -O insecure_private_key https://raw.githubusercontent.com/hashicorp/vagrant/master/keys/vagrant, and we must apply permissions for access only with the current user ($ chmod 600 insecure_private_key).

Virtual Machine Template

The third element to create the virtual machine once the last two are created is a virtual machine template, which for a recent libvirt release will have the following form:

<domain type='kvm'>
  <memory unit='MB'>VM_MEMORY</memory>
    <bootmenu enable='no'/>
    <boot dev='hd'/>
  <clock offset='utc'/>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='IMAGE_STORAGE_PATH/IMAGE_NAME.qcow2'/>
      <target dev='vda' bus='virtio'/>
    <interface type='network'>
      <source network='NETWORK_NAME'/>
      <model type='virtio'/>
    <console type='pty'>
      <target type='serial' port='0'/>
    <input type='mouse' bus='ps2'>
      <alias name='input0'/>
    <input type='keyboard' bus='ps2'>
      <alias name='input1'/>
    <graphics type='spice' port='5900' autoport='yes' listen=''>
      <listen type='address' address=''/>
      <image compression='off'/>
      <model type='cirrus' vram='16384' heads='1' primary='yes'/>
      <alias name='video0'/>
    <memballoon model='virtio'/>

Whose elements are:

  • VM_NAME: The descriptive name that will allow us to identify the virtual machine
  • VM_MEMORY: How much memory in MB we allocate to the virtual machine
  • CPU_NUMBER: How many virtual CPUs will the virtual machine have
  • IMAGE_STORAGE_PATH: The path where we have saved the virtual machine, the standard is /var/lib/libvirt/images
  • IMAGE_NAME: A descriptive name we have given to the image in the previous stage
  • NETWORK_NAME: A descriptive name have we given to the network that will manage the virtual machines.

Deploying Virtual Machines

Once the templates are created and the image is deployed, we execute:

# sudo virsh net-create plantilla_red.xml
# sudo virsh create plantilla_máquina_virtual.xml

Finally we must check that it works, first we identifyif the virtual machines are running with # sudo virsh list, second we need to get the ip in use with # sudo virsh net-dhcp-leases NETWORK_NAME and finally we access the assigned IP with # ssh -i insecure_private_key vagrant @IP

Posts navigation

1 2
Scroll to top