Kubernetes: adventures and misadventures of patching (kubectl patch).

    • Kubernetes is a powerful container orchestration tool where many different objects are executed and at some point in time we will be interested in modifying.

    For this, Kubernetes offers us an interesting mechanism: patch, we are going to explain how to patch and we will see that this tool is far from being enough tool as would be desirable.

    Patching operations in Kubernetes

    According to the Kubernetes documentation and the Kubernetes API rules, three types are defined (with –type):

    Strategic

    This is the type of patch that Kubernetes uses by default and is a native type, defined in the SIG. It follows the structure of the original object but indicating the changes (by default join: merge, that’s why it is known as strategic merge patch) in a yaml file. For example: if we have the following service (in service.yml file):

    apiVersion: v1
kind: Service
metadata:
  labels:
    app: traefik
  name: traefik
  namespace: kube-system
spec:
  clusterIP: 10.43.122.171
  externalTrafficPolicy: Cluster
  ports:
  - name: http
    nodePort: 30200
    port: 80
    protocol: TCP
    targetPort: http
  - name: https
    nodePort: 31832
    port: 443
    protocol: TCP
    targetPort: https
  selector:
    app: traefik
  type: LoadBalancer

    We are going to use the command kubectl patch -f service.yml --type="strategic" -p "$(cat patch.yml)" --dry-run -o yaml to allow us to perform tests on objects without the danger of modifying its content in the Kubernetes cluster.

    In this case, if we want this service to listen for an additional port, we will use the “merge” strategy and apply the following patch (patch.yml):

    spec:
  ports:
  - name: dashboard
    port: 8080
    protocol: TCP
    targetPort: dashboard

    As we can see, the patch only follows the service object as far as we want to make the change (the “ports” array) and being a “strategic merge” type change it will be added to the list as seen in the command dump:

    ...
spec:
  clusterIP: 10.43.122.171
  externalTrafficPolicy: Cluster
  ports:
  - name: dashboard
    port: 8080
    protocol: TCP
    targetPort: dashboard
  - name: http
    nodePort: 30200
    port: 80
    protocol: TCP
    targetPort: http
...

    But if instead of “merge” we use “replace” what we do is eliminate all the content of the subtree where we are indicating the label “$patch: replace” and instead, directly put the content of the patch. For example to change the content of the array we use the file “patch.yml”:

    spec:
  ports:
  - $patch: replace
  - name: dashboard
    port: 8080
    protocol: TCP
    targetPort: dashboard

    In this example, the entire contents of “ports:” are deleted and instead the object defined after the “$patch: replace” tag is left, although the order is not important, the tag can go back and has the same effect . The result of the above is:

    ...
  ports:
  - name: dashboard
    port: 8080
    protocol: TCP
    targetPort: dashboard
  selector:
...

    Finally, “delete” indicated by “$patch: delete” deletes the content of the subtree, even if there are new content, it is not added.

    spec:
  $patch: delete

    The result will be the empty spec content:

    apiVersion: v1
kind: Service
metadata:
  labels:
    app: traefik
  name: traefik
  namespace: kube-system
spec: {}

    Merge

    This type of patch is a radical change compared to “strategic” because it requires the use of JSON Merge Patch (RFC7386), it can be applied as yaml or json and is based on a procedure of changes application from a pair of target and content.

    If the target exists and the value is not “null” it replaces it, if the value is “null” it eliminates the target. If the target does not exist, it creates it. The rest of the content is unchanged.

    To clearly see the differences if we apply the following patch to the service.yml file

    spec:
  ports:
  - name: dashboard
    port: 8080
    protocol: TCP
    targetPort: dashboard

    With kubectl patch -f service.yml --type="merge" -p "$(cat patch.yml)" --dry-run -o yaml we will get the same result as with the type “stategic replace”, since the pointed object (spec.ports) is replaced by the choosen value.

    To delete a target (for example, remove the selector) we will usekubectl patch -f service.yml --type="merge" -p '{"spec": {"selector": null}}' --dry-run -o yaml

    And to modify the selector we can also use a json as kubectl patch -f service.yml --type="merge" -p '{"spec": {"selector": "newtraefik"}}' --dry-run -o yaml

    JSON

    The two types of patches above are interesting for simple applications such as modifying a name, changing or deleting a tag or adding a check to a deployment without editing it.

    But if we need to work with lists, we find them very limited and in addition many objects are not well prepared to be able to understand these patches (for example the Ingress object suffers from list problem) and instead of modifying the lists it always replaces.

    That is why this third type of patch exists, more efficiently than the previous two, since it has more operations (“add”, “remove”, “replace”, “move”, “copy”, or “test”), and a more precise pointer to the object we want to modify (defined with JavaScript Object Notation (JSON) Pointer RFC6901).

    • “JSON Pointer“

    It is the string of objects separated by “/” targetting to the object we want to manipulate. The characters “/” and “~” must be encoded as “~0” and “~1”.

    It is especially interesting for arrays, since it can refer to an element of an array indicating its position with a number or indicate a non-existent element with “-“. Unlike JSONPath it is not able to select an element of an Array using a search, so although this method is a better one it still has important limitations.

    The patch will have the format:
    ‘{“op”: “OPERATION”, “PARAMETER1”, “VALUE1”, …}
    it will usually have at least two components: operation “op” and path “path”. The following are the operations with this type of patch:

    • Add
      • Operation parameter: “add”
      • Path parameter: “path”
      • Parámetro valor: “value”

    If the object targetted by route exists is replaced, if it does not exist it creates it and if it is an Array it creates it in the indicated position in the Array, and the element “-” is used to add new elements to an Array.

    • Remove
      • Operation parameter: “remove”
      • Path parameter : “path”

    If the object pointed by the route exists, it is deleted, along with the corresponding branch.

    • Replace
      • Operation parameter: “replace”
      • Path parameter : “path”
      • Parámetro valor: “value”

    If the object pointed by the route exists, replace it with the one indicated in value.

    • Move
      • Operation parameter: “move”
      • Source path parameter: “from”
      • Destination path parameter: “path”

    If both routes exist, it will proceed to copy the object pointed by the origin route to the destination route and delete the one that exists in the origin route.

    • Copy
      • Operation parameter: “copy”
      • Source path parameter: “from”
      • Destination path parameter: “path”

    If both routes exist, it will proceed to copy the object pointed by the origin route to the destination route.

    • Test
      • Operation parameter: “test”
      • Path parameter: “path”
      • Value parameter: “value”

    Compare the objectpassed as value with the one that exists pointed by the path, if the path does not exist or the object is different it will return an error.

    Applying a JSON patch to a Kubernetes Ingress object

    To understand all better we will take as an example an Ingress element whose main function is to work as a reverse proxy from the outside towards the services that run on the platform.

    Kubernetes: adventures and misadventures of patching (kubectl patch). 1
    It is always interesting to publish web services to outside through an Ingress

    In the graph we see an Ingress above all, functioning as a reverse proxy basically needs to know three parameters: the domain through which it is published, the route through which it is published and what service we publish.

    Kubernetes: adventures and misadventures of patching (kubectl patch). 2

    This Ingress will have a yaml like the following:

    apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: traefik
  namespace: external-dns
spec:
  rules:
  - host: powerdns-admin.disasterproject.com
    http:
      paths:
      - backend:
          serviceName: powerdns-admin
          servicePort: 80
        path: /

    As we can see the specification format of an Ingress includes at least two lists, one of services and another of service routes. Let’s see that these types of objects even with the “json” type patch are of complex manipulation.

    We can use “add” to create new “hosts”, for example a new service for the powerdns-api:

    kubectl -n external-dns patch ingress traefik --type "json" -p '[{"op":"add","path":"/spec/rules/-","value":{"host":"powerdns-api.disasterproject.com","http":{"paths":[{"backend":{"serviceName":"powerdns-api","servicePort":8081 },"path":"/"}]} } }]'

    As we can see, we add a new host at the end of the rules with the result:

    Kubernetes: adventures and misadventures of patching (kubectl patch). 3

    Now we have two elements in the Array of rules and within the Array of backends, we have an important problem now because JSON Pointer is not able to search and you can only indicate an object in an Array by its order (0,1,2,3 ,etc). Now, there only a solution: to do a search with jsonpath and number the results, for example with:

    $ kubectl -n external-dns get ingress/traefik -o jsonpath='{.spec.rules[*].host}'|tr ' ' '\n'|cat --number
     1  powerdns-admin.disasterproject.com
     2  powerdns-api.disasterproject.com

    If we want to modify the path of powerdns-api “/” to “/api” we already have the order number for the JSONPointer (the order number in the array starts with 0, so we must subtract one):

    $ kubectl -n external-dns patch ingress/traefik --type "json" -p '[{"op":"replace","path":"/spec/rules/1/http/paths/0/path","value":"/api" }]'

$ kubectl -n external-dns get ingress/traefik -o yaml
...
  - host: powerdns-api.disasterproject.com
    http:
      paths:
      - backend:
          serviceName: powerdns-api
          servicePort: 8081
        path: /api
...

    But what if I just want to modify the domain where the “/api” route is serving? JSON Pointer shows important limitations at this stage, since we have to look (surely with scripts) for each array the order it occupies in it. It clearly lacks the power of jsonpath that allows queries such as the following that will be returned to us by the backend of the powedns-api.disasterproject.com domain that has a “/api” path:

    $ kubectl get -n external-dns ingress traefik -o jsonpath='{.spec.rules[?(@.host=="powerdns-api.disasterproject.com")].http.paths[?(@.path=="/api")]}'

map[backend:map[serviceName:powerdns-api servicePort:8081] path:/api]

    In this jsonpath we see two searchs @.host==”powerdns-api.disasterproject.com” and @.path==”/api”, not available in any of the patching strategies. The most advanced JSON patch is the test operation, but it suffers from the problems of JSON Pointer:

    $ kubectl -n external-dns patch ingress traefik --type="json" -p '[{"op": "test","path": "/metadata/name","value": "traefik2"}]' --dry-run -o yaml
error: Testing value /metadata/name failed

    And therefore it is inefficient in scripts, leaving as the only jsonpath solution to list all the objects, filter the results, take the obtained index and perform the operation again for all arrays that the object we are going to patch has.

    Referencias

    All tests have been performed in the test environment defined inhttps://www.disasterproject.com/index.php/2019/07/12/entorno-minimo-para-demos-de-kubernetes/ and in https://www.disasterproject.com/index.php/2019/07/18/kubernetes-mas-simple-k3s/

    Scroll to top