Kubernetes

This article is a written companion to my KubeCon EU 2026 talk of the same name. It covers four stories from five years of running a Kubernetes platform at PostFinance, a systemic Swiss financial institution: SLOs as a reliability driver, open-source monitoring tools, continuous end-to-end testing, and an interactive debugging session tracking down rare 502 errors. The interactive visualizations below (hash ring, race condition sequence diagram) are ported from the Slidev presentation so you can explore them at your own pace. ...

The Weighbridge Next to the actual farm with the milking cows, the farm is also constituted of a biogas plant. Taking advantage of the facilities there (trucks, buildings, etc.), my wife’s family have been collecting “green waste” for years now, and up until 2024, the cost for taking care of that waste was being paid for by a “per-habitant” tax paid by the town. Recently however, due to the so-called “principe de causalité”, in place of a tax/fee per capita, people bringing green waste to the biogas plant will have to pay for the amount they brought. As a result, a weighbridge had to be installed, which is only one part of the equation. ...

A little context I work at PostFinance, taking care of the Linux systems and of the Open-Source Kubernetes platform we are running to support all sorts of banking workloads. Aside from running the platform, we also take in user support issues (where users are internal developers/colleagues), and this blog article covers an issue named “Ingress gets HTTP 502 errors on high load”. You can take this article as an SRE exercise: I’ll provide the same data I received in the support issue, in the same order, and you should try to discover the actual issue as soon as possible. Good luck ;) ...

TLDR: Kubenurse is the Swiss army knife for Kubernetes network monitoring. It will help you pinpoint bottlenecks and know the latency in your network identify nodes with network issues (packet drops, slow connection, etc.) uncover issues like DNS failures, broken sockets, or interrupted TLS negotiations Description Kubenurse is a Kubernetes network monitoring tool developed and open-sourced by PostFinance (a Swiss Banking Institution), which acts like an in-cluster doctor, continuously checking the health of your pod-to-pod, pod-to-service, and pod-to-ingress connections. ...

Alongside my work as a System Engineer (with a focus on Kubernetes) at PostFinance, I’m married to a farmer in Switzerland, and live with her and her family on the family farm. This is quite different from my daily work, and I sometimes have the opportunity to help by, for example, feeding calves during milking, using my skills to install surveillance cameras, deploying a long-distance WiFi network across the farm, or modernizing the milking monitoring. It’s this latter point that I’m detailing today (without all the technical details, which are covered in the README of the open-source project I’ve created for this purpose). ...

Hosting MariaDB on Kubernetes proved so far a quite good experience: using the Bitnami Helm Chart to host a “standalone” instance (i.e. without replication, as replication already happens on the storage layer, and because simplicity is more valuable than a complex HA setup like Galera) of MariaDB worked out quite well. Being cautious, I had configured a daily backup to S3, using a tool found on Github, but when it came to restoring data dumped with this tool, which uses a pretty old mysqldump binary, I was stuck and couldn’t restore 😅 For some reason, the default config of the tool didn’t bother to escape quotes and other sensitive types of chars, and as a result I had to resort to restoring my daily velero backup of my MariaDB instance in another namespace to make a proper export from there and to finally restore my data. ...

Graceful leader changes When needing to restart some Kubernetes control-plane nodes on which etcd also happens to be running, you will prefer a graceful transfer of the leadership of the etcd cluster, to reduce the transition period that comes with a leader election. This can be achieved with the following script, provided you specify the adequate environment variables in /etc/profile.d/etcd-all file. set -o pipefail && \ source /etc/profile.d/etcd-all && \ AM_LEADER=$(etcdctl endpoint status | grep $(hostname) | cut -d ',' -f 5 | tr -d ' ') && \ if [[ $AM_LEADER = "true" ]] then NEW_LEADER=$(etcdctl endpoint status | grep -v $(hostname) | cut -d ',' -f 2 | tr -d ' ' | tail -n '-1') && \ etcdctl move-leader $NEW_LEADER && sleep 15 fi Info: the following environment variables need to be set, for example through a file such as: /etc/profile.d/etcd-all ...

A few months ago, I had to understand in detail how Container Network Interface (CNI) is implemented to, well, simply get a chaos testing solution working on a bare-metal installation of Kubernetes. At that time, I found a few resources that helped me understand how this was implemented, mainly Kubernetes’ official documentation on the topic, and the official CNI specification. And yes, this specification simply consists of a Markdown document, which I needed to invest a consequent amount of energy to digest and process. ...