Articles on n8r | Clément

Copy Fail: From Unprivileged Pod to Kubernetes Node Root

Fri, 01 May 2026 10:00:00 +0200

This article covers two complementary paths: the CNI wrapper staging chain, and the fully autonomous operator-SA compromise that eliminates the external trigger dependency. Both are proven on Talos Linux v1.12.4, Cilium v1.18.x, kernel 6.18.9.

Update (May 5th): code and building blocks on GitHub: https://github.com/clementnuss/copyfail-cve-exploits

Context

I work at PostFinance, where we run a Kubernetes platform supporting banking workloads. Our production clusters run Debian 12 with kernel 6.1.158+, which happens to be not vulnerable to CVE-2026-31431 (more on that at the end).

Banking on Reliability: Cloud Native SRE Practices in Financial Services

Thu, 26 Mar 2026 10:00:00 +0100

This article is a written companion to my KubeCon EU 2026 talk of the same name. It covers four stories from five years of running a Kubernetes platform at PostFinance, a systemic Swiss financial institution: SLOs as a reliability driver, open-source monitoring tools, continuous end-to-end testing, and an interactive debugging session tracking down rare 502 errors.

The interactive visualizations below (hash ring, race condition sequence diagram) are ported from the Slidev presentation so you can explore them at your own pace.

Adding PrometheusHistograms support to VictoriaMetrics/metrics

Sat, 12 Jul 2025 06:50:54 +0200

TL;DR: I added support for PrometheusHistograms (those with le buckets) to the VictoriaMetrics/metrics package (a lightweight alternative to prometheus/client_golang), which allows me to:

Switch to the more lightweight VictoriaMetrics/metrics library in my open-source projects, which I find simpler to use
Make it possible to choose between classical Prometheus histograms or VictoriaMetrics histograms (much more precise) with a flag
Maintain compatibility with existing Prometheus-based monitoring setups

Problem

While working on kubenurse, I wanted to switch from the heavier prometheus/client_golang library to the more lightweight VictoriaMetrics/metrics package. However, there was one significant blocker: the VictoriaMetrics library only supported their own log-based histogram format, not the traditional Prometheus histograms with static le buckets.

A connected farm, part 3 - weighbridge automation

Tue, 25 Feb 2025 18:34:07 +0100

This article covers a topic related to my wife's family farm, namely the brand new weighbridge installed on the biogas plant.

HTTP 502 - Upstream errors with nginx

Mon, 17 Feb 2025 11:30:51 +0100

A little context

I work at PostFinance, taking care of the Linux systems and of the Open-Source Kubernetes platform we are running to support all sorts of banking workloads.

Aside from running the platform, we also take in user support issues (where users are internal developers/colleagues), and this blog article covers an issue named “Ingress gets HTTP 502 errors on high load”.

You can take this article as an SRE exercise: I’ll provide the same data I received in the support issue, in the same order, and you should try to discover the actual issue as soon as possible. Good luck ;)

A Connected Farm, part 2 - Remote Controlled Fence ⚡️

Sat, 11 May 2024 06:17:14 +0200

This article again covers a topic related to my wife’s family farm, but this time, instead of exporting milking data to Grafana, I will detail my usage of Michael Stapelberg’s amazing gokrazy project, which made it possible to reliably develop Go software to control fences around the farm.

Fences and Cows 🐄

The farm is distributed on 2 sites, and on each site there are rather long electric fences, in which the cows happily pasture during the day (and for the heifer’s fence, also during the night).
To prevent the cows from escaping the fences and e.g. eat our neighbour’s grass (which is always greener, as we all know), the fences are electrified ⚡️ with high voltage (6000V) impulsions every second.

Kubenurse: The In-Cluster Doctor Making Network Rounds

Sun, 07 Apr 2024 12:12:16 +0000

TLDR: Kubenurse is the Swiss army knife for Kubernetes network monitoring. It will help you

pinpoint bottlenecks and know the latency in your network
identify nodes with network issues (packet drops, slow connection, etc.)
uncover issues like DNS failures, broken sockets, or interrupted TLS negotiations

Description

Kubenurse is a Kubernetes network monitoring tool developed and open-sourced by PostFinance (a Swiss Banking Institution), which acts like an in-cluster doctor, continuously checking the health of your pod-to-pod, pod-to-service, and pod-to-ingress connections.

A Connected Farm, part 1 - Milking 🐄 🥛

Sat, 17 Feb 2024 17:00:14 +0200

Alongside my work as a System Engineer (with a focus on Kubernetes) at PostFinance, I’m married to a farmer in Switzerland, and live with her and her family on the family farm.
This is quite different from my daily work, and I sometimes have the opportunity to help by, for example, feeding calves during milking, using my skills to install surveillance cameras, deploying a long-distance WiFi network across the farm, or modernizing the milking monitoring.
It’s this latter point that I’m detailing today (without all the technical details, which are covered in the README of the open-source project I’ve created for this purpose).

Backing up MariaDB on Kubernetes

Wed, 27 Dec 2023 05:12:16 +0000

Hosting MariaDB on Kubernetes proved so far a quite good experience: using the Bitnami Helm Chart to host a “standalone” instance (i.e. without replication, as replication already happens on the storage layer, and because simplicity is more valuable than a complex HA setup like Galera) of MariaDB worked out quite well.

Being cautious, I had configured a daily backup to S3, using a tool found on Github, but when it came to restoring data dumped with this tool, which uses a pretty old mysqldump binary, I was stuck and couldn’t restore 😅 For some reason, the default config of the tool didn’t bother to escape quotes and other sensitive types of chars, and as a result I had to resort to restoring my daily velero backup of my MariaDB instance in another namespace to make a proper export from there and to finally restore my data.

Advent of Code 🎄 - an eBPF take 🐝

Sat, 09 Dec 2023 15:58:38 +0200

It’s that period of the year already ! With December comes the Advent of Code programming challenge, and its daily mental workout.

Advent of Code is an Advent calendar of small programming puzzles for a variety of skill sets and skill levels that can be solved in any programming language you like.

The complexity level of the programming challenges increase every day, and tend to be notoriously hard during the last few days. However, as of writing this article, it’s only day 9, and there were a few problems that didn’t require too much processing cycles, provided you spent enough mathematical effort and didn’t come up with only the brute-force solution.

DNS servers monitoring

Mon, 31 Jul 2023 16:23:32 +0100

A few months ago, I found myself needing to know about the reliability of some internal DNS provider’s servers, after getting a series of hardly trackable random network issues, aka “It’s always DNS”.

More specifically, I needed to know about the following:

number of errors/timeouts
capability to query over TCP or UDP
capability to monitor multiple DNS servers at once
return codes received in the answer (i.e. NOERROR, SERVFAIL, NXDOMAIN, you name it)

Minimal downtime when rebooting etcd nodes

Fri, 07 Jul 2023 08:17:40 +0100

Graceful leader changes

When needing to restart some Kubernetes control-plane nodes on which etcd also happens to be running, you will prefer a graceful transfer of the leadership of the etcd cluster, to reduce the transition period that comes with a leader election.

This can be achieved with the following script, provided you specify the adequate environment variables in /etc/profile.d/etcd-all file.

set -o pipefail && \
source /etc/profile.d/etcd-all && \
AM_LEADER=$(etcdctl endpoint status | grep $(hostname) | cut -d ',' -f 5 | tr -d ' ') && \
if [[ $AM_LEADER = "true" ]]
then
  NEW_LEADER=$(etcdctl endpoint status | grep -v $(hostname) | cut -d ',' -f 2 | tr -d ' ' | tail -n '-1') && \
  etcdctl move-leader $NEW_LEADER && sleep 15
fi

Info: the following environment variables need to be set, for example through a file such as: /etc/profile.d/etcd-all

Kubernetes CNI — deconstructed

Mon, 29 Mar 2021 05:42:38 +0100

A few months ago, I had to understand in detail how Container Network Interface (CNI) is implemented to, well, simply get a chaos testing solution working on a bare-metal installation of Kubernetes.

At that time, I found a few resources that helped me understand how this was implemented, mainly Kubernetes’ official documentation on the topic, and the official CNI specification. And yes, this specification simply consists of a Markdown document, which I needed to invest a consequent amount of energy to digest and process.