Nginx Ingress Controller Upgrade

PublishedJuly 26, 2025

I like learning new stuff - anything, including technology. I love tinkering with new tools, systems and services, especially open source projects

Comments

Join the discussion

No comments yet. Be the first to comment.

More from this blog

inodes

inodes is a concept in Linux. Oh wait…no Fun fact that I learned while experimenting on my macOS - I can see that the term and concept of inodes exists in the context of macOS too Looks like it’s a “Unix” thing and Linux and Darwin, both are Unix bas...

Jan 4, 20262 min read47

Kubernetes Preemption Event

You can look for Kubernetes Preemption Events in your observability system assuming you are exporting your Kubernetes Events to some store like some time series DB or similar In our case, we use Prometheus and we have an exporter for exporting the Ku...

Dec 24, 20251 min read36

Speedscope: Performance Data Visualization

Recently, when I discovered the py-spy profiler took for Python, I also discovered Speedscope, which is visualization tool for visualizing performance data (performance profile etc). This is a flamegraph Visualization. I have something of this sort w...

Dec 19, 20251 min read87

Speedscope: Performance Data Visualization

Profilers!

So, today, we had an issue in one of our internal systems called API Tester. It was very slow. Only today it was slow, and the CPU usage was very high according to our monitoring systems, especially since today morning. Before noticing the CPU usage,...

Dec 18, 20254 min read25

Kubernetes Features Enabled

If you have Prometheus running and scraping metrics - You can find Kubernetes list of features enabled information for every feature using kubernetes_feature_enabled metric which gives build information kubernetes_feature_enabled{} The name of the f...

Dec 18, 20251 min read32

Karuppiah's Blog

57 posts

Things to ensure:

Understand if you want to upgrade the nginx too or not. Look at compatibility between the nginx ingress controller versions and the nginx versions

Anything and everything related to Nginx Upgrade in general, in the case of Standalone Nginx. Refer Nginx Upgrade or the Nginx Upgrade post

Everything works as is and smoothly, with the latest version

Metrics to understand everything is working as is or better, with no degradation in performance and NO errors. At least NO new errors. For any existing / old errors, while we have an older version - let’s ensure we fix those or get rid of them or ignore them. Preferably fix them or get rid of them (in case of any dead code related errors)

Do Load Testing to check performance and do benchmarking. Do it on both the new version and the old version and check the difference in numbers (performance numbers) when it comes to performance. Ensure no issues like memory leaks etc or unnecessary huge increase in resource usage - CPU, RAM, Disk, and Network too

As minimal downtime as possible - preferably 0 downtime. Since nginx and the nginx ingress controllers are stateless services, 0 downtime should be possible. More like, MUST be possible. Through rolling update deployment strategy. And automation and observability helps here

Metrics to understand any possible downtime - from the client’s perspective. The client can be a human, or a service too. So, get metrics from the client perspective too (usage perspective) and NOT just server perspective

Smooth and Easy upgrade. It should be easy to do the upgrade. Simple and easy. Automation helps here

Everything automated. Any and all steps automated

No gaps or space for human errors to occur. Testing ensures this and automation helps here

Thorough testing before doing upgrade

In testing environments and staging (which is close to production environment)

Understand the criticality of each of the nginx ingress controller instances

This will help us to rollout the upgrade for less critical ones first and check / test for any issues / errors / problems / bugs
This will help us form a plan of - in which order we do the upgrade

Smooth and Easy downgrade. It should be easy to downgrade. Simple and easy. Automation helps here

Have all kinds of data related to the existing nginx and nginx ingress controllers and the new nginx and the new ingress controller version we are going to upgrade to

Find Versions of all the existing nginx ingress controllers
- This probably involves two versions - one is the nginx ingress controller itself and then the nginx. Since nginx is also available as a standalone software. Understand any nuances here and understand if it’s just one version - the version of the nginx ingress controller or if different versions of nginx ingress controller can be used with different versions of nginx (understand compatibility between nginx ingress controller and nginx)
Find Version of the nginx ingress controller we are going to upgrade to
- This probably involves two versions - one is the nginx ingress controller itself and then the nginx. Since nginx is also available as a standalone software. Understand any nuances here and understand if it’s just one version - the version of the nginx ingress controller or if different versions of nginx ingress controller can be used with different versions of nginx. So, check this clearly (understand compatibility between nginx ingress controller and nginx)
- Choose this carefully and meticulously. We need a new or latest version but also a stable version and something that has some amount of long term support
- Ensure the stability of the new version by checking any data around it - issues raised, including security issues, performance issues, correctness issues.
Check compatibility between the versions of the different softwares involved at play over here
- Nginx Ingress Controller version
- Kubernetes version (API version, control plane version etc)
- Nginx version
  - Versions of the Nginx plugins
  - Versions of the Nginx Modules
- Nginx Ingress Controller Image’s Base image Operating System version
  - Compatibility between nginx and the base image operating system version
  - Compatibility between ingress controller and the base image operating system version
- Helm Chart version
Difference between the older versions (the ones we have, the existing versions) and the new version we want to upgrade to
- This involves both the nginx ingress controller version and the nginx version
Also, check the difference between the older versions (the ones we have, the existing versions) and the newer versions (a few recent ones if not all) apart from just the one we want to upgrade to, just to understand what kind of changes are happening - to be aware
Data around the nginx ingress controllers
- How many ingresses (ingress configurations) are there?
- How many nginx ingress controllers are there?
- Who uses the nginx ingress controllers?
- Who manages the nginx ingress controllers?
- How many ingresses are being managed by each of the nginx ingress controllers?
- Are there any other ingress controllers other than nginx? Are they managing any ingresses?
- Understand how the ingress is connected to the ingress controller using configuration
  - This can be through annotations - which mentions ingress class name
  - This can be through ingress class name field in the spec
  - [EXTRA] Try to change the ingresses too if they use annotations to mention ingress class name and don’t have any ingress class name field in the spec. This is important for the future because depending on annotations to mention critical information is bad
- Check how many default ingress controllers are there
  - [EXTRA] Ensure there’s only one default ingress controller. Or, there are no default ingress controllers. Maybe no default ingress controllers is a good idea - that way, users are required to put a particular ingress class name or else their ingress will never be used. But for this to happen, we need to fill up any missing fields in existing ingresses which map to the default since they don’t have any annotations to mention the ingress class name, or any ingress class name spec field
- Understand what features of the nginx ingress controller we use - this refers to both the ingress controller and the nginx
  - For example, nginx has features to run plugins to extend nginx, using lua programming language source code etc. Check if any nginx plugins are being used
  - nginx has features to extend nginx using nginx modules, written in C programming language. Check if any nginx modules are being used
  - Understand any customisations done to nginx, any custom features used in nginx etc
- Understand what features of the ingress resource we use
- Understand the ingress resource version we use. Version as in - API Group and it’s Version, that is, the apiVersion field in Kubernetes resources. For example networking.k8s.io API Group and v1 Version, under which we have Ingress or ingress resource, that is, kind field in Kubernetes resources

Metrics around the nginx ingress controllers

How much traffic is currently coming
- Requests per second
Network usage of the nginx - inbound and outbound traffic
Resource Usage apart from Network Usage - CPU and RAM. And any Disk Usage
- Ensure that there’s NO disk usage (internal/container/ephemeral/volatile or non-volatile/external-disk etc) proportional to the traffic - as much as possible. Ideally the disk usage should stay almost constant, given it’s a stateless service. For logs etc, it should log to standard output and that should be enough. No log files etc

Modern Trends

Understand the modern trends. For the future

For example, people are moving away from Ingress resource and Ingress Controllers to Gateway API
- [EXTRA] We can see if we need Gateway API or if we want it, and accordingly see if we can / want to migrate the Ingress resources to Gateway API resources

Command Palette

Comments

More from this blog

Things to ensure: