Upgrading a NATS Cluster in Production

UpdatedOctober 16, 2023

I like learning new stuff - anything, including technology. I love tinkering with new tools, systems and services, especially open source projects

Comments

Join the discussion

No comments yet. Be the first to comment.

More from this blog

inodes

inodes is a concept in Linux. Oh wait…no Fun fact that I learned while experimenting on my macOS - I can see that the term and concept of inodes exists in the context of macOS too Looks like it’s a “Unix” thing and Linux and Darwin, both are Unix bas...

Jan 4, 20262 min read47

Kubernetes Preemption Event

You can look for Kubernetes Preemption Events in your observability system assuming you are exporting your Kubernetes Events to some store like some time series DB or similar In our case, we use Prometheus and we have an exporter for exporting the Ku...

Dec 24, 20251 min read36

Speedscope: Performance Data Visualization

Recently, when I discovered the py-spy profiler took for Python, I also discovered Speedscope, which is visualization tool for visualizing performance data (performance profile etc). This is a flamegraph Visualization. I have something of this sort w...

Dec 19, 20251 min read87

Speedscope: Performance Data Visualization

Profilers!

So, today, we had an issue in one of our internal systems called API Tester. It was very slow. Only today it was slow, and the CPU usage was very high according to our monitoring systems, especially since today morning. Before noticing the CPU usage,...

Dec 18, 20254 min read25

Kubernetes Features Enabled

If you have Prometheus running and scraping metrics - You can find Kubernetes list of features enabled information for every feature using kubernetes_feature_enabled metric which gives build information kubernetes_feature_enabled{} The name of the f...

Dec 18, 20251 min read32

Karuppiah's Blog

57 posts

Downgrade compatibility note 2.10.x brings on-disk storage changes which bring significant performance improvements. Upgrade existing server versions will handle the new storage format transparently. However, if a downgrade from 2.10.x occurs, the old version will not understand the format on disk with the exception 2.9.22 and any subsequent patch releases for 2.9. So if you upgrade from 2.9.x to 2.10.0 and then need to downgrade for some reason, it must be back to 2.9.22+ to ensure the stream data can be read correctly.

# SSH into the NATS server # go to your home directory 🏠 🏡 cd $HOME # check the NATS server version nats-server --version # download the NATS server tar ball for the version you want to upgrade to. # for example, to upgrade to v2.10.1 , do this wget https://github.com/nats-io/nats-server/releases/download/v2.10.1/nats-server-v2.10.1-linux-amd64.tar.gz # extract the tar ball tar xvzf nats-server-v2.10.1-linux-amd64.tar.gz # use the below to find the existing NATS server binary location ls -al $(which nats-server) # move the new version of NATS server binary to the appropriate location where existing NATS server binary is there. For example, for v2.10.1 and /usr/bin , do this sudo mv nats-server-v2.10.1-linux-amd64/nats-server /usr/bin # check the status of the existing version of NATS server systemctl status nats-server.service # restart the NATS server - so that it can use the new version of the NATS server binary sudo systemctl restart nats-server.service # check the status of the new version of NATS server systemctl status nats-server.service

$ sudo tail -n 100 -f /data/nats-cluster/logs/nats-server.log .... [12052] 2023/09/27 08:22:48.561553 [INF] Server Exiting.. [5436] 2023/09/27 08:22:48.586120 [INF] Starting nats-server [5436] 2023/09/27 08:22:48.586241 [INF] Version: 2.10.1 [5436] 2023/09/27 08:22:48.586246 [INF] Git: [d3ef745] [5436] 2023/09/27 08:22:48.586249 [INF] Cluster: togai-nats-cluster [5436] 2023/09/27 08:22:48.586272 [INF] Name: togai-nats-cluster-nats-server-1 [5436] 2023/09/27 08:22:48.586277 [INF] Node: 2M6l87aT [5436] 2023/09/27 08:22:48.586281 [INF] ID: <id> [5436] 2023/09/27 08:22:48.586288 [WRN] Plaintext passwords detected, use nkeys or bcrypt [5436] 2023/09/27 08:22:48.586295 [INF] Using configuration file: /data/nats-cluster/nats-server.conf [5436] 2023/09/27 08:22:48.587988 [INF] Starting http monitor on 0.0.0.0:8222 [5436] 2023/09/27 08:22:48.588076 [INF] Starting JetStream [5436] 2023/09/27 08:22:48.588238 [INF] _ ___ _____ ___ _____ ___ ___ _ __ __ [5436] 2023/09/27 08:22:48.588245 [INF] _ | | __|_ _/ __|_ _| _ \\ __| /_\\ | \\/ | [5436] 2023/09/27 08:22:48.588248 [INF] | || | _| | | \\__ \\ | | | / _| / _ \\| |\\/| | [5436] 2023/09/27 08:22:48.588251 [INF] \\__/|___| |_| |___/ |_| |_|_\\___/_/ \\_\\_| |_| [5436] 2023/09/27 08:22:48.588254 [INF] [5436] 2023/09/27 08:22:48.588257 [INF] <https://docs.nats.io/jetstream> [5436] 2023/09/27 08:22:48.588260 [INF] [5436] 2023/09/27 08:22:48.588263 [INF] ---------------- JETSTREAM ---------------- [5436] 2023/09/27 08:22:48.588272 [INF] Max Memory: 953.67 MB [5436] 2023/09/27 08:22:48.588276 [INF] Max Storage: 4.66 GB [5436] 2023/09/27 08:22:48.588300 [INF] Store Directory: "/data/nats-cluster/jetstream-store/jetstream" [5436] 2023/09/27 08:22:48.588304 [INF] ------------------------------------------- [5436] 2023/09/27 08:22:48.590676 [INF] Starting restore for stream '$G > revenue_events' [5436] 2023/09/27 08:22:48.591326 [INF] Restored 0 messages for stream '$G > revenue_events' in 1ms [5436] 2023/09/27 08:22:48.591380 [INF] Recovering 1 consumers for stream - '$G > revenue_events' [5436] 2023/09/27 08:22:48.592916 [INF] Starting JetStream cluster [5436] 2023/09/27 08:22:48.592926 [INF] Creating JetStream metadata controller [5436] 2023/09/27 08:22:48.596012 [INF] JetStream cluster recovering state [5436] 2023/09/27 08:22:48.597572 [INF] Listening for client connections on 0.0.0.0:4222 [5436] 2023/09/27 08:22:48.597726 [INF] Server is ready

Command Palette

Comments

More from this blog