This is what my potential first customers saw when I launched the first email campaign for my latest project, all due to a crash that was not only unpredictable, but entirely avoidable. And it could happen to you too.

What happened

I have Postgres running as a Kubernetes deployment. In my values.yaml, I had set the following:

postgresql:
  image:
    tag: 15

It worked perfectly for months until this morning, when I woke up to my database in a crashloop and my service completely unavailable. Why? It turns out K8s had to restart the PostgreSQL pod, which then loaded the latest 15.* version of the Docker image. Unfortunately for me, that image was buggy and sent my database down to into oblivion.

The impact

Half of the recipients of my email campaign opened the link and were greeted with a 500 error before I could identify and fix the issue. Apart from the fact that it is embarrassing, I probably lost a few potential customers due to this.

Why you should care and what you can do

Pin. Your. Dependencies.

The main thing here is that you don't want any devops or dependency management systems to have control over which versions are pulled into your system. No matter if it's Docker, Python, Maven, Gradle, NPM or whatever system you are using to manage your dependencies - Specify all versions down to the PATCH version.

Semantic versioning

If you don't know what semantic versioning is, I suggest you read up on it. In a nutshell, it is a standard/guideline for defining version numbers and how to increase them:

Given a version number MAJOR.MINOR.PATCH, increment the:

1. MAJOR version when you make incompatible API changes
2. MINOR version when you add functionality in a backward compatible manner
3. PATCH version when you make backward compatible bug fixes Additional labels for pre-release and build metadata are available as extensions to the MAJOR.MINOR.PATCH format.

"Pinning" and why it matters

In my case, by setting the Docker image version to 15, I gave Kubernetes the freedom to download any minor or patch version of it - at any time, without warning. By "pinning" the version - in other words specifying MAJOR, MINOR and PATCH version explicitly - I regained control and the ability to update the image version on my own terms and test it thoroughly in my setup.

To summarize, this is bad:

postgresql:
  image:
    # Will potentially crash your application at any point
    tag: 15

Stay safe by only allowing the latest PATCH version:

postgresql:
  image:
    # This will pull the latest 15.7.x patch version
    tag: 15.7

Final thoughts

Even with 15 years of experience, rookie mistakes creep in. Not pinning your dependencies is an easy mistake to make, but can have dramatic consequences for your product and your company.

Ironically, the product I was about to launch could have actually helped me catch this issue before merging it into production 🤦‍♂️ Check out PR Manager, where you can generate reviews for your code changes before merging them. It allows you to catch issues like the one in this post and lets you fix them with the click of a button.