Logo

dev-resources.site

for different kinds of informations.

Hidden dependencies and the Fastly outage

Published at
6/9/2021
Categories
fastly
outage
postmortem
dependencies
Author
jhall
Author
5 person written this
jhall
open
Hidden dependencies and the Fastly outage

In IT we talk a lot about dependencies. We depend on NPM modules in our code. Our microservices have dependencies on each other. We depend on certain operating systems and versions to run our microservices.

A lot of effort goes into managing these dependencies effectively and efficiently.

Even so, dependencies almost inevitably leak through the cracks, and we can end up with hidden or invisible dependencies.

Yesterday served as a big reminder of this for a huge part of the Internet when Fastly deployed a bug.

Immediately, many of us began to see panic from colleagues and online friends blaming StackOverflow, or The Verge, or some other online property for being down, when in fact the problem was an unseen (to the user, at least) dependency on Fastly.

What’s more, Fastly’s own network status page was victim of the outage*, making it impossible to even read about Fastly’s ongoing attempts to resolve the problem.

What hidden and invisible dependencies does your project rely on? Obviously, you don’t know. But next time you have an unexpected outage (perhaps caused by Fastly), take note, document it, and consider coming up with a mitigation strategy (an incident postmortem can be a great tool for this). Most of our hidden dependencies are not managed by a company as responsive as Fastly, and could result in days, or months of downtime, or even bankruptcy in extreme cases.


*at least it was unreachable for me, although I've seen others claim they could reach it.


If you enjoyed this message, subscribe to The Daily Commit to get future messages to your inbox.

postmortem Article's
29 articles in total
Favicon
Postmortem: The Popcorn Panic
Favicon
How I stopped RSpec from spiking to 2x runtime
Favicon
The Day the Web Stood Still: A Firewall Configuration Catastrophe
Favicon
Why I decided to get bad grades in college
Favicon
Zuri Booking Engine Outage - Incident Report and Recovery Analysis
Favicon
Postmortem: Outage Incident on Thavmasios Online Store
Favicon
Postmortem: Nginx Server Failure
Favicon
SRE book notes: Postmortem Culture
Favicon
Postmortem reports: How to get the most from failure for massive growth
Favicon
Post-mortem: 1h30 downtime on a Saturday morning
Favicon
Incident report (Postmortem)
Favicon
What can we learn from the Facebook outage?
Favicon
Retrospectives or postmortems?
Favicon
Where to start with DevOps
Favicon
Incident Retro: Failing Comment Creation + Erroneous Push Notifications
Favicon
Hidden dependencies and the Fastly outage
Favicon
Gamedev.js Jam 2021 post mortem
Favicon
How to do a postmortem without any preparation
Favicon
A Star Trek Postmortem
Favicon
Duplicate Digest Email Incident Retro From January
Favicon
Post-Mortem: Outbreak Database
Favicon
What I've learned from my 2nd Game | Teddy's Crew
Favicon
What I’ve learned from my first game | R0d3nt
Favicon
Project Nodetree recap ~ AoaH Eight
Favicon
Postmortem of Incident on 08 June 2020
Favicon
Postmortem of Root Certificate Expiration from 30 May 2020
Favicon
40,000+ Users in 3 months... Story of a Product I built
Favicon
Post-Mortem: LinkedIn Talent Intelligence Experience
Favicon
Maximize learnings from a Kubernetes cluster failure

Featured ones: