What happened

On February 28, from approximately 17:35 UTC (09:35 PST) to 21:59 UTC (12:59 PST) the Amazon S3 service in the US-East-1 region in Northern Virginia, US, became inaccessible. Due to the reliance of Contentful on S3, the outage affected certain parts of our infrastructure:

1. Users attempting to create, edit or publish entries via the web app or the Content Management API (CMA) encountered errors with HTTP 500 Internal Server Error status.
2. Clients attempting to fetch or transform non-cached images via the Images API encountered errors with HTTP 500 Internal Server Error status. Images previously cached on the CDN edge node near the client continued to be served uninterrupted.

The Delivery API was unaffected throughout.

Timeline of events, all times in UTC:

Time Event 17:35 AWS confirms elevated error rates in the S3 service in the US-East region. 17:42 Our monitoring systems detect an issue in the Management API and notify the on-call engineer. Contentful's Engineering team begins to investigate the issue. 19:10 We begin implementation of the most viable mitigation strategy for the Images API deliverability. 20:15 Images API re-routing code change pushed to a small percentage of traffic. 20:53 Images API traffic fully re-routed for all traffic. 21:49 Amazon S3 considered fully recovered by AWS. 21:59 Our on-call engineer certifies that Management API operation has returned to normal.

Why did the Availability metric on the Contentful Status Page not indicate any issues?

The published metric for the Management API tests an endpoint that touches all components operated by Contentful - our servers, caches, databases, and so on. This particular endpoint did not interact with S3. We have now added an additional metric to our Status Page from a probe that specifically makes calls to S3.

The Component Status for the Management API in our Status Page was correctly set to Major Outage by our monitoring at 17:45 UTC.

Our internal monitoring picked up the issue immediately.

What did Contentful do to mitigate the impact of the outage?

Once we detected the S3 outage we investigated possible mitigation options. After validating such an option, we deployed a code change that re-routed Image API requests to a backup S3 bucket located in a different S3 region. This action lowered the rate of errors issued by the Images API, but did not eliminate errors completely since the Images API transformation service was also affected by the S3 outage.

Contentful APIs are designed to be a collection of independent services residing on separate infrastructure as much as possible. As a result of this design, the S3 outage has not affected customers' published content, which continued to be accessible via the Content Delivery API.

What steps are we taking to mitigate the impact of similar outages in the future?

To handle potential failures of S3 in the primary region, we will review our current failsafe mechanism in order to be able to re-route incoming Images API requests in a timely and fast manner. Despite this incident, because of the overall reliability of the S3 service, its great track record, and its current market position as leading service provider, we currently do not have plans to replace our S3 storage layer with alternative providers.