... .  .    .       .joeberkovitz.com

S3 ‘n’ Me, or, My Lost Semi-Weekend

Yesterday I was busy playing around with my new web application, when all of a sudden it froze up.  I was a little worried but, hey, it’s sort of in a super-early preview so some bugs are expected.  I logged into the Noteflight Amazon EC2 instance to check out the production logs, and found a “Connection Reset By Peer” error.  The peer was Amazon’s S3 (Simple Storage Service), on which we completely rely at this point for all data persistence needs.  My web app couldn’t communicate with the S3 service to fetch or store data — kind of an essential function.

Next I fired up an S3 client on my machine to look at the data from another vantage point. It couldn’t connect successfully.  Uh oh.  With a little more digging I found my way to Amazon’s Services Health Dashboard, which showed that there was in fact some kind of service disruption.  “Elevated error rates,” it said.  A few minutes later, this turned into “Service Disrupted”.  All over the web, sites relying on Amazon S3 were either not showing vital data or crashing and burning.

In the end, the outage lasted almost 7 hours.  Every other discussion group has someone flaming about how they’re going to have to find some other solution, that S3 and “the cloud” won’t cut it after all, and so on.

Me?  I’m not so upset.  I think it’s not so surprising that there should be a major outage with something as new and complicated as this, and Amazon provided updates every 20 minutes or so for the duration of the downtime.  If anything, it’s good when we’re more aware of our dependencies, and act with full knowledge of what could happen.  I’m not at all happy about my site being unavailable because of Amazon’s problems (and I am anxious to see a real explanation posted, not just status updates), but I compare that with the bad situations I’ve had with some other hosting providers, and I feel I am still getting my money’s worth.  Is it perfect?  No. Does it enable businesses like mine to scale without huge up-front capital investments?  Yes… and that’s why I picked AWS, accepting the risk and the dependency that goes with that.  We’ll have to see where they take it from here.