Amazon S3 keeps getting better, now supports versioning

A good thing really can get better! Amazon S3, perhaps the most well-known cloud computing infrastructure service, just got another upgrade. The simple storage service now supports versioning:

Versioning provides an additional layer of protection for your S3 objects. You can easily recover from unintended user errors or application failures. You can also use Versioning for data retention and archiving.

This new feature will give the thousands of websites and services using S3 a quick and easy to way to support undo or file revision histories, among other things. It kind of moves S3 “up the stack” a little, in that it can now do something that developers could have built themselves, but in a simple and easy-to-use way.

Combine this powerful new functionality with Import/Export that launched last year and a couple of recent price drops, and it’s easy to see why Amazon continues to lead the way. Developers continue to make extensive use of the service too. At the end of Q3 2009, there were over 82 billion objects stored in Amazon S3. Just incredible.

I remember when S3 launched back in March 2006, when I was building Podcast Spot, a hosting service for podcasters. It completely changed our business. Global, scalable storage with Amazon worrying about all the details? And for such a small cost? It seemed too good to be true. I’m thrilled to see that S3 just keeps getting better, with relatively frequent price reductions too.

Mountains of data, right at your fingertips

Last week, two announcements caught my eye. The first was from Amazon.com, which announced that there is now more than 1 TB of public data available to developers through its Public Data Sets on AWS project. The second was from the New York Times, which announced its Newswire API, providing access all NYTimes articles as they are published.

This is a big deal. Never before has so much data been so readily available to anyone. The AWS data is particularly interesting. All of a sudden, any developer in the world has cost-effective access to all publicly available DNA sequences (including the entire Human Genome), an entire dump of Wikipedia, US Census data, and much more. Perhaps most importantly, the data is in machine-readable formats. It’s relatively easy for developers to tap into the data sources for cross-referencing, statistical analysis, and who knows what else.

The Newswire API is also really intriguing. It’s part of a growing set of APIs that the New York Times has made available. With the Newswire API, developers can get links and metadata for new articles the minute they are published. What will developers do with this data? Again, who knows. Imagination is the only limitation now that everyone can have immediate access.

Both of these projects remove barriers and will help foster invention, innovation, and discovery. I hope they are part of a larger trend, where simple access to data becomes the norm. Google’s mission might be to organize the world’s information and make it universally accessible and useful, but it’s projects like these that are making that vision a reality. I can’t wait to see what comes next!

Amazon Web Services: Still getting better

aws logo I often think back to 2006 when Dickson and I were in the midst of the VenturePrize business plan competition. It was around that time that Amazon.com launched their first web service, the Simple Storage Service (S3). It had a huge impact on our business, and we’ve been extremely happy customers ever since.

Over the last couple of years, Amazon has introduced a number of additional web services, the most well-known of which might be the Elastic Compute Cloud (EC2). You can think of it like an on-demand computer in the cloud. I had a quick look at it when it launched, but being a Windows shop, we really didn’t have time to invest the extra effort necessary to get it running. Now, Amazon has announced that EC2 will support Windows:

Starting later this Fall, Amazon Elastic Compute Cloud (Amazon EC2) will offer the ability to run Microsoft Windows Server or Microsoft SQL Server. Our goal is to support any and all of the programming models, operating systems and database servers that you need for building applications on our cloud computing platform. The ability to run a Windows environment within Amazon EC2 has been one of our most requested features, and we are excited to be able to provide this capability. We are currently operating a private beta of Amazon EC2 running Windows Server and SQL Server.

Very cool news for Windows developers. It should put some extra pressure on Microsoft too – though apparently they are getting ready to launch something. Watch for more news on that at PDC.

Another interesting new service that Amazon is introducing is a Content Delivery Service:

This new service will provide you a high performance method of distributing content to end users, giving your customers low latency and high data transfer rates when they access your objects. The initial release will help developers and businesses who need to deliver popular, publicly readable content over HTTP connections.

It will run atop S3, so anything that currently exists there can easily be added to the new content delivery network. This is very cool, and will finally bring world-class CDN infrastructure to small businesses. I wish they had introduced this two years ago!

Those are both very important improvements to AWS. Amazon is raising the bar, again. When will Microsoft, Google, and others answer?

Also – I just noticed recently that Amazon has redesigned the AWS website. It looks fantastic, in my opinion, and is much easier to navigate. Keep the positive improvements coming!

What if Twitter had been built by Amazon.com’s Web Services team?

twitter by aws? I’ve been using Twitter for a long time now, and I can’t remember a period of downtime quite as bad as the current one. Features have been disabled, and there’s no ETA for when everything will be back to normal. Who knows, maybe it won’t ever be. Which got me wondering about why Twitter’s reliability is so terrible. Is it the nature of the application, or is it something to do with the people behind Twitter?

What if Twitter had been built by a different team, a team with a pretty good track record for high-availability services? What if Twitter had been built by the Web Services team at Amazon.com?

I think it’s safe to say that things would be quite different:

  1. Reliable, redundant infrastructure
    Twitter would be run inside Amazon’s high-availability data centers. We would never know (or care) that Twitter’s main database was named db006, nor would we ever wonder whether it has a good backup. We’d just know that if it’s good enough for Amazon, it’s good enough for us.
  2. No wondering, “is Twitter working?”
    Instead of wondering if Twitter is working correctly or waiting for Twitter messages or blog posts that explain what the problem is, Twitter would be part of the AWS Service Health Dashboard. We’d be able to see, at a glance, how Twitter is working now, and how well it has worked for the last month. This is what transparency is all about.
  3. Twitter wouldn’t be free, but we’d be cool with that
    Twitter would have had a business model from day one, and we’d all be cheering about how affordable it is. A pay-as-you-go model like all the other web services from Amazon would work quite well for Twitter. You get what you pay for, right?
  4. Premium Support and SLAs
    Speaking of getting what you pay for, Amazon would likely have realized that there are lots of different types of users, and they’d react accordingly. We’d probably have Premium Support for Twitter, to service support requests more efficiently. We’d also have Service Level Agreements.
  5. We wouldn’t call it Twitter…
    Of course, the service wouldn’t be called Twitter. In keeping with Amazon’s other services it would probably have a name like “Amazon Simple Messaging Service”, or SMS for short. Though I suppose that acronym is already taken!

I am a huge Twitter fan, and I really do hope that Ev, Biz, Jack, and the rest of the team get things working and fixed. With every passing hour of downtime though, I lose a little bit of faith. I wonder if Twitter would be better off in someone else’s hands.

Of course, if Twitter really had been built by AWS, there would be far more differences than just the items in my list above. The service may not be recognizable as Twitter!

That doesn’t mean that they couldn’t adopt some of these items as improvements, however. I’d love to see an official Twitter health dashboard, for instance. One can hope.

Maybe Microsoft should buy Amazon instead

The Microsoft-Yahoo deal continues to be the hot topic in the blogosphere right now, with Techmeme still dominated by related discussion. The latest news is that Google has posted an official response to the proposed takeover. In general, discussion has moved from “can you believe what just happened” to “this deal with fail/succeed because…” If you read only two posts on the topic, read this one from Fake Steve Jobs and this one from Henry Blodget.

I really have no idea how this is going to play out. Based on what I’ve read, it seems pretty likely that Microsoft will successfully acquire Yahoo. Many think the deal is as good as done. Far less certain, however, is whether they can make the acquisition a success. It could go either way.

I think what’s clear is that this is a strategy change for Microsoft. A bold recognition that they need to succeed on the web. They trail Google in both search and advertising, and it makes a certain amount of sense that combining with Yahoo will create a stronger competitor.

Microsoft is a platform company. Their cash cow is Windows, the most widely used technology platform in history. They are good at platforms. If the strategy shift is to the web, shouldn’t it be slanted towards a great platform?

amazonawsSuch as Amazon’s Web Services platform. On Wednesday Amazon announced their fourth quarter earnings, and shared this tidbit about Amazon Web Services (AWS):

Adoption of Amazon Elastic Compute Cloud (EC2) and Amazon Simple Storage Service (S3) continues to grow. As an indicator of adoption, bandwidth utilized by these services in fourth quarter 2007 was even greater than bandwidth utilized in the same period by all of Amazon.com’s global websites combined.

ReadWriteWeb has a good discussion of what this means.

Obviously Microsoft isn’t a retailer, and owning Amazon.com itself probably isn’t in the company’s best interests. It could acquire the company for AWS and spin off the rest, however. I suppose Microsoft could just try to duplicate what Amazon has already accomplished with AWS, but why bother? Grab the early market leader and take it to the next level.

I think AWS is an indication of what the platform of the future will look like. Microsoft would be wise to pay attention.

More on the Amazon Kindle

Post ImageNow that Amazon’s Kindle ebook reader has actually launched, there is a lot more information available out on the web. And after reading a bunch of it, I am less excited than I was yesterday. Here are a few links that may be of interest if you’re curious about the Kindle:

I dunno. Crippled wireless, lousy document support, DRM, a $400 price tag, and it’s still ugly. The Kindle sounds less and less impressive with each article I read.

Here’s to hoping that version 2 is better!

The Amazon Kindle

Post ImageAmazon.com is venturing into the hardware industry on Monday with the launch of their new ebook reading device, Kindle. The image I have included to the right comes from a September 2006 post at Endgadget, so I have no idea if that is just a prototype or if it is a reasonable representation of the shipping product. I hope just a prototype, because it’s kind of ugly. The device is 4.9 inches by 7.5 inches by 0.7 inches and weighs 10.2 ounces (so slightly larger than the iPod classic, and double the weight).

Newsweek has a seven page article up all about the device, and after reading it, I’m pretty excited (though still distraught over the look…beige is so pre-Internet). I encourage you to go read it for yourself, but here’s the gist of the article:

  • The Amazon Kindle will sell for $399 USD.
  • It has no back-light, and utilizes E Ink technology that mimics the readability of ink on paper.
  • Battery will last 30 hours and will fully charge in just 2.
  • Wireless connectivity via Wi-Fi and EVDO.
  • Does not require a computer. You can buy the books on the device with one-touch and start reading.
  • There will be 88,000 books available at launch, for $9.99 each.
  • You can subscribe to newspapers like the New York Times and Wall Street Journal.
  • You get a private Kindle email address. Send a PDF to it, and the document is automatically added to your library, ready to read on the Kindle.

The Newsweek article then goes into the history of ebooks, and provides a pretty good analysis of how the Kindle could transform both reading and writing.

Don’t you wish you had one of these? Or maybe one of the second generation Kindles, with color screens, a sleeker design, and a lower price? I do. I know ebooks have been talked about for years, but the Kindle could be the device that finally opens the floodgates. The killer feature, as far as I am concerned, is the inclusion of wireless connectivity. It’s a big, big deal.

You don’t always have access to your computer, and even if you do, connecting a device to it is an annoying step that needs to go away forever. With a wireless connection, the Kindle can do everything on its own, without the need for a computer. Score one for the inclusion of wireless.

As the Newsweek article states, the Kindle is "the first ‘always-on book". That could transform the way books are published entirely! I read a lot of non-fiction, and I like to buy the books when they are brand new. For instance, I bought Freakonomics as soon as I heard about it. The problem is that these books almost always have a second "revised and expanded" edition! There’s no way I’m going to buy the book again. With the Kindle however, I could subscribe to the book. The author could update the book on the fly, and I’d see the changes instantly, wherever I am. How cool is that?!

Another key advantage to the Kindle’s wireless feature is the ability to venture out onto the web. You can look up something on Wikipedia for instance, and then capture passages to your Kindle library "with an electronic version of a highlight pen." Combined with the fact that you can send documents to the Kindle, it could become your hub for all kinds of reading. Books, newspapers, blogs, documents, and web pages.

There will always be critics and individuals who say nay to the idea of ebooks, but it’s a losing battle for them. From the Newsweek article:

"I’ve actually asked myself, ‘Why do I love these physical objects?’ " says [Amazon.com founder Jeff Bezos]. " ‘Why do I love the smell of glue and ink?’ The answer is that I associate that smell with all those worlds I have been transported to. What we love is the words and ideas."

Microsoft’s Bill Hill has a riff where he runs through the energy-wasting, resource-draining process of how we make books now. We chop down trees, transport them to plants, mash them into pulp, move the pulp to another factory to press into sheets, ship the sheets to a plant to put dirty marks on them, then cut the sheets and bind them and ship the thing around the world. "Do you really believe that we’ll be doing that in 50 years?" he asks.

Amazon’s Kindle attempts to solve the first problem – the affinity for the physical book – by being a device that disappears as you read. The print is clear and crisp, the device mimics the size of a paperback, and there isn’t much to distract you from reading (I guess that’s one advantage to the less than sleek look). It solves the second problem – environmental waste and inefficiencies – by getting rid of the need for paper altogether.

I think the Kindle could be good. Read the Newsweek article, and also this post by the author, Steven Levy. In it, he addresses the "ugly" reaction that bloggers like me have had. There’s even more on the device at Techmeme. Maybe in a few years you’ll be reading this on the Kindle!

Oh, and don’t let the $399 price tag get in the way of realizing how much potential the Kindle has. After all, when the iPod launched in 2001, it was priced at $399 too.

Read: Newsweek

Amazon now offers an SLA for S3

Post ImageAmazon announced on Monday the launch of an SLA, or Service Level Agreement, for the S3 web service. The lack of an SLA has always been cited as a “shortcoming” of S3, but I don’t know exactly how many customers have requested it. Enough for them to offer it I guess:

Basically, we commit to 99.9% uptime, measured on a monthly basis. If an S3 call fails (by returning a ServiceUnavailable or InternalError result) this counts against the uptime. If the resulting uptime is less than 99%, you can apply for a service credit of 25% of your total S3 charges for the month. If the uptime is 99% but less than 99.9%, you can apply for a service credit of 10% of your S3 charges.

The SLA is effective as of October 1st, 2007. Jeff makes it sound like they had planned to have an SLA for a long time, but I’m not so sure that’s the case. Doesn’t matter now, they have one!

I think SmugMug’s Don MacAskill makes a good point:

Everything fails sometimes.

The SLA payment is rarely comparable to the pain and suffering your customers had to deal with.

Very true. From my perspective, the SLA isn’t a big deal. I hope it helps Amazon land some more customers though!

Read: Amazon

New Pricing for Amazon S3

Post ImageLate last night Amazon sent an email to S3 customers announcing an upcoming pricing change. Storage costs will remain the same, but the price for bandwidth is going to change:

Current bandwidth price (through May 31, 2007)
$0.20 / GB – uploaded
$0.20 / GB – downloaded

New bandwidth price (effective June 1, 2007)
$0.10 per GB – all data uploaded

$0.18 per GB – first 10 TB / month data downloaded
$0.16 per GB – next 40 TB / month data downloaded
$0.13 per GB – data downloaded / month over 50 TB

$0.01 per 1,000 PUT or LIST requests
$0.01 per 10,000 GET and all other requests

They claim that if the pricing had been applied to usage for March 2007, about 75% of customers would have seen their bill decrease. In some cases however, the price change makes things significantly more expensive, as this thread points out:

Uploading 1GB of 4K files will cost $2.72 instead of $0.20

We haven’t yet figured out how Podcast Spot will be affected, but I suspect we’ll see a slight decrease. I’m also interested to hear from Don MacAskill on SmugMug.

UPDATE: Don talks about the new pricing model here and says they’ll save money.

Read: S3 Forums

Amazon EC2

Post ImageI’ve been meaning to post about this for some time now, but haven’t had a chance. I was really excited last Thursday when I read about Amazon’s new web service called “Elastic Compute Cloud” or EC2 for short. After seeing what they did with S3, I was particularly interested in the how EC2 would fit in. And boy does it ever fit in:

Create an Amazon Machine Image (AMI) containing your applications, libraries, data and associated configuration settings. Or use our pre-configured, templated images to get up and running immediately. Upload the AMI into Amazon S3. Amazon EC2 provides tools that make storing the AMI simple. Amazon S3 provides a safe, reliable and fast repository to store your images.

Nicely integrated with S3. The other great feature? Bandwidth between EC2 and S3 is FREE. I cannot even imagine how much cost savings that could equate to. With EC2, you pay only for instance hours used. Each machine instance is equivalent to “a 1.7Ghz Xeon CPU, 1.75GB of RAM, 160GB of local disk, and 250Mb/s of network bandwidth”. Pretty darn sweet.

I’m already thinking of ways we could integrate this into Podcast Spot (we’re already using and loving S3). I’ve only taken a cursory glance at the forums, API and other documentation, but it seems to me there are two missing features that are extremely desirable: persistent storage and support for Windows (currently it only supports Linux). The AWS guys seem to be pretty on top of things though, so if enough people request them, I’m sure the features will get implemented.

I can’t wait to see what Amazon releases next!

Read: TechCrunch