Amazon S3 keeps getting better, now supports versioning

A good thing really can get better! Amazon S3, perhaps the most well-known cloud computing infrastructure service, just got another upgrade. The simple storage service now supports versioning:

Versioning provides an additional layer of protection for your S3 objects. You can easily recover from unintended user errors or application failures. You can also use Versioning for data retention and archiving.

This new feature will give the thousands of websites and services using S3 a quick and easy to way to support undo or file revision histories, among other things. It kind of moves S3 “up the stack” a little, in that it can now do something that developers could have built themselves, but in a simple and easy-to-use way.

Combine this powerful new functionality with Import/Export that launched last year and a couple of recent price drops, and it’s easy to see why Amazon continues to lead the way. Developers continue to make extensive use of the service too. At the end of Q3 2009, there were over 82 billion objects stored in Amazon S3. Just incredible.

I remember when S3 launched back in March 2006, when I was building Podcast Spot, a hosting service for podcasters. It completely changed our business. Global, scalable storage with Amazon worrying about all the details? And for such a small cost? It seemed too good to be true. I’m thrilled to see that S3 just keeps getting better, with relatively frequent price reductions too.

Tech stuff I’ve been learning about lately

I haven’t done one of these lists in quite a while, so I figured it was time for an update. Here are some of the geeky tech things I’m currently playing with/learning more about:

Facebook Connect
I’ve added support for Facebook Connect to ShareEdmonton, and was surprised at how straightforward it was. Little bit of a learning curve initially, but nothing major. I’m impressed with the way it works.

OAuth
Looking into this one for use with Twitter and Brightkite. I’m not very far along yet, but it also seems fairly straightforward. Check out this ReadWriteWeb article on how a recent OAuth security flaw was addressed by the community.

MSMQ
I’ve been using this since January, actually. It works great, and I wouldn’t hesitate to use it again. I want to explore some of the more advanced uses now.

OpenCalais
You might consider it a semantic web service, because Calais is all about giving meaning to data. I came across it via bit.ly, which uses Calais to automatically extract names, places, and other metadata from the links you shorten.

Google Maps API
Again, this one is fairly straightforward. Lately I’ve been looking into the non-visual services it provides. For example, geocoding and reverse-geocoding in the background, without necessarily having a map on the page.

Public Data Sets on AWS
This one I’ve looked into the least so far, but I’m keen to check it out. In particular, I’d like to explore working with the Freebase data dump. It’s still amazing to me that you can tap into such a vast amount of data with relative ease.

As you can probably guess, almost all of these are related to work I’m doing with ShareEdmonton. I should have an update on that soon.

Mountains of data, right at your fingertips

Last week, two announcements caught my eye. The first was from Amazon.com, which announced that there is now more than 1 TB of public data available to developers through its Public Data Sets on AWS project. The second was from the New York Times, which announced its Newswire API, providing access all NYTimes articles as they are published.

This is a big deal. Never before has so much data been so readily available to anyone. The AWS data is particularly interesting. All of a sudden, any developer in the world has cost-effective access to all publicly available DNA sequences (including the entire Human Genome), an entire dump of Wikipedia, US Census data, and much more. Perhaps most importantly, the data is in machine-readable formats. It’s relatively easy for developers to tap into the data sources for cross-referencing, statistical analysis, and who knows what else.

The Newswire API is also really intriguing. It’s part of a growing set of APIs that the New York Times has made available. With the Newswire API, developers can get links and metadata for new articles the minute they are published. What will developers do with this data? Again, who knows. Imagination is the only limitation now that everyone can have immediate access.

Both of these projects remove barriers and will help foster invention, innovation, and discovery. I hope they are part of a larger trend, where simple access to data becomes the norm. Google’s mission might be to organize the world’s information and make it universally accessible and useful, but it’s projects like these that are making that vision a reality. I can’t wait to see what comes next!

Amazon Web Services: Still getting better

aws logo I often think back to 2006 when Dickson and I were in the midst of the VenturePrize business plan competition. It was around that time that Amazon.com launched their first web service, the Simple Storage Service (S3). It had a huge impact on our business, and we’ve been extremely happy customers ever since.

Over the last couple of years, Amazon has introduced a number of additional web services, the most well-known of which might be the Elastic Compute Cloud (EC2). You can think of it like an on-demand computer in the cloud. I had a quick look at it when it launched, but being a Windows shop, we really didn’t have time to invest the extra effort necessary to get it running. Now, Amazon has announced that EC2 will support Windows:

Starting later this Fall, Amazon Elastic Compute Cloud (Amazon EC2) will offer the ability to run Microsoft Windows Server or Microsoft SQL Server. Our goal is to support any and all of the programming models, operating systems and database servers that you need for building applications on our cloud computing platform. The ability to run a Windows environment within Amazon EC2 has been one of our most requested features, and we are excited to be able to provide this capability. We are currently operating a private beta of Amazon EC2 running Windows Server and SQL Server.

Very cool news for Windows developers. It should put some extra pressure on Microsoft too – though apparently they are getting ready to launch something. Watch for more news on that at PDC.

Another interesting new service that Amazon is introducing is a Content Delivery Service:

This new service will provide you a high performance method of distributing content to end users, giving your customers low latency and high data transfer rates when they access your objects. The initial release will help developers and businesses who need to deliver popular, publicly readable content over HTTP connections.

It will run atop S3, so anything that currently exists there can easily be added to the new content delivery network. This is very cool, and will finally bring world-class CDN infrastructure to small businesses. I wish they had introduced this two years ago!

Those are both very important improvements to AWS. Amazon is raising the bar, again. When will Microsoft, Google, and others answer?

Also – I just noticed recently that Amazon has redesigned the AWS website. It looks fantastic, in my opinion, and is much easier to navigate. Keep the positive improvements coming!

What if Twitter had been built by Amazon.com's Web Services team?

twitter by aws? I’ve been using Twitter for a long time now, and I can’t remember a period of downtime quite as bad as the current one. Features have been disabled, and there’s no ETA for when everything will be back to normal. Who knows, maybe it won’t ever be. Which got me wondering about why Twitter’s reliability is so terrible. Is it the nature of the application, or is it something to do with the people behind Twitter?

What if Twitter had been built by a different team, a team with a pretty good track record for high-availability services? What if Twitter had been built by the Web Services team at Amazon.com?

I think it’s safe to say that things would be quite different:

  1. Reliable, redundant infrastructure
    Twitter would be run inside Amazon’s high-availability data centers. We would never know (or care) that Twitter’s main database was named db006, nor would we ever wonder whether it has a good backup. We’d just know that if it’s good enough for Amazon, it’s good enough for us.
  2. No wondering, “is Twitter working?”
    Instead of wondering if Twitter is working correctly or waiting for Twitter messages or blog posts that explain what the problem is, Twitter would be part of the AWS Service Health Dashboard. We’d be able to see, at a glance, how Twitter is working now, and how well it has worked for the last month. This is what transparency is all about.
  3. Twitter wouldn’t be free, but we’d be cool with that
    Twitter would have had a business model from day one, and we’d all be cheering about how affordable it is. A pay-as-you-go model like all the other web services from Amazon would work quite well for Twitter. You get what you pay for, right?
  4. Premium Support and SLAs
    Speaking of getting what you pay for, Amazon would likely have realized that there are lots of different types of users, and they’d react accordingly. We’d probably have Premium Support for Twitter, to service support requests more efficiently. We’d also have Service Level Agreements.
  5. We wouldn’t call it Twitter…
    Of course, the service wouldn’t be called Twitter. In keeping with Amazon’s other services it would probably have a name like “Amazon Simple Messaging Service”, or SMS for short. Though I suppose that acronym is already taken!

I am a huge Twitter fan, and I really do hope that Ev, Biz, Jack, and the rest of the team get things working and fixed. With every passing hour of downtime though, I lose a little bit of faith. I wonder if Twitter would be better off in someone else’s hands.

Of course, if Twitter really had been built by AWS, there would be far more differences than just the items in my list above. The service may not be recognizable as Twitter!

That doesn’t mean that they couldn’t adopt some of these items as improvements, however. I’d love to see an official Twitter health dashboard, for instance. One can hope.

Maybe Microsoft should buy Amazon instead

The Microsoft-Yahoo deal continues to be the hot topic in the blogosphere right now, with Techmeme still dominated by related discussion. The latest news is that Google has posted an official response to the proposed takeover. In general, discussion has moved from “can you believe what just happened” to “this deal with fail/succeed because…” If you read only two posts on the topic, read this one from Fake Steve Jobs and this one from Henry Blodget.

I really have no idea how this is going to play out. Based on what I’ve read, it seems pretty likely that Microsoft will successfully acquire Yahoo. Many think the deal is as good as done. Far less certain, however, is whether they can make the acquisition a success. It could go either way.

I think what’s clear is that this is a strategy change for Microsoft. A bold recognition that they need to succeed on the web. They trail Google in both search and advertising, and it makes a certain amount of sense that combining with Yahoo will create a stronger competitor.

Microsoft is a platform company. Their cash cow is Windows, the most widely used technology platform in history. They are good at platforms. If the strategy shift is to the web, shouldn’t it be slanted towards a great platform?

amazonawsSuch as Amazon’s Web Services platform. On Wednesday Amazon announced their fourth quarter earnings, and shared this tidbit about Amazon Web Services (AWS):

Adoption of Amazon Elastic Compute Cloud (EC2) and Amazon Simple Storage Service (S3) continues to grow. As an indicator of adoption, bandwidth utilized by these services in fourth quarter 2007 was even greater than bandwidth utilized in the same period by all of Amazon.com’s global websites combined.

ReadWriteWeb has a good discussion of what this means.

Obviously Microsoft isn’t a retailer, and owning Amazon.com itself probably isn’t in the company’s best interests. It could acquire the company for AWS and spin off the rest, however. I suppose Microsoft could just try to duplicate what Amazon has already accomplished with AWS, but why bother? Grab the early market leader and take it to the next level.

I think AWS is an indication of what the platform of the future will look like. Microsoft would be wise to pay attention.

Amazon now offers an SLA for S3

Post ImageAmazon announced on Monday the launch of an SLA, or Service Level Agreement, for the S3 web service. The lack of an SLA has always been cited as a “shortcoming” of S3, but I don’t know exactly how many customers have requested it. Enough for them to offer it I guess:

Basically, we commit to 99.9% uptime, measured on a monthly basis. If an S3 call fails (by returning a ServiceUnavailable or InternalError result) this counts against the uptime. If the resulting uptime is less than 99%, you can apply for a service credit of 25% of your total S3 charges for the month. If the uptime is 99% but less than 99.9%, you can apply for a service credit of 10% of your S3 charges.

The SLA is effective as of October 1st, 2007. Jeff makes it sound like they had planned to have an SLA for a long time, but I’m not so sure that’s the case. Doesn’t matter now, they have one!

I think SmugMug’s Don MacAskill makes a good point:

Everything fails sometimes.

The SLA payment is rarely comparable to the pain and suffering your customers had to deal with.

Very true. From my perspective, the SLA isn’t a big deal. I hope it helps Amazon land some more customers though!

Read: Amazon

Facebook continues to strengthen The Platform

Back in July I wrote about Microsoft’s so-called Cloud OS. There hasn’t been any Microsoft-specific news since then (that I’ve come across anyway) but more and more companies seem to be gearing up to offer cloud infrastructure services. Take Nirvanix for instance, an Amazon S3 competitor that launched earlier this month with some impressive features.

And today, the blogosphere is buzzing about Facebook potentially getting into the cloud services game (some might argue that they already are). Rev2 reports that Facebook is preparing to offer data storage services:

At this stage it seems unclear as to what the precise data storage offering from Facebook is going to be. The Developer wiki indicates that the new service is in Beta, however, there are no indications around more specific details such as space limitations. Costs are also not revealed so one could assume that the data storage offered may be free for a while whilst the service is still in Beta.

AllFacebook has some interesting discussion on the topic, and Read/WriteWeb notes that the service is somewhat in line with Facebook’s earlier acquisition of Parakey.

This is pretty intriguing news on it’s own, but it gets better. At the TechCrunch40 conference today, Facebook CEO Mark Zuckerberg announced that the company is launching a venture fund called fbFund:

The size of the fund will be $10 million with anywhere between $25 to $250 thousand in grants available for each selected startup dedicated to developing Facebook applications. Founders Fund and Accel will get the right of first refusal for the first round of financing of any company in the fund.

Facebook created The Platform, and everyone went crazy. Anyone could develop an application that would run on The Platform, as long as they invested their own infrastructure, time, and money. Today Facebook took steps to eliminate two of those hurdles. Pretty soon, all you’ll need to invest is time.

I don’t think it’s wise to base your entire business around a Facebook application, but people will do it, and quite a few people will probably make money from it. The really good applications may even be able to transcend Facebook’s walled garden.

Looks like The Platform is just getting started.

Read: Rev2

Amazon FPS: another US-only payments service

Post ImageAmazon launched another web service on Friday, called the Amazon Flexible Payments Service (FPS). The interesting thing about the launch is that Jeff Barr was teasing everyone on Twitter, building up the anticipation. I was eagerly watching for updates! I wish more product launches happened in a similar fashion.

Anyway, here is how Jeff described FPS:

We’ve taken all that we know about dealing with credit cards, bank accounts, fraud checking and customer service and wrapped it all up into one convenient package.

In much the same way that S3 and EC2 allow developers to forget about leasing space in data centers, buying servers and negotiating for bandwidth, FPS shields developers from many of the messy and complex issues which arise when dealing with money. Once again, we take care of the “muck” and developers get to focus on being innovative and creative.

As you know, I love S3. It shouldn’t be much of a surprise then when I say that FPS gets me really, really excited. I haven’t looked at it too closely, but even a quick glance suggests that FPS is amazing. PayPal is the giant of the payments space, but they badly need some competition. Unfortunately, no one seems prepared to provide it. No, not even FPS.

The problem is geography. There’s competition for US merchants, but everyone else is left out in the cold. Just like Google Checkout, FPS has launched with support limited to companies in the US (though buyers can be anywhere…and Google has since added UK merchant support but that’s it). There are certainly a lot of Internet users in the United States, but they still make up less than a quarter of the world total, and that percentage drops every day. There is a ton of e-commerce taking place outside the US.

PayPal is supported in over 190 countries and regions. There are 14 countries that PayPal describes as “localized” and another 21 that support withdrawals to local bank accounts. For merchants in 34 of these countries (ignoring the US), Google Checkout and Amazon FPS simply are not options. PayPal is the only viable choice.

And don’t get me wrong, I quite like PayPal, but there are definitely things that need to be improved. The FPS feature set addresses a wide range of these issues (such as support for micro-payments). But Amazon FPS and Google Checkout will simply not beat PayPal until they are supported just as widely around the world. Maybe they don’t want to take on PayPal, you say? Bullshit, I reply. PayPal is the target, even if Amazon and Google will only admit that behind closed doors.

I would suggest there are only five companies that could build a payment system to rival PayPal. The first two are eBay and Amazon, the giants of e-commerce. eBay owns PayPal, and Amazon now has FPS. The other three are the usual suspects: Google, Yahoo, and Microsoft. Google has Checkout, and Yahoo seems to be fine with PayPal. That leaves Microsoft. A small part of me is hopeful for a wicked payments service as part of Windows Live Core, but Microsoft’s track record with launching worldwide services isn’t so great either.

Will there ever be a viable, worldwide competitor to PayPal? I hope so, but I’m not holding my breath.

Read: Amazon FPS

Look out Amazon: Here comes Microsoft's Cloud OS

Post ImageIf you’ve been following the news about Windows Live Core, also referred to as “Cloud OS”, then today’s article at News.com probably is nothing new. Still, it’s a good overview of what is known thus far:

In addition to making available its existing services, such as mail and instant messaging, Microsoft also will create core infrastructure services, such as storage and alerts, that developers can build on top of. It’s a set of capabilities that have been referred to as a “Cloud OS,” though it’s not a term Microsoft likes to use publicly.

It’s about time.

Just as Windows handles memory management, storage, and other “fundamentals” as Microsoft likes to call them, for your computer, Cloud OS would handle the fundamentals for distributed web apps. Developers can spend less time worrying about how the application will run, and more time building out what the application should do. That’s a win-win for everyone.

The most visible company doing this today is Amazon.com with their web services. Services like S3 for storage and EC2 for computing power are tremendously valuable for developers. As I’ve said before, S3 made a big difference for Podcast Spot, and we’d like to use more services if only they existed.

I know there’s going to be pushback, simply because it’s Microsoft, but I am happy to finally see more news on Cloud OS. I can’t wait until the services are available – competition will explode.

It’s a buzzword, and not a very meaningful one at that, but I think this guy has it right:

IMHO, the cloud is the true Web 3.0 – the distributed, ubiquitous web.

Let the Web 3.0 wars being! Microsoft, Google, Amazon, maybe even Yahoo, all competing to provide the best cloud infrastructure for developers to build amazing applications and experiences on top of.

Read: News.com