Amazon CloudFront: Awards

Posted in Uncategorized on October 26, 2010 by swaminathans

Amazon CloudFront is a web service for content delivery that allows you to deliver static and streaming content using a network of edge locations.  Often, these systems are called Content Deliver Networks or CDNs (for more general survey of CDN techniques, see here).

I was fortunate to have had the opportunity to work with folks like Brad Marshall and David Richardson during the initial days of CloudFront, and even more excited to see what they have done in the past couple of years such as adding the support for streaming media and HTTPS support.

Last week, I was excited to see CloudFront winning couple of prestigious awards: Streaming Media – 2010 Editor’s Pick and European Readers Choice awards. I’m glad to see Cloudfront’s contributions being recognized and our customers find it extremely useful. Congratulations, CloudFront team!

On that note, CloudFront team is actively looking for extremely smart engineers who are passionate about large scale distributed systems and networks. If you’re interested, please contact David Richardson.


Taking on a new role in AWS Database Services

Posted in Cloud Computing, NoSQL on October 12, 2010 by swaminathans

As many of you know me and my work, I’ve always been passionate about building large scale distributed systems. I’m glad to have had the opportunity to work in teams that built great systems like Amazon Dynamo, Amazon CloudFront, and Amazon RDS. Moreover, I had the opportunity to learn from some exceptionally smart people like Werner Vogels, Al Vermuellen and James Hamilton.

I am personally thrilled to see the momentum Dynamo has created in the NoSQL world and was personally excited to talk about in the NoSQL meetups. Similarly, I have been amazed at the rapid adoption of RDS and its Multi-AZ features.

So far, in AWS, I’ve been working as a Principal Engineer (aka Architect) in charge of the architecture and implementation of these systems. Recently, I took on a new role to manage and lead the non-relational database services team in AWS in a leadership role. I’ll be leading the SimpleDB team and couple of other internal teams and looking at ways to build scalable data access primitives. I’m super excited to to join this incredibly smart team and deliver some great systems.

On this note, I’m actively hiring for my teams, see here. If you’re interested in joining a team that is building what will be the blueprint for scalable datastores, then please send me a note ( I’m looking for highly talented engineers.

WebApps 11

Posted in Distributed Systems on October 6, 2010 by swaminathans

As many of you, the past 4 years in Amazon, I’ve spent on building out various parts of Amazon and AWS infrastructure that will enable Amazon and other AWS users to build highly available and scalable applications.

This also gives me immense pleasure to be part of the program committee for Usenix WebApps 2011. This conference is primarily focused on building highly scalable and Web applications. Looking forward to reviewing some great submissions and the conference.

I highly recommend submitting your ideas.

CFP details below:

WebApps ’11 Call for Papers

2nd USENIX Conference on Web Application Development (WebApps ’11)

June 15–16, 2011
Portland, OR

WebApps ’11 will take place during USENIX Federated Conferences Week, June 12–17, 2011.

Important Dates

  • Submissions due: January 21, 2011, 11:59 p.m. PST (hard deadline, no extensions, no exceptions, so don’t ask)
  • Notification to authors: March 17, 2011
  • Final files due: May 3, 2011

Conference Organizers

Program Chair
Armando Fox, University of California, Berkeley

Program Committee
Adam Barth, Google Inc.
Abdur Chowdhury, Twitter
Jon Howell, Microsoft Research
Collin Jackson, Carnegie Mellon University
Bobby Johnson, Facebook
Emre Kıcıman, Microsoft Research
Michael E. (“Max”) Maximilien, IBM Research
Owen O’Malley, Yahoo! Research
John Ousterhout, Stanford University
Swami Sivasubramanian, Amazon Web Services
Geoffrey M. Voelker, University of California, San Diego
Nickolai Zeldovich, Massachusetts Institute of Technology


The Web is now the dominant platform for delivering interactive applications to hundreds of millions of users. Such applications are now expected to scale effortlessly from tens of users to tens of millions of users in a single day while providing a responsive “always-on” experience. These demands, as well as the new possibilities opened by the proliferation of Web-capable mobile devices, requires that Web apps’ design and operation be elastic, failure-tolerant, and seamlessly scalable, supporting multiple devices and access methods.

Like the inaugural WebApps ’10, WebApps ’11 seeks to attract cutting-edge research that advances the state of the art, not only on novel Web applications but also on infrastructure, tools, and techniques that support the development, analysis/testing, operation, or deployment of those applications.


Possible topics include but are not limited to:

  • Storage for Web-scale applications
  • Techniques for testing and debugging
  • Novel strategies for fault tolerance or high availability in Web apps
  • The Web as an emerging platform in new application areas
  • HCI techniques related specifically to Web apps
  • Measurement, modeling, workload generation, and other tools to aid experimental research on Web apps
  • New and unusual app features or implementation techniques
  • Media delivery applications and infrastructure
  • Client-side libraries, toolkits, plug-ins
  • Server-side frameworks
  • Languages and language engineering advances relevant to Web app development
  • Deployment substrates and technologies (cloud computing, infrastructure as a service, testing as a service, etc.)

More details:


CFP online and in PDF format:

NoSQL meetup

Posted in NoSQL on August 24, 2010 by swaminathans

This week, I’ll be giving a talk to the NoSQL Seattle community about “Building Scalable Distributed Systems”. I’m super-excited to meet this really active community, which is passionate about building and running large scale systems.  I’m also curious to hear about Ben about Riak and about the Hue framework. More details about the meetup can be found here.

See you there on the NoSQL meetup this Wednesday evening!

Just Launched: Amazon RDS Multi-AZ

Posted in Cloud Computing, Distributed Systems on May 18, 2010 by swaminathans

Today, my colleagues and I launched a major feature in Amazon RDS called Multi-AZ DBInstances. A Multi-AZ DB Instance performs synchronous replication of data across replicas located in different availability zones.

What benefits does Multi-AZ DB Instance provide?

–          Higher Durability: Your data is synchronously replicated across different availability zones. This guarantees higher levels of durability even in the wake of a disk, instance or availability zone failures.

–          Higher Availability: With Multi-AZ instances, we perform master-slave replication and when we detect that the master is unavailable, we automatically failover to the slave replica. This guarantees higher levels of availability. Contrary to MySQL’s asynchronous replication, since the data is synchronously replicated, when the replica fails over to the secondary replica you will experience no data loss.

You can learn more about all the resilient goodness of RDS Multi-AZ deployments here.

Building Scalable Social Gaming Platform using Clouds

Posted in Cloud Computing, Distributed Systems on February 18, 2010 by swaminathans

This week, I ran into many social gaming related posts.  First thing that surprised me was the social games are not played predominantly by teens and the average social gamer is around 40s.  This illustrates why these social games attract tens of millions of players every day.

When I read the high scalability article about How Zynga scaled Farmville to Harvest 75 Million Players a month, I was intrigued by their scaling challenges and requirements:

* Read-write ratio: Interactive games are write heavy unless traditional web applications. Seems intuitive when you think about the fact every move is recorded in a datastore.
* Users are disturbed by high latencies and variability in latencies:  Note that I called high latencies and latency variations as two different entities. As we noted in Dynamo, Amazon also cares about the variability in latencies (percentiles) and build our services to make sure we can constantly keep the variability in control.
* Dealing with latency and failure characteristics of external dependencies: These applications need to deal with external platforms like Facebook which may or may not be available all the time.

I like the way Zynga approached this problem:

* For solving heavy writes, looks like they have partitioned heavily. Seems reasonable – however, I’m curious to see how they handled “hot spots” (wherein the most active users are the ones constantly generating more data) and whether simple hash-based partitioning is good enough to spread the hot spots.

* For handling latency variations, they went for isolating each component and built graceful degradation at each layer. This is a common practice in building large scale systems . The thing what I would be curious to see is how “gracefully” does their datastore degrades and also which datastore they are using for that.

* Finally, to deal with failures of external dependancies and still meet latency SLAs, looks like they cache the responses of external dependancies.

These are very good lessons for building scalable systems.

An interesting followup to this, I saw that Rightscale (which is apparently helping Zynga run on top of AWS) is using its expertise to offer a social gaming platform for other aspiring Zyngas out there! Seems like an exciting internet industry at its really nascent stages.

Amazon RDS

Posted in Cloud Computing, Distributed Systems on October 27, 2009 by swaminathans

Today, my team launched Amazon RDS, a new AWS service that offers relational database in the AWS Cloud. Amazon RDS provides a managed relational database in the cloud and the service does the heavy lifting done by DBAs such as provisioning, monitoring database health, managing automated backups, point-in-time recovery, creating/restoring snapshots, adding storage space on the fly, changing compute power for your database, etc.., all through simple Web service calls. Get started in few minutes!

A Word about Persistence: One size does not fit all.

The scalability, reliability and performance of a platform is heavily dependent on how it manages their data. In Amazon, we believe that when it comes to persistence: one size does not fit all. Pick the right data management platform based on your application’s needs. For people, who require a highly available and highly durable key-value store, we have Amazon S3. For people requiring raw disk in the cloud – we have Amazon EBS. For, people who want to store and query structured data in the cloud and don’t want to actively manage its scalability, we have Amazon SimpleDB.

One of the biggest features our customer has been asking us has been to provide a managed relational database service in the cloud. The need for their relational database can be due to various reasons: programming familiarity, dealing with existing applications already built for relational databases, need for complex transactions and joins which are relational databases’ forte. If you fall in this category, you will be happy with Amazon RDS.

For more details on Amazon RDS, take a look at: