Archive for the Uncategorized Category

AWS re:Invent

Posted in Uncategorized on November 10, 2014 by swaminathans


This week is the busiest week for many of us in AWS – this is the re:Invent time. Our annual cloud conference that will be packed with series of announcements and some of the best talks.  I just wanted to highlight some of the talks:

  • Keynotes: Needless to say, Wednesday and Thursday keynotes by Andy and Werner will be filled with customer highlights, announcements etc.
  • Spotlight talks: Spotlight talks are an exciting set of talks. These talks are meant to be technically deep and informative for attendees. Some of the speakers are our Distinguished engineers like James Hamilton, Allan Vermuellen (Creator of S3). A few spotlight talks to watch out (note all are live streamed, please register here):
    • SPOT 301: James Hamilton’s talk on AWS Pace of Innovation. James is an awesome speaker and he is going to talk some of the recent innovations in Networking and databases.
    • SPOT 302:  (Shameless plug) This is a talk I am giving with Allan Vermuellen (VP/Distinguished Engineer). Allan built Amazon S3 and is a good friend. We will be talking about core distributed systems primitives that form the basis for various AWS services. Al and I have worked together now for 8 years and built various core distributed systems together and will be fun to talk with him on distributed systems – if you are interested in large scale systems stop by!
    • SPOT 303: This is a conversation talk where we talk to some of our top AWS customers about their best database practices. If you want to hear from how some of our top AWS customers are building stateful services on AWS, their best practices on how they choose databases, their reliability, scalability, and operational best practices. I expect this to be a good learning experience for both experts and beginning customers.
    • SPOT 305: This is a talk I am very excited to see where Khawaja (who is an awesome speaker) and Marvin will be talking through building event driven applications on AWS. They have a cool demo as well!

There are still too many talks I couldn’t list here – otherwise this post will run for multiple pages :). Take a look at detailed agenda here.

If you are in reinvent and want to chat about AWS, Databases or Big data, or just want to meet my team that builds DynamoDB or ElastiCache to just say “Hi”  – stop by the AWS Databases booth or please email me:

Amazon’s 1st Annual Ph.D. Symposium

Posted in Uncategorized on October 2, 2013 by swaminathans

The Amazon’s University Programs is pleased to announce the first annual PhD Research Symposium! Within Amazon there are many applications for research science including Operations Research, Distributed Systems, Storage, Machine Learning and Computer Vision.

The Symposium will be held at the Amazon headquarters in Seattle, WA on Friday, November 22nd. The conference will feature talks by Amazon researchers and PhD students and a poster session for students to exhibit their research. The day will conclude with an evening mixer and dinner. The audience will include Amazon Researchers and members of Amazon’s Principal Engineering Community.

Amazon groups presenting will include:

·         Machine Learning

·         Supply Chain

·         Transaction Risk Management

·         Digital Products

·         Amazon Web Services

The symposium is currently accepting applications from students wishing to attend. Travel and hotel expenses will be covered. The deadline for submission is October 9th.

You are invited to apply to attend the conference and also provide a 60 minute talk or a poster for the afternoon poster session.

Application Requirements:

Applications should be no longer than 300 words

·         A brief description of why you would like to attend the symposium (no longer than 300 words)

·         If you would like to participate in delivering a lecture (60 minutes in length),  include a brief abstract of the lecture topic

·         If you would like to display a poster, include an abstract of the lecture topic

·         A current CV no longer than 6 pages including publication history and links to papers if available

Applications should be sent to Please submit your application in the form of a pdf or MS Word document. Please also add the following fields to the subject line of your email: (1) “Application for PhD Symposium” (2) Your current field of study, (3) Your current School.

Accepted attendees will receive notice by October 18th.

A quick recap..

Posted in Uncategorized on April 19, 2013 by swaminathans

I got a few emails from friends asking why I am not blogging anymore. So, I owe folks an explanation:).

Yes, it has been 10 months since I blogged. Time flies given how busy things are in Amazon. Moreover, given how bad my back got injured, I had to take a month off to recover hence completely had to take my hands off the keyboard for a while. 

I wanted to recap a few significant features my team launched in the past few months:

DynamoDB Local secondary indexes: Starting today, DynamoDB provides our customers the ability to query on non-primary key attributes (within a hash bucket). We call this feature, local secondary indices. Folks who have built partitioned datastore on relational databases using hash based partitioning will be familiar with the pattern of running scoped complex queries within a hash partition. Local secondary indexes let you do just that. We have seen a huge amount of developers migrating their database from relational databases to DynamoDB and this feature will help this migration lot sooner.

DynamoDB Price reduction: We dropped a 3 datacenter replicated SSD-backed storage device by 75% to $0.25 per GB-month and the throughput pricing by 85%. I am excited to be part of the team that has been working hard over the past year to improve storage density and bring down the costs of our underlying hardware platform. We have also made significant improvements to our software by optimizing our storage engine, replication system and various other internal components. 


Auto-discovery in Elasticache:  A common challenge in distributed systems is discovery of new members. For instance, in the Dynamo paper, I talked about using gossip based protocols for discovery. However, one of the things we found when we talked to our customers using memcache (in our Elasticache service) is that the problem of auto discovery is not solved for caching layers. For instance, a cache cluster membership is hardcoded in the form of config files in client boxes. When a cache node is added or removed, the onus is on customers to update the membership configuration files. To get around this manual step, a few folks resorted to running some memcache proxy like moxi which introduced possibly additional latency and bottlenecks. Instead, systems like Dynamo (see my earlier paper) use client libraries that “learn” the latest membership and appropriately reroute. We employed a similar technique in Elasticache where we built a smart membership solution that periodically exchanges this information with its cache clients. This enables customers to automatically “discover” change in cache membership. 

Long Polling with SQS: The Amazon Simple Queue Service (SQS) is a highly scalable, reliable and elastic queuing service that ‘just works’. It has been widely used by our customers and has been a critical building block for building asynchronous distributed systems. With long polling, SQS instead waits for a message to become available and sends it to the client if the message arrives within a customer-defined time period.

I am really proud of what my team has done in the past few months and I am really excited given what we have coming in the next few months. These are exciting times!




Amazon DynamoDB

Posted in Uncategorized on February 16, 2012 by swaminathans

I’m four weeks late to blog about this as I was busy with launching this service but wanted to cover this in my blog:

I’m super excited to tell folks that we launched Amazon DynamoDB, a fully managed NoSQL database service that provides fast and predictable performance with seamless scalability. We have lots of interesting materials and getting started guides. A few things I encourage to take a look at are:

Detail page:

Introduction video:

Werner’s blog:

Hadoop/EMR integration HOWTOs:

I don’t want to rehash things stated in these links but wanted to make sure I provide a simple link.

We are also talking about Amazon DynamoDB in a few events in the next couple of weeks:

1. NoSQL meetup in Seattle on Feb 22

2. Strata 2012.

Hope to see many of you in these venues!



Synchronous vs. Asynchronous replication strategy: Which one is better?

Posted in Uncategorized on September 22, 2011 by swaminathans

If you’re a developer who is looking to pick a highly available database or datastore for your application, you’ve to think of various parameters to make the right choice for your application. Some of the questions you need to hash out include “What query model do I want and what does the datastore offer?”, “What is its consistency guarantees?”, “What is its reliability and availability SLA?” and “What are its scaling dimensions?”. While the process of picking the right datastore warrants a post on its own, I wanted to focus on one specific parameter of this question for this post.

The focus of this post is: “How does a datastore replicate data? Is replication done lazily or synchronously?

One might ask: Why should an application developer (or any datastore consumer) care how a datastore replicates data? The reality is that how a system replicates data has great impact on the datastore’s durability and availability.

Traditionally there are two kinds of replication techniques: synchronous replication and asynchronous replication. A datastore using synchronous replication does not acknowledge a write call until it is acknowledged by its replicas (usually a majority of the replicas).  Examples of pratical datastore that does synchronous replication include Amazon Dynamo (and many of its variants like Cassandra, Voldermort and Riak), Amazon SimpleDB, Amazon RDS Multi-AZ MySQL and Google AppEngine’s High Replication Datastore. On the other hand, a datastore that replicates data asynchronously propagates data to its replica, i.e., when one of the replica gets a write call it will acknowledge to the client right away and in the background it will propagate the writes to its replicas. Examples of systems that use this model include MySQL replication, MongoDB and Google AppEngine’s Master-slave datastore.

Which one is better? Well, it really depends on what you’re planning to use it for. Datastores that synchronously replicates data ensures provides higher durability and availability. Why? Consider the failure scenario where the replica that acknowledged that even if the replica that took a write dies as soon as it acknowledged to the clients, the data is not lost. Clients can read their last acknowledged writes by accessing from the other available replicas. However, in an asynchronously replicated systems, it is not possible for clients to know if their writes were propagated to the other replicas before the failure of the replica that coordinated the writes. If the replica has failed permanently (due to a permanent node failure or so), then you can experience a data loss where you lost the latest updates (durability issue). If the replica has failed temporarily, then clients can still access an old version of the data from the other available replicas. However, they will not be able to perform any writes unless they are willing to deal with merging conflicting updates when the replica comes back online.

So, why would anyone run the risk of picking a lower durable datastore and pick an asynchronously replicated datastore? The reason is because asynchronously replicated datastore provides reduced write latency (since a write is acknowledged locally by a single before being written to other replicas, the write latency is lower) and better read scaling (as adding more replicas does not impact your latency like in the case of synchronously replicated systems).

So, in a nutshell, the rule of thumb I tend to use in picking between these two techniques is: If your application requires high durability and availability, pick a synchronously replicated datastore solution. If your application is OK with losing a small % of writes (e.g., a soft-state or caching system) due to a failed replica (usually the master replica), then you are OK with asynchronous replication system. A great example is how MySQL users use read replicas to do read scaling. Another great example for using asynchronous replication is to use your asynchronous replica as the backup copy (used for disaster recovery).

Filtering effects in cache hierarchies

Posted in Uncategorized on August 31, 2011 by swaminathans

A friend of mine was building a multi-tiered application with a cache fronting a database and was investigating why his database memory hit ratio was not as high as he expected it to be and how to optimize it. This matters only when you are paranoid about latency percentiles at TP99 or TP99.9s. One of the things, he found after a deeper investigation is what he called as “cherry picking” done by the caching tier that the next level did not see requests with high cache locality. A similar observation was made in Web cache research and was referred to filtering effects.

One of the important works done in this area is by Carey Williamson on “On Filter Effects in Web cache hierarchies”. This paper uses trace driven simulation to look into effects of what happens to workload locality when it travels through various cache hierarchies. Important results to call out:
– Proxy caches filter out the most popular documents from a workload. The filtering effect typically changes the Zipf-like document popularity profile of the input stream into a two-part piecewise-linear document popularity profile for the output stream. The upper leftmost part of the plot is significantly flattened.
– The percentage of unique documents referenced and the percentage of onetimer documents tend to increase as the request stream passes through a cache.
– Among the caches in a caching hierarchy, the first-level cache has the most pronounced filtering effect on the workload.

Even though this is a 10 year old work, highly recommended read for folks trying to understand the effects on caching on their systems.

Amazon ElastiCache

Posted in Uncategorized on August 23, 2011 by swaminathans

Last night, my sister team in AWS launced a service I’m very excited about: Amazon Elasticache. Historically, caching has been the one of the most widely used techniques to build scalable web applications where the caches store the most often accessed computation results which take longer (or is harder) to re-compute in the source. In-memory caches are normally used to front databases so that often accessed results can be retrieved from memory faster (see examples of how to use MySQL and memcache together, here). However, to ensure that in-memory cache do not become a scalability bottleneck themselves, distributed cache clusters use techniques like distributed hash tables (DHTs) to ensure that cache cluster can be “scaled out”. As the scale of caching system becomes harder, it is a challenge to manage them in a large scale environment.

Today, AWS has made the process of running a cache cluster easier with a new managed cache offering called Elasticache. A quote from the detail page sums it up well:

“Amazon ElastiCache is a web service that makes it easy to deploy, operate, and scale an in-memory cache in the cloud. Amazon ElastiCache is protocol-compliant with Memcached, a widely adopted memory object caching system, so code, applications, and popular tools that you use today with existing Memcached environments will work seamlessly with the service. Amazon ElastiCache simplifies and offloads the management, monitoring, and operation of in-memory cache environments, enabling you to focus on the differentiating parts of your applications.”

Congratulations team!