Archive for April, 2013

A quick recap..

Posted in Uncategorized on April 19, 2013 by swaminathans

I got a few emails from friends asking why I am not blogging anymore. So, I owe folks an explanation:).

Yes, it has been 10 months since I blogged. Time flies given how busy things are in Amazon. Moreover, given how bad my back got injured, I had to take a month off to recover hence completely had to take my hands off the keyboard for a while. 

I wanted to recap a few significant features my team launched in the past few months:

DynamoDB Local secondary indexes: Starting today, DynamoDB provides our customers the ability to query on non-primary key attributes (within a hash bucket). We call this feature, local secondary indices. Folks who have built partitioned datastore on relational databases using hash based partitioning will be familiar with the pattern of running scoped complex queries within a hash partition. Local secondary indexes let you do just that. We have seen a huge amount of developers migrating their database from relational databases to DynamoDB and this feature will help this migration lot sooner.

DynamoDB Price reduction: We dropped a 3 datacenter replicated SSD-backed storage device by 75% to $0.25 per GB-month and the throughput pricing by 85%. I am excited to be part of the team that has been working hard over the past year to improve storage density and bring down the costs of our underlying hardware platform. We have also made significant improvements to our software by optimizing our storage engine, replication system and various other internal components. 


Auto-discovery in Elasticache:  A common challenge in distributed systems is discovery of new members. For instance, in the Dynamo paper, I talked about using gossip based protocols for discovery. However, one of the things we found when we talked to our customers using memcache (in our Elasticache service) is that the problem of auto discovery is not solved for caching layers. For instance, a cache cluster membership is hardcoded in the form of config files in client boxes. When a cache node is added or removed, the onus is on customers to update the membership configuration files. To get around this manual step, a few folks resorted to running some memcache proxy like moxi which introduced possibly additional latency and bottlenecks. Instead, systems like Dynamo (see my earlier paper) use client libraries that “learn” the latest membership and appropriately reroute. We employed a similar technique in Elasticache where we built a smart membership solution that periodically exchanges this information with its cache clients. This enables customers to automatically “discover” change in cache membership. 

Long Polling with SQS: The Amazon Simple Queue Service (SQS) is a highly scalable, reliable and elastic queuing service that ‘just works’. It has been widely used by our customers and has been a critical building block for building asynchronous distributed systems. With long polling, SQS instead waits for a message to become available and sends it to the client if the message arrives within a customer-defined time period.

I am really proud of what my team has done in the past few months and I am really excited given what we have coming in the next few months. These are exciting times!