Amazon DynamoDB

Last week, Amazon announced the launch of a new product, DynamoDB. Within the same day, Mitch Garnaat quickly released support for DynamoDB in Boto. I quickly worked with Mitch to add on some additional features, and work out some of the more interesting quirks that DynamoDB has, such as the provisioned throughput, and what exactly it means to read and write to the database.

One very interesting and confusing part that I discovered was how Amazon actually measures this provisioned throughput. When creating a table (or at any time in the future), you set up a provisioned amount of "Read" and "Write" units individually. At a minimum, you must have at least 5 Read and 5 Write units partitioned. What isn't as clear, however, is that read and write units are measured in terms of 1KB operations. That is, if you're reading a single value that's 5KB, that counts as 5 Read units (same with Write). If you choose to operate in eventually consistent mode, you're charged for half of a read or write operation, so you can essentially get double your provisioned throughput if you're willing to put up with only eventually consistent operations.

Ok, so read operations are essentially just look-up operations. This is a database after all, so we're probably not just going to be looking at looking up items we know, right?

Wrong.

Amazon does offer a "Scan" operation, but they state that it is very "expensive". This isn't just in terms of speed, but also in terms of partitioned throughput. A scan operation iterates over every item in the table, It then filters out the returned results, based on some very crude filtering options which are not full SQL-like, (nothing close to what SDB or any relational database offers). What's worse, a single Scan operation can operate on up to 1MB of data at a time. Since Scan operates only in eventually consistent mode, that means it will use up to 500 Read units in a single operation (1,000KB items/2 (eventually consistent) = 500). If you have 5 provisioned Read units per second, that means you're going to have to wait 100 seconds (almost 2 minutes) before you can perform another Read operation of any sort again.

So, if you have 1 Million 1KB records in your Table, that's approximately 1,000 Scan operations to perform. Assuming you provisioned 1,000 Read operations per second, that's roughly 17 minutes to iterate through the entire database. Now yes, you could easily increase your read operations to cut that time down significantly, but lets assume that at a minimum it takes at least 10ms for a single scan operation. That still means the fastest you could get through your meager 1 Million records is 10 seconds. Now extend that out to a billion records. Scan just isn't effective.


So what's the alternative? Well there's this other very obscure ability that DynamoDB has, you may set your Primary Key to a Hash and Range key. You always need to provide your Hash Key, but you may also provide the Range Key as either Greater then, Less then, Equal To,  Greater then or equal to, Less then or equal to, Between, or Starts With using the Query operation.

Unlike Scan, Query only operates on matching records, not all records. This means that you only pay for the throughput of the items that match, not for everything scanned.

So how do you effectively use this operation? Simply put, you have to build your own special indexes. This lends itself to the concept of "Ghost Records", which simply point back to the original record, letting you keep a separate index of the original for specific attributes. Lets assume we're dealing with a record representing a Person. This Person may have several things that identify it, but lets use a unique identifier as their Hash key, with no Rage key. Then we'll create several separate Ghost records, in a different table. Lets call this table "PersonIndex".

Now if we want to search for someone by their First Name, we simply issue a query with a Hash Key of property = "First Name", and a range Key of the first name we're looking for, or even "Starts With" to match things like "Sam" to match "Samuel". We can also insert "alias" records, for things like "Dick" to match "Richard". Once we retrieve the Index Record, we can use the "Stories" property to go back and retrieve the Person records.

So now to search for a record it takes us  Read operation to search, and 1 Read operation for each matching record, which is a heck of a lot cheaper then one million! The only negative is that you also have to maintain this secondary table of Indexes. Keeping these indexes up to date is the hardest part of maintaining your own separate indexes. however, if you can do this, you can search and return records within milliseconds instead of seconds, or even minutes.


How are you using or planning to use Amazon DynamoDB?

What's coming up for Amazon Web Services Cloud?

Amazon announced today the new upcoming Live streaming event for What's up and coming in AWS for 2012. It's not surprising since they've recently hired a ton of new people, including Mitch Garnaat, creator of Boto and former co-worker of mine. They've recently been launching new services all over the globe, including new regions and CloudFront/Route53 Endpoints. With the additional new Direct Connect locations, it's also easier then ever to build a Hybrid cloud.

They've also been launching new services, such as ElastiCache, a memcached service offered up directly from Amazon. And of course it wouldn't be amazon without dozens of improvements to existing services, such as finally enabling console logins for IAM users, and vast improvements to Amazon SNS. 

So what then really can we expect at this new announcement? A few things have been on the radar for quite a while now:

• Location-Aware Route53 (DNS) - Make sure that users get the closest server to where they are
• Two-Factor Auth for IAM Users
• APN (Apple Push Notifications) support for SNS
• "Pass Through" option for CloudFront (i.e. don't cache certain URLs)
• Lots of improvements for SimpleDB, such as SQL functions and cross-domain joins.

What do you think? What other improvements have you been waiting on for AWS?

Monitor your SDB Domains

This recently came up in the Boto-Users mailing list, so I thought I'd post here a few quick details on how to monitor your SimpleDB domains to prevent them from hitting maximum capacity before you know it.

As you should be aware, SimpleDB has a limit of 10GB per domain. This limit is calculated as a sum of the bytes used by Item Names, Attribute Names (unique), and Attribute Values. Fortunately, Attribute Names are only stored once per name, so you don't pay for each name being used multiple times, they just charge you per unique name.

You can get all of the Usage information about your SDB Domain using the get_metadata function of a boto domain.

>>> import boto
>>> sdb = boto.connect_sdb()
>>> db = sdb.lookup("my-domain")
>>> md = db.get_metadata()


This "md" object then contains the following elements:


md.item_names_size
md.attr_names_size
md.attr_values_size
md.item_count


I wrote a simple script to check my domains, which takes an optional list of arguments for domain names to check. If you dont' pass in a domain name, it will iterate over all of them and show you any domain that uses more then 3GB:



#!/usr/bin/env python
"""
Check script to make sure none of our domains are close to the size limit
"""
import boto
if __name__ == "__main__":
   import sys
   sdb = boto.connect_sdb()
   if len(sys.argv) > 1:
      query = [sdb.lookup(n) for n in sys.argv[1:]]
      limit = 0
   else:
      query = sdb.get_all_domains()
      limit = 3000000000
   for db in query:
      md = db.get_metadata()
      total = int(md.item_names_size) + int(md.attr_names_size) + int(md.attr_values_size)
      if total > limit:
         print db.name
         print "\tItems:", md.item_count
         print "\tItem Name Size:", md.item_names_size
         print "\tAttribute Name Size:", md.attr_names_size
         print "\tAttribute Values Size:", md.attr_values_size
         print "\t---------------------------------------------"
         print "\tTOTAL:", total




If your domain is using more then 10GB of space, you can use this to track down what's using a lot of space. In my case, I was adding a lot of unnecessary items that were almost completely blank, so my item_count was huge, and my item_names_size was over 7GB. 


Of course, if you do have need for all these items, you should consider Sharding your domain into multiple sub-domains. This process is usually handled by taking one attribute that nicely splits your items into different segments, and using that value as the domain name. Unfortunately, you can not query across multiple domains, so you have to be very careful what you choose as your Shard Key.