{"id":18219,"date":"2017-08-11T12:15:18","date_gmt":"2017-08-11T09:15:18","guid":{"rendered":"https:\/\/www.webcodegeeks.com\/?p=18219"},"modified":"2017-08-11T10:09:57","modified_gmt":"2017-08-11T07:09:57","slug":"working-with-dynamodb","status":"publish","type":"post","link":"https:\/\/www.webcodegeeks.com\/web-development\/working-with-dynamodb\/","title":{"rendered":"Working with DynamoDB"},"content":{"rendered":"

Recently, I worked on an IoT-based project where we had to store time-series data coming from multiple sensors and show real-time data to an enduser. Further, we needed to generate reports and analyze gathered information.<\/p>\n

To deal with the continuous data flowing from the sensors, we chose DynamoDB for storage. DynamoDB promises to handle large data with single-digit millisecond latency, at any scale. Since it\u2019s a fully managed database service, we never had to worry about scaling, architecture, and hardware provisioning. Plus, we were using AWS IoT for sensors, so choosing a NoSQL database from AWS services like DynamoDB was the right decision.<\/p>\n

Here is what Amazon says about DynamoDB:<\/p>\n

Fully managed NoSQL database service that provides fast and predictable performance with seamless scalability.<\/p><\/blockquote>\n

The Basics<\/h2>\n

DynamoDB tables are made of items<\/em>, which are similar to rows<\/em> in relational databases, and each item<\/em> can have more than one attribute. These attributes can be either scalar types or nested.<\/p>\n

Everything starts with the Primary Key Index<\/h3>\n

Each item has one attribute as the Primary Key Index that uniquely identifies the item. The Primary Key is made of two parts: the Partition Key (Hash Key) and the Sort Key (Range Key); where the Range Key is optional. DynamoDB doesn\u2019t just magically spread the data into multiple different servers to boost performance, it relies on partitioning to achieve that.<\/p>\n

Partitioning is similar to the concept of sharding seen in MongoDB and other distributed databases where data is spread across different database servers to distribute load across multiple servers to give consistent high performance. Now think of partitions as similar to shards and the Hash Key<\/em> specified in the Primary Key determines in which shard the item will be stored.<\/p>\n

In order to determine in which partition the item will be stored, the Hash Key is passed to a special hash function which ensures that all items are evenly spread across all partitions. This also explains why it is called a Partition Key or Hash Key. The Sort Key on the other hand, determines the order of items being stored and allows DynamoDB to have more than one item with the same Hash Key.<\/p>\n

The Sort Key, when present and combined with the Partition Key (Hash Key) forms the Primary Key Index, which is used to uniquely identify a particular item.<\/p>\n

This is very useful for time series data such as the price of stocks, where the price of one stock item changes over time and you need to track the price in comparison to the stock. In such cases, the stock name can be the Partition Key and the date can be used as a Range Key to sort data according to time.<\/p>\n

Secondary indexes<\/h3>\n

Primary indexes are useful to identify items and allow us to store infinitely large amounts of data without having to worry about performance or scaling, but soon you will realize that querying data becomes extremely difficult and inefficient.<\/p>\n

Having worked with relational databases mostly, querying with DynamoDB was the most confusing aspect. You don\u2019t have joins or views as in relational databases; denormalization helps, but not much.<\/p>\n

Secondary indexes in DynamoDB follow the same structure as the Primary Key Index, where one part is the Partition Key and the second part is the Sort Key, which is optional. Two types of secondary indexes are supported by DynamoDB: the Local Secondary Index and the Global Secondary Index.<\/p>\n

Local Secondary Index (LSI):<\/h3>\n

The Local Secondary Index is a data structure that shares the Partition Key defined in the Primary Index, and allows you to define the Sort Key with an attribute other than the one defined in the Primary Index. The Sort Key attribute must be of scalar type.<\/p>\n

While creating the LSI, you define attributes to be projected other than the Partition Key and the Sort Key, and the LSI maintains projected attributes along with the Partition Key and Sort Key. The LSI data and the table data for each item is stored inside the same partition.<\/p>\n

Global Secondary Index (GSI):<\/h3>\n

You will need to query data from a different attribute than the Partition Key. You can achieve this by creating a Global Secondary Index for that attribute. GSI follows the same structure as the Primary Key, though it has a different Partition Key than the Primary Index and can optionally have one Sort Key.<\/p>\n

Similar to the LSI, attributes to be projected need to be specified while creating the GSI. Both the Partition Key attribute and Sort Key attribute need to be scalar.<\/p>\n

You definitely should look up the official documentation for GSI<\/a> and LSI<\/a> to understand how indexes work.<\/p>\n

!Sign up for a free Codeship Account<\/p>\n

Setup for DynamoDB<\/h2>\n

DynamoDB doesn\u2019t require special setup, as it is a web service fully managed by AWS. You just need API credentials to start working with DynamoDB. There are two primary ways you can interact with DynamoDB, using AWS SDK for Ruby<\/a> or Dynamoid<\/a>.<\/p>\n

Both libraries are quite good, and Dynamoid offers an Active Record<\/em> kind of interface. But to get an overview of how DynamoDB works, it\u2019s better to start with AWS SDK for Ruby<\/a>.<\/p>\n

In Gemfile,<\/p>\n

gem 'aws-sdk', '~> 2'<\/pre>\n
First of all, you need to initialize a DynamoDB client, preferably via an initializer so as to avoid instantiating a new client for every request you make to DynamoDB.<\/p>\n
# dynamodb_client.rb\r\n$ddb = Aws::DynamoDB::Client.new({\r\n    access_key_id: ENV['AWS_ACCESS_KEY_ID'],\r\n    secret_access_key: ENV['AWS_SECRET_ACCESS_KEY'],\r\n    region: ENV['AWS_REGION']\r\n})<\/pre>\n
AWS provides a downloadable version of DynamoDB, \u2018DynamoDB local\u2019, which can be used for development and testing. First, download<\/a> the local version and follow the steps specified in the documentation to set up and run it on a local machine.<\/p>\n
To use it, just specify an endpoint in the DynamoDB client initialization hash as shown below:<\/p>\n
# dynamodb_client.rb\r\n$ddb = Aws::DynamoDB::Client.new({\r\n    access_key_id: ENV['AWS_ACCESS_KEY_ID'],\r\n    secret_access_key: ENV['AWS_SECRET_ACCESS_KEY'],\r\n    region: ENV['AWS_REGION'],\r\n    endpoint: 'http:\/\/localhost:8000'\r\n})<\/pre>\n
The DynamoDB local server runs on port 8000 by default.<\/p>\n
Put it all together with this simple example<\/h2>\n
Suppose you need to store the information of users along with shipping addresses for an ecommerce website. The users table will hold information such as first_name<\/code>, last_name<\/code>, email<\/code>, profile_pictures<\/code>, authentication_tokens<\/code>, addresses<\/code>, and much more.<\/p>\n
In relational databases, a users table might look like this:<\/p>\n
\"\"<\/a><\/p>\n
And addresses and authentication tokens will need to be placed separately in other tables with the id of a user as Foreign Key:<\/p>\n
Addresses: \"\"<\/a><\/p>\n
Authentication Tokens: \"\"<\/a><\/p>\n
In DynamoDB, there is no concept of Foreign Keys and no joins. There are ways to reference data related to an item from another table as we do in relational databases, but it\u2019s not efficient. A better way would be to denormalize data into a single users table. As DynamoDB is a key value store, each item in a users<\/code> table would look as shown below:<\/p>\n
{\r\n  \"first_name\": \"string\",\r\n  \"last_name\": \"string\",\r\n  \"email\": \"string\",\r\n  \"created_at\": \"Date\",\r\n  \"updated_at\": \"Date\",\r\n  \"authentication_tokens\": [\r\n    {\r\n      \"token\": \"string\",\r\n      \"last_used_at\": \"Date\"\r\n    }\r\n  ],\r\n  \"addresses\": [\r\n    {\r\n      \"city\": \"string\",\r\n      \"state\": \"string\",\r\n      \"country\": \"string\" \r\n    }\r\n  ]\r\n}<\/pre>\n
Make email<\/code> as the Partition Key of the Primary Key and leave the Range Key optional, as each user will have a unique email id, and we definitely need to look up a user having a particular email id.<\/p>\n
In the future, you might need to search users with first_name<\/code> or last_name<\/code>. This requirement makes first_name<\/code> and last_name<\/code> ideal candidates for the Range Key. Additionally, you may want to get users registered on a particular date or updated on a particular date, which can be found with created_at<\/code> and updated_at fields<\/code>, making them ideal for the Partition Key in the Global Secondary Index.<\/p>\n
For now, we will make one Global Secondary Index (GSI), where created_at<\/code> will be the Partition Key and first_name<\/code> will be Range Key, allowing you to run queries like:<\/p>\n
select users.* where users.created_at = ' xxx ' and users.first_name starts_with(' xxx ')<\/pre>\n
Basic CRUD Operations<\/h2>\n
All logic related to persistence and querying stays in the model, so following that convention, first create the class User<\/code> and include ActiveModel::Model<\/a> and ActiveModel::Serialization<\/a> modules inside the class.<\/p>\n
ActiveModel::Model<\/code> adds callbacks, validations, and an initializer. The main purpose for adding it is to initialize the User<\/code> model with parameters hash like Active Record does.<\/p>\n
ActiveModel::Serialization<\/code> provides serialization helper methods such as to_json<\/code>, as_json<\/code>, and serializable_hash<\/code> for objects of the User<\/code> class. After adding these two modules, you can specify the attributes related to the User<\/code> model with attr_accessor<\/code> method. At this point, the User<\/code> model looks like this:<\/p>\n
# models\/user.rb\r\nclass User \r\n    include include ActiveModel::Model, ActiveModel::Serialization\r\n    \r\n    attr_accessor :first_name, :last_name, :email, :addresses, :authentication_tokens, :created_at, :updated_at\r\nend<\/pre>\n
You can create User<\/code> objects with User.new<\/code>, pass parameters hash, serialize, and deserialize it, but you cannot persist them in DynamoDB. To be able to persist the data, you will need to create a table in DynamoDB and allow the model to know about and access that table.<\/p>\n
I prefer to create a migrate_table!<\/code> class method where I put the logic required for table creation. If the table already exists, it will be recreated, and the application should wait until the table is created as table creation on DynamoDB takes around a few minutes.<\/p>\n
# models\/user.rb\r\n\r\n...\r\ndef self.migrate_table\r\n      $ddb.delete_table(table_name: table_name) if $ddb.list_tables.table_names.include?(table_name)\r\n      create_table_params = {\r\n        table_name: table_name,\r\n        # array of attributes name and their type that describe schema for the Table and Indexes\r\n        attribute_definitions: [\r\n          {\r\n            attribute_name: \"first_name\",\r\n            attribute_type: \"S\"\r\n          },\r\n          {\r\n            attribute_name: \"created_at\",\r\n            attribute_type: \"S\"\r\n          },\r\n          {\r\n            attribute_name: \"email\",\r\n            attribute_type: \"S\"\r\n          },\r\n        ],\r\n\r\n        # key_schema specifies the attributes that make up the primary key for a table\r\n        # HASH - specifies Partition Key\r\n        # RANGE - specifies Range Key\r\n        # key_type can be either HASH or RANGE\r\n        key_schema: [\r\n          {\r\n            attribute_name: \"email\",\r\n            key_type: \"HASH\",\r\n          }\r\n        ],\r\n\r\n        # global_secondary_indexes array specifies one or more keys that makes up index,\r\n        # with name of index and provisioned throughput for global secondary indexes\r\n        global_secondary_indexes: [\r\n\r\n          index_name: \"created_at_first_name_index\",\r\n          key_schema: [\r\n              {\r\n                  attribute_name: \"created_at\",\r\n                  key_type: \"HASH\"\r\n              },\r\n              {\r\n                  attribute_name: \"first_name\",\r\n                  key_type: \"RANGE\"\r\n              }\r\n          ],\r\n\r\n          # Projection - Specifies attributes that are copied (projected) from the table into the index.\r\n          # Allowed values are - ALL, INCLUDE, KEYS_ONLY\r\n          # KEYS_ONLY - only the index and primary keys are projected into the index.\r\n          # ALL - All of the table attributes are projected into the index.\r\n          # INCLUDE - Only the specified table attributes are projected into the index. The list of projected attributes are then needs to be specified in non_key_attributes array\r\n          projection: {\r\n              projection_type: \"ALL\"\r\n          },\r\n\r\n          # Represents the provisioned throughput settings for specified index.\r\n          provisioned_throughput: {\r\n              read_capacity_units: 1,\r\n              write_capacity_units: 1\r\n          }\r\n        ],\r\n\r\n        # Represents the provisioned throughput settings for specified table.\r\n        provisioned_throughput: {\r\n          read_capacity_units: 1,\r\n          write_capacity_units: 1,\r\n        }\r\n      }\r\n      $ddb.create_table(create_table_params)\r\n\r\n      # wait till table is created\r\n      $ddb.wait_until(:table_exists, {table_name: \"movies\"})\r\n    end\r\n    \r\n...<\/pre>\n
Creating an Item<\/h2>\n
DynamoDB provides the #put_item<\/code> method, which creates a new item with passed attributes and the Primary Key. If an item with the same Primary Key exists, it is replaced with new item attributes.<\/p>\n
# models\/user.rb\r\nclass User\r\n    ...\r\n    def save\r\n        item_hash = instance_values\r\n        begin\r\n          resp = $ddb.put_item({\r\n            table_name: self.table_name,\r\n            item: item_hash,\r\n            return_values: 'NONE'\r\n          })\r\n          resp.successful?\r\n        rescue Aws::DynamoDB::Errors::ServiceError => e\r\n          false\r\n        end\r\n    end\r\n    ...\r\nend<\/pre>\n
The instance method save<\/code> simply saves an item and returns either true or false depending upon the response. The instance_values<\/code> method returns hash of all the attr_accessor<\/code> fields, which is passed to the item<\/code> key as item_hash<\/code>.<\/p>\n
The return_values<\/code> option inside the put_item<\/code> request determines whether you want to receive a saved item or not. We are just interested in knowing whether an item is saved successfully or not, hence \u2018NONE\u2019 was passed.<\/p>\n
Reading an item<\/h3>\n
Getting an item from DynamoDB with the Primary Key is similar to the way records are found by ids by Active Record in relational databases.<\/p>\n
The #get_item<\/code> method is used to fetch a single item for a given Primary Key. If no item is found, then the method returns nil in the item\u2019s element of response.<\/p>\n
# models\/user.rb\r\n...\r\ndef self.find(email)\r\n    if email.present?\r\n      begin\r\n        resp = $ddb.get_item({\r\n          table_name: self.table_name,\r\n          key: {\r\n            email: email\r\n          }\r\n        })\r\n      resp.item\r\n      rescue Aws::DynamoDB::Errors::ServiceError => e\r\n        nil\r\n      end\r\n    else\r\n      nil\r\n    end\r\n  end\r\n ...<\/pre>\n
Updating an item<\/h3>\n
An item is updated with the #update_item<\/code> method, which behaves more like the upsert (update or insert) of PostgreSQL. In other words, it updates an item with given attributes. But in case no item is found with those attributes, a new item is created. This might sound similar to how #put_item<\/code> works, but the difference is that #put_item<\/code> replaces an existing item, whereas #update_item<\/code> updates an existing item.<\/p>\n
# models\/user.rb\r\n  def update(attrs)\r\n    item_hash = attrs\r\n    item_hash['updated_at'] = DateTime.current.to_s\r\n    item_hash.keys.each do |key|\r\n      item_hash[key] = {\r\n        'value' => item_hash[key],\r\n        'action' => 'PUT'\r\n      }\r\n    end\r\n    begin\r\n      resp = $ddb.update_item({\r\n        table_name: self.class.table_name,\r\n        key: {\r\n          email: email\r\n        },\r\n        attribute_updates: item_hash\r\n      })\r\n      resp.successful?\r\n    rescue Aws::DynamoDB::Errors::ServiceError => e\r\n      false\r\n    end\r\n  end<\/pre>\n
While updating an item, you need to specify the Primary Key of that item and whether you want to replace or add new values to an existing attribute.<\/p>\n
Look closely at how item_hash<\/code> is formed. The attribute hash is passed normally to update the method, which is processed further to add two fields \u2014 value<\/code> and action<\/code> \u2014 in place of simple key => value<\/code> pairs. This modifies a given simple hash as shown below to attribute_updates<\/code> key compatible values.<\/p>\n
Deleting an item<\/h3>\n
The #delete_item<\/code> method deletes items with a specified Primary Key. If the item is not present, it doesn\u2019t return an error.<\/p>\n
# models\/user.rb\r\n  def delete\r\n    if email.present?\r\n      begin\r\n        resp = $ddb.delete_item({\r\n          table_name: self.class.table_name,\r\n          key: {\r\n            email: email\r\n          }\r\n        })\r\n        resp.successful?\r\n      rescue Aws::DynamoDB::Errors::ServiceError => e\r\n        false\r\n      end\r\n    else\r\n      false\r\n    end\r\n  end<\/pre>\n
Conditional Writes<\/h2>\n
All DynamoDB operations can be categorized into two types: read<\/code> operations, such as get_item<\/code>, and write operations, such as put_item<\/code>, update_item<\/code>, and delete_item<\/code>.<\/p>\n
These write operations can be constrained with specified conditions, such as put_item<\/code>, and should be performed only if a certain item with the same Primary Key does not exist. All write operations support these kinds of conditional writes.<\/p>\n
For example, if you want to create an item only if it isn\u2019t present, you can add attribute_not_exists(attribute_name)<\/code> as a value of the condition_expression<\/code> key in the #put_item<\/code> method params.<\/p>\n
def save!\r\n    item_hash = instance_values\r\n    item_hash['updated_at'] = DateTime.current.to_s\r\n    item_hash['created_at'] = DateTime.current.to_s\r\n    begin\r\n      resp = $ddb.put_item({\r\n        table_name: self.class.table_name,\r\n        item: item_hash,\r\n        return_values: 'NONE',\r\n        condition_expression: 'attribute_not_exists(email)'\r\n      })\r\n      resp.successful?\r\n    rescue Aws::DynamoDB::Errors::ServiceError => e\r\n      false\r\n    end\r\n  end<\/pre>\n
Ruby SDK v2<\/a> provides an interface to interact with DynamoDB, and you can read more about the methods provided in the given SDK documentation<\/a>.<\/p>\n
Batch Operations<\/h2>\n
Apart from four basic CRUD operations, DynamoDB provides two types of batch operations:<\/p>\n