Node.js – Path of the saint coder…

Amazon Web Services, Javascript, Lambda, Node.js, Serverless

Serverles AWS Lambda – Part 2: Retrieve data from AWS DynamoDB

In this past article , we learned how to create our 1st AWS Lambda service through using Serverless framework. Our current Function in the project, currently exposing GET HTTP verb and when it is invoked, it returns a list of harcoded blog objects. In this arcitle, we are going to extends its capability, by refactoring this part, to not returning hardcoded array. Instead, we are going to return an array of objects where are stored in AWS Data Storage service.

DynamoDB vs SimpleDB vs RDS (Relational Database Service)

AWS offers 3 Database Services to customers as follows: DynamoDB (a NoSQL database solution) , RDS ( Relational Database service, hosted & handled by AWS) and SimpleDB ( a similar NoSQL database to DynamoDB, yet AWS seems `hide` it from customers, but it’s accessible ). Among of these 3 options, I rule out RDS, because its pricing is the most expensive compared to the others (https://aws.amazon.com/rds/pricing/).

So we have SimpleDB vs DynamoDB now. DynamoDB is popular and widely used. However, in terms of query speed & price, SimpleDB is more attractive compared to DynamoDB on a certain case. I am tempted to choose SimpleDB over DynamoDB, but since in this sample, we are going to build the backend API for a kind of blog application, DynamoDB is mentioned as suitable choice for this case. However, in future articles, I will cover the SimpleDB version of this sample, because it’s still interesting to me.

Initialise blog table on AWS DynamoDB

Open Blog project’s serverless.yml file and then add these resources entries.

15442222_10209531126492450_6453505499330815730_n — Add resources block in serveless.yml

The resources section we added in serverless.yml file in there, is a way of telling serverless for creating a new DynamoDB table on AWS. In the section we define the table’s name, a String typed Attribute that we defined as the Primary key for this table and initial Read & Write Capacity units of the table. As for other attributes, we will add them when we are going to create records later.

Once we have saved the changes in serverless.yml file, let’s go back to command terminal and invoke serverless deploy command for deploying the service. You may want to remove prior deployed service by running serverless remove command 1st.

15492048_10209533696276693_8348512342170787345_n — Running *serverless deploy* command

Now we are going to check the created table and add records on it. Let’s login into AWS Web Console using your Account and look for DynamoDB home page.

15542284_10209526893986640_8278991969420752178_n — Accessing DynamoDB’s Home Page

Go to DynamoDB Tables’s page. Confirm that you notice the Blogs table appear on screen. Select it and then click Items tab.

15440518_10209527049990540_3733155806861019371_o

On the Items tab view, click Create item button and confirm that a modal dialog as in this following screenshot appears.

15439827_10209527249635531_182565783167740202_n — Access DynamoDB Create Item page

15439976_10209527261395825_443327492936059599_n — DynamoDB Create Item Modal

Through accessing the popup menu, add 3 more attributes (columns) and fill them with strings as their values.

15578480_10209527865330923_6722338196689040398_n — Add more attributes with values

As for the id, we are going to assign it with UUID. To generate the UUID, we can use available tool such as in this site.

15492602_10209528130057541_4408324398395814107_n — Assign UUID on id attribute

Repeat prior steps to add as many records as you want. You could go to Actions menu -> click Duplicate button for doing this.

15577903_10209530053905636_5985571610743560938_o — Created Blog records

Setup AWS SDK for Node.JS

AWS provides SDK for developer which contains various APIs for accessing their services, including DynamoDB. We need to use the SDK for Node.JS in our Lambda function for accessing the DynamoDB Table we created in previously.

In the lambda project, ensure that it has package.json file. Otherwise, we will need to create it through running npm init command.

15590010_10209534771183565_2910924853169733744_n — package.json file

15590464_10209534788944009_2455790222455992124_n — running *npm init* command for creating *package.json* file, in proper way

Once we have finished prior step, we’ll install the AWS SDK inside our project through running npm install aws-sdk –save command. The extra –save argument on the command will add an entry in the package.json file, to ensure that when CloudFormation building our lambda, and run npm install command, npm would install the AWS SDK library into this project.

15578542_10209536533947633_7211788418072510870_n — Installing AWS SDK

15492538_10209536541747828_8470975299309866365_n — AWS SDK entry in dependencies section of package.json file

Ensure that you have setup your IAM account’s Access & Secret Keys. If you are not sure with this, open ~/.aws/credentials file and ensure that there are lines which define these keys entries. If not, follow guide in this site.

15590609_10209535294556649_42198199827940078_n — content of *~/.aws/credential* file

Refactor Blogs resource’s GET verb – Phase 1

As we have done in prior article, we created retrieveBlogs helper method which returns a list of hardcoded blog objects. We are going to create a new method for replacing this retrieveBlog method. Here is the steps of how we are going to do this.

Remove the retrieveBlogs method from handler JS file. Move the removed method’s hardcoded lines into a class, we name it as DynamoDbDataService class. Change the handler JS file to instantiate the new class and call its getAll method.

15578204_10209543119832276_3459169278569822042_o — Moving *retrieveBlogs* method into the new *DynamoDbDataService* class

Then, in the Handler JS file, we change the code by importing our new class, instantiate & initialise the new class and call its getAll method for retrieving the Blogs items from AWS DynamoDB.

15338663_10209543486281437_4971861985644717314_n — Changed Handler JS file to use our new *DynamoDbDataService* class

Before we move to next refactoring phase, we’ll invoke serverless invoke local command 1st to ensure that there are no errors in our new code.

15492125_10209535931132563_9057149045344926516_n — Test our refactored lambda function in our local machine

Implement calls to AWS DynamoDB using AWS-SDK

Let’s go back to DynamoDbDataService class. On the early lines (below ‘use strict’ line), we’ll put a statement to import AWS-SDK library.

15621720_10209543639045256_8136405371073962034_n — Import AWS SDK

Moving to the constructor part, we write lines to initialise the AWS’s configuration property. We want to tell AWS SDK which AWS Region that our Lambda Service is deployed to. We could hardcode it to a specific region such as us-east-1, ap-southeast-1 , etc. But, we won’t do this way. Because we don’t want to change this line in the future if we want to deploy the Service to different region.
AWS has provided an environment variable AWS_DEFAULT_REGION which filled with correct AWS Region of where our Lambda service deployed to, in the cloud environment. Therefore, we are going to get the AWS Region from AWS_DEFAULT_REGION environment variable, instead of hardcoded it.

15492400_10209543745727923_6070001576467098088_n — Initialise AWS SDK & Region inside constructor

Removed all hardcoded lines inside getAll’s returned Promise object. Then, we’ll start with instantiating AWS.DynamoDB.DocumentClient type. This type expose methods which one of them can be used for pulling data from a DynamoDB table – the scan method. scan method takes TableName as a required parameter. We build the scan’s parameters which consist of TableName & Limit(define maximum number of returned records). We assign the tableName & numberOfItems property’s values into these parameters. Next, we call the documentClient instance’s scan method and pass the params as its argument. When the call is finished, the callback in the method’s 2nd argument is triggered. Inside the callback method, we check whether the calling process is ended as giving error or result. Should it is ended as error (err is not null), we call the promise’s reject method and takes the err object as its argument. Otherwise, we call resolve method and takes the result (data) as its argument.

15622538_10209543768088482_3759355797743570048_n — Re-implement the *getAll* method

Testing the changes in local development machine

At this point, we should be ready to test our changes. Before we deploy our code to AWS, it’s good thing to do if we test it first in our local machine. As usual, to do this, we will invoke this command to invoke the service, AWS_DEFAULT_REGION=<AWS Region of where your lambda sits on> serverless invoke local -f <lambda function’s name> . In our case, we will invoke the command as AWS_DEFAULT_REGION=us-east-1 serverless invoke local -f blogsFetch
Invoke the command and confirm that we got the result with status code is 200 and the body contains stringified retrieved data, came from our AWS DynamoDB’s table.

15589635_10209549143102854_3206144194476542933_n — Invoke changed Lambda function in local against AWS DynamoDB

Deploy to AWS and testing it on API Gateway test page

All should be still fine. It’s time for uploading our changed Lambda function to AWS. This time, instead of running serverless deploy -s dev -r <aws-region> command, we call this command for deploying only the Lambda code only (node.js code): serverless deploy -f <function’s name> -s <stage> -r <region>.We use this command because we don’t want to rebuild other resources such as the DynamoDB when deploying our updated Lambda code.

15578735_10209549236505189_1908013641050570633_n — Deploy the updated lambda using -f argument (deploy per function)

Although it was working fine when we tested our lambda in local environment, we still need to test the deployed Lambda function. This time, instead of using Postman for testing our API, we will do it in different way. We are going to test it through using API Gateway’s Test Page. Go back to our AWS Console page then go to AWS API Gateway service page. On the page, click Resources link in left menu. Then on Resources pane, click the GET verb. On right pane, click a Blinking Dot with label TEST link. This will bring the API Test page when we clicked it.

15585075_10209549347147955_5725502401346282511_o — Accessing API Test Page

Confirm that the right pane is refreshed and display the test page. On the page, there is blue coloured with thunder icon button, the test button. Click this button. This will invoke our blogs/fetch API.

15171049_10209549411429562_7284752223752980644_n

When the call is finished, we received “Internal server error”. We will cover how we are going to fixing these issues.

15665887_10209549448350485_8850368016774724947_n — “Internal server error” when testing ther API GET verb

On the displayed Logs, we could not find any useful information which explains why the error happened. To look for what was going on and the cause of this error, we could see it on CloudWatch Logs window. Open the Cloud Watch page then click Logs item on left menu. Confirm that the right pane refresh and display a list of Log Group item. Click the item whose name is matched to our blogs/lambda service.

15590038_10209549566393436_7830682212492007648_n — CloudWatch page with displayed Log Groups list

When we clicked one of displayed Log Groups item, the right pane is refreshed again and displays a list of Log Streams. Click the one with latest Last Event Time.

15622007_10209550862505838_7253757097566019991_n — CloudWatch page with displayed Log Streams list

In the next page, expand the item that looks like explain this error. Notice the errorMessage & errorType, it seems that we have not authorise the Lambda Function to do Scan operation against the designated DynanmoDB table.

15589616_10209550976148679_2309715960096197322_n

Fixing the unauthorised access error

To fix the previous error, we need to give authorisation access to our Lambda function for performing scan operation against the DynamoDB Table. The way to do this is by adding DynamoDBIamPolicy entry in the serverless.yml file, under Resources entry as follow:

15665701_10209551212474587_6687703448620478353_n — Updated serverless.yml with DynamoDBIamPolicy entry

Save the changed serverless.yml file and then re-run the serverless deploy -f <function’s name> -s <stage> -r <region> command again.

Retesting the Lambda on API Gateway test page

Once we have redployed our lambda function, go back to AWS Web Console’s API Gateway Test page of our deployed Lambda function. Then, press the Test button. Noticed that we do not receive error anymore. Instead, we should see records from DynamoDB Blogs table are retrieved and displayed as follow.

15589549_10209551406519438_828441359589004905_n — Returned response from testing the GET API

Conclusion

At the end of this article, we have learned a number of key things to get our Lambda function able to pull data from AWS DynamoDB. First, we define the DynamoDB table we want to create through adding Resources section in serverless.yml file. Once we redeployed our Lambda to AWS, AWS CloudFormation will create the DynamoDB Table, beside our Lambda function & its API Gateway endpoint. Then, we filled the created table with several items through AWS DynamoDB Web Console.

On the Lambda function’s handler code, we structured our code by moving retrieveBlogs method into an ES6 class (the DynamoDbDataService class) and wrap the method’s body with ES6 Promise (because we want the records retrieval to be an asynchronous process). Then, we replaced the hardcoded lines with logic for Initialising AWS SDK’s Document Client class and calling its scan method for retrieving records from Blogs table.

Aside from these, we learned how to get the detailed error log in case a request to our deployed Lambda’s endpoints return “Internal Server Error”,through looking at AWS CloudWatch web page. We also learned that the API Gateway page has a section that allow us to Test our Lambda’s HTTP Endpoint.The source code of this article can be found in this link.

In the next article, we will add more verbs on the Lambda function so that it would provide complete CRUD endpoints.

Amazon Web Services, Javascript, Lambda, Node.js, S3, Serverless

Serverless AWS Lambda – Part 1: A Quickstart for Beginners

This article covers steps for creating your first AWS Lambda service & functions through using Serveless framework. Before following these steps, ensure that you have already had AWS Account.

Pre-requisites:

Ensure that you have installed latest LTS version of Node.JS. Visit this link to finding out how to install it in your machine: https://nodejs.org/en/
Ensure that you have installed Serverless framework. If not, run this command for installing it:
```
sudo npm install serverless -g
```
Then, run this command to check whether serverless has been installed successfully or not:
```
serverless -v
```
Create a new or reuse existing IAM User account. Ensure that you have given AdministratorAccess to the user account. If you are not sure how to do this, follow the guidance of how to do it in this document
Upon created a new IAM User account, take note the displayed API Key & Secret Key of the new IAM Account.
Follow the guidance in this document to configure the aws credentials (API Key & Secret Key) that you have noted in prior step. Serverless framework need this information so that it could deploy.

Create your first service:

Once we have completed all of required pre-requisites, create a new folder, go into the new folder then run this command to begin creating your 1st service:
```
serverless create --template  --path
```
. Example:
```
serverless create --template aws-nodejs --path blog
```
Creating 1st Service

When creating a new service is finished, we will see file structure in the project folder, as shown in this following screenshot:
- serverless.yml – a YAML file where we will define configurations for our service, such as AWS Resources (S3, DynamoDB, etc), Region, Nodejs Runtime, we want to use and also our service’s functions configurations.
- handler.js – Initial Javascript file , created by serverless, that is supposed to be the place where we will write our function’s logic. Rename the file’s name with name of entity that our function interacts with (e.g. blog, product, task, etc).

serverless_initial_project_structure — Initial Project’s Structure

Open the serverless.yml file and edit these Configuration sections: lambda function’s name, handler method’s name, associated HTTP path & verb.

serverless_configure_function_1

serverless_configure_function_2 — serverless.yml – Configure service’s function

Open the handler javascript file. Let’s write code inside the exported function whose the logic is simple – just returning an array of JSON objects

serverless_handler_get_function_1 — handler node.js – It retrieves the data, wrap the result in response’s body and return

serverless_handler_get_function — handler node.js code – helper method that is supposed for retrieving the data from storage

Before we deploy the lambda function, let’s invoke it in our local machine through executing this command:
```
serverless invoke local --function
```
```
 e.g. serverless invoke local --function blogs 
```
Ensure that no error happens and we notice correct result is printed on terminal.

Deploy the service to AWS Lambda

We have implemented simple logic inside Lambda service’s handler and then invoke it locally using serverless invoke command. Now, we need to deploy our lambda through running this command in terminal :

serverless deploy --stage  --region .

Example:

serverless deploy --stage dev --region ap-shouteast-1

15585216_10209556096116675_171074867079464586_o — Deploying Lambda to AWS through calling *serverless deploy* command

When deployment is successful, we should get the URL Endpoint of our deployed service and there should be no error message appear. If we don’t see the URL Endpoint, there should be a typo inside serverless.yaml (check the events section , if you type it as event, this issue occurs).

This is result that you should get when we browse the endpoint or invoke it using Postman

serverless_invoke_lambda_in_postman — Invoke the Lambda in Postman

How the Serverless deploys our Lambda function to AWS

When we invoked serverless deploy command, serverless zipped our function file(s) and also created a file for configuring AWS CloudFormation stack setting (cloud formation template). Serverless also created a new AWS S3 bucket using our AWS API Key & Secret Key, then upload the zip file & cloudformation setting file into the created AWS S3 Bucket.

serverless_files_in_s3 — Lambda files uploaded by Serverless in AWS S3 Bucket

Once the file uploading process is done, Serverless manage the creation of our Lambda function & its API Gateway Endpoint through AWS CloudFormation service. This is done based on the uploaded cloud formation template file. Upon finished the deployment, you could see the created AWS Resources which build up your deployed service through browsing the Lambda , CloudFormation, API Gateway section pages, on your AWS web console.

Removing Deployed Service

In case you need to destroy your deployed lambda service, the common way to do this is through destroying the resources that built your service, through AWS Web console page and then do these procedures: Open S3 page and destroy the Bucket which build the service, Destroy CloudFormation stack, Destroy the Lambda & then the related API Gateway. Serverless provides a quickest way for destroying our service along with its AWS resources. We can do this through invoking serverless remove command, inside the serverless project folder.

serverless_remove_lambda — Removing Lambda function and its claimed AWS Resource

Conclusion

Serverless has simplified the efforts of writing AWS Lambda Function, deploying & hosting the function as an API Gateway resource endpoint. By hosting our node.js-based API on AWS Lambda, we do not need to setup an EC2 instance or other kind of virtual private server just for hosting our code. Through using Serverless & AWS Lambda, we shift this responsibility to AWS and thus, free us from responsibility of setup our own server. In the future article, we will cover steps of how to integrate our Lambda service with AWS Data storage services such as SimpleDB or DynamoDB.

Amazon Web Services, Javascript, Linux, Node.js, Sails.JS, Ubuntu

Deploying Sails.js Web Application on AWS EC2 Instance

Sails.js is a Web MVC Framework, built on the top of Node.js platfom, that is interesting to me. Beside the framework leverages MVC pattern and offers blazing fast performance during runtime (thanks to Node.js), it also came in with a built in HTTP server. When i am going to run my web application, I do not need to compile, package & deploy my code into a separated web server. Instead, it just requires me to invoke a line of command to get my web application up & running:

 sails lift MyAwesomeWebApplication

Also, it just need me to press CTL+C keys in the terminal, to stop the running web application. This is a similar feature that exists in Play.

In the meantime, I was thinking about what if I deploy & run a sails web application on a cloud environment, let’s say, Amazon Web Service. So, i made the first attempt by deploying a simple sails web app on AWS through Elastic Beanstalk (EB). The result is my web app’s homepage did not show in my browser. Instead, it displays EB App’s default homepage.

Then, I took the other route. I create an EC2 Instance (It’s a virtual private server) using AMI (Amazon Machine Image) with pre-installed Ubuntu 14.04 OS. I installed required softwares in the created EC2 instance then lift my sails web app on it. Then, check it using my browser & it confirmed me that my sails web app was up & running.

In this article, I would like to share you the steps that I did to get my sails app running in a Ubuntu AWS EC2 instance. And, here they are:

Create a new AWS Instance using Ubuntu 64 AMI

Browse to AWS console at https://console.aws.amazon.com & click EC2 link.
On the EC2 Dashboard, click IMAGES->AMIs menu link.
On the Filter menu bar, modify the options to Public images | 64-bit images | Ubuntu.
On the returned filtered result, tick a desired AMI (e.g. ubuntu/images/ebs/ubuntu-trusty-14.04-amd64-server).
Click [Launch] button.
On the “Step 2: Choose an Instance Type” page, tick a desired EC2 Instance Type (e.g. Micro Instances) then click [Next: Configure Instance Details] button.
On the “Step 3: Configure Instance Details” page, leave the default settings and click [Next: Add Storage] button.
On the “Step 4: Add Storage” page, adjust the Root’s storage size or just leave the default settings then click [Next: Tag Instance] button.
On the “Step 5: Tag Instance” page, leave the default settings and click [Next: Configure Security Group] button.
On the “Step 6: Configure Security Group” page:
- Select “Create a new security group” option.
- Enter name & description for the new security group.
- Click [Add Rule] button & on the new entry, enter: Type = Custom TCP Rule, Protocol = TCP, Port Range = 1337, Source = Anywhere.
- Click [Review and Launch] button.
On the “Step 7: Review Instance Launch”, click [Launch] button.
On the displayed “Select an existing key pair of create a new key pair” dialog, select “Create a new key pair” on the combo field, enter Key pair name then click [Download Key Pair] & [Launch Instances] buttons.
Save the downloaded .pem file into somewhere within your home directory & restrict its access by running this command in terminal:
```
chmod 400 yourdownloadedkeypair.pem
```
Go back to browser, click [View Instance] button. Notice that the browser redirects to Instances dashboard page and the new AMI Instance is shown in the Instances list. Give the new Instance a name if you like ( by clicking the new instance’s empty Name cell and type the name on it). Make notes on the new Instance’s Public IP or Public DNS fields.

Connecting to the created AMI Instance using SSH

On the Instances Dashboard page, click [Connect] button. A dialog would appear, showing 2 options for connecting to the Instance. Select “A standalone SSH client”, block and copy the command written under ‘Example’ section.
Open the command line Terminal box, move to the directory that has the downloaded .pem file and run the command written in the earlier instructions dialog.
```
ssh -i downloaded_keypair.pem ubuntu@new_instance_public_ip_or_dns
```
Confirm that you have logged in successfully.
Set the root’s password by running these commands:
```
sudo su
passwd
```

Setup required softwares on the created AMI Instance (as root)

In the SSH terminal connected to the new Instance, run these following commands to update the new Instance’s software repositories:
```
apt-get upgrade && apt-get dist-upgrade && apt-get update && apt-get autoclean
```
Run these commands to set up the git client:
```
apt-get install build-essential git
```
Create a new directory & pull the latest Node.js source code in this directory by running this command ( using git client ):
```
git clone https://github.com/joyent/node.git
```
Change directory to the cloned Node.js source directory, then run these commands to compile & install the Node.js:
```
sudo ./configure && make && make install
```
Confirm that the compilation process is finished successfully.
Install sail.js web MVC framework by running this command:
```
npm -g install sails
```
Confirm that the installation is finished successfully

Deploy a sails.js app into the created Instance

Ensure that you have put the sails.js app source code in your git account (e.g. github, bitbucket, etc ).
On the created Instance, create a new directory and do git clone the sails.js app source code into this directory.
Change directory to the cloned source code’s directory and run this command to install node module dependencies referenced by your sails.js app
```
npm install
```
Run this command to lift the sails.js app online on the created instance:
```
sails lift
```
Confirm that the sails.js app is lifted successfully
Go back to your internet browser and browse to your instance’s public IP address , port 1337.
Confirm that the lifted sails.js app’s home page is displayed

It’s alive now ! But, wait..

When I close my current SSH session that was connected to the EC2 instance & refreshed the sails app’s page in my browser, I noticed that it returns 404 error. My sails web app was offline. Apparently, each processes that have been started during SSH session within the EC2 instance, would be shutdown when the SSH connection is closed. Somehow, I need to prevent the running sails app from being closed even when the SSH session is ended.

Fortunately, the solution for this is already suggested in the documentation of sails.js. The document suggests us to install & start a sails.js app by using forever. Forever prevents any running scripts from being closed during SSH session by running them as a Daemon (*nix service). Then, I tried the solution and it worked well. I would explain the steps of how to forever my sails app in EC2, in the next section,

Run the deployed app as a running daemon in the EC 2 Instance

Install forever globally:
```
npm -g install forever
```
In the terminal connected to the EC2 Instance, change directory to the sails.js app’s root folder then run this command:
```
forever start -ae errors.log app.js --dev --port 1337
```
OR, run this command if you wish to run the production version:
```
forever start -ae errors.log app.js --prod --port 80
```
If you write your controllers as coffee script files, open the errors.log file. Notice that there is error message written in it (http://tinyurl.com/p8y4ovl). This means the sails.js app is failing to be lifted by running prior command. This is a known issue in Sails.js version 0.9.16. This issue has been raised to balderdashy and it can be seen in this link, along with the temporary workaround as well : http://tinyurl.com/m2jyx24).
Logout or disconnect from the EC2 instance’s SSH session and then browse to your lifted sails.js app’s url. Confirm that your lifted app is still up & running now.

The previous section marks the end of this article. I hope this ‘how-to’ guide would help you deploying your sails.js web app on your AWS account. Happy sailing in your AWS cloud.