lab
lab title
Programming Amazon SQS and SNS using the AWS
NodeJS SDK
V1.04
Course title
AWS Certified Developer Associate
AWS Certified Developer Associate
Table of Contents
Contents
Table of Contents ........................................................................................................................................................ 1
About the Lab .............................................................................................................................................................. 2
Creating an SQS Queue using the AWS NodeJS SDK ............................................................................................... 3
Creating SQS Messages using the AWS NodeJS SDK .............................................................................................. 9
Sending Messages with sendMessage.................................................................................................................. 9
Increasing Throughput with sendMessageBatch ............................................................................................... 11
Processing SQS Messages using the NodeJS SDK ................................................................................................. 12
Subscribing an SQS Queue to an SNS Topic ........................................................................................................... 12
1 Copyright 2015 all rights reserved - BackSpace.Academy
AWS Certified Developer Associate
About the Lab
These lab notes are to support the instructional videos on Programming Amazon SQS and SNS using the AWS
NodeJS SDK in the BackSpace AWS Certified Developer course.
In this lab we will then:
• Create an SQS queue using the AWS NodeJS SDK.
• Create SQS messages to the queue.
• Create SQS messages to the queue using the batch method.
• Process and delete SQS messages.
• Create an SNS topic.
• Create SNS messages to the SQS queue.
Please refer to the AWS JavaScript SDK documentation at:
http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/SQS.html
and
http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/SNS.html
Please note that AWS services change on a weekly basis and it is extremely important
you check the version number on this document to ensure you have the lastest version
with any updates or corrections.
Copyright 2015 all rights reserved - BackSpace.Academy 2
AWS Certified Developer Associate
Creating an SQS Queue using the
AWS NodeJS SDK
In this section we will use the AWS NodeJS SDK to create an SQS Queue.
Please note these lab notes have been updated for the Cloud9 IDE due to problems with the Atom Remote-Edit
package. You can still use an IDE such as Atom if you like but, you will need to upload files to your EC2 instance using
FileZilla.
Go to Services - Cloud9 from the console
Click Create environment
Give your environment a name
Click Next step
Leave default settings
Click Next step
Select Amazon Linux
3 Copyright 2015 all rights reserved - BackSpace.Academy
AWS Certified Developer Associate
Click Next Step
Click Create environment
When your environment is ready select File-New File
Copyright 2015 all rights reserved - BackSpace.Academy 4
AWS Certified Developer Associate
Copy the following code and paste into the new file (ctrl-v to paste):
// Load the AWS SDK for Node.js
var AWS = require('aws-sdk');
/**
* Don't hard-code your credentials!
* Create an IAM role for your EC2 instance instead.
*/
// Set your region
AWS.config.region = 'us-east-1';
var sqs = new AWS.SQS();
//Create an SQS Queue
var queueUrl;
var params = {
QueueName: 'backspace-lab', /* required */
Attributes: {
ReceiveMessageWaitTimeSeconds: '20',
VisibilityTimeout: '60'
}
};
sqs.createQueue(params, function(err, data) {
if (err) console.log(err, err.stack); // an error occurred
else {
console.log('Successfully created SQS queue URL '+ data.QueueUrl); //
successful response
}
});
5 Copyright 2015 all rights reserved - BackSpace.Academy
AWS Certified Developer Associate
Click + to save the file as index.js
From the Bash console at the bottom of the screen enter:
npm install aws-sdk
Now that you have installed the AWS SDK you can run the app
node index.js
Copyright 2015 all rights reserved - BackSpace.Academy 6
AWS Certified Developer Associate
Now go to the SQS console and see your newly created SQS queue
Click on the queue to see its details including the visibility timeout and receive message wait time we specified in our
code.
7 Copyright 2015 all rights reserved - BackSpace.Academy
AWS Certified Developer Associate
Copyright 2015 all rights reserved - BackSpace.Academy 8
AWS Certified Developer Associate
Creating SQS Messages using the
AWS NodeJS SDK
In this section we will create and add messages to our SQS queue using sendMessage
asynchronously and also with sendMessageBatch.
Important: These lab notes have been updated for the new built in async features of NodeJS version 8+. The code is
slightly different to that on the video and does not require an npm package and will not work with the npm async
package.
This part of the lab will require the new async capabilities of NodeJS version 8+
You can check the current version is 8 or higher:
node --version
Sending Messages with sendMessage
Add a createMessages call in the sqs.createQueue method callback:
sqs.createQueue(params, function(err, data) {
if (err) console.log(err, err.stack); // an error occurred
else {
console.log('Successfully created SQS queue URL '+ data.QueueUrl); //
successful response
createMessages(data.QueueUrl);
}
});
9 Copyright 2015 all rights reserved - BackSpace.Academy
AWS Certified Developer Associate
Now create the createMessages function:
// Create 50 SQS messages
async function createMessages(queueUrl){
var messages = [];
for (var a=0; a<50; a++){
messages[a] = 'This is the content for message '+ a + '.';
}
// Asynchronously deliver messages to SQS queue
for (const message of messages){
console.log('Sending message: '+ message)
params = {
MessageBody: message, /* required */
QueueUrl: queueUrl /* required */
};
await sqs.sendMessage(params, function(err, data) { // Wait until callback
if (err) console.log(err, err.stack); // an error occurred
else console.log(data); // successful response
});
}
}
Click + to save to the EC2 instance.
Now run index.js
node index.js
It has now created and sent 50 messages.
Copyright 2015 all rights reserved - BackSpace.Academy 10
AWS Certified Developer Associate
Now go to the SQS console and you will see the messages have been added to the queue.
Increasing Throughput with sendMessageBatch
If the maximum total payload size (i.e., the sum of all a batch's individual message lengths) is 256 KB (262,144 bytes)
or less, we can use a single sendMessageBatch call. This reduces our number of calls and resource costs.
Now let’s use sendMessageBatch to do send up to 10 messages at a time.
Change createMessages to:
// Create 50 SQS messages
async function createMessages(queueUrl){
var messages = [];
for (var a=0; a<5; a++){
messages[a] = [];
11 Copyright 2015 all rights reserved - BackSpace.Academy
AWS Certified Developer Associate
for (var b=0; b<10; b++){
messages[a][b] = 'This is the content for message '+ (a*10+b) + '.';
}
}
// Asynchronously deliver messages to SQS queue
for (const message of messages){
console.log('Sending message: '+ message)
params = {
Entries: [],
QueueUrl: queueUrl /* required */
};
for (var b=0; b<10; b++){
params.Entries.push({
MessageBody: message [b],
Id: 'Message'+ (messages.indexOf(message)*10+b)
});
}
await sqs.sendMessageBatch(params, function(err, data) { // Wait until callback
if (err) console.log(err, err.stack); // an error occurred
else console.log(data); // successful response
});
}
}
Click + to save to the EC2 instance.
Now run index.js
It has now created 50 messages but this time using only 5 calls to SQS instead of 50.
You will also see an empty array returned for failed messages.
Copyright 2015 all rights reserved - BackSpace.Academy 12
AWS Certified Developer Associate
Now go to the SQS console and you will see the messages have been added to the queue.
13 Copyright 2015 all rights reserved - BackSpace.Academy
AWS Certified Developer Associate
Processing SQS Messages using the
NodeJS SDK
In this section we will use the NodeJS SDK to read, process then delete messages from an SQS
queue.
First let’s create a polling function with 1 second interval.
In the sqs.createQueue method success callback save the queue URL and change waitingSQS to false.
sqs.createQueue(params, function(err, data) {
if (err) console.log(err, err.stack); // an error occurred
else {
console.log('Successfully created SQS queue URL '+ data.QueueUrl); //
successful response
queueUrl = data.QueueUrl;
waitingSQS = false;
createMessages(queueUrl);
}
});
After the sqs.createQueue method call place the following code for polling SQS
Copyright 2015 all rights reserved - BackSpace.Academy 14
AWS Certified Developer Associate
// Poll queue for messages then process and delete
var waitingSQS = false;
var queueCounter = 0;
setInterval(function(){
if (!waitingSQS){ // Still busy with previous request
if (queueCounter <= 0){
receiveMessages();
}
else --queueCounter; // Reduce queue counter
}
}, 1000);
Now create a function to read up to 10 messages (the max allowed) from the SQS queue. The function halts further
calls to it while it is waiting for SQS to respond. It will also halt polling for 60 seconds when the queue is empty.
// Receive messages from queue
function receiveMessages(){
var params = {
QueueUrl: queueUrl, /* required */
MaxNumberOfMessages: 10,
VisibilityTimeout: 60,
WaitTimeSeconds: 20 // Wait for messages to arrive
};
waitingSQS = true;
sqs.receiveMessage(params, function(err, data) {
if (err) {
waitingSQS = false;
console.log(err, err.stack); // an error occurred
}
else{
waitingSQS = false;
if ((typeof data.Messages !== 'undefined')&&(data.Messages.length !== 0)) {
console.log('Received '+ data.Messages.length
+ ' messages from SQS queue.'); // successful response
}
else {
queueCounter = 60; // Queue empty back of for 60s
15 Copyright 2015 all rights reserved - BackSpace.Academy
AWS Certified Developer Associate
console.log('SQS queue empty, waiting for '+ queueCounter + 's.');
}
}
});
}
Click + to save to the EC2 instance.
Now run index.js
You can see it is receiving messages but not always 10 messages. This is normal.
Press + to stop the application.
Now update receiveMessages with a call to processMessages in the callback
// Receive messages from queue
function receiveMessages(){
var params = {
QueueUrl: queueUrl, /* required */
MaxNumberOfMessages: 10,
VisibilityTimeout: 60,
WaitTimeSeconds: 20 // Wait for messages to arrive
};
waitingSQS = true;
Copyright 2015 all rights reserved - BackSpace.Academy 16
AWS Certified Developer Associate
sqs.receiveMessage(params, function(err, data) {
if (err) {
waitingSQS = false;
console.log(err, err.stack); // an error occurred
}
else{
waitingSQS = false;
if ((typeof data.Messages !== 'undefined')&&(data.Messages.length !== 0)) {
console.log('Received '+ data.Messages.length
+ ' messages from SQS queue.'); // successful response
processMessages(data.Messages);
}
else {
queueCounter = 60; // Queue empty back of for 60s
console.log('SQS queue empty, waiting for '+ queueCounter + 's.');
}
}
});
}
Now add the function to asynchronously process and delete messages from the queue.
// Process and delete messages from queue
async function processMessages(messagesSQS){
for (const item of messagesSQS){
await console.log('Processing message: '+ item.Body); // Do something with the
message
var params = {
QueueUrl: queueUrl, /* required */
ReceiptHandle: item.ReceiptHandle /* required */
}
await sqs.deleteMessage(params, function(err, data) { // Wait until callback
if (err) console.log(err, err.stack); // an error occurred
else {
console.log('Deleted message RequestId: '
+ JSON.stringify(data.ResponseMetadata.RequestId)); // successful
response
}
})
}
}
17 Copyright 2015 all rights reserved - BackSpace.Academy
AWS Certified Developer Associate
Click + to save to the EC2 instance.
Now run the application.
You will see the messages being processed and deleted from the queue after processing.
After the SQS WaitTimeSeconds of 20 seconds has expired the SQS queue empty message will appear.
Press + to stop the application.
Copyright 2015 all rights reserved - BackSpace.Academy 18
AWS Certified Developer Associate
Subscribing an SQS Queue to an SNS
Topic
In this section we will create and subscribe our application to an SNS topic. We will then use the
NodeJS SDK to send SNS messages and then read, process and delete the messages from the SQS
queue.
We will sending messages to the queue from the SNS service. We won’t need to call createMessages.
Comment out the call to createMessages
sqs.createQueue(params, function(err, data) {
if (err) console.log(err, err.stack); // an error occurred
else {
console.log('Successfully created SQS queue URL '+ data.QueueUrl); //
successful response
queueUrl = data.QueueUrl;
waitingSQS = false;
// createMessages(data.QueueUrl);
}
});
Now let’s create an SNS topic.
Go to the SNS console.
Click “Create Topic”
19 Copyright 2015 all rights reserved - BackSpace.Academy
AWS Certified Developer Associate
Give it the topic name backspace-lab, and display name backspace
Click “Create Topic”
Click “Create subscription”
Select “Amazon SQS” for protocol
Copy and paste the SQS ARN from the SQS console for the endpoint.
Copyright 2015 all rights reserved - BackSpace.Academy 20
AWS Certified Developer Associate
Now go back to the SQS console.
Select “Queue Actions” “Subscribe queue to SNS topic”
Select the topic and click “Subscribe”
Click “Subscribe”
Now click on the permissions tab at the bottom of the screen to see the permission for SNS created.
21 Copyright 2015 all rights reserved - BackSpace.Academy
AWS Certified Developer Associate
Now go back to the SNS console.
Select “Topics”
Select the topic and click “Publish to topic”
Create a message.
Copyright 2015 all rights reserved - BackSpace.Academy 22
AWS Certified Developer Associate
Now run your app again.
You will see the message has been delivered to SQS and processed by your app.
23 Copyright 2015 all rights reserved - BackSpace.Academy
AWS Certified Developer Associate
Now we will send an SNS message using the NodeJS SDK
Uncomment the createMessages call from createQueue
sqs.createQueue(params, function(err, data) {
if (err) console.log(err, err.stack); // an error occurred
else {
console.log('Successfully created SQS queue URL '+ data.QueueUrl); //
successful response
queueUrl = data.QueueUrl;
waitingSQS = false;
createMessages(data.QueueUrl);
}
});
Replace the createMessages code with (make sure to replace YOUR_SNS_ARN with the SNS topic arn):
// Create an SNS messages
var sns = new AWS.SNS();
function createMessages(){
var message = 'This is a message from Amazon SNS';
console.log('Sending messages: '+ message);
sns.publish({
Message: message,
Copyright 2015 all rights reserved - BackSpace.Academy 24
AWS Certified Developer Associate
TargetArn: 'arn:aws:sns:us-east-1:950302654420:backspace-lab' }, function(err,
data) {
if (err) {
console.log(err.stack);
}
else{
console.log('Message sent by SNS: '+ data);
}
});
}
Click + to save to the EC2 instance.
Now run index.js again
Press + to stop the application.
25 Copyright 2015 all rights reserved - BackSpace.Academy
