B.

TECH - ARTIFICIAL INTELLIGENCE AND DATA SCIENCE

SEMESTER 05 – REGULATION 2021

CCS335 CLOUD COMPUTING

 BONAFIDE CERTIFICATE

UNIVERSITY REGISTER NUMBER

Certified that this is bonafide record of practical work done by Mr./Mrs.

………………………………… of …………………. Department in the
…………………………………………………….. Laboratory during the semester ...................... Year
… ............................and submitted for the University Practical Examination conducted
on ………………….. at the CHRIST THE KING ENGINEERING COLLEGE, KARAMADAI,
COIMBATORE-604 104.

Lab in-charge Head of the Department

Internal Examiner External Examiner

LAB MANNERS

• Students must be present in proper dress code and wear the ID card.

• Students should enter the log-in and log-out time in the log register
without fail.
• Students are not allowed to download pictures, music, videos or files
withoutthe permission of respective lab in-charge.
• Students should wear their own lab coats and bring observation note
books tothe laboratory classes regularly.
• Record of experiments done in a particular class should be submitted
in the next labclass.
• Students who do not submit the record note book in time will not be
allowed to do thenext experiment and will not be given attendance
for that laboratory class.
• Students will not be allowed to leave the laboratory until they
complete the experiment.
• Students are advised to switch-off the Monitors and CPU when they
leave the lab.
• Students are advised to arrange the chairs properly when they leave
the lab.
 College

Vision
To improve the quality of human life through multi-disciplinary programs in Engineering,
architecture and management that are internationally recognized and would facilitate
research work to incorporate social economical and environmental development.

Mission
• To create a vibrant atmosphere that creates competent engineers, innovators,scientists,
entrepreneurs, academicians and thinkers of tomorrow.
• To establish centers of excellence that provides sustainable solutions to industryand society.
• To enhance capability through various value added programs so as to meet thechallenges of
dynamically changing global needs.

Department

Vision

To achieve the outstanding standards of high quality of education and to impart

discipline and technically strong to meet the global challenges by using the latest
technology, fostering collective culture and distributing innovations to relevant areas of
academia and industry to welfare for the society.

Mission
• To produce young Engineers with a sound knowledge in the area of Artificial Intelligence
and Data Science.
• To empower Students to become an eminent in the industries, high rank officials in the
government and academia in the world wide.
• To attain the demand of the experts around the world wide in the fields of Artificial
Intelligence, Machine Learning and Data Science.
 Program Outcomes(POs)
Engineering knowledge: Apply the knowledge of mathematics, science,
engineering fundamentals, and an engineering specialization to the solution of
PO1
complex engineering problems.
Problem analysis: Identify, formulate, review research literature, and
analyze complex engineering problems reaching substantiated conclusions
PO2 using first principles of mathematics, natural sciences, and engineering
sciences
Design/development of solutions: Design solutions for complex
engineering problems and design system components or processes that meet
PO3 the specified needs with appropriate consideration for the public health and
safety, and the cultural, societal, and environmental Considerations.
Conduct investigations of complex problems: Use research-based
knowledge
PO4 and research methods including design of experiments, analysis and
interpretation of data,and synthesis of the information to provide valid
conclusions.
Modern tool usage: Create, select, and apply appropriate techniques,
resources, and modern engineering and IT tools including prediction and
PO5 modeling to complex engineering activities with an understanding of the
limitations.

The engineer and society: Apply reasoning informed by the contextual

knowledge to assess societal, health, safety, legal and cultural issues and the
PO6
consequent responsibilities relevant to the professional engineering practice
Environment and sustainability: Understand the impact of the professional
engineering solutions in societal and environmental contexts, and
PO7 demonstrate the knowledge of, and need for sustainable development

Ethics: Apply ethical principles and commit to professional ethics and

PO8 responsibilities and norms of the engineering practice.

Individual and team work: Function effectively as an individual, and as a

PO9 member or leader in diverse teams, and in multidisciplinary settings.

Communication: Communicate effectively on complex engineering activities

with the engineering community and with society at large, such as, being able
PO10
to comprehend and write effective reports and design documentation, make
effective presentations, and give and receive clear instructions.
Project management and finance: Demonstrate knowledge and
understanding of the engineering and management principles and apply
PO11
these to one’s own work, as a member and leader in a team, to manage
projects and in multidisciplinary environments.

Life-long learning: Recognize the need for, and have the preparation and
ability to engage in independent and life-long learning in the broadest
PO12
context of technological change.
 PROGRAM SPECIFIC OUTCOMES (PSOs)

PO1 Describe big data and use cases from selected business
domains.
PO2 Explain NoSQL big data management.

PO3 Install, configure, and run Hadoop and HDFS.

PO4 Perform map-reduce analytics using Hadoop .

PO5 Use Hadoop-related tools such as HBase, Cassandra, Pig, and

Hive for big data analytics.
 lOMoARcPSD|319 796 21

lOMoARcPSD|319 796 21

LIST OF EXPERIMENTS

1. Install Virtualbox /VMware Workstation with

different flavours of linux orwindows OS ontop of
windows7 or 8.
2. Install a C compiler in the virtual machine
created using virtual boxand executeSimple
Programs.
3. Install Google App Engine. Create hello world
app and other simple webapplicationsusing
python/java.
4. Use GAE launcher to launch the web applications.

5. Simulate a cloud scenario using

CloudSim and run a scheduling
algorithm that is notpresent in
CloudSim.
6. Find a procedure to transfer the files
from one virtual machine toanother
virtualmachine.
7. Install Hadoop single node cluster and run simple applications like word
count.

8. Creating and Executing Your First Container Using Docker.

9. Run a Container from Docker Hub.

 lOMoARcPSD|319 796 21

INDEX

Ex.No Date List of Experiments Mark Signature

1 Install Virtualbox / VMwareWorkstation with

different flavours of linux or windows OS on top
of windows7 or 8.

2 Install a C compiler in the virtualmachine created

using virtual box andexecute Simple Programs

3
Install Google App Engine. Create hello world app
and other simple webapplications using
python/java.

4
Use GAE launcher to launch the web
applications.

5 Simulate a cloud scenario using CloudSim and run

a scheduling algorithm that is not present in
CloudSim
6 Find a procedure to transfer the files from one
virtual machine to another virtual machine.

7 Install Hadoop signature node cluster and run

Simple applications like wordcount

8 Creating and Executing Your First Container

Using Docker

9 Run a Container from Docker Hub

 lOMoARcPSD|319 796 21

Ex.No :1 Install Virtualbox / VMware Workstation with different flavours

of linux or windows OS on top of windows7 or 8.
Date :

Aim:
To Install Virtualbox / VMware Workstation with different flavours of linux or windows OS on
top ofwindows7 or 8.

PROCEDURE:

Steps to install Virtual Box:

1. Download the Virtual box exe and click the exe file…and select next button..

2. Click the next button..

 lOMoARcPSD|319 796 21

3. Click the next button

4. Click the YES button..

 lOMoARcPSD|319 796 21

5. Click the install button…

6. Then installation was completed..the show virtual box icon on desktop screen….
 lOMoARcPSD|319 796 21

Steps to import Open nebula sandbox:

1. Open Virtual box
2. File import Appliance
3. Browse OpenNebula-Sandbox-5.0.ova file
4. Then go to setting, select Usb and choose USB 1.1
5. Then Start the Open Nebula
6. Login using username: root, password:opennebula
 lOMoARcPSD|319 796 21

Steps to create Virtual Machine through opennebula

1. Open Browser, type localhost:9869
2. Login using username: oneadmin, password: opennebula
3. Click on instances, select VMs then follow the steps to create Virtaul machine
a. Expand the + symbol
b. Select user oneadmin
c. Then enter the VM name,no.of instance, cpu.
d. Then click on create button.
e. Repeat the steps the C,D for creating more than one VMs.
lOMoARcPSD|319 796 21
 lOMoARcPSD|319 796 21

APPLICATIONS:

There are various applications of cloud computing in today’s network world. Many search engines and
social websites are using the concept of cloud computing like www.amazon.com, hotmail.com, facebook.com,
linkedln.com etc. the advantages of cloud computing in context to scalability is like reduced risk , low cost
testing ,ability to segment the customer base and auto-scaling based on application load.

RESULT:
Thus the procedure to run the virtual machine of different configuration.
 lOMoARcPSD|319 796 21

Ex.No :2 Install a C compiler in the virtual machine created using virtual box and
execute Simple Programs
Date :

Aim:
To Install a C compiler in the virtual machine created using virtual box and
execute Simple Programs`

PROCEDURE:

Steps to import .ova file:

1. Open Virtual box
2. File import Appliance
3. Browse ubuntu_gt6.ova file
4. Then go to setting, select Usb and choose USB 1.1
5. Then Start the ubuntu_gt6
6. Login using username: dinesh, password:99425.

Steps to run c program:

1. Open the terminal

2. Type cd /opt/axis2/axis2-1.7.3/bin then press enter
3. gedit hello.c
4. gcc hello.c
5. ./a.out
 lOMoARcPSD|319 796 21

1. Type cd /opt/axis2/axis2-1.7.3/bin then press enter

2. Type gedit first.c

 lOMoARcPSD|319 796 21

3. Type the c program

4. Running the C program

 lOMoARcPSD|319 796 21

5. Display the output:

APPLICATIONS:
Simply running all programs in grid environment.

RESULT:

Thus the simple C programs executed successfully.

 lOMoARcPSD|319 796 21

Ex.No :3 Install Google App Engine. Create hello world app and other simple web
applications using python/java.
Date :

Aim:
To Install Google App Engine. Create hello world app and other simple web applications using
python/java.
Procedure:

1. Install Google Plugin for Eclipse

Read this guide – how to install Google Plugin for Eclipse. If you install the Google App Engine Java SDK
together with “Google Plugin for Eclipse“, then go to step 2, Otherwise, get the Google App Engine Java
SDKand extract it.

2. Create New Web Application Project

In Eclipse toolbar, click on the Google icon, and select “New Web Application Project…”

Figure – New Web Application Project

 lOMoARcPSD|319 796 21

Click finished, Google Plugin for Eclipse will generate a sample project automatically.

3. Hello World
Review the generated project directory.
 lOMoARcPSD|319 796 21

Nothing special, a standard Java web project structure.

HelloWorld/ src/
...Java source code...
META-INF/
...other configuration...
war/
...JSPs, images, data files...
WEB-INF/
...app configuration...
lib/
...JARs for libraries...
classes/
...compiled classes...
 lOMoARcPSD|319 796 21

Copy
The extra is this file “appengine-web.xml“, Google App Engine need this to run and deploy the application.

File : appengine-web.xml

<?xml version="1.0" encoding="utf-8"?>

<appengine-web-app xmlns="http://appengine.google.com/ns/1.0">
<application></application>
<version>1</version>

<!-- Configure java.util.logging -->

<system-properties>
<property name="java.util.logging.config.file" value="WEB-INF/logging.properties"/>
</system-properties>

</appengine-web-app>
Copy

4. Run it local
Right click on the project and run as “Web Application“.

Eclipse console :

//...
INFO: The server is running at http://localhost:8888/
30 Mac 2012 11:13:01 PM com.google.appengine.tools.development.DevAppServerImpl start INFO: The
admin console is running at http://localhost:8888/_ah/admin
Copy
Access URL http://localhost:8888/, see output

also the hello world servlet – http://localhost:8888/helloworld

 lOMoARcPSD|319 796 21

5. Deploy to Google App Engine

Register an account on https://appengine.google.com/, and create an application ID for your web application.

In this demonstration, I created an application ID, named “mkyong123”, and put it in appengine- web.xml.

File : appengine-web.xml

<?xml version="1.0" encoding="utf-8"?>

<appengine-web-app xmlns="http://appengine.google.com/ns/1.0">
<application>mkyong123</application>
<version>1</version>

<!-- Configure java.util.logging -->

<system-properties>
<property name="java.util.logging.config.file" value="WEB-INF/logging.properties"/>
</system-properties>

</appengine-web-app>
Copy

To deploy, see following steps:

Figure 1.1 –Click on GAE deploy button on the toolbar.

 lOMoARcPSD|319 796 21

Figure 1.3 – If everything is fine, the hello world web application

Result:
Thus the simple application was created successfull
 lOMoARcPSD|319 796 21

Ex.No :4

Date :
Use GAE launcher to launch the web
applications.

Aim:
To Use GAE launcher to launch the web applications.

Steps:

Making your First Application

Now you need to create a simple application. We could use the “+” option to have the
launcher make us an application – but instead we will do it by hand to get a better sense
of what is going on.

Make a folder for your Google App Engine applications. I am going to make the Folder on
my Desktop called “apps” – the path to this folder is:

C:\Documents and Settings\csev\Desktop\apps

And then make a sub--‐folder in within apps called “ae--01--trivial” – the path to
this folder would be:

C:\ Documents and Settings \csev\Desktop\apps\ae--01--trivial

Using a text editor such as JEdit (www.jedit.org), create a file called app.yaml in the ae--01-
-trivial folder with the following contents:
application: ae-01-trivial version: 1
runtime: python api_version: 1 handlers:- url:
/.* script: index.py
Note: Please do not copy and paste these lines into your text editor – you might
end up with strange characters – simply type them into your editor.
Then create a file in the ae--01--trivial folder called index.py with three lines in it:
print 'Content-Type: text/plain'
print ' '
print 'Hello there Chuck'
Then start the GoogleAppEngineLauncher program that can be found under
Applications. Use the File --> Add Existing Application command and navigate
into the apps directory and select the ae--01--trivial folder. Once youhave added
the application, select it so that you can control the application using
 lOMoARcPSD|319 796 21

the launcher.

Once you have selected your application and press Run. After a few moments your application will
start and the launcher will show a little green icon next to your application. Then press Browse to
open a browser pointing at your application which is running at http://localhost:8080/

Watching the Log

You can watch the internal log of the actions that the web server is performing when you are
interacting with your application in the browser. Select your application in the Launcher and press
the Logs button to bring up a log window:
 lOMoARcPSD|319 796 21

Each time you press Refresh in your browser – you can see it retrieving the output with a GET
Request.

Dealing With Errors

With two files to edit, there are two general categories of errors that you may encounter. If
You make a mistake onthe app.yamlfile, the App Engine willnotstart and your launcher will show
a yellow icon near your application:

To get more detail on what is going wrong, take a look at the log for the application:
 lOMoARcPSD|319 796 21

The error you need to see is likely to be the last few lines of the output – in this case I made a
Python syntax error on line one of our one-•‐line application.
Reference: http://en.wikipedia.org/wiki/Stack_trace
When you make a mistake in the app.yaml file – you must the fix the mistake and attempt to
startthe application again.
If you make a mistake in a file like index.py, you can simply fix the file and press refresh in your
browser – there is no need to restart the server.
Shutting Down the Server
To shut down the server, use the Launcher, select your application and press the Stop button.

Result:

Thus the GAE web applications was created.

 lOMoARcPSD|319 796 21

Ex.No :5
Simulate a cloud scenario using CloudSim and
Date : run ascheduling algorithm that is not present in
CloudSim.

Aim
To Simulate a cloud scenario using CloudSim and run a scheduling algorithm that isnot
present in CloudSim.

Steps:

How to use CloudSim in Eclipse

CloudSim is written in Java. The knowledge you need to use CloudSim is basic Java programming
andsome basics about cloud computing. Knowledge of programming IDEs such as Eclipse or
NetBeans isalso helpful. It is a library and, hence, CloudSim does not have to be installed. Normally,
you can unpack the downloaded package in any directory, add it to the Java classpath and it is ready
to be used.Please verify whether Java is available on your system.

To use CloudSim in Eclipse:

1. Download CloudSim installable files
from https://code.google.com/p/cloudsim/downloads/list and unzip
2. Open Eclipse
3. Create a new Java Project: File -> New
4. Import an unpacked CloudSim project into the new Java Project
The first step is to initialise the CloudSim package by initialising the CloudSim library, as follows
CloudSim.init(num_user, calendar, trace_flag)
5. Data centres are the resource providers in CloudSim; hence, creation of data centres is a
second step. To create Datacenter, you need the DatacenterCharacteristics object that stores
the properties of a data centre such as architecture, OS, list of machines, allocation policy that
covers the time or spaceshared, the time zone and its price:
Datacenter datacenter9883 = new Datacenter(name, characteristics, new
VmAllocationPolicySimple(hostList), s
6. The third step is to create a broker:
DatacenterBroker broker = createBroker();
7. The fourth step is to create one virtual machine unique ID of the VM, userId ID of the
VM’s owner, mips, number Of Pes amount of CPUs, amount of RAM, amount of bandwidth,
amount of storage, virtual machine monitor, and cloudletScheduler policy for cloudlets:
Vm vm = new Vm(vmid, brokerId, mips, pesNumber, ram, bw, size, vmm, new
CloudletSchedulerTimeShared())
 lOMoARcPSD|319 796 21

8. Submit the VM list to the broker:

broker.submitVmList(vmlist)
9. Create a cloudlet with length, file size, output size, and utilisation model:
Cloudlet cloudlet = new Cloudlet(id, length, pesNumber, fileSize, outputSize, utilizationModel, utilizationM ode
10. Submit the cloudlet list to the broker:
broker.submitCloudletList(cloudletList) Sample
Output from the Existing Example:
Starting
CloudSimExample1...
Initialising...
Starting CloudSim version
3.0 Datacenter_0 is
starting...
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>null
Broker is
starting... Entities
started.
: Broker: Cloud Resource List received with 1
resource(s) 0.0: Broker: Trying to Create VM #0 in
Datacenter_0
: Broker: VM #0 has been created in Datacenter #2, Host #0
0.1: Broker: Sending cloudlet 0 to VM #0
400.1: Broker: Cloudlet 0 received
: Broker: All Cloudlets executed.
Finishing ...... 400.1: Broker: Destroying
VM #0
Broker is shutting down...
Simulation: No more future events
Cloud Information Service: Notify all CloudSim entities for shutting down.Datacenter_0
is shutting down...
Broker is shutting down
Simulation completed.
Simulation completed.
 lOMoARcPSD|319 796 21

========== OUTPUT ==========

Cloudlet ID STATUS Data center ID VM ID Time Start Time
Finish Time 0 SUCCESS 2 0 400
0.1 400.1
*****Datacenter:
Datacenter_0***** User id
Debt
3 35.6

CloudSimExample1 finished!

RESULT:

The simulation was successfully executed.

 lOMoARcPSD|319 796 21

Ex.No :6
Find a procedure to transfer the files from one virtual machine
Date : to another virtual machine.

Aim:
To Find a procedure to transfer the files from one virtual machine to another virtual machine.

Steps:

1. You can copy few (or more) lines with copy & paste mechanism.
For this you need to share clipboard between host OS and guest OS, installing Guest
Addition on both the virtual machines (probably setting bidirectional and restarting them).
You copy from guest OS in the clipboard that is shared with the host OS.
Then you paste from the host OS to the second guest OS.
2. You can enable drag and drop too with the same method (Click on the machine,
settings, general, advanced, drag and drop: set to bidirectional )
3. You can have common Shared Folders on both virtual machines and use one of
the directory shared as buffer to copy.
Installing Guest Additions you have the possibility to set Shared Folders too. As you put
a file in a shared folder from host OS or from guest OS, is immediately visible to the
other. (Keep in mind that can arise some problems for date/time of the files when there
are different clock settings on the different virtual machines).
If you use the same folder shared on more machines you can exchange files directly copying
them in this folder.
4. You can use usual method to copy files between 2 different computer with client-server
application. (e.g. scp with sshd active for linux, winscp... you can get some info about SSH
servers e.g. here)
You need an active server (sshd) on the receiving machine and a client on the sending
machine. Of course you need to have the authorization setted (via password or, better,
via an automatic authentication method).
Note: many Linux/Ubuntu distribution install sshd by default: you can see if it is running
with pgrep sshd from a shell. You can install with sudo apt-get install openssh-server.
5. You can mount part of the file system of a virtual machine via NFS or SSHFS on
the other, or you can share file and directory with Samba. You may find
interesting the article Sharing files between guest and host without VirtualBox
shared folders with detailed step by step instructions.
You should remember that you are dialling with a little network of machines with different
operative systems, and in particular:
• Each virtual machine has its own operative system running on and acts as a
physical machine.
• Each virtual machine is an instance of a program owned by an user in the hosting
operative system and should undergo the restrictions of the user in the hosting OS.
 lOMoARcPSD|319 796 21

E.g Let we say that Hastur and Meow are users of the hosting machine, but they did not
allow each other to see their directories (no read/write/execute authorization). When each
of them run a virtual machine, for the hosting OS those virtual machine are two normal
programs owned by Hastur and Meow and cannot see the private directory of the other
user. This is a restriction due to the hosting OS. It's easy to overcame it: it's enough to give
authorization to read/write/execute to a directory or to chose a different directory in which
both users can read/write/execute.
• Windows likes mouse and Linux fingers. :-)
I mean I suggest you to enable Drag & drop to be cosy with the Windows machines and the
Shared folders or to be cosy with Linux.
When you will need to be fast with Linux you will feel the need of ssh-keygen and
to Generate once SSH Keys to copy files on/from a remote machine without writing password anymore. In
this way it functions bash auto-completion remotely too!

PROCEDURE:
Steps:
1. Open Browser, type localhost:9869
2. Login using username: oneadmin, password: opennebula
3. Then follow the steps to migrate VMs
a. Click on infrastructure
b. Select clusters and enter the cluster name
c. Then select host tab, and select all host
d. Then select Vnets tab, and select all vnet
e. Then select datastores tab, and select all datastores
f. And then choose host under infrastructure tab
g. Click on + symbol to add new host, name the host then click on create.
4. on instances, select VMs to migrate then follow the stpes
a. Click on 8th icon ,the drop down list display
b. Select migrate on that ,the popup window display
c. On that select the target host to migrate then click on migrate.
 lOMoARcPSD|319 796 21

Before migration
Host:SACET

Host:one-sandbox
lOMoARcPSD|319 796 21
 lOMoARcPSD|319 796 21

After Migration:

Host:one-sandbox
 lOMoARcPSD|319 796 21

Host:SACET

APPLICATIONS:

Easily migrate your virtual machine from one pc to another.

Result:
Thus the file transfer between VM was successfully completed…..
 lOMoARcPSD|319 796 21

Ex.No :7 Install Hadoop signature node cluster and run

Date :
Simple applications like wordcount

Aim:
To Install Hadoop single node cluster and run simple
applications like wordcount.

Steps:
Install Hadoop
Step 1: Click here to download the Java 8 Package. Save this file in your home directory.
Step 2: Extract the Java Tar File.
Command: tar -xvf jdk-8u101-linux-i586.tar.gz

Fig: Hadoop Installation – Extracting Java Files

Step 3: Download the Hadoop 2.7.3 Package.

Command: wget- https://archive.apache.org/dist/hadoop/core/hadoop-2.7.3/hadoop-

2.7.3.tar.gz

Fig: Hadoop Installation – Downloading Hadoop

Step 4: Extract the Hadoop tar File.

Command: tar -xvf hadoop-2.7.3.tar.gz

Fig: Hadoop Installation – Extracting Hadoop Files

 lOMoARcPSD|319 796 21

Step 5: Add the Hadoop and Java paths in the bash file (.bashrc). Open. bashrc

file. Now, add Hadoop and Java Path as shown below.

Command: vi .bashrc

Fig: Hadoop Installation – Setting Environment Variable

Then, save the bash file and close it.

For applying all these changes to the current Terminal, execute the source command.
Command: source .bashrc

Fig: Hadoop Installation – Refreshing environment variables

To make sure that Java and Hadoop have been properly installed on your system and can be
accessed through the Terminal, execute the java -version and hadoop version commands.

Command: java -version

Fig: Hadoop Installation – Checking Java Version
 lOMoARcPSD|319 796 21

Command: hadoop version

Fig: Hadoop Installation – Checking Hadoop Version

Step 6: Edit the Hadoop Configuration files.

Command: cd hadoop-2.7.3/etc/hadoop/

Command: ls

All the Hadoop configuration files are located in hadoop-2.7.3/etc/hadoop directory as you can
see in the snapshot below:

Fig: Hadoop Installation – Hadoop Configuration Files

 lOMoARcPSD|319 796 21

Step 7: Open core-site.xml and edit the property mentioned below inside configuration tag:

core-site.xml informs Hadoop daemon where NameNode runs in the cluster. It contains
configuration settings of Hadoop core such as I/O settings that are common to HDFS &
MapReduce.
Command: vi core-site.xml

Fig: Hadoop Installation – Configuring core-site.xml

Step 8: Edit hdfs-site.xml and edit the property mentioned below inside
configuration tag:

hdfs-site.xml contains configuration settings of HDFS daemons (i.e. NameNode, DataNode,

Secondary NameNode). It also includes the replication factor and block size of HDFS.
Command: vi hdfs-site.xml

Fig: Hadoop Installation – Configuring hdfs-site.xml

 Step 9: Edit the mapred-site.xml file and edit the property mentioned below

inside configuration tag:

mapred-site.xml contains configuration settings of MapReduce application like number of JVM

that can run in parallel, the size of the mapper and the reducer process, CPU cores available for a
process, etc.

In some cases, mapred-site.xml file is not available. So, we have to create the mapred- site.xml
file using mapred-site.xml template.

Command: cp mapred-site.xml.template mapred-site.xml

Command: vi mapred-site.xml.

Fig: Hadoop Installation – Configuring mapred-site.xml

 lOMoARcPSD|319 796 21

1
<?xml version="1.0" encoding="UTF-8"?>
2 <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
3 <configuration>
4 <property>
5 <name>mapreduce.framework.name</name>
6 <value>yarn</value>
</property>
7 </configuration>

Step 10: Edit yarn-site.xml and edit the property mentioned below inside
configuration tag:
yarn-site.xml contains configuration settings of ResourceManager and NodeManager like
application memory management size, the operation needed on program & algorithm, etc.
Command: vi yarn-site.xml

Fig: Hadoop Installation – Configuring yarn-site.xml

Step 11: Edit hadoop-env.sh and add the Java Path as mentioned below:
1
2
<?xml version="1.0">
3 <configuration>
4 <property>
5 <name>yarn.nodemanager.aux-services</name>
6 <value>mapreduce_shuffle</value>
</property>
7 <property>
8 <name>yarn.nodemanager.auxservices.mapreduce.shuffle.class</ name>
9 <value>org.apache.hadoop.mapred.ShuffleHandler</value>
1 </property>
0
1

hadoop-env.sh contains the environment variables that are used in the script to run Hadoop
like Java home path, etc.
 lOMoARcPSD|319 796 21

Command: vi hadoop–env.sh

Fig: Hadoop Installation – Configuring hadoop-env.sh Step

12: Go to Hadoop home directory and format the NameNode.

Command: cd

Command: cd hadoop-2.7.3

Command: bin/hadoop namenode -format

Fig: Hadoop Installation – Formatting NameNode

This formats the HDFS via NameNode. This command is only executed for the first time.
Formatting the file system means initializing the directory specified by the dfs.name.dir
variable.

Never format, up and running Hadoop filesystem. You will lose all your data stored in the
HDFS.

Step 13: Once the NameNode is formatted, go to hadoop-2.7.3/sbin directory and start all the daemons.

Command: cd hadoop-2.7.3/sbin

Either you can start all daemons with a single command or do it individually.

Command: ./start-all.sh

The above command is a combination of start-dfs.sh, start-yarn.sh & mr-jobhistory-

daemon.sh

Or you can run all the services individually as below:

 lOMoARcPSD|319 796 21

Start NameNode:

The NameNode is the centerpiece of an HDFS file system. It keeps the directory tree of all files
stored in the HDFS and tracks all the file stored across the cluster.

Command: ./hadoop-daemon.sh start namenode

Fig: Hadoop Installation – Starting NameNode

Start DataNode:
On startup, a DataNode connects to the Namenode and it responds to the requests from the Namenode for different
operations.

Command: ./hadoop-daemon.sh start datanode

Fig: Hadoop Installation – Starting DataNode

 lOMoARcPSD|319 796 21

Start ResourceManager:

ResourceManager is the master that arbitrates all the available cluster resources and thus
helps in managing the distributed applications running on the YARN system. Its work
is to manage each NodeManagers and the each application’s ApplicationMaster.

Command: ./yarn-daemon.sh start resourcemanager

Fig: Hadoop Installation – Starting ResourceManager

Start NodeManager:

The NodeManager in each machine framework is the agent which is responsible for
managing containers, monitoring their resource usage and reporting the same to the
ResourceManager.

Command: ./yarn-daemon.sh start nodemanager

 lOMoARcPSD|319 796 21

See Batch Details

Fig: Hadoop Installation – Starting NodeManager

Start JobHistoryServer:

JobHistoryServer is responsible for servicing all job history related requests from client.

Command: ./mr-jobhistory-daemon.sh start historyserver

Step 14: To check that all the Hadoop services are up and running, run the
below command.

Command: jps

Fig: Hadoop Installation – Checking Daemons

 lOMoARcPSD|319 796 21

Step 15: Now open the Mozilla browser and go

to localhost:50070/dfshealth.html to check the NameNode interface.

Fig: Hadoop Installation – Starting WebUI

Congratulations, you have successfully installed a single node Hadoop cluster

Result:
Thus the Hadoop one cluster was installed and simple applications executed successfully.
 lOMoARcPSD|319 796 21

Ex.No :9
Run a Container from Docker Hub
Date :

AIM:
To write a program to run a container from Docker hub.
PROCEDURE:
Run a container from docker hub

Rundocker -h,

$ docker -h
Flag shorthand -h has been deprecated, please use --help Usage:

docker [OPTIONS] COMMAND

A self-sufficient runtime for containers

...

Management Commands:
builder Manage builds
config Manage Docker configs
container Manage containers
engine
Manage the docker
engineimage Manage images
network
Manage networks
node Manage Swarm nodes
plugin Manage plugins
secret Manage Docker secrets
service Manage
services
stack Manage Docker stacks
swarm Manage Swarm
system Manage Docker
trust Manage trust on Docker images
volume Manage volumes

The Docker command line can be used to manage several features of the Docker Engine. In this lab, we will
mainly focus on the container command.
 lOMoARcPSD|319 796 21

If podman is installed, you can run the alternative command for comparison. sudo

podman -h

You can additionally review the version of your Docker installation,

docker

version

Client:
Version: 19.03.6
...

Server: Docker Engine -

CommunityEngine
Version: 19.03.5
...

sudo podman version --events-

backend=noneVersion: 2.1.1
API Version:
2.0.0 Go Version:
go1.15.2
Built: Thu Jan 1 00:00:00 1970
OS/Arch:
linux/amd64

Step 1: Run your first container

We are going to use the Docker CLI to run our first container.

Opena terminal on your local computer

Run docker container run -t ubuntu top
Use the docker container run command to run a container with the ubuntu image using
the top command. The -t flags allocate a pseudo-TTY which we need for the top to work correctly.
$ docker container run -it ubuntu top Unableto
find image 'ubuntu:latest' locally latest:
Pulling from library/ubuntu aafe6b5e13de:
Pull complete 0a2b43a72660: Pull complete
18bdd1e546d2: Pull complete 8198342c3e05:
Pull complete f56970a44fd4: Pull complete
Digest: sha256:f3a61450ae43896c4332bda5e78b453f4a93179045f20c8181043b26b5e79028 Status:
Downloaded newer image for ubuntu:latest
The docker run command will result first in a docker pull to download the ubuntu image onto your host.
 lOMoARcPSD|319 796 21

Once it is downloaded, it will start the container. The output for the running container should look like
this:
top - 20:32:46 up 3 days, 17:40, 0 users, load average: 0.00, 0.01, 0.00
 lOMoARcPSD|319 796 21

Tasks: 1 total, 1 running, 0 sleeping, 0 stopped, 0 zombie

%Cpu(s): 0.0 us, 0.1 sy, 0.0 ni, 99.9 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st

KiB Mem : 2046768 total, 173308 free, 117248 used, 1756212 buff/cache
KiB Swap: 1048572 total, 1048572 free, 0 used. 1548356 avail
Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+

COMMAND 1 root 20 0 36636 3072 2640 R 0.3 0.2 0:00.04 top
Inspect the container with docker container exec
The docker container exec command is a way to "enter" a running container's namespaces with a new
process.
Open a new terminal. On cognitiveclass.ai, select Terminal > New Terminal.
Using play-with-docker.com, to open a new terminal connected to node1, click "Add New Instance" on
the lefthand side, then ssh from node2 into node1 using the IP that is listed by 'node1 '. For example:
[node2] (local) [email protected] ~
$ ssh 192.168.0.18
[node1] (local) [email protected] ~
$
In the new terminal, use the docker container ls command to get the ID of the running container you just
created.
$ docker container ls
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS
NAMES
b3ad2a23fab3 ubuntu "top" 29 minutes ago Up 29 mi
goofy_nobel

And Voila! We just used the docker container exec command to "enter" our container's namespaces with
our bash process. Using docker container exec with bash is a common pattern to inspect a docker
container.
Notice the change in the prefix of your terminal. e.g. root@b3ad2a23fab3:/. This is an indication that we
are running bash "inside" of our container.
From the same termina, run ps -ef to inspect the running processes.
root@b3ad2a23fab3:/# ps -ef UID PID PPID C STIME TTY TIME CMD
root 1 0 0 20:34 ? 00:00:00 top
root 17 0 0 21:06 ? 00:00:00 bash
root 27 17 0 21:14 ? 00:00:00 ps -ef
 lOMoARcPSD|319 796 21

You should see only the top process, bash process and our ps process.
root@b3ad2a23fab3:/# exit
exit
$ ps -ef

# Lots of processes!
docker ps -a

docker rm <CONTAINER ID>

Step 2: Run Multiple Containers

Explore the Docker Hub
The Docker Hub is the public central registry for Docker images, which contains community and official images.
Run an Nginx server
Let's run a container using the official Nginx image from the Docker Hub.
$ docker container run --detach --publish 8080:80 --name nginx nginx
Unable to find image 'nginx:latest' locally
latest: Pulling from library/nginx
36a46ebd5019: Pull complete 57168433389f:
Pull complete 332ec8285c50: Pull complete
Digest: sha256:c15f1fb8fd55c60c72f940a76da76a5fccce2fefa0dd9b17967b9e40b0355316 Status:
Downloaded newer image for nginx:latest

5e1bf0e6b926bd73a66f98b3cbe23d04189c16a43d55dd46b8486359f6fdf048 Nginx is a
lightweight web server. You can access it on port 8080 on your localhost.
Access the nginx server on localhost:8080. curl
localhost:8080
will return the HTML home page of Nginx,
<!DOCTYPE html>
<html>
<head>

<title>Welcome to nginx!</title>
<style>
body
{
width: 35em; margin:
0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
 lOMoARcPSD|319 796 21

</style>
</head>
<body>

<h1>Welcome to nginx!</h1>
If you are using play-with-docker, look for the 8080 link near the top of the page, or if you run a Docker
client with access to a local browser,
Run a mongo DB server
Now, run a mongoDB server. We will use the official mongoDB image from the Docker Hub. Instead of
using the latest tag (which is the default if no tag is specified), we will use a specific version of the mongo
image.
$ docker container run --detach --publish 8081:27017 --name mongo mongo:4.4 Unable
to find image mongo:4.4 locally
4.4: Pulling from library/mongo
d13d02fa248d: Already exists
bc8e2652ce92: Pull complete
3cc856886986: Pull complete
c319e9ec4517: Pull complete
b4cbf8808f94: Pull complete
cb98a53e6676: Pull complete
f0485050cd8a: Pull complete
ac36cdc414b3: Pull complete
61814e3c487b: Pull complete
523a9f1da6b9: Pull complete
3b4beaef77a2: Pull complete
Digest: sha256:d13c897516e497e898c229e2467f4953314b63e48d4990d3215d876ef9d1fc7c Status:
Downloaded newer image for mongo:4.4
d8f614a4969fb1229f538e171850512f10f490cb1a96fca27e4aa89ac082eba5
Access localhost:8081 to see some output from mongo.
curl localhost:8081
which will return a warning from MongoDB,

It looks like you are trying to access MongoDB over HTTP on the native driver port. If
you are using play-with-docker, look for the 8080 link near the top of the page.
 lOMoARcPSD|319 796 21

Check your running containers with docker container ls

$ docker container ls

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

d6777df89fea nginx "nginx -g 'daemon ..." Less than a second ago Up 2 seconds
0.0.0.0:8080-
>80/tcp nginx
ead80a0db505 mongo "docker-entrypoint..." 17 seconds ago Up 19 seconds 0.0.0.0:8081-
>27017/tcp mongo
af549dccd5cf ubuntu "top" 5 minutes ago Up 5 minutes priceless_kepler

Step 3: Clean Up
First get a list of the containers running using docker container ls.
$ docker container ls
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
d6777df89fea nginx "nginx -g 'daemon ..." 3 minutes ago Up 3 minutes
0.0.0.0:8080-
>80/tcp nginx
ead80a0db505 mongo "docker-entrypoint..." 3 minutes ago
Up 3 minutes 0.0.0.0:8081->27017/tcp mongo
af549dccd5cf ubuntu "top" 8 minutes ago
Up 8 minutes priceless_kepler
Next, run docker container stop [container id] for each container in the list. You can also use the
names of the containers that you specified before.
$ docker container stop
d67 ead af5 d67
ea
d

af5
 lOMoARcPSD|319 796 21

Remove the stopped containers

docker system prune is a really handy command to clean up your system. It will remove any
stopped containers, unused volumes and networks, and dangling images.
$ docker system
prune WARNING!
This will remove:
- all stopped containers
- all volumes not used by at least one container
- all networks not used by at least one container
- all dangling images

Are you sure you want to

continue? [y/N] y Deleted

- Containers:
7872fd96ea4695795c41150a06067d605f69702dbcb9ce49492c9029f0e1b44b

60abd5ee65b1e2732ddc02b971a86e22de1c1c446dab165462a08b037ef7835c
31617fdd8e5f584c51ce182757e24a1c9620257027665c20be75aa3ab6591740

Total reclaimed space: 12B

lOMoARcPSD|319 796 21