Install Cloudera Manager using AMI on AWS EC2:
Prerequisites:
Region- N.Virginia (You need to select N.Virginia as the region as this is the cheapest
region & our AMI is available on N.Virginia)
AMI Name :-Cloudera Manager_upgrad
AMI ID-: ami-0603830fb9820a125
1. Log in to the AWS console home and click on ‘EC2’ under the Comput Section inside
the N.Virginia Region.
2. Click on ‘AMIs’ on the left side.
3. Click on ‘Public Images’ option in the drop-down menu(top left of the main section)
found left of search tab.
4. Inside the search, tab enter our AMI ID:ami-0603830fb9820a125 and Select our AMI
and click ‘Launch’ button on the top left corner.
5. Select instance type ‘m4.xlarge’ (please select very carefully), and click on ‘Next
Configure Instance details’.
6. We recommend you to select your vpc name and enable auto-assign public ip.
Number of instances- 1
VPC - my_vpc
Auto-assign Public ip- Enable
and click on ‘Add Storage’ button (bottom right).
7. We recommend you to add 60GB of storage, and volume type should be ‘Magnetic’
for lower prices. (as pricing is based on the VM usage and configuration). Finally click
on ‘Next: Add Tags’ button.
8. Now click on ‘click to add a Name tag’ as shown in the image below.
9. Give a name in the cell under ‘Value’. In our case, we used Cloudera Box.
10. Click on ‘Next: Configure Security Group’.and select the ‘ Select an existing
security group’ and click on “Review and launch”
Example: Security group name- cloudera
11. Click on ‘Review and Launch’. The following page will appear as shown in the
image below. Click on ‘Continue with Magnetic as the boot volume for this
instance’. Then click ‘Next’
12. Click on ‘Launch’ button.
13. Now select ‘Choose an existing key pair’ from the drop-down menu and select
your key pair name (in our case, RHEL). Tick the 'I acknowledge….’ and click on
‘Launch Instances’.
14. Click on ‘View Instances’ and your instances will appear on the screen.
15. Wait until the status check is 2/2 (under ‘Status Checks’ column)..
16. Now Access/ log in the instance using putty for window users and install the
Hadoop(CDH).
For window users: need .ppk file (convert your RHEL.pem to name.ppk file using
PuTTYgen software).
Note:- if you have already .ppk file please ignore it.
17. Now, open your EC2 dashboard and select your instance. Copy your ‘Public DNS
(IPv4)’ information as shown in the screenshot.
18. Open putty:
Under ‘Host Name’ section, paste the public DNS information of your instance that you
just copied.
19. On the left-hand side panel, click on ‘Connection’. Then click on ‘SSH’ followed by
‘Auth’. In the private key field, click on ‘Browse’.
20. Select the .ppk file(RHEL.ppk) you generated using PuTTYgen and click on ‘Open’.
Then, click on open in the PuTTY software.
21. Click on ‘Yes’ and login with: ec2-user
22. Switch to root user using the command
‘sudo -i ’ ( enter all commands without quotes unless it is mentioned).
23. Run the command ‘ls’ and verify the ‘cloudera bin’ is available or not.
24. Next, run the bin file using the command given below and press the enter
‘ ./cloudera-manager-installer.bin ’
25. Cloudera Manager will start, and the screen will look like the image below. Select
‘next’ and press the enter.
26. Select next and press enter :
27. Finally, to accept the licence - select yes
28. Click on Next and Accept the Oracle Binary licence
29. Wait until the Cloudera manager is installed and click on ok. (as shown in below
image)
30. Now open a web browser on your local machine; go to the following address
<public-domain-of-ec2>:7180. This way you will be able to access the Cloudera
Manager service that’s running on the EC2 instance.
● To get the public domain of ec2, go to the EC2 dashboard, and click on your
instance. Then, copy your ‘Public DNS’ or Public IP.
31. Put the copied public IP in place of <public-domain-of-ec2> in
<public-domain-of-ec2>:7180, and open it in your browser. The following page will
appear after a few minutes.
32. Now, log in to Cloudera Manager using username: admin and password: admin.
33. Accept the terms and conditions, and then, keep clicking on ‘Continue’.
34. Choose “Cloudera Express for free” and click on continue
35. Click on ‘Continue’:
36. When you’re asked to search for the instance, give your ‘Private IP’.
37. After this, the following screen will appear. Type your private IP in the box, and click on
‘Search’
38. Select your private IP from the search results and click on ‘Continue’
39. Click on ‘More option’:
Remove the {latest_supported} from the “Remote Parcel Repository URLs” and fix the version
5.15.1 and remove other URL using Minus box and click on “save changes”
Finally below screen like looks.
Now select the CDH Version and click on continue.
40. Select the JDK and encryption policy then click on ‘Continue’.
41. Click ‘Continue’ again.
42. After this, the following screen will appear. Select Another User and type ec2-user.
Select ‘All hosts accept same private key’ as authentication method. Browse the
.pem file in the Private Key File option, and click on ‘Continue’.
● Our case: RHEL.pem
- Click on ‘Continue’
- Click on ‘continue’:
44. Click on ‘finish’.
Note: Do not add the spark service- later we will add the spark 2.3.0 Service.
45. choose Core with HBase.
- Click on ‘Continue’
46. Select ‘use custom data bases’.
Note :- we already installed MySQL on that AMI with the databases metastore
rman hue and oozie.
47. Please keep all the usernames and passwords for all the services: Hive, Hue, etc. in
a safe place. After paste/ enter these database name and passwords, scroll down, and
then, click on ‘Test Connections’.
Port NUMBER is 3306.
For hive service:- database name: metastore and password is admin.
For rman service:- database name: rman and password is admin
For hue service:- database name: hue and password is admin
For oozie service :- database name: oozie and password is admin
- Click on Continue:
- Deploying all the services takes some time. After this is complete, click on
Continue
- The following screen will appear on the successful set-up of the cluster. Then,
click on Finish.
48. After this, the following image will appear. Kindly check whether all the services are
green. Please note that it takes some time for all the services to be active
49. Click on ‘Critical Health Issue(s)’ (if any).
50. Click on ‘suppress’:
- Click on Confirm:
- Click on Close.
- Now all services are green
51. Set the java path for cloudera-scm-manager or other users:
Again login to ec2 instance using putty and switch to root users;
ls /usr/java/jdk1.7.0_67-cloudera/ ( enter command without quotes).
vi /etc/default/cloudera-scm-server
Note :- vi takes us to vim text editor. To enter any text we need to enter into insert mode
which is achieved by pressing ‘i’ and to ‘save and quit’ the doc, we need to hit ‘esc’
followed by :wq! and hit enter.
Then, press ‘ i ’ (without quotes) for insert mode, and paste this command as shown:
export JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera/
For others users: open /etc/profile and go to end of the file and insert the variable
value.
vi /etc/profile (command)
export JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera/
export JRE_HOME=/usr/java/jdk1.7.0_67-cloudera/jre/
export PATH=$JAVA_HOME/bin:$PATH
Then update the /etc/profile using the below commands:
source /etc/profile
Verify the java and JRE version using the below commands as shown in the image
below:
java -version
echo $JAVA_HOME
-----
Note: Please stop the instance if you are not using from the ec2 dashboard.
If your instance is running then the cost is Rs.19/hours
After I stopped my instance and restarted it, my Cloudera management services and
other services are stopped into my browser. The following screen appeared.
A. Start the instance from the EC2 dashboard and wait until the status check is 2/2.
B. Copy the public IP and paste into the browser
http://publicip:7180
To solve this problem, go to the page above, scroll down, and click on ‘Cloudera
Management Service’; then, click on ‘Restart’
Click on ‘Restart’:
Click on ‘Close’:
Now Restart the cluster1 services:
Click on Restart:
Click on ‘Close’.
All services in green.
