Scribd
Upload
20 views5 pages

Hadoop Commands

The document details a series of command line operations executed in a Windows environment for a Hadoop matrix multiplication project. It includes steps for starting Hadoop services, compiling Java classes, creating HDFS directories, and running a MapReduce job. The job successfully processes input data and outputs results to HDFS, demonstrating the use of Hadoop for matrix multiplication.

Uploaded by

abhishekbvabhi2004
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
20 views5 pages

Hadoop Commands

The document details a series of command line operations executed in a Windows environment for a Hadoop matrix multiplication project. It includes steps for starting Hadoop services, compiling Java classes, creating HDFS directories, and running a MapReduce job. The job successfully processes input data and outputs results to HDFS, demonstrating the use of Hadoop for matrix multiplication.

Uploaded by

abhishekbvabhi2004
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd

Microsoft Windows [Version 10.0.26100.

4652]
(c) Microsoft Corporation. All rights reserved.

C:\Windows\System32>[Link]

C:\Windows\System32>[Link]
starting yarn daemons

C:\Windows\System32>jps
14756 DataNode
14536 NameNode
11292 ResourceManager
13980 Jps
14476 NodeManager

C:\Windows\System32>D:

D:\>cd MatrixMultiplicationProject

D:\MatrixMultiplicationProject>hdfs dfs -mkdir -p /user/hadoop/matrix_input

D:\MatrixMultiplicationProject>hdfs dfs -put matrix_data.txt /user/hadoop/matrix_input/


put: `/user/hadoop/matrix_input/matrix_data.txt': File exists

D:\MatrixMultiplicationProject>for /f "usebackq delims=" %i in (`hadoop classpath`) do set


HADOOP_CLASSPATH=%i

D:\MatrixMultiplicationProject>set
HADOOP_CLASSPATH=D:\Hadoop\hadoop-3.3.6\etc\hadoop;D:\Hadoop\hadoop-3.3.6\share\h
adoop\common;D:\Hadoop\hadoop-3.3.6\share\hadoop\common\lib\*;D:\Hadoop\hadoop-3.3.6\
share\hadoop\common\*;D:\Hadoop\hadoop-3.3.6\share\hadoop\hdfs;D:\Hadoop\hadoop-3.3.6\
share\hadoop\hdfs\lib\*;D:\Hadoop\hadoop-3.3.6\share\hadoop\hdfs\*;D:\Hadoop\hadoop-3.3.6\
share\hadoop\yarn;D:\Hadoop\hadoop-3.3.6\share\hadoop\yarn\lib\*;D:\Hadoop\hadoop-3.3.6\s
hare\hadoop\yarn\*;D:\Hadoop\hadoop-3.3.6\share\hadoop\mapreduce\*

D:\MatrixMultiplicationProject>echo %HADOOP_CLASSPATH%
D:\Hadoop\hadoop-3.3.6\etc\hadoop;D:\Hadoop\hadoop-3.3.6\share\hadoop\common;D:\Hadoo
p\hadoop-3.3.6\share\hadoop\common\lib\*;D:\Hadoop\hadoop-3.3.6\share\hadoop\common\*;
D:\Hadoop\hadoop-3.3.6\share\hadoop\hdfs;D:\Hadoop\hadoop-3.3.6\share\hadoop\hdfs\lib\*;D:
\Hadoop\hadoop-3.3.6\share\hadoop\hdfs\*;D:\Hadoop\hadoop-3.3.6\share\hadoop\yarn;D:\Had
oop\hadoop-3.3.6\share\hadoop\yarn\lib\*;D:\Hadoop\hadoop-3.3.6\share\hadoop\yarn\*;D:\Had
oop\hadoop-3.3.6\share\hadoop\mapreduce\*
D:\MatrixMultiplicationProject>javac -classpath %HADOOP_CLASSPATH% -d classes
[Link] [Link] [Link]

D:\MatrixMultiplicationProject>jar -cvf [Link] -C classes/ .


added manifest
adding: [Link](in = 1916) (out= 1009)(deflated 47%)
adding: [Link](in = 2775) (out= 1204)(deflated 56%)
adding: [Link](in = 2562) (out= 1162)(deflated 54%)

D:\MatrixMultiplicationProject>hdfs dfs -mkdir -p /user/hadoop/matrix_input

D:\MatrixMultiplicationProject>hdfs dfs -put matrix_data.txt /user/hadoop/matrix_input/


put: `/user/hadoop/matrix_input/matrix_data.txt': File exists

D:\MatrixMultiplicationProject>hadoop jar [Link] MatrixMultiplyDriver


/user/hadoop/matrix_input /user/hadoop/matrix_output 2 2 2
Exception in thread "main" [Link]: MatrixMultiplyDriver has
been compiled by a more recent version of the Java Runtime (class file version 55.0), this
version of the Java Runtime only recognizes class file versions up to 52.0
at [Link].defineClass1(Native Method)
at [Link]([Link])
at [Link]([Link])
at [Link]([Link])
at [Link]$100([Link])
at [Link]$[Link]([Link])
at [Link]$[Link]([Link])
at [Link](Native Method)
at [Link]([Link])
at [Link]([Link])
at [Link]([Link])
at [Link].forName0(Native Method)
at [Link]([Link])
at [Link]([Link])
at [Link]([Link])

D:\MatrixMultiplicationProject>javac -classpath %HADOOP_CLASSPATH% -source 1.8 -target


1.8 -d classes [Link] [Link] [Link]
warning: [options] bootstrap class path not set in conjunction with -source 8
1 warning

D:\MatrixMultiplicationProject>jar -cvf [Link] -C classes/ .


added manifest
adding: [Link](in = 1916) (out= 1010)(deflated 47%)
adding: [Link](in = 2511) (out= 1087)(deflated 56%)
adding: [Link](in = 2562) (out= 1162)(deflated 54%)

D:\MatrixMultiplicationProject>hdfs dfs -mkdir -p /user/hadoop/matrix_input

D:\MatrixMultiplicationProject>hdfs dfs -put matrix_data.txt /user/hadoop/matrix_input/


put: `/user/hadoop/matrix_input/matrix_data.txt': File exists

D:\MatrixMultiplicationProject>hdfs dfs -rm /user/hadoop/matrix_input/matrix_data.txt


Deleted /user/hadoop/matrix_input/matrix_data.txt

D:\MatrixMultiplicationProject>hdfs dfs -put matrix_data.txt /user/hadoop/matrix_input/

D:\MatrixMultiplicationProject>hdfs dfs -rm -r /user/hadoop/matrix_output


rm: `/user/hadoop/matrix_output': No such file or directory

D:\MatrixMultiplicationProject>hadoop jar [Link] MatrixMultiplyDriver


/user/hadoop/matrix_input /user/hadoop/matrix_output 2 2 2
2025-08-07 [Link],924 INFO [Link]: Connecting to
ResourceManager at /[Link]:8032
2025-08-07 [Link],278 WARN [Link]: Hadoop command-line
option parsing not performed. Implement the Tool interface and execute your application with
ToolRunner to remedy this.
2025-08-07 [Link],380 INFO [Link]: Disabling Erasure Coding
for path: /tmp/hadoop-yarn/staging/cse/.staging/job_1754550781839_0001
2025-08-07 [Link],716 INFO [Link]: Total input files to process : 1
2025-08-07 [Link],993 INFO [Link]: number of splits:1
2025-08-07 [Link],168 INFO [Link]: Submitting tokens for job:
job_1754550781839_0001
2025-08-07 [Link],169 INFO [Link]: Executing with tokens: []
2025-08-07 [Link],277 INFO [Link]: [Link] not found
2025-08-07 [Link],277 INFO [Link]: Unable to find '[Link]'.
2025-08-07 [Link],625 INFO [Link]: Submitted application
application_1754550781839_0001
2025-08-07 [Link],656 INFO [Link]: The url to track the job:
[Link]
2025-08-07 [Link],657 INFO [Link]: Running job: job_1754550781839_0001
2025-08-07 [Link],813 INFO [Link]: Job job_1754550781839_0001 running in uber
mode : false
2025-08-07 [Link],816 INFO [Link]: map 0% reduce 0%
2025-08-07 [Link],895 INFO [Link]: map 100% reduce 0%
2025-08-07 [Link],969 INFO [Link]: map 100% reduce 100%
2025-08-07 [Link],005 INFO [Link]: Job job_1754550781839_0001 completed
successfully
2025-08-07 [Link],133 INFO [Link]: Counters: 54
File System Counters
FILE: Number of bytes read=198
FILE: Number of bytes written=556117
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=199
HDFS: Number of bytes written=28
HDFS: Number of read operations=8
HDFS: Number of large read operations=0
HDFS: Number of write operations=2
HDFS: Number of bytes read erasure-coded=0
Job Counters
Launched map tasks=1
Launched reduce tasks=1
Data-local map tasks=1
Total time spent by all maps in occupied slots (ms)=2189
Total time spent by all reduces in occupied slots (ms)=2551
Total time spent by all map tasks (ms)=2189
Total time spent by all reduce tasks (ms)=2551
Total vcore-milliseconds taken by all map tasks=2189
Total vcore-milliseconds taken by all reduce tasks=2551
Total megabyte-milliseconds taken by all map tasks=2241536
Total megabyte-milliseconds taken by all reduce tasks=2612224
Map-Reduce Framework
Map input records=8
Map output records=16
Map output bytes=160
Map output materialized bytes=198
Input split bytes=127
Combine input records=0
Combine output records=0
Reduce input groups=4
Reduce shuffle bytes=198
Reduce input records=16
Reduce output records=4
Spilled Records=32
Shuffled Maps =1
Failed Shuffles=0
Merged Map outputs=1
GC time elapsed (ms)=56
CPU time spent (ms)=841
Physical memory (bytes) snapshot=647811072
Virtual memory (bytes) snapshot=1043279872
Total committed heap usage (bytes)=588251136
Peak Map Physical memory (bytes)=347095040
Peak Map Virtual memory (bytes)=528916480
Peak Reduce Physical memory (bytes)=300716032
Peak Reduce Virtual memory (bytes)=514457600
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=72
File Output Format Counters
Bytes Written=28

D:\MatrixMultiplicationProject>hdfs dfs -ls /user/hadoop/matrix_output


Found 2 items
-rw-r--r-- 1 cse supergroup 0 2025-08-07 12:49 /user/hadoop/matrix_output/_SUCCESS
-rw-r--r-- 1 cse supergroup 28 2025-08-07 12:49 /user/hadoop/matrix_output/part-r-00000

D:\MatrixMultiplicationProject>hdfs dfs -cat /user/hadoop/matrix_output/part-r-00000


0,0 19
0,1 22
1,0 43
1,1 50

D:\MatrixMultiplicationProject>

You might also like