Hadoop Commands
hadoop@aceec:~$ hadoop
Usage: hadoop [OPTIONS] SUBCOMMAND [SUBCOMMAND OPTIONS]
or hadoop [OPTIONS] CLASSNAME [CLASSNAME OPTIONS]
where CLASSNAME is a user-provided Java class
OPTIONS is none or any of:
buildpaths attempt to add class files from build tree
--config dir Hadoop config directory
--debug turn on shell script debug mode
--help usage information
hostnames list[,of,host,names] hosts to use in slave mode
hosts filename list of hosts to use in slave mode
loglevel level set the log4j level for this command
workers turn on worker mode
SUBCOMMAND is one of:
Admin Commands:
daemonlog get/set the log level for each daemon
Client Commands:
archive create a Hadoop archive
checknative check native Hadoop and compression libraries availability
classpath prints the class path needed to get the Hadoop jar and the
required libraries
conftest validate configuration XML files
credential interact with credential providers
distch distributed metadata changer
distcp copy file or directories recursively
dtutil operations related to delegation tokens
envvars display computed Hadoop environment variables
fs run a generic filesystem user client
gridmix submit a mix of synthetic job, modeling a profiled from
production load
jar <jar> run a jar file. NOTE: please use "yarn jar" to launch YARN
applications, not this command.
jnipath prints the java.library.path
kdiag Diagnose Kerberos Problems
kerbname show auth_to_local principal conversion
key manage keys via the KeyProvider
rumenfolder scale a rumen input trace
rumentrace convert logs into a rumen trace
s3guard manage metadata on S3
trace view and modify Hadoop tracing settings
version print the version
Daemon Commands:
kms run KMS, the Key Management Server
registrydns run the registry DNS server
SUBCOMMAND may print help when invoked w/o parameters or with -h.
hadoop@aceec:~$ jps
3088 Jps
hadoop@aceec:~$ start-all.sh
WARNING: Attempting to start all Apache Hadoop daemons as hadoop in 10 seconds.
WARNING: This is not a recommended production deployment configuration.
WARNING: Use CTRL-C to abort.
Starting namenodes on [localhost]
Starting datanodes
Starting secondary namenodes [aceec]
Starting resourcemanager
Starting nodemanagers
hadoop@aceec:~$ jps
4449 Jps
3330 NameNode
3458 DataNode
4082 NodeManager
3672 SecondaryNameNode
3948 ResourceManager
hadoop@aceec:~$ hadoop dfsadmin -safemode get
WARNING: Use of this script to execute dfsadmin is deprecated.
WARNING: Attempting to execute replacement "hdfs dfsadmin" instead.
Safe mode is OFF
hadoop@aceec:~$ hadoop dfsadmin -safemode enter
WARNING: Use of this script to execute dfsadmin is deprecated.
WARNING: Attempting to execute replacement "hdfs dfsadmin" instead.
Safe mode is ON
hadoop@aceec:~$ hadoop fs -mkdir /PADMINI_HDFS
mkdir: Cannot create directory /PADMINI_HDFS. Name node is in safe mode.
hadoop@aceec:~$ hadoop dfsadmin -safemode leave
WARNING: Use of this script to execute dfsadmin is deprecated.
WARNING: Attempting to execute replacement "hdfs dfsadmin" instead.
Safe mode is OFF
hadoop@aceec:~$ hadoop fs -mkdir /PADMINI_HDFS
hadoop@aceec:~$ hadoop fs -ls /
Found 3 items
drwxr-xr-x - hadoop supergroup 0 2023-03-16 14:12 /PADMINI_HDFS
drwxrwxr-x - hadoop supergroup 0 2023-02-11 16:04 /tmp
drwxr-xr-x - hadoop supergroup 0 2023-02-11 16:16 /user
hadoop@aceec:~$ hadoop fs -put input.log /PADMINI_HDFS
hadoop@aceec:~$ hadoop fs -ls /
Found 3 items
drwxr-xr-x - hadoop supergroup 0 2023-03-16 14:12 /PADMINI_HDFS
drwxrwxr-x - hadoop supergroup 0 2023-02-11 16:04 /tmp
drwxr-xr-x - hadoop supergroup 0 2023-02-11 16:16 /user
hadoop@aceec:~$ hadoop fs -ls /PADMINI_HDFS
/* Create a file in folder PADMINI with name inputfile.log before executing the following
command*/
hadoop@aceec:~$ cd PADMINI/
hadoop@aceec:~/PADMINI$ hadoop fs -put inputfile.log /PADMINI_HDFS
hadoop@aceec:~/PADMINI$ cd\
>
hadoop@aceec:~$ hadoop fs -ls /
Found 3 items
drwxr-xr-x - hadoop supergroup 0 2023-03-16 14:32 /PADMINI_HDFS
drwxrwxr-x - hadoop supergroup 0 2023-02-11 16:04 /tmp
drwxr-xr-x - hadoop supergroup 0 2023-02-11 16:16 /user
hadoop@aceec:~$ hadoop fs -ls /PADMINI_HDFS
Found 1 items
-rw-r--r-- 1 hadoop supergroup 107 2023-03-16 14:32 /PADMINI_HDFS/inputfile.log
hadoop@aceec:~$ hadoop fs -cat /PADMINI_HDFS/inputfile.log
This is Big Data Sample File.
Hadoop is used to maintain Big Data.
After Saving file Editing the document.
hadoop@aceec:~$ cd PADMINI/
hadoop@aceec:~/PADMINI$ hadoop fs -moveFromLocal inputNew.log /PADMINI_HDFS
hadoop@aceec:~/PADMINI$ cd\
>
hadoop@aceec:~$ hadoop fs -ls /PADMINI_HDFS
Found 2 items
-rw-r--r-- 1 hadoop supergroup 107 2023-03-16 14:43 /PADMINI_HDFS/inputNew.log
-rw-r--r-- 1 hadoop supergroup 107 2023-03-16 14:32 /PADMINI_HDFS/inputfile.log
hadoop@aceec:~$ cd PADMINI/
hadoop@aceec:~/PADMINI$ hadoop fs -copyFromLocal Sample.log /PADMINI_HDFS
hadoop@aceec:~/PADMINI$ pwd
/home/hadoop/PADMINI
hadoop@aceec:~/PADMINI$ cd\
>
hadoop@aceec:~$ hadoop fs -ls /PADMINI_HDFS
Found 3 items
-rw-r--r-- 1 hadoop supergroup 0 2023-03-16 14:44 /PADMINI_HDFS/Sample.log
-rw-r--r-- 1 hadoop supergroup 107 2023-03-16 14:43 /PADMINI_HDFS/inputNew.log
-rw-r--r-- 1 hadoop supergroup 107 2023-03-16 14:32 /PADMINI_HDFS/inputfile.log
hadoop@aceec:~$ pwd
/home/Hadoop
hadoop@aceec:~$ hadoop fs -mkdir /PADMINI_HDFS1
hadoop@aceec:~$ hadoop fs -mv /PADMINI_HDFS/Sample.log /PADMINI_HDFS1
hadoop@aceec:~$ hadoop fs -ls /PADMINI_HDFS1
Found 1 items
-rw-r--r-- 1 hadoop supergroup 0 2023-03-16 14:44 /PADMINI_HDFS1/Sample.log
hadoop@aceec:~$ hadoop fs -ls /PADMINI_HDFS
Found 2 items
-rw-r--r-- 1 hadoop supergroup 107 2023-03-16 14:43 /PADMINI_HDFS/inputNew.log
-rw-r--r-- 1 hadoop supergroup 107 2023-03-16 14:32 /PADMINI_HDFS/inputfile.log
hadoop@aceec:~$ hadoop dfsadmin -report
WARNING: Use of this script to execute dfsadmin is deprecated.
WARNING: Attempting to execute replacement "hdfs dfsadmin" instead.
Configured Capacity: 260152410112 (242.29 GB)
Present Capacity: 218281312374 (203.29 GB)
DFS Remaining: 218281263104 (203.29 GB)
DFS Used: 49270 (48.12 KB)
DFS Used%: 0.00%
Replicated Blocks:
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
Missing blocks (with replication factor 1): 0
Low redundancy blocks with highest priority to recover: 0
Pending deletion blocks: 0
Erasure Coded Block Groups:
Low redundancy block groups: 0
Block groups with corrupt internal blocks: 0
Missing block groups: 0
Low redundancy blocks with highest priority to recover: 0
Pending deletion blocks: 0
-------------------------------------------------
Live datanodes (1):
Name: 127.0.0.1:9866 (localhost)
Hostname: aceec
Decommission Status : Normal
Configured Capacity: 260152410112 (242.29 GB)
DFS Used: 49270 (48.12 KB)
Non DFS Used: 28581552010 (26.62 GB)
DFS Remaining: 218281263104 (203.29 GB)
DFS Used%: 0.00%
DFS Remaining%: 83.91%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 0
Last contact: Thu Mar 16 14:51:01 IST 2023
Last Block Report: Thu Mar 16 14:09:52 IST 2023
Num of Blocks: 2
hadoop@aceec:~$ hadoop fs -touch /PADMINI_HDFS/MyFile.txt
hadoop@aceec:~$ hadoop fs -ls /PADMINI_HDFS
Found 3 items
-rw-r--r-- 1 hadoop supergroup 0 2023-03-16 14:52 /PADMINI_HDFS/MyFile.txt
-rw-r--r-- 1 hadoop supergroup 107 2023-03-16 14:43 /PADMINI_HDFS/inputNew.log
-rw-r--r-- 1 hadoop supergroup 107 2023-03-16 14:32 /PADMINI_HDFS/inputfile.log
hadoop@aceec:~$ hadoop fs -ls /PADMINI_HDFS1
Found 1 items
-rw-r--r-- 1 hadoop supergroup 0 2023-03-16 14:44 /PADMINI_HDFS1/Sample.log
hadoop@aceec:~$ hadoop fs -mv /PADMINI_HDFS/MyFile.txt /PADMINI_HDFS1
hadoop@aceec:~$ hadoop fs -ls /PADMINI_HDFS
Found 2 items
-rw-r--r-- 1 hadoop supergroup 107 2023-03-16 14:43 /PADMINI_HDFS/inputNew.log
-rw-r--r-- 1 hadoop supergroup 107 2023-03-16 14:32 /PADMINI_HDFS/inputfile.log
hadoop@aceec:~$ hadoop fs -ls /PADMINI_HDFS1
Found 2 items
-rw-r--r-- 1 hadoop supergroup 0 2023-03-16 14:52 /PADMINI_HDFS1/MyFile.txt
-rw-r--r-- 1 hadoop supergroup 0 2023-03-16 14:44 /PADMINI_HDFS1/Sample.log
hadoop@aceec:~$
Copying a file from local disk to HDFS is done with the hadoop command:
$ hadoop fs -put local-file.txt hdfs-file.txt
The behavior of the Hadoop -put command differs from the Linux cp command—in Linux if the
destination already exists, it is overwritten; in Hadoop the copy fails with an error:
put: `hdfs-file.txt': File exists
The -f option must be added to force the file to be overwritten:
$ hadoop fs -put -f local-file.txt hdfs-file.txt
Much like with the Linux cp command, multiple files can be copied using the same command. In
this case, the final argument must be the directory in HDFS into which the local files are copied:
$ hadoop fs -put local-file1.txt local-file2.txt /hdfs/dest/
