In this blog post I will explain different HDFS commands to access HDFS which are commonly used while working as a Big Data Developer Training.<\/p>\n
[npntraining@centos8 Desktop]$ hdfs -help\nUsage: hdfs [--config confdir] [--loglevel loglevel] COMMAND\n where COMMAND is one of:\n dfs run a filesystem command on the file systems supported in Hadoop.\n classpath prints the classpath\n namenode -format format the DFS filesystem\n secondarynamenode run the DFS secondary namenode\n namenode run the DFS namenode\n journalnode run the DFS journalnode\n zkfc run the ZK Failover Controller daemon\n datanode run a DFS datanode\n dfsadmin run a DFS admin client\n haadmin run a DFS HA admin client\n fsck run a DFS filesystem checking utility\n balancer run a cluster balancing utility\n jmxget get JMX exported values from NameNode or DataNode.\n mover run a utility to move block replicas across\n storage types\n oiv apply the offline fsimage viewer to an fsimage\n oiv_legacy apply the offline fsimage viewer to an legacy fsimage\n oev apply the offline edits viewer to an edits file\n fetchdt fetch a delegation token from the NameNode\n getconf get config values from configuration\n groups get the groups which users belong to\n snapshotDiff diff two snapshots of a directory or diff the\n current directory contents with a snapshot\n lsSnapshottableDir list all snapshottable dirs owned by the current user\n Use -help to see options\n portmap run a portmap service\n nfs3 run an NFS version 3 gateway\n cacheadmin configure the HDFS cache\n crypto configure HDFS encryption zones\n storagepolicies list\/get\/set block storage policies\n version print the version\n<\/code><\/pre>\nList of all HDFS Commands<\/strong><\/p>\n[naveen@npntraining ~]$ hdfs dfs -help\nUsage: hadoop fs [generic options]\n [-appendToFile <localsrc> ... <dst>]\n [-cat [-ignoreCrc] <src> ...]\n [-checksum <src> ...]\n [-chgrp [-R] GROUP PATH...]\n [-chmod [-R] <MODE[,MODE]... | OCTALMODE> PATH...]\n [-chown [-R] [OWNER][:[GROUP]] PATH...]\n [-copyFromLocal [-f] [-p] [-l] <localsrc> ... <dst>]\n [-copyToLocal [-p] [-ignoreCrc] [-crc] <src> ... <localdst>]\n [-count [-q] [-h] <path> ...]\n [-cp [-f] [-p | -p[topax]] <src> ... <dst>]\n [-createSnapshot <snapshotDir> [<snapshotName>]]\n [-deleteSnapshot <snapshotDir> <snapshotName>]\n [-df [-h] [<path> ...]]\n [-du [-s] [-h] <path> ...]\n [-expunge]\n [-find <path> ... <expression> ...]\n [-get [-p] [-ignoreCrc] [-crc] <src> ... <localdst>]\n [-getfacl [-R] <path>]\n [-getfattr [-R] {-n name | -d} [-e en] <path>]\n [-getmerge [-nl] <src> <localdst>]\n [-help [cmd ...]]\n [-ls [-d] [-h] [-R] [<path> ...]]\n [-mkdir [-p] <path> ...]\n [-moveFromLocal <localsrc> ... <dst>]\n [-moveToLocal <src> <localdst>]\n [-mv <src> ... <dst>]\n [-put [-f] [-p] [-l] <localsrc> ... <dst>]\n [-renameSnapshot <snapshotDir> <oldName> <newName>]\n [-rm [-f] [-r|-R] [-skipTrash] <src> ...]\n [-rmdir [--ignore-fail-on-non-empty] <dir> ...]\n [-setfacl [-R] [{-b|-k} {-m|-x <acl_spec>} <path>]|[--set <acl_spec> <path>]]\n [-setfattr {-n name [-v value] | -x name} <path>]\n [-setrep [-R] [-w] <rep> <path> ...]\n [-stat [format] <path> ...]\n [-tail [-f] <file>]\n [-test -[defsz] <path>]\n [-text [-ignoreCrc] <src> ...]\n [-touchz <path> ...]\n [-truncate [-w] <length> <path> ...]\n [-usage [cmd ...]]<\/code><\/pre>\n-ls<\/h2>\n\nThis command is used for listing the directories and files present under under a specific directory in an HDFS.<\/p>\n<\/blockquote>\n
Usage<\/em><\/strong> : hdfs dfs [generic options] -ls [-d] [-h] [-R] [ \u2026]<\/p>\n[npntraining ~]$ hdfs dfs -ls \/user\/$USERNAME\/hdfs_commands<\/code><\/pre>\nNote<\/strong><\/p>\n\n- -d is used to list the directories as plain files.<\/li>\n
- -h is used to print file size in human readable format.<\/li>\n
- -R is used to recursively list the content of the directories.<\/li>\n<\/ul>\n
-mkdir<\/h2>\n\nThis command is similar to that of Unix mkdir and is used to create a directory in HDFS.<\/p>\n<\/blockquote>\n
Usage<\/strong> : <\/p>\nhdfs dfs -mkdir [-p] \/hdfs-path<\/code><\/pre>\n\n\n\nOptions<\/strong><\/th>\nDescription<\/th>\n<\/tr>\n<\/thead>\n \n\n-p<\/td>\n Mention not to fail if the directory already exists.<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n[npntraining ~]$ hdfs dfs -mkdir \/user\/$USER\/hdfs_commands<\/code><\/pre>\nNote<\/strong><\/p>\n\n- If the directory already exists or if intermediate directories doesn\u2019t exists then it will throws an error. In order to overcome that error we will be using -p<\/strong> (parent), which not only ignores if the directory already exists but also create the intermediate directories if they doesn\u2019t exists<\/li>\n<\/ul>\n
[npntraining ~]$ hdfs dfs -ls \/<\/code><\/pre>\n-cat<\/h2>\n\nThis command is used for displaying the contents of a file on the console.<\/p>\n<\/blockquote>\n
Usage<\/strong> : hdfs dfs [-cat [-ignoreCrc] \u2026]<\/p>\nNote<\/strong><\/p>\n\n- -ignoreCrc option will disable the checksum verification.<\/li>\n<\/ul>\n
-copyFromLocal &<\/h2>\n\nThis command is used to copy files from the local file system to the HDFS file system.<\/p>\n<\/blockquote>\n
Usage<\/strong><\/p>\nhdfs dfs [generic options] -copyFromLocal [-f] [-p] [-l] \u2026 <\/p>\n\n- -f overwrites the destination if it already exists.<\/li>\n
- -p preserves access and modification times, ownership and the permissions<\/li>\n
- -d : Skip creation of temporary file with the suffix .COPYING<\/em>.<\/li>\n<\/ul>\n
Using -copyFromLocal<\/u><\/p>\n[npntraining ~]$ hdfs dfs -copyFromLocal <localfile_path1> <localfile_path2> \/<hdfs-path><\/code><\/pre>\n-put<\/h2>\n
Usage<\/strong> : <\/p>\nhdfs dfs -put [-f] [-p] [-l] [-d] [ \u2014 | .. ]. <\/p>\n\n- -f overwrites the destination if it already exists.<\/li>\n
- -p preserves access and modification times, ownership and the permissions<\/li>\n
- -d skips creation of temporary file with the suffix .COPYING<\/em>.<\/li>\n
- -l allows Data Node to lazily persist the file to disk, Forces a replication factor of 1.<\/li>\n<\/ul>\n
[npntraining~]$> hdfs dfs -put file1.txt hdfs:\/\/localhost.localdomain:9000\/file1.txt<\/code><\/pre>\nImportant Note:<\/strong><\/p>\nThe fundamental difference between -copyFromLocal and -put is the <\/p>\n
\n- -put can take input from stdin.<\/li>\n<\/ol>\n
[npntraining]$> echo "Hello Naveen" | hdfs dfs -put - \/file1.txt<\/code><\/pre>\n-moveFromLocal<\/h2>\n\nThis command will move the file from local file system to HDFS. It ensures that the local copy is deleted.<\/p>\n<\/blockquote>\n
Usage<\/strong><\/p>\nhdfs dfs -moveFromLocal <\/p>\n[npntraining]$> hdfs dfs -mv \/<hdfs-path> \/<hdfs-path><\/code><\/pre>\nSimilar to \u2013copyFromLocal command, but it follows cut and paste approach but within a file system<\/p>\n
[npntraining]$ > hdfs dfs -mv file1.txt \/file1.txt<\/code><\/pre>\n-get<\/h2>\n\nThis command is used to copy files from HDFS to the local file system.<\/p>\n<\/blockquote>\n
Usage<\/strong><\/p>\nhdfs dfs [generic options] -get [-f][-p] [-ignoreCrc] [-Crc] \u2026 <\/p>\n[npntraining]$> hdfs dfs -get \/<hdfs-path> <localfile_path1><\/code><\/pre>\n[npntraining]$> hdfs dfs -copyToLocal \/<hdfs-path> <localfile_path1><\/code><\/pre>\n-copyToLocal<\/h2>\n\nThis command is similar to get command except that destination is restricted to the local file system. This command is used to move the file from HDFS to local file system.<\/p>\n<\/blockquote>\n
Usage<\/strong><\/p>\nhdfs dfs -copyToLocal [-f] [-p][-ignoreCrc] [-Crc] URI <\/p>\n-rm<\/h2>\n\nThis command is used to remove a file or directory from HDFS.<\/p>\n<\/blockquote>\n
Usage<\/strong><\/p>\nhdfs dfs -rm [-f] [-r |-R] [-skipTrash] [-safely] URI [URI \u2026]<\/p>\n
\n- \u2013rm option will remove only files but directories can\u2019t be deleted by this command.<\/li>\n
- \u2013skipTrash option is used to bypass the trash then it immediately deletes the source.<\/li>\n
- \u2013f option is used to mention that if there is no file existing.<\/li>\n
- \u2013r option is used to recursively delete directories<\/li>\n
- -safely option will require safety confirmation before deleting directory with total number of files greater than hadoop.shell.delete.limit.num.files (in core-site.xml, default: 100). It can be used with -skipTrash to prevent accidental deletion of large directories.<\/li>\n<\/ul>\n
[npntraining]$> hdfs dfs \u2013rm \/file.txt<\/code><\/pre>\n[npntraining]$> hdfs dfs \u2013rmr \/directory<\/code><\/pre>\n-setrep<\/h2>\n\nThis command changes the replication factor for the particular file\/s or directory recursively. <\/p>\n<\/blockquote>\n
Usage<\/strong><\/p>\nhdfs dfs -setrep [-R] [-w] <numReplicas> <path><\/code><\/pre>\n\n- \u2013w option will wait till replication process to complete because replication will normally take long time to complete.<\/li>\n
- -R option is accepted for backwards compatibility. It has no effect.<\/li>\n<\/ul>\n
[npntraining]$ > hdfs dfs -setrep 4 \/file1.txt<\/code><\/pre>\nWhenever we change replication factor automatic cluster re-balancing is going to happen.<\/p>\n
-touch<\/strong>z<\/h2>\n\nThis command is used to create a file of zero length. An error is returned if the file exists with non-zero length.<\/p>\n<\/blockquote>\n
Usage<\/strong><\/p>\nhdfs dfs -touchz URI [URI \u2026]<\/code><\/pre>\n[npntraining]$ > hdfs dfs -touchz \/file1.txt<\/code><\/pre>\n-stat<\/strong><\/h2>\nPrint statistics of the file\/directory specified in the path with format. Format may contain following options:<\/p>\n
\n- \n
%b: Filesize in blocks.<\/p>\n<\/li>\n
- \n
%F: File type.<\/p>\n<\/li>\n
- \n
%g: group name of owner.<\/p>\n<\/li>\n
- \n
%n: name of the file.<\/p>\n<\/li>\n
- \n
%o: block size occupied by the file\/directory.<\/p>\n<\/li>\n
- \n
%r: number of replicas.<\/p>\n<\/li>\n
- \n
%u: username of owner.<\/p>\n<\/li>\n<\/ol>\n
[npntraining]$ > hdfs dfs -stat "%n %o %u %g %r" \/file.txt<\/code><\/pre>\n-getmerge<\/strong><\/h2>\nTakes a source directory files as input and concatenates files in src into the destination local file.<\/p>\n
[npntraining]$ > hdfs dfs -getmerge \/<hdfsdir> file.txt<\/code><\/pre>\n-appendToFile<\/strong><\/h2>\nAppends the content of local files to HDFS files<\/p>\n
[npntraining]$ > hdfs dfs -appendToFile \/<local_file> \/<local_file> \/<hdfs-file><\/code><\/pre>\n-text<\/strong><\/p>\n-test [-ezd]<\/strong><\/h2>\nThe -test is used for file test operations<\/p>\n
\n- \u2013e: If the path exists, returns 0.<\/li>\n
- \u2013z: If the file is zero length, returns 0<\/li>\n
- \u2013d:If the path is directory, return 0;<\/li>\n<\/ol>\n
[npntraining]$ > hdfs dfs -test -e \/file.txt\n[npntraining]$ > echo $?<\/code><\/pre>\n-du<\/strong><\/h2>\nThis command is used to check disk usage<\/p>\n
hdfs dfs -du -h \/<\/code><\/pre>\n","protected":false},"excerpt":{"rendered":"In this blog post I will explain different HDFS commands to access HDFS which are commonly used while working as a Big Data Developer Training. Hadoop provides command line interface…<\/p>\n","protected":false},"author":1,"featured_media":5603,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[58,59,60,65,66],"tags":[],"class_list":["post-5602","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-hadoop","category-hive","category-java-programming","category-selenium","category-statistics"],"_links":{"self":[{"href":"https:\/\/www.npntraining.com\/blog\/wp-json\/wp\/v2\/posts\/5602","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.npntraining.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.npntraining.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.npntraining.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.npntraining.com\/blog\/wp-json\/wp\/v2\/comments?post=5602"}],"version-history":[{"count":21,"href":"https:\/\/www.npntraining.com\/blog\/wp-json\/wp\/v2\/posts\/5602\/revisions"}],"predecessor-version":[{"id":7851,"href":"https:\/\/www.npntraining.com\/blog\/wp-json\/wp\/v2\/posts\/5602\/revisions\/7851"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.npntraining.com\/blog\/wp-json\/wp\/v2\/media\/5603"}],"wp:attachment":[{"href":"https:\/\/www.npntraining.com\/blog\/wp-json\/wp\/v2\/media?parent=5602"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.npntraining.com\/blog\/wp-json\/wp\/v2\/categories?post=5602"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.npntraining.com\/blog\/wp-json\/wp\/v2\/tags?post=5602"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}