如何查看HDFS文件系统空间使用情况

这篇文章主要介绍如何查看HDFS文件系统空间使用情况，文中介绍的非常详细，具有一定的参考价值，感兴趣的小伙伴们一定要看完！

1、查看集群的空间使用状态

[hduser@master1 bin]$ ./hdfs dfsadmin -report

Configured Capacity: 8453768839168 (7.69 TB)

Present Capacity: 7959274510394 (7.24 TB)

DFS Remaining: 4128882869718 (3.76 TB)

DFS Used: 3830391640676 (3.48 TB)

DFS Used%: 48.12%

Under replicated blocks: 18

Blocks with corrupt replicas: 0

Missing blocks: 0

Missing blocks (with replication factor 1): 0

-------------------------------------------------

Live datanodes (5):

Name: 192.168.11.137:50010 (slave5)

Hostname: slave5

Decommission Status : Normal

Configured Capacity: 1056617381888 (984.05 GB)

DFS Used: 736041209958 (685.49 GB)

Non DFS Used: 66876043079 (62.28 GB)

DFS Remaining: 253700128851 (236.28 GB)

DFS Used%: 69.66%

DFS Remaining%: 24.01%

Configured Cache Capacity: 0 (0 B)

Cache Used: 0 (0 B)

Cache Remaining: 0 (0 B)

Cache Used%: 100.00%

Cache Remaining%: 0.00%

Xceivers: 30

Last contact: Tue Jul 03 15:06:51 CST 2018

Name: 192.168.11.134:50010 (slave2)

Hostname: slave2

Decommission Status : Normal

Configured Capacity: 2113511358464 (1.92 TB)

DFS Used: 749551308902 (698.07 GB)

Non DFS Used: 120793546401 (112.50 GB)

DFS Remaining: 1243166503161 (1.13 TB)

DFS Used%: 35.46%

DFS Remaining%: 58.82%

Configured Cache Capacity: 0 (0 B)

Cache Used: 0 (0 B)

Cache Remaining: 0 (0 B)

Cache Used%: 100.00%

Cache Remaining%: 0.00%

Xceivers: 23

Last contact: Tue Jul 03 15:06:50 CST 2018

Name: 192.168.11.133:50010 (slave1)

Hostname: slave1

Decommission Status : Normal

Configured Capacity: 2113511358464 (1.92 TB)

DFS Used: 792363753676 (737.95 GB)

Non DFS Used: 118847225403 (110.69 GB)

DFS Remaining: 1202300379385 (1.09 TB)

DFS Used%: 37.49%

DFS Remaining%: 56.89%

Configured Cache Capacity: 0 (0 B)

Cache Used: 0 (0 B)

Cache Remaining: 0 (0 B)

Cache Used%: 100.00%

Cache Remaining%: 0.00%

Xceivers: 21

Last contact: Tue Jul 03 15:06:50 CST 2018

Name: 192.168.11.135:50010 (slave3)

Hostname: slave3

Decommission Status : Normal

Configured Capacity: 2113511358464 (1.92 TB)

DFS Used: 752023101542 (700.38 GB)

Non DFS Used: 121481747963 (113.14 GB)

DFS Remaining: 1240006508959 (1.13 TB)

DFS Used%: 35.58%

DFS Remaining%: 58.67%

Configured Cache Capacity: 0 (0 B)

Cache Used: 0 (0 B)

Cache Remaining: 0 (0 B)

Cache Used%: 100.00%

Cache Remaining%: 0.00%

Xceivers: 26

Last contact: Tue Jul 03 15:06:50 CST 2018

Name: 192.168.11.136:50010 (slave4)

Hostname: slave4

Decommission Status : Normal

Configured Capacity: 1056617381888 (984.05 GB)

DFS Used: 800412266598 (745.44 GB)

Non DFS Used: 66495765928 (61.93 GB)

DFS Remaining: 189709349362 (176.68 GB)

DFS Used%: 75.75%

DFS Remaining%: 17.95%

Configured Cache Capacity: 0 (0 B)

Cache Used: 0 (0 B)

Cache Remaining: 0 (0 B)

Cache Used%: 100.00%

Cache Remaining%: 0.00%

Xceivers: 30

2、分析192.168.11.133

$ df -h

Filesystem            Size  Used Avail Use% Mounted on

/dev/mapper/VolGroup00-LVroot

                       17G   12G  4.9G  70% /

tmpfs                 7.8G     0  7.8G   0% /dev/shm

/dev/sda1             190M  125M   56M  70% /boot

/dev/mapper/VolGroup00-LVhome

                      477M  2.4M  449M   1% /home

/dev/mapper/VolGroup00-LVcloud

                      2.9G  247M  2.5G   9% /opt/cloud

/dev/mapper/VolGroup00-LVtmp

                      7.8G   19M  7.4G   1% /tmp

/dev/mapper/VolGroup00-LVvar

                       25G  786M   23G   4% /var

/dev/mapper/VolGroup1-LVdata1

                      985G  384G  551G  42% /data

/dev/mapper/VolGroup2-LVdata2

                      985G  365G  570G  40% /data2

数据统计：

/data和/data2为数据存放目录

总的容量 = 985G + 985G = 1970G = 1.92T

已使用容量 = 384G + 365G = 749G

未使用容量 = 551G + 570G = 1121G

未显示容量 = 1970G - （749G + 1121G）= 100G

/data目录中的hadoop和hbase的解压文件占用10G左右

dfs文件系统分析：

Name: 192.168.11.133:50010 (slave1)

Hostname: slave1

Decommission Status : Normal

Configured Capacity: 2113511358464 (1.92 TB)

DFS Used: 792363753676 (737.95 GB)

Non DFS Used: 118847225403 (110.69 GB)

DFS Remaining: 1202300379385 (1.09 TB)

DFS Used%: 37.49%

DFS Remaining%: 56.89%

Configured Cache Capacity: 0 (0 B)

Cache Used: 0 (0 B)

Cache Remaining: 0 (0 B)

Cache Used%: 100.00%

Cache Remaining%: 0.00%

Xceivers: 21

Last contact: Tue Jul 03 15:06:50 CST 2018

说明解析：

Configured Capacity:配置容量

DFS Used:DFS已使用的容量

Non DFS Used:非DFS使用的容量

DFS Remaining:DFS剩余的容量

数据统计：

Configured Capacity = 1.92 TB = 1970 GB

DFS Used = 737.95 GB

Non DFS Used = 110.69 GB

DFS Remaining = 1.09 TB = 1116.16 GB

DFS Used + Non DFS Used + DFS Remaining = 737.95 GB +  110.69 GB +  1116.16 GB = 1964.8 GB = Configured Capacity

所以得出结果：

du -h和./hdfs dfsadmin -report显示的容量对应关系为：

总的容量 ---> Configured Capacity

已使用容量 ---> DFS Used

未使用容量 ---> DFS Remaining

未显示容量 + /data下非dfs文件 ---> Non DFS Used

非dfs文件指的是不是同过dfs文件系统上传的文件，通过mv、cp等方式传输的文件。

统计192.168.11.136和192.168.11.133类似。

3、寻找消失的100G

Non DFS Used占用了110.69GB，/data下非dfs文件有10GB左右，那么剩下的100GB去哪了？

查看保留空间

# tune2fs -l /dev/mapper/VolGroup1-LVdata1

Reserved block count:     13107148

13107148 * 4 / 1024 /1024 = 49.99

# tune2fs -l /dev/mapper/VolGroup2-LVdata2

Reserved block count:     13107148

13107148 * 4 / 1024 /1024 = 49.99

消失的100G就是两个分区的保留空间之和

以上是“如何查看HDFS文件系统空间使用情况”这篇文章的所有内容，感谢各位的阅读！希望分享的内容对大家有帮助，更多相关知识，欢迎关注亿速云行业资讯频道！