这篇文章主要介绍了HDFS Balancer负载均衡器及语法怎么使用的相关知识，内容详细易懂，操作简单快捷，具有一定借鉴价值，相信大家阅读完这篇HDFS Balancer负载均衡器及语法怎么使用文章都会有所收获，下面我们一起来看看吧。
当我们的hadoop集群运行了一段时间之后，各个DataNode上的数据分布并不一定是均匀分布的。比如说： 我们向现有集群中添加了一个新的DataNode。
此处是我自己的一个简单的理解
所谓的平衡指的是 每个DataNode的利用率 与 集群的利用率 之间相差不超过给定的阈值百分比。此处的平衡 指的是各个DataNode之间的平衡，同一个DataNode之间的各个磁盘是不会平衡的。
DataNode的利用率= dfs已用的空间 / 分配给dfs的空间。
注意： 分配给dfs的空间 不是磁盘的总空间。
集群的利用率= 各datanode dfs已使用的空间 / 各datanode总空间
假设平衡的阈值是 5%，集群的利用率是 37.5，那么每个节点的利用率在32.5%到42.5%之间都认为是均衡的。也就是说，极端情况下，DataNode的利用率最大相差10%。
[hadoopdeploy@hadoop01 ~]$ hdfs balancer --help Usage: hdfs balancer [-policy <policy>] the balancing policy: datanode or blockpool [-threshold <threshold>] Percentage of disk capacity [-exclude [-f <hosts-file> | <comma-separated list of hosts>]] Excludes the specified datanodes. [-include [-f <hosts-file> | <comma-separated list of hosts>]] Includes only the specified datanodes. [-source [-f <hosts-file> | <comma-separated list of hosts>]] Pick only the specified datanodes as source nodes. [-blockpools <comma-separated list of blockpool ids>] The balancer will only run on blockpools included in this list. [-idleiterations <idleiterations>] Number of consecutive idle iterations (-1 for Infinite) before exit. [-runDuringUpgrade] Whether to run the balancer during an ongoing HDFS upgrade.This is usually not desired since it will not affect used space on over-utilized machines. [-asService] Run as a long running service. Generic options supported are: -conf <configuration file> specify an application configuration file -D <property=value> define a value for a given property -fs <file:///|hdfs://namenode:port> specify default filesystem URL to use, overrides 'fs.defaultFS' property from configurations. -jt <local|resourcemanager:port> specify a ResourceManager -files <file1,...> specify a comma-separated list of files to be copied to the map reduce cluster -libjars <jar1,...> specify a comma-separated list of jar files to be included in the classpath -archives <archive1,...> specify a comma-separated list of archives to be unarchived on the compute machines The general command line syntax is: command [genericOptions] [commandOptions]
| 参数 | 描述 |
|---|---|
| threshold | 磁盘容量的百分比。默认值为10%，表示上下浮动10%。 |
| policy | 平衡策略。 datanode（默认）：当每一个DataNode是平衡的时候，集群就是平衡的。 blockpool：当每一个DataNode中的blockpool是平衡的，集群就是平衡的。 |
| exclude | 不参与平衡的DataNode节点 |
| include | 参与平衡的DataNode节点 |
| source | 仅选取指定的数据节点作为源节点 |
| blockpools | Balancer仅在指定的blockpools中运行 |
| idleiterations | 退出前的连续空闲迭代次数（-1表示无限） |
| -runDuringUpgrade | 是否在正在进行的HDFS升级过程中运行平衡器。通常不需要这样做，因为这不会影响过度使用的计算机上的已用空间。 |
| -asService | 作为长期运行的服务运行 |
[hadoopdeploy@hadoop01 ~]$ hdfs dfsadmin -setBalancerBandwidth 10485760 Balancer bandwidth is set to 10485760 [hadoopdeploy@hadoop01 ~]$
当我们的集群负载需要调低这个值，当我们的集群负载较低时，可以适当调高这个值。
[hadoopdeploy@hadoop01 ~]$ hdfs balancer -policy datanode -threshold 5 2023-03-26 14:10:09,785 INFO balancer.Balancer: Using a threshold of 5.0 2023-03-26 14:10:09,786 INFO balancer.Balancer: namenodes = [hdfs://hadoop01:8020] 2023-03-26 14:10:09,786 INFO balancer.Balancer: parameters = Balancer.BalancerParameters [BalancingPolicy.Node, threshold = 5.0, max idle iteration = 5, #excluded nodes = 0, #included nodes = 0, #source nodes = 0, #blockpools = 0, run during upgrade = false] 2023-03-26 14:10:09,786 INFO balancer.Balancer: included nodes = [] 2023-03-26 14:10:09,786 INFO balancer.Balancer: excluded nodes = [] 2023-03-26 14:10:09,786 INFO balancer.Balancer: source nodes = [] Time Stamp Iteration# Bytes Already Moved Bytes Left To Move Bytes Being Moved NameNode 2023-03-26 14:10:09,787 INFO balancer.NameNodeConnector: getBlocks calls for hdfs://hadoop01:8020 will be rate-limited to 20 per second 2023-03-26 14:10:10,392 INFO balancer.Balancer: dfs.namenode.get-blocks.max-qps = 20 (default=20) 2023-03-26 14:10:10,392 INFO balancer.Balancer: dfs.balancer.movedWinWidth = 5400000 (default=5400000) 2023-03-26 14:10:10,392 INFO balancer.Balancer: dfs.balancer.moverThreads = 1000 (default=1000) 2023-03-26 14:10:10,392 INFO balancer.Balancer: dfs.balancer.dispatcherThreads = 200 (default=200) 2023-03-26 14:10:10,392 INFO balancer.Balancer: dfs.balancer.getBlocks.size = 2147483648 (default=2147483648) 2023-03-26 14:10:10,392 INFO balancer.Balancer: dfs.balancer.getBlocks.min-block-size = 10485760 (default=10485760) 2023-03-26 14:10:10,392 INFO balancer.Balancer: dfs.datanode.balance.max.concurrent.moves = 100 (default=100) 2023-03-26 14:10:10,392 INFO balancer.Balancer: dfs.datanode.balance.bandwidthPerSec = 104857600 (default=104857600) 2023-03-26 14:10:10,395 INFO balancer.Balancer: dfs.balancer.max-size-to-move = 10737418240 (default=10737418240) 2023-03-26 14:10:10,395 INFO balancer.Balancer: dfs.blocksize = 134217728 (default=134217728) 2023-03-26 14:10:10,401 INFO net.NetworkTopology: Adding a new node: /default-rack/192.168.121.141:9866 2023-03-26 14:10:10,401 INFO net.NetworkTopology: Adding a new node: /default-rack/192.168.121.140:9866 2023-03-26 14:10:10,401 INFO net.NetworkTopology: Adding a new node: /default-rack/192.168.121.142:9866 2023-03-26 14:10:10,402 INFO balancer.Balancer: 0 over-utilized: [] 2023-03-26 14:10:10,402 INFO balancer.Balancer: 0 underutilized: [] 2023-3-26 14:10:10 0 0 B 0 B 0 B 0 hdfs://hadoop01:8020 The cluster is balanced. Exiting... 2023-3-26 14:10:10 Balancing took 810.0 milliseconds [hadoopdeploy@hadoop01 ~]$
关于“HDFS Balancer负载均衡器及语法怎么使用”这篇文章的内容就介绍到这里，感谢各位的阅读！相信大家对“HDFS Balancer负载均衡器及语法怎么使用”知识都有一定的了解，大家如果还想学习更多知识，欢迎关注亿速云行业资讯频道。
免责声明：本站发布的内容（图片、视频和文字）以原创、转载和分享为主，文章观点不代表本网站立场，如果涉及侵权请联系站长邮箱：is@yisu.com进行举报，并提供相关证据，一经查实，将立刻删除涉嫌侵权内容。
