resource manager HA是hadoop自从2.4之后推出的功能,以Active/Standby的方式提供冗余,目的是为了消除单点失败的风险。
1、总体架构:
2、故障切换:有自动和手动两种形式。
手动:如果以手动形式切换,使用yarn haadmin命令首先将Active节点转为standby,再将standby节点转为active。
自动:RM有基于zookeeper的节点选举机制决定哪一个是活动节点。不需要像HDFS一样部署一个zkfc守护进程,因为RM内嵌了这样的功能。
做了rm的HA之后,所有节点和客户端都要列出所有RM节点,连接时会用轮询的方式遍历,直到找到一个active的节点。如果活动节点down了,它们会继续轮询。这一动作被实现为org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider类。可以通过重新实现该类,并在yarn.client.failover-proxy-provider 配置项中指定新的类名来改写这一行为逻辑。
3、配置
Configuration Properties | Description |
---|---|
yarn.resourcemanager.zk-address | Address of the ZK-quorum. Used both for the state-store and embedded leader-election. |
yarn.resourcemanager.ha.enabled | Enable RM HA. |
yarn.resourcemanager.ha.rm-ids | List of logical IDs for the RMs. e.g., “rm1,rm2”. |
yarn.resourcemanager.hostname.rm-id | For each rm-id, specify the hostname the RM corresponds to. Alternately, one could set each of the RM’s service addresses. |
yarn.resourcemanager.address.rm-id | For each rm-id, specify host:port for clients to submit jobs. If set, overrides the hostname set in yarn.resourcemanager.hostname.rm-id. |
yarn.resourcemanager.scheduler.address.rm-id | For each rm-id, specify scheduler host:port for ApplicationMasters to obtain resources. If set, overrides the hostname set in yarn.resourcemanager.hostname.rm-id. |
yarn.resourcemanager.resource-tracker.address.rm-id | For each rm-id, specify host:port for NodeManagers to connect. If set, overrides the hostname set in yarn.resourcemanager.hostname.rm-id. |
yarn.resourcemanager.admin.address.rm-id | For each rm-id, specify host:port for administrative commands. If set, overrides the hostname set in yarn.resourcemanager.hostname.rm-id. |
yarn.resourcemanager.webapp.address.rm-id | For each rm-id, specify host:port of the RM web application corresponds to. You do not need this if you set yarn.http.policy to HTTPS_ONLY. If set, overrides the hostname set in yarn.resourcemanager.hostname.rm-id. |
yarn.resourcemanager.webapp.https.address.rm-id | For each rm-id, specify host:port of the RM https web application corresponds to. You do not need this if you set yarn.http.policy to HTTP_ONLY. If set, overrides the hostname set in yarn.resourcemanager.hostname.rm-id. |
yarn.resourcemanager.ha.id | Identifies the RM in the ensemble. This is optional; however, if set, admins have to ensure that all the RMs have their own IDs in the config. |
yarn.resourcemanager.ha.automatic-failover.enabled | Enable automatic failover; By default, it is enabled only when HA is enabled. |
yarn.resourcemanager.ha.automatic-failover.embedded | Use embedded leader-elector to pick the Active RM, when automatic failover is enabled. By default, it is enabled only when HA is enabled. |
yarn.resourcemanager.cluster-id | Identifies the cluster. Used by the elector to ensure an RM doesn’t take over as Active for another cluster. |
yarn.client.failover-proxy-provider | The class to be used by Clients, AMs and NMs to failover to the Active RM. |
yarn.client.failover-max-attempts | The max number of times FailoverProxyProvider should attempt failover. |
yarn.client.failover-sleep-base-ms | The sleep base (in milliseconds) to be used for calculating the exponential delay between failovers. |
yarn.client.failover-sleep-max-ms | The maximum sleep time (in milliseconds) between failovers. |
yarn.client.failover-retries | The number of retries per attempt to connect to a ResourceManager. |
yarn.client.failover-retries-on-socket-timeouts | The number of retries per attempt to connect to a ResourceManager on socket timeouts. |
4、示例(最小配置)
<property>
<name>yarn.resourcemanager.ha.enabled</name>
<value>true</value>
</property>
<property>
<name>yarn.resourcemanager.cluster-id</name>
<value>cluster1</value>
</property>
<property>
<name>yarn.resourcemanager.ha.rm-ids</name>
<value>rm1,rm2</value>
</property>
<property>
<name>yarn.resourcemanager.hostname.rm1</name>
<value>master1</value>
</property>
<property>
<name>yarn.resourcemanager.hostname.rm2</name>
<value>master2</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address.rm1</name>
<value>master1:8088</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address.rm2</name>
<value>master2:8088</value>
</property>
<property>
<name>yarn.resourcemanager.zk-address</name>
<value>zk1:2181,zk2:2181,zk3:2181</value>
</property>
5、管理命令
查看节点状态:
$ yarn rmadmin -getServiceState rm1
active
$ yarn rmadmin -getServiceState rm2
standby
故障切换:
$ yarn rmadmin -transitionToStandby rm1
$ yarn rmadmin -transitionToActive rm2
注意:
开启自动故障切换后,系统为防止造成脑裂或其它不一致的状态,会拒绝人为管理HA状态。如果非常清楚自己的行为,可以在切换命令中指定-forcemanual选项。
免责声明:本站发布的内容(图片、视频和文字)以原创、转载和分享为主,文章观点不代表本网站立场,如果涉及侵权请联系站长邮箱:is@yisu.com进行举报,并提供相关证据,一经查实,将立刻删除涉嫌侵权内容。