这篇文章给大家介绍如何搭建BeeGFS开发环境，内容非常详细，感兴趣的小伙伴们可以参考借鉴，希望对大家能有所帮助。
我的测试环境使用CentOS 7， 安装BeeGFS的V6版本。
使用下载好的RPM包进行安装：
$ rpm -ivh beegfs_rpm/*.rpm $ rpm -qa | grep bee beegfs-meta-6.18-el7.x86_64 beegfs-admon-6.18-el7.x86_64 beegfs-utils-6.18-el7.x86_64 beegfs-helperd-6.18-el7.x86_64 beegfs-storage-6.18-el7.x86_64 beegfs-opentk-lib-6.18-el7.x86_64 beegfs-client-6.18-el7.noarch beegfs-common-6.18-el7.noarch beegfs-mgmtd-6.18-el7.x86_64
使用Yum安装：
$ wget https://www.beegfs.io/release/beegfs_6/dists/beegfs-rhel7.repo -O /etc/yum.repos.d/beegfs-rhel7.repo $ yum install -y beegfs-mgmtd beegfs-meta libbeegfs-ib beegfs-storage libbeegfs-ib beegfs-client beegfs-helperd beegfs-utils beegfs-admon $ yum install -y mdadm
sacd01 $ rm -rfv /beegfs_mgmtd sacd01 $ /opt/beegfs/sbin/beegfs-setup-mgmtd -p /beegfs_mgmtd Preparing storage directory: /beegfs_mgmtd * Creating format.conf file... Updating config file: /etc/beegfs/beegfs-mgmtd.conf * Setting storage directory in config file... * Disabling usage of uninitialized storage directory in config file... All done.
两个元数据节点的ID分别选择2和3：
sacd01 $ rm /beegfs_meta/meta* -rf sacd01 $ /opt/beegfs/sbin/beegfs-setup-meta -p /beegfs_meta/meta1 -s 2 -m 172.29.39.125 Preparing storage directory: /beegfs_meta/meta1 * Creating format.conf file... * Creating server numeric ID file: /beegfs_meta/meta1/nodeNumID Updating config file: /etc/beegfs/beegfs-meta.conf * Setting management host: 172.29.39.125 * Setting storage directory in config file... * Disabling usage of uninitialized storage directory in config file... * Setting usage of extended attributes to: true All done. sacd02 $ rm /beegfs_meta/meta* -rf sacd02 $ /opt/beegfs/sbin/beegfs-setup-meta -p /beegfs_meta/meta2 -s 3 -m 172.29.39.125 Preparing storage directory: /beegfs_meta/meta2 * Creating format.conf file... * Creating server numeric ID file: /beegfs_meta/meta2/nodeNumID Updating config file: /etc/beegfs/beegfs-meta.conf * Setting management host: 172.29.39.125 * Setting storage directory in config file... * Disabling usage of uninitialized storage directory in config file... * Setting usage of extended attributes to: true All done.
sacd01 $ rm /beegfs_data/data* -rf
sacd01 $ /opt/beegfs/sbin/beegfs-setup-storage -p /beegfs_data/data1 -s 4 -i 401 -m 172.29.39.125
Preparing storage target directory: /beegfs_data/data1
* Creating format.conf file...
* Creating chunks directory...
* Creating buddymir directory...
* Creating target numeric ID file: /beegfs_data/data1/targetNumID
* Creating server numeric ID file: /beegfs_data/data1/nodeNumID
Updating config file: /etc/beegfs/beegfs-storage.conf
* Setting management host: 172.29.39.125
* Appending to target directory list in config file...
* WARNING: Skipping append. Target directory seems to be included in current targets list already. ('-f' disables this check.)
* Disabling usage of uninitialized storage targets in config file...
All done.
sacd01 $ /opt/beegfs/sbin/beegfs-setup-storage -p /beegfs_data/data2 -s 4 -i 402 -m 172.29.39.125
Preparing storage target directory: /beegfs_data/data2
* Creating format.conf file...
* Creating chunks directory...
* Creating buddymir directory...
* Creating target numeric ID file: /beegfs_data/data2/targetNumID
* Creating server numeric ID file: /beegfs_data/data2/nodeNumID
Updating config file: /etc/beegfs/beegfs-storage.conf
* Setting management host: 172.29.39.125
* Appending to target directory list in config file...
* WARNING: Skipping append. Target directory seems to be included in current targets list already. ('-f' disables this check.)
* Disabling usage of uninitialized storage targets in config file...
All done.
sacd02 $ rm /beegfs_data/data* -rf
sacd02 $ /opt/beegfs/sbin/beegfs-setup-storage -p /beegfs_data/data3 -s 5 -i 501 -m 172.29.39.125
Preparing storage target directory: /beegfs_data/data3
* Creating format.conf file...
* Creating chunks directory...
* Creating buddymir directory...
* Creating target numeric ID file: /beegfs_data/data3/targetNumID
* Creating server numeric ID file: /beegfs_data/data3/nodeNumID
Updating config file: /etc/beegfs/beegfs-storage.conf
* Setting management host: 172.29.39.125
* Appending to target directory list in config file...
* WARNING: Skipping append. Target directory seems to be included in current targets list already. ('-f' disables this check.)
* Disabling usage of uninitialized storage targets in config file...
All done.
sacd02 $ /opt/beegfs/sbin/beegfs-setup-storage -p /beegfs_data/data4 -s 5 -i 502 -m 172.29.39.125
Preparing storage target directory: /beegfs_data/data4
* Creating format.conf file...
* Creating chunks directory...
* Creating buddymir directory...
* Creating target numeric ID file: /beegfs_data/data4/targetNumID
* Creating server numeric ID file: /beegfs_data/data4/nodeNumID
Updating config file: /etc/beegfs/beegfs-storage.conf
* Setting management host: 172.29.39.125
* Appending to target directory list in config file...
* WARNING: Skipping append. Target directory seems to be included in current targets list already. ('-f' disables this check.)
* Disabling usage of uninitialized storage targets in config file...
All done.sacd03 $ /opt/beegfs/sbin/beegfs-setup-client -m 172.29.39.125 Updating config file: /etc/beegfs/beegfs-client.conf * Setting management host: 172.29.39.125 All done.
让让整个存储集群使用指定的网口进行通信：
sacd01 $ echo bond1.1039 > /etc/beegfs/network/beegfs-network-interface.conf sacd02 $ echo bond1.1039 > /etc/beegfs/network/beegfs-network-interface.conf sacd03 $ echo bond1.1039 > /etc/beegfs/network/beegfs-network-interface.conf # Management sacd01 $ vi /etc/beegfs/beegfs-mgmtd.conf connInterfacesFile = /etc/beegfs/network/beegfs-network-interface.conf # Metadata sacd01 $ vi /etc/beegfs/beegfs-meta.conf connInterfacesFile = /etc/beegfs/network/beegfs-network-interface.conf sacd02 $ vi /etc/beegfs/beegfs-meta.conf connInterfacesFile = /etc/beegfs/network/beegfs-network-interface.conf # Storage sacd01 $ vi /etc/beegfs/beegfs-storage.conf connInterfacesFile = /etc/beegfs/network/beegfs-network-interface.conf sacd02 $ vi vi /etc/beegfs/beegfs-storage.conf connInterfacesFile = /etc/beegfs/network/beegfs-network-interface.conf # Client sacd03 $ vi /etc/beegfs/beegfs-client.conf connInterfacesFile = /etc/beegfs/network/beegfs-network-interface.conf
sacd01 $ systemctl start beegfs-mgmtd sacd01 $ systemctl start beegfs-meta sacd02 $ systemctl start beegfs-meta sacd01 $ systemctl start beegfs-storage sacd02 $ systemctl start beegfs-storage sacd03 $ beegfs-ctl --listtargets --nodetype=metadata --state TargetID Reachability Consistency NodeID ======== ============ =========== ====== 2 Online Good 2 3 Online Good 3 sacd03 $ beegfs-ctl --listtargets --nodetype=storage --state TargetID Reachability Consistency NodeID ======== ============ =========== ====== 401 Online Good 4 402 Online Good 4 501 Online Good 5 502 Online Good 5
sacd03 $ beegfs-ctl --addmirrorgroup --automatic --nodetype=meta New mirror groups: BuddyGroupID Node type Node ============ ========= ==== 1 primary 2 @ beegfs-meta sacd01 [ID: 2] secondary 3 @ beegfs-meta sacd02 [ID: 3] Mirror buddy group successfully set: groupID 1 -> target IDs 2, 3 sacd03 $ beegfs-ctl --listmirrorgroups --nodetype=meta BuddyGroupID PrimaryNodeID SecondaryNodeID ============ ============= =============== 1 2 3 sacd03 $ beegfs-ctl --mirrormd sacd01 $ systemctl restart beegfs-meta sacd02 $ systemctl restart beegfs-meta
sacd03 $ vi /etc/beegfs/beegfs-client.conf logClientID = true logLevel = 5 #sysMountSanityCheckMS = 11000 sysMountSanityCheckMS = 0 sacd03 $ systemctl start beegfs-helperd && systemctl start beegfs-client sacd03 $ systemctl status beegfs-helperd && systemctl status beegfs-client sacd01 $ systemctl enable beegfs-mgmtd beegfs-meta beegfs-storage sacd02 $ systemctl enable beegfs-meta beegfs-storage sacd03 $ systemctl enable beegfs-client
$ beegfs-ctl --listnodes --nodetype=management --nicdetails sacd01 [ID: 1] Ports: UDP: 8008; TCP: 8008 Interfaces: + bond1.1039[ip addr: 172.29.39.125; type: TCP] Number of nodes: 1 $ beegfs-ctl --listnodes --nodetype=metadata --nicdetails sacd01 [ID: 2] Ports: UDP: 8005; TCP: 8005 Interfaces: + bond1.1039[ip addr: 172.29.39.125; type: TCP] sacd02 [ID: 3] Ports: UDP: 8005; TCP: 8005 Interfaces: + bond1.1039[ip addr: 172.29.39.126; type: TCP] Number of nodes: 2 Root: 2 $ beegfs-ctl --listnodes --nodetype=storage --nicdetails sacd01 [ID: 4] Ports: UDP: 8003; TCP: 8003 Interfaces: + bond1.1039[ip addr: 172.29.39.125; type: TCP] sacd02 [ID: 5] Ports: UDP: 8003; TCP: 8003 Interfaces: + bond1.1039[ip addr: 172.29.39.126; type: TCP] Number of nodes: 2 $ beegfs-ctl --listnodes --nodetype=client --nicdetails C26A-5CD18762-sacd03 [ID: 5] Ports: UDP: 8004; TCP: 0 Interfaces: + bond1.1039[ip addr: 172.29.39.133; type: TCP] Number of nodes: 1
启动客户端服务时，可能出现如下错误：
$ systemctl start beegfs-client Job for beegfs-client.service failed because the control process exited with error code. See "systemctl status beegfs-client.service" and "journalctl -xe" for details. $ systemctl status beegfs-client.service -l ● beegfs-client.service - Start BeeGFS Client Loaded: loaded (/usr/lib/systemd/system/beegfs-client.service; enabled; vendor preset: disabled) Active: failed (Result: exit-code) since Tue 2019-05-07 16:37:01 CST; 17s ago Process: 134148 ExecStart=/etc/init.d/beegfs-client start (code=exited, status=1/FAILURE) Main PID: 134148 (code=exited, status=1/FAILURE) May 07 16:37:00 sacd03 systemd[1]: Starting Start BeeGFS Client... May 07 16:37:00 sacd03 beegfs-client[134148]: Starting BeeGFS Client: May 07 16:37:00 sacd03 beegfs-client[134148]: - Loading BeeGFS modules May 07 16:37:00 sacd03 beegfs-client[134148]: - BeeGFS module autobuild May 07 16:37:01 sacd03 beegfs-client[134148]: Makefile:117: *** Linux kernel build directory not found. Please check if the kernel module development packages are installed for the current kernel version. (RHEL: kernel-devel; SLES: linux-kernel-headers, kernel-source; Debian: linux-headers). Stop. May 07 16:37:01 sacd03 beegfs-client[134148]: make: *** [auto_rebuild] Error 2 May 07 16:37:01 sacd03 systemd[1]: beegfs-client.service: main process exited, code=exited, status=1/FAILURE May 07 16:37:01 sacd03 systemd[1]: Failed to start Start BeeGFS Client. May 07 16:37:01 sacd03 systemd[1]: Unit beegfs-client.service entered failed state. May 07 16:37:01 sacd03 systemd[1]: beegfs-client.service failed.
这是由于没有安装kernel-devel，没有内核头文件引起的。
或者是下面这个错误：
$ vi /var/log/beegfs-client.log Message from syslogd@sacd03 at May 8 09:44:12 ... dracut:dracut: creation of /boot/initramfs-3.10.0-957.12.1.el7.x86_64.tmp failed (5) May08 09:44:33 *df(63614) [C206-5CD22EA8-sacd03: FhgfsOps_getattr] >> called. Path: /; EntryID: root (0) May08 09:44:33 *df(63614) [C206-5CD22EA8-sacd03: Messaging (RPC node)] >> Invalid mirror buddy group ID: 0 (5) May08 09:44:33 *df(63614) [C206-5CD22EA8-sacd03: FhgfsOps_statfs] >> called. Path: n/a (no dentry); EntryID: n/a (no inode) (1) May08 09:44:33 *df(63614) [C206-5CD22EA8-sacd03: Remoting (stat storage targets)] >> No storage targets known. (5) May08 09:45:04 *df(65354) [C206-5CD22EA8-sacd03: FhgfsOps_getattr] >> called. Path: /; EntryID: root (0) May08 09:45:04 *df(65354) [C206-5CD22EA8-sacd03: Messaging (RPC node)] >> Invalid mirror buddy group ID: 0 (5) May08 09:45:04 *df(65354) [C206-5CD22EA8-sacd03: FhgfsOps_statfs] >> called. Path: n/a (no dentry); EntryID: n/a (no inode) (1) May08 09:45:04 *df(65354) [C206-5CD22EA8-sacd03: Remoting (stat storage targets)] >> No storage targets known. $ ls /mnt/beegfs/ ls: reading directory /mnt/beegfs/: Not a directory
这是BeeGFS 6.18在RHEL 3.10.0-862版本内核上的一个Bug引起的：
在所有节点上安装kernel-3.10.0-693版本的内核和开发包即可解决问题：
$ yum list kernel
Loaded plugins: fastestmirror, langpacks
Loading mirror speeds from cached hostfile
* base: ap.stykers.moe
* epel: mirrors.tuna.tsinghua.edu.cn
* extras: ap.stykers.moe
* updates: mirrors.tuna.tsinghua.edu.cn
Installed Packages
kernel.x86_64 3.10.0-957.5.1.el7 @updates
Available Packages
kernel.x86_64 3.10.0-957.12.1.el7 updates
$ wget ftp://ftp.pbone.net/mirror/ftp.scientificlinux.org/linux/scientific/7.0/x86_64/updates/security/kernel-3.10.0-693.el7.x86_64.rpm
$ wget ftp://ftp.pbone.net/mirror/ftp.scientificlinux.org/linux/scientific/7.0/x86_64/updates/security/kernel-devel-3.10.0-693.el7.x86_64.rpm
# wget ftp://ftp.pbone.net/mirror/ftp.redhat.com/pub/redhat/linux/enterprise/7Server/en/RHEV/SRPMS/kernel-3.10.0-693.el7.src.rpm
$ rpm -ivh kernel* --force
$ grep "menuentry " /boot/grub2/grub.cfg | awk -F"'" '{print $2}'
CentOS Linux (3.10.0-693.el7.x86_64) 7 (Core)
CentOS Linux (3.10.0-957.12.1.el7.x86_64) 7 (Core)
CentOS Linux (0-rescue-e3a40c9c3add41e3b2eb72e341b52ec1) 7 (Core)
$ vi /etc/default/grub
GRUB_DEFAULT="CentOS Linux (3.10.0-693.el7.x86_64) 7 (Core)"
$ grub2-mkconfig -o /boot/grub2/grub.cfg
$ reboot
$ rpm -evh kernel-3.10.0-957.12.1.el7.x86_64
$ grep "menuentry " /boot/grub2/grub.cfg | awk -F"'" '{print $2}'
CentOS Linux (3.10.0-693.el7.x86_64) 7 (Core)
CentOS Linux (0-rescue-e3a40c9c3add41e3b2eb72e341b52ec1) 7 (Core)关于如何搭建BeeGFS开发环境就分享到这里了，希望以上内容可以对大家有一定的帮助，可以学到更多知识。如果觉得文章不错，可以把它分享出去让更多的人看到。
免责声明：本站发布的内容（图片、视频和文字）以原创、转载和分享为主，文章观点不代表本网站立场，如果涉及侵权请联系站长邮箱：is@yisu.com进行举报，并提供相关证据，一经查实，将立刻删除涉嫌侵权内容。
