使用 kubeadm 在 VMW 部署 k8s 集群之部署 GlusterFS 分布式存储集群

经过前一篇文章的设置,我们已经配置好基础环境,现在可以来搭建GlusterFS分布式集群,用来做k8s存储。

以下操作没有特殊说明的,即在所有节点上执行

GlusterFS 安装部署

安装 glusterfs

1
2
yum -y install centos-release-gluster
yum -y install glusterfs-server

启用 glusterd 服务

1
2
systemctl enable glusterd
systemctl start glusterd

配置信任池(在k8s-master上操作)

1
2
3
4
gluster peer probe 192.168.1.71
gluster peer probe 192.168.1.72
# 查看状态
gluster peer status

创建挂载目录

1
mkdir -p /data/glusterfs/gv0/brick1

创建测试卷(在k8s-master上操作)

1
2
3
4
5
6
# 创建卷gv0
gluster volume create gv0 replica 3 192.168.1.70:/data/glusterfs/gv0/brick1 192.168.1.71:/data/glusterfs/gv0/brick1 192.168.1.72:/data/glusterfs/gv0/brick1
# 激活卷
gluster volume start gv0
# 查看卷信息
gluster volume info

挂载测试

1
mount -t glusterfs 192.168.1.70:/gv0 /mnt/glusterfs

删除测试卷(在k8s-master上操作)

1
2
gluster volume stop gv0
gluster volume delete gv0

Heketi Server 安装部署

关于Heketi的官网释义

Heketi provides a RESTful management interface which can be used to manage the life cycle of GlusterFS volumes. With Heketi, cloud services like OpenStack Manila, Kubernetes, and OpenShift can dynamically provision GlusterFS volumes with any of the supported durability types. Heketi will automatically determine the location for bricks across the cluster, making sure to place bricks and its replicas across different failure domains. Heketi also supports any number of GlusterFS clusters, allowing cloud services to provide network file storage without being limited to a single GlusterFS cluster.

Heketi提供一个RESTful管理界面,可用于管理GlusterFS存储卷的生命周期。通过Heketi,就可以让k8s动态配置GlusterFS存储卷。

以下操作在k8s-master上执行

安装 Heketi

1
yum -y install heketi heketi-client

配置 heketi.json

1
vim /etc/heketi/heketi.json
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
......
# 修改端口,防止端口冲突
"port": "18080",
......
# 允许认证
"use_auth": true,
......
# 设置 admin 用户 key
"admin": {
"key": "adminkey"
},
......
# 修改执行插件为 ssh,并配置ssh所需证书,以实现对集群中的机器免密ssh登陆,使用ssh-copy-id把公钥copy到每台glusterfs服务器上
"executor": "ssh",

"_sshexec_comment": "SSH username and private key file information",
"sshexec": {
"keyfile": "/etc/heketi/heketi_key",
"user": "root",
"port": "22",
"fstab": "/etc/fstab"
},
......
# 定义heketi数据库文件公交车
"db": "/var/lib/heketi/heketi.db",
......
# 日志输出级别
"loglevel" : "warning"

创建 SSH 密钥

1
2
3
4
5
6
ssh-keygen -t rsa -q -f /etc/heketi/heketi_key -N ''
chown -R heketi:heketi /etc/heketi
# 拷贝公钥到每台节点上
ssh-copy-id -i /etc/heketi/heketi_key.pub root@192.168.1.70
ssh-copy-id -i /etc/heketi/heketi_key.pub root@192.168.1.71
ssh-copy-id -i /etc/heketi/heketi_key.pub root@192.168.1.72

启用 heketi 服务

1
2
systemctl enable heketi
systemctl start heketi

验证

1
2
3
curl http://localhost:18080/hello
# 显示以下信息代表安装成功
Hello from Heketi

设置拓扑

1
cat /etc/heketi/topology.json
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
# 这边的`devices`用的是第二块磁盘`/dev/sdb`,磁盘必须是原始格式,没有格式化过。
{
"clusters": [
{
"nodes": [
{
"devices": [
{
"destroydata": false,
"name": "/dev/sdb"
}
],
"node": {
"hostnames": {
"manage": [
"192.168.1.70"
],
"storage": [
"192.168.1.70"
]
},
"zone": 1
}
},
{
"devices": [
{
"destroydata": false,
"name": "/dev/sdb"
}
],
"node": {
"hostnames": {
"manage": [
"192.168.1.71"
],
"storage": [
"192.168.1.71"
]
},
"zone": 1
}
},
{
"devices": [
{
"destroydata": false,
"name": "/dev/sdb"
}
],
"node": {
"hostnames": {
"manage": [
"192.168.1.72"
],
"storage": [
"192.168.1.72"
]
},
"zone": 1
}
}
]
}
]
}
1
2
3
4
alias heketi-cli="heketi-cli --user admin --secret adminkey --server http://192.168.1.70"
heketi-cli topology load --json=/etc/heketi/topology.json
# 验证是否加载成功
heketi-cli topology info

设置拓扑(另一种方式)

1
2
3
4
5
6
7
8
# 设置别名
alias heketi-cli="heketi-cli --user admin --secret adminkey --server http://192.168.1.70"
# 创建 cluster
heketi-cli cluster create
# 添加节点到 cluster
heketi-cli node add --cluster "cluster id" --management-host-name 节点ip --storage-host-name 节点ip --zone 1
# 添加 device
heketi-cli device add --name "/dev/sdb" --node "node id"

创建卷

1
2
3
4
# size 是以 GiB 为单位
heketi-cli volume create --size=3
# 查看卷列表
heketi-cli volume list

到此,我们已经把GlusterFS分布式存储给部署好了,下一篇将讲如何使用kubeadm部署k8s集群。

一分、两分都是爱!