安装Swift（对象存储服务）、安装Cinder（块存储服务）

  • 安装Swift（对象存储服务）
    • 控制节点安装和配置
    • 对象存储节点安装和配置
    • Create and distribute initial rings
    • 配置与启动
    • 验证
  • 安装Cinder（块存储服务）
    • 控制节点安装和配置
    • 块存储节点安装和配置
      • 使用LVM存储
      • 使用NFS存储
    • 块存储备份安装和配置
      • 使用Swift备份
      • 使用NFS备份
    • 验证

安装Swift（对象存储服务）

控制节点安装和配置

  1. 创建服务凭证
# openstack user create --domain default --password-prompt swift

输入密码：SWIFT_PASS
确认密码：SWIFT_PASS

# openstack role add --project service --user swift admin# openstack service create --name swift --description "OpenStack Object Storage" object-store
  1. 创建服务API接入点
# openstack endpoint create --region RegionOne object-store public http://controller:8080/v1/AUTH_%\(project_id\)s# openstack endpoint create --region RegionOne object-store internal http://controller:8080/v1/AUTH_%\(project_id\)s# openstack endpoint create --region RegionOne object-store admin http://controller:8080/v1
  1. 安装和配置
# yum install openstack-swift-proxy python-swiftclient python-keystoneclient python-keystonemiddleware memcached -y

配置，下载最新配置模板：

# curl -o /etc/swift/proxy-server.conf https://opendev.org/openstack/swift/raw/branch/master/etc/proxy-server.conf-sample

修改配置文件 /etc/swift/proxy-server.conf，在对应分组【[组名]】下注释原有配置并添加配置：
[DEFAULT]
bind_port = 8080
user = swift
swift_dir = /etc/swift
[pipeline:main]
pipeline = catch_errors gatekeeper healthcheck proxy-logging cache container_sync bulk ratelimit authtoken keystoneauth container-quotas account-quotas slo dlo versioned_writes proxy-logging proxy-server
[app:proxy-server]
use = egg:swift#proxy
account_autocreate = True
[filter:keystoneauth]
use = egg:swift#keystoneauth
operator_roles = admin,user
[filter:authtoken]
paste.filter_factory = keystonemiddleware.auth_token:filter_factory
www_authenticate_uri = http://controller:5000
auth_url = http://controller:5000
memcached_servers = controller:11211
auth_type = password
project_domain_id = default
user_domain_id = default
project_name = service
username = swift
password = SWIFT_PASS
delay_auth_decision = True
[filter:cache]
use = egg:swift#memcache
memcache_servers = controller:11211
4. 说明

  • 详细说明参考官网：
    https://docs.openstack.org/swift/train/install/controller-install-rdo.html

对象存储节点安装和配置

  1. 安装存储设备及同步服务
# yum install xfsprogs rsync -y

格式化（需要存在盘/dev/sdb、/dev/sdc）：

# mkfs.xfs /dev/sdb# mkfs.xfs /dev/sdc

创建挂载目录：

# mkdir -p /srv/node/sdb# mkdir -p /srv/node/sdc

创建LVM卷组：

# vgcreate cinder-volumes /dev/sdb

配置，修改配置文件 /etc/fstab，添加配置：
/dev/sdb /srv/node/sdb xfs noatime,nodiratime,logbufs=8 0 2
/dev/sdc /srv/node/sdc xfs noatime,nodiratime,logbufs=8 0 2

挂载设备：

# mount /srv/node/sdb# mount /srv/node/sdc

配置，修改配置文件 /etc/rsyncd.conf，添加配置：
uid = swift
gid = swift
log file = /var/log/rsyncd.log
pid file = /var/run/rsyncd.pid
address = 192.168.0.31

[account]
max connections = 2
path = /srv/node/
read only = False
lock file = /var/lock/account.lock

[container]
max connections = 2
path = /srv/node/
read only = False
lock file = /var/lock/container.lock

[object]
max connections = 2
path = /srv/node/
read only = False
lock file = /var/lock/object.lock

设置开机启动并启动服务：

# systemctl enable rsyncd.service# systemctl start rsyncd.service
  1. 安装和配置
# yum install openstack-cinder targetcli python-keystone -y

配置，下载最新配置模板：

# curl -o /etc/swift/account-server.conf https://opendev.org/openstack/swift/raw/branch/master/etc/account-server.conf-sample# curl -o /etc/swift/container-server.conf https://opendev.org/openstack/swift/raw/branch/master/etc/container-server.conf-sample# curl -o /etc/swift/object-server.conf https://opendev.org/openstack/swift/raw/branch/master/etc/object-server.conf-sample

修改配置文件 /etc/swift/account-server.conf，在对应分组【[组名]】下注释原有配置并添加配置：
[DEFAULT]
bind_ip = 192.168.0.31
bind_port = 6202
user = swift
swift_dir = /etc/swift
devices = /srv/node
mount_check = True
[pipeline:main]
pipeline = healthcheck recon account-server
[filter:recon]
use = egg:swift#recon
recon_cache_path = /var/cache/swift

修改配置文件 /etc/swift/container-server.conf，在对应分组【[组名]】下注释原有配置并添加配置：
[DEFAULT]
bind_ip = 192.168.0.31
bind_port = 6201
user = swift
swift_dir = /etc/swift
devices = /srv/node
mount_check = True
[pipeline:main]
pipeline = healthcheck recon container-server
[filter:recon]
use = egg:swift#recon
recon_cache_path = /var/cache/swift

修改配置文件 /etc/swift/object-server.conf，在对应分组【[组名]】下注释原有配置并添加配置：
[DEFAULT]
bind_ip = 192.168.0.31
bind_port = 6200
user = swift
swift_dir = /etc/swift
devices = /srv/node
mount_check = True
[pipeline:main]
pipeline = healthcheck recon object-server
[filter:recon]
use = egg:swift#recon
recon_cache_path = /var/cache/swift
recon_lock_path = /var/lock

目录授权：

# chown -R swift:swift /srv/node

创建目录并授权：

# mkdir -p /var/cache/swift# chown -R root:swift /var/cache/swift# chmod -R 775 /var/cache/swift
  1. 说明
  • 详细说明参考官网：
    https://docs.openstack.org/swift/train/install/storage-install-rdo.html
  • 此手册将计算计算作为块存储节点，安装多个对象存储节点需要调整IP。

Create and distribute initial rings

在控制节点进行操作。

  1. Create account ring
    切换目录：
# cd /etc/swift/

Create the base account.builder file:

# swift-ring-builder account.builder create 10 3 1

Add each storage node to the ring:

# swift-ring-builder account.builder add \--region 1 --zone 1 --ip 192.168.0.31 --port 6202 --device sdb --weight 100# swift-ring-builder account.builder add \--region 1 --zone 1 --ip 192.168.0.31 --port 6202 --device sdc --weight 100# swift-ring-builder account.builder add \--region 1 --zone 2 --ip 192.168.0.32 --port 6202 --device sdb --weight 100# swift-ring-builder account.builder add \--region 1 --zone 2 --ip 192.168.0.32 --port 6202 --device sdc --weight 100

Verify the ring contents:

# swift-ring-builder account.builder


Rebalance the ring:

# swift-ring-builder account.builder rebalance
  1. Create container ring
    切换目录：
# cd /etc/swift/

Create the base container.builder file:

# swift-ring-builder container.builder create 10 3 1

Add each storage node to the ring:

# swift-ring-builder container.builder add \--region 1 --zone 1 --ip 192.168.0.31 --port 6201 --device sdb --weight 100# swift-ring-builder container.builder add \--region 1 --zone 1 --ip 192.168.0.31 --port 6201 --device sdc --weight 100# swift-ring-builder container.builder add \--region 1 --zone 2 --ip 192.168.0.32 --port 6201 --device sdb --weight 100# swift-ring-builder container.builder add \--region 1 --zone 2 --ip 192.168.0.32 --port 6201 --device sdc --weight 100

Verify the ring contents:

# swift-ring-builder container.builder

Rebalance the ring:

# swift-ring-builder container.builder rebalance
  1. Create object ring
    切换目录：
# cd /etc/swift/

Create the base object.builder file:

# swift-ring-builder object.builder create 10 3 1

Add each storage node to the ring:

# swift-ring-builder object.builder add \--region 1 --zone 1 --ip 192.168.0.31 --port 6200 --device sdb --weight 100# swift-ring-builder object.builder add \--region 1 --zone 1 --ip 192.168.0.31 --port 6200 --device sdc --weight 100# swift-ring-builder object.builder add \--region 1 --zone 2 --ip 192.168.0.32 --port 6200 --device sdb --weight 100# swift-ring-builder object.builder add \--region 1 --zone 2 --ip 192.168.0.32 --port 6200 --device sdc --weight 100

Verify the ring contents:

# swift-ring-builder object.builder

Rebalance the ring:

# swift-ring-builder object.builder rebalance

4.Distribute ring configuration files
拷贝 /etc/swift目录中的文件account.ring.gz、container.ring.gz、object.ring.gz到存储节点的 /etc/swift目录下：

# scp /etc/swift/account.ring.gz root@192.168.0.31:/etc/swift# scp /etc/swift/account.ring.gz root@192.168.0.32:/etc/swift# scp /etc/swift/container.ring.gz root@192.168.0.31:/etc/swift# scp /etc/swift/container.ring.gz root@192.168.0.32:/etc/swift# scp /etc/swift/object.ring.gz root@192.168.0.31:/etc/swift# scp /etc/swift/object.ring.gz root@192.168.0.32:/etc/swift

配置与启动

  1. 配置（控制节点）
    配置，下载最新配置模板：
# curl -o /etc/swift/swift.conf https://opendev.org/openstack/swift/raw/branch/master/etc/swift.conf-sample

修改配置文件 /etc/swift/swift.conf，在对应分组【[组名]】下注释原有配置并添加配置：
[swift-hash]
swift_hash_path_suffix = HASH_PATH_SUFFIX
swift_hash_path_prefix = HASH_PATH_PREFIX
[storage-policy:0]
name = Policy-0
default = yes

拷贝 /etc/swift/swift.conf到存储节点的 /etc/swift目录下：

# scp /etc/swift/swift.conf root@192.168.0.31:/etc/swift# scp /etc/swift/swift.conf root@192.168.0.32:/etc/swift
  1. 配置文件夹授权（控制节点+存储节点）
# chown -R root:swift /etc/swift
  1. 开机启动并启动服务（控制节点）
# systemctl enable openstack-swift-proxy.service memcached.service# systemctl start openstack-swift-proxy.service memcached.service
  1. 开机启动并启动服务（存储节点）
# systemctl enable openstack-swift-account.service openstack-swift-account-auditor.service openstack-swift-account-reaper.service openstack-swift-account-replicator.service# systemctl start openstack-swift-account.service openstack-swift-account-auditor.service openstack-swift-account-reaper.service openstack-swift-account-replicator.service# systemctl enable openstack-swift-container.service openstack-swift-container-auditor.service openstack-swift-container-replicator.service openstack-swift-container-updater.service# systemctl start openstack-swift-container.service openstack-swift-container-auditor.service openstack-swift-container-replicator.service openstack-swift-container-updater.service# systemctl enable openstack-swift-object.service openstack-swift-object-auditor.service openstack-swift-object-replicator.service openstack-swift-object-updater.service# systemctl start openstack-swift-object.service openstack-swift-object-auditor.service openstack-swift-object-replicator.service openstack-swift-object-updater.service

验证

  1. 文件夹授权（存储节点）
# chcon -R system_u:object_r:swift_data_t:s0 /srv/node

1.5.5.2检查服务状态

# swift stat

创建容器（容器名：container1）：

# openstack container create container1

上传文件到容器（测试文件：account.ring.gz）：

# openstack object create container1 account.ring.gz

说明：
创建容器或上传文件报错，查看日志用：

# journalctl -f

暂时发现上传文件报编码错误：

解决方式：
将提示编码错误py文件 /usr/lib64/python2.7/logging/init.py前面加上编码设置：

安装Cinder（块存储服务）

控制节点安装和配置

  1. 数据库创建、创建用户并授权
# mysql
MariaDB [(none)]> CREATE DATABASE cinder;MariaDB [(none)]> GRANT ALL PRIVILEGES ON cinder.* TO 'cinder'@'localhost' \IDENTIFIED BY 'CINDER_DBPASS';MariaDB [(none)]> GRANT ALL PRIVILEGES ON cinder.* TO 'cinder'@'%' \IDENTIFIED BY 'CINDER_DBPASS';
  1. 创建服务凭证
# openstack user create --domain default --password-prompt cinder

输入密码：CINDER_PASS
确认密码：CINDER_PASS

# openstack role add --project service --user cinder admin# openstack service create --name cinderv2 --description "OpenStack Block Storage" volumev2# openstack service create --name cinderv3 --description "OpenStack Block Storage" volumev3
  1. 创建服务API接入点
# openstack endpoint create --region RegionOne volumev2 public http://controller:8776/v2/%\(project_id\)s# openstack endpoint create --region RegionOne volumev2 internal http://controller:8776/v2/%\(project_id\)s# openstack endpoint create --region RegionOne volumev2 admin http://controller:8776/v2/%\(project_id\)s# openstack endpoint create --region RegionOne volumev3 public http://controller:8776/v3/%\(project_id\)s# openstack endpoint create --region RegionOne volumev3 internal http://controller:8776/v3/%\(project_id\)s# openstack endpoint create --region RegionOne volumev3 admin http://controller:8776/v3/%\(project_id\)s
  1. 安装和配置
# yum install openstack-cinder -y

配置，修改配置文件 /etc/cinder/cinder.conf，在对应分组【[组名]】下添加配置：
[database]
connection = mysql+pymysql://cinder:CINDER_DBPASS@controller/cinder
[DEFAULT]
transport_url = rabbit://openstack:RABBIT_PASS@controller
auth_strategy = keystone
my_ip = 192.168.0.11
[keystone_authtoken]
www_authenticate_uri = http://controller:5000
auth_url = http://controller:5000
memcached_servers = controller:11211
auth_type = password
project_domain_name = default
user_domain_name = default
project_name = service
username = cinder
password = CINDER_PASS
[oslo_concurrency]
lock_path = /var/lib/cinder/tmp

初始化数据库：

# su -s /bin/sh -c "cinder-manage db sync" cinder

配置，修改配置文件 /etc/nova/nova.conf，在对应分组【[组名]】下添加配置：
[cinder]
os_region_name = RegionOne

设置开机启动并启动服务：

# systemctl restart openstack-nova-api.service# systemctl enable openstack-cinder-api.service openstack-cinder-scheduler.service# systemctl start openstack-cinder-api.service openstack-cinder-scheduler.service
  1. 说明
  • 详细说明参考官网：
    https://docs.openstack.org/cinder/train/install/cinder-controller-install-rdo.html
  • 安装mysql时做了安装初始化，将密码设置为无，如有设置密码，连接时请加账号密码。

块存储节点安装和配置

使用LVM存储

  1. 安装存储设备
# yum install lvm2 device-mapper-persistent-data -y

设置开机启动并启动服务：

# systemctl enable lvm2-lvmetad.service# systemctl restart lvm2-lvmetad.service

创建LVM物理卷（需要存在盘/dev/sdb）：

# pvcreate /dev/sdb

创建LVM卷组：

# vgcreate cinder-volumes /dev/sdb#vgs

配置，修改配置文件 /etc/lvm/lvm.conf，在devices中的添加：
filter = [ “a/sdb/”, “r/.*/”] //不一定要添加

  1. 安装和配置
# yum install openstack-cinder targetcli python-keystone -y

配置，修改配置文件 /etc/cinder/cinder.conf，在对应分组【[组名]】下添加配置：
[database]
connection = mysql+pymysql://cinder:CINDER_DBPASS@controller/cinder
[DEFAULT]
transport_url = rabbit://openstack:RABBIT_PASS@controller
auth_strategy = keystone
my_ip = 192.168.0.11
enabled_backends = lvm
glance_api_servers = http://controller:9292
[keystone_authtoken]
www_authenticate_uri = http://controller:5000
auth_url = http://controller:5000
memcached_servers = controller:11211
auth_type = password
project_domain_name = default
user_domain_name = default
project_name = service
username = cinder
password = CINDER_PASS
[lvm]
volume_backend_name = lvm
volume_driver = cinder.volume.drivers.lvm.LVMVolumeDriver
volume_group = cinder-volumes
target_protocol = iscsi
target_helper = lioadm
[oslo_concurrency]
lock_path = /var/lib/cinder/tmp

设置开机启动并启动服务：

# systemctl enable openstack-cinder-volume.service target.service# systemctl start openstack-cinder-volume.service target.service

验证点

# openstack volume service list


确保state都为up。
3. 说明

  • 详细说明参考官网：
    https://docs.openstack.org/cinder/train/install/cinder-storage-install-rdo.html
  • 此手册将控制节点（controller）作为块存储节点，配置与【控制节点安装和配置】中有交集则将多的添加进去。

使用NFS存储

  1. 安装NFS
# yum install nfs-utils -y

创建存储目录：

# mkdir -p /home/nfs/data

配置，修改配置文件 /etc/exports，添加一行配置：
/home/nfs/data 192.168.0.0/24(rw,async,no_root_squash,no_all_squash)

设置开机启动并启动服务：

# systemctl enable rpcbind# systemctl restart rpcbind# systemctl enable nfs# systemctl restart nfs

校验：

# showmount -e 192.168.0.11
  1. 安装和配置
# yum install openstack-cinder targetcli python-keystone -y

配置，添加nfs配置：

# cat >> /etc/cinder/nfs_shares << EOF192.168.0.11:/home/nfs/dataEOF# chown root:cinder /etc/cinder/nfs_shares# chmod 0640 /etc/cinder/nfs_shares

配置，修改配置文件 /etc/cinder/cinder.conf，在对应分组【[组名]】下添加配置：
[database]
connection = mysql+pymysql://cinder:CINDER_DBPASS@controller/cinder
[DEFAULT]
transport_url = rabbit://openstack:RABBIT_PASS@controller
auth_strategy = keystone
my_ip = 192.168.0.11
enabled_backends = nfs
glance_api_servers = http://controller:9292
[keystone_authtoken]
www_authenticate_uri = http://controller:5000
auth_url = http://controller:5000
memcached_servers = controller:11211
auth_type = password
project_domain_name = default
user_domain_name = default
project_name = service
username = cinder
password = CINDER_PASS
[nfs]
volume_driver = cinder.volume.drivers.nfs.NfsDriver
nfs_shares_config = /etc/cinder/nfs_shares
volume_backend_name = nfs
[oslo_concurrency]
lock_path = /var/lib/cinder/tmp

设置开机启动并启动服务：

# systemctl enable openstack-cinder-volume.service target.service# systemctl restart openstack-cinder-volume.service target.service

验证块存储支持nfs

# openstack volume service list

  1. 说明
  • 此手册将控制节点（controller）作为块存储节点，配置与【控制节点安装和配置】中有交集则将多的添加进去。
  • 参照https://docs.openstack.org/cinder/train/admin/blockstorage-nfs-backend.html

块存储备份安装和配置

使用Swift备份

  1. 安装Swift（上面已安装）
  2. 块存储节点安装和配置
# yum install openstack-cinder -y

配置，修改配置文件 /etc/cinder/cinder.conf，在对应分组【[组名]】下添加配置：
[DEFAULT]
backup_driver = cinder.backup.drivers.swift.SwiftBackupDriver
backup_swift_url = http://controller:8080/v1/AUTH_

设置开机启动并启动服务：

# systemctl enable openstack-cinder-backup.service# systemctl restart openstack-cinder-backup.service
  1. 控制节点安装和配置（Horizon安装主机）
    配置，修改配置文件 /etc/openstack-dashboard/local_settings，添加配置：
    OPENSTACK_CINDER_FEATURES = {
    ‘enable_backup’: True,
    }

设置开机启动并启动服务：

# systemctl restart httpd.service memcached.service

使用NFS备份

  1. 备份节点安装和配置（上面已安装）
  2. 块存储节点安装和配置
# yum install openstack-cinder -y

配置，修改配置文件 /etc/cinder/cinder.conf，在对应分组【[组名]】下添加配置：
[DEFAULT]
backup_driver = cinder.backup.drivers.nfs.NFSBackupDriver
backup_share = 172.16.1.11:/home/nfs/data

设置开机启动并启动服务：

# systemctl enable openstack-cinder-backup.service# systemctl restart openstack-cinder-backup.service
  1. 控制节点安装和配置（Horizon安装主机）
    配置，修改配置文件 /etc/openstack-dashboard/local_settings，添加配置：
    OPENSTACK_CINDER_FEATURES = {
    ‘enable_backup’: True,
    }

设置开机启动并启动服务：

# systemctl restart httpd.service memcached.service
  1. 说明
  • 此手册将控制节点（controller）作为备份节点，配置与【控制节点安装和配置】中有交集则将多的添加进去。

验证

使用lvm作为存储卷状态是否正常：

# openstack volume service list

使用nfs作为存储卷状态是否正常：

# openstack volume service list

说明：
可以采用多种类型的存储卷，例如需要同时使用lvm与nfs，配置（/etc/cinder/cinder.conf）如下：