Open-Local简介
Open-local 是阿里巴巴开源，由多个组件构成的本地磁盘管理系统，目标是解决当前kubernetes本地存储能力缺失问题。
Open-Local包含四大类组件：
•Scheduler-Extender: 作为 Kubernetes Scheduler 的扩展组件，通过 Extender 方式实现，新增本地存储调度算法
•CSI: 按照 CSI(Container Storage Interface) 标准实现本地磁盘管理能力
•Agent: 运行在集群中的每个节点，根据配置清单初始化存储设备，并通过上报集群中本地存储设备信息以供 Scheduler-Extender 决策调度
•Controller: 获取集群存储初始化配置，并向运行在各个节点的 Agent 下发详细的配置清单
架构图
Open-Local部署
方式1、Ack-distro 部署k8s，自带open-local（阿里巴巴二次开发的）
前期准备：（前期准备过程略）
基于redhat和debian的linux发行版（本次实验使用centos7.9）
LVM2+
docker-ce版本：19.03.15
集群中至少有一个空闲块设备用来测试
Ack-distro
Ack-distro作为完整的kubernetes发行版，通过阿里巴巴开源的应用打包交付工具sealer，可以简单，快速的交付到离线环境，帮助使用者更简单、敏捷地管理自己的集群。Ack-distro 部署时会默认安装open-local。
安装sealer
（前提：需要docker-ce版本：19.03.15）
上传压缩包，解压到/usr/bin目录 （压缩包需要github或者网上找）
tar -zxvf sealer-latest-linux-amd64.tar.gz
cp sealer /usr/bin/
上传sealer部署k8s集群所需的镜像包ackdistro.tar （压缩包需要github或者网上找）
sealer load -i ackdistro.tar
部署k8s
sealer run ack-agility-registry.cn-shanghai.cr.aliyuncs.com/ecp_builder/ackdistro:v1-20-4-ack-5 -m 10.0.101.89 -p asdfasdf..
-m 后面参数是k8s的masterIP
-p 后面参数是masterIP的root密码
（如果报错时间同步有问题，可以用：
sealer run ack-agility-registry.cn-shanghai.cr.aliyuncs.com/ecp_builder/ackdistro:v1-20-4-ack-5 -m 10.0.101.89 -p asdfasdf.. –env IgnoreErrors=”OS;TimeSyncService”）
完成之后，可以看到版本
查看open-local
支持的存储驱动
使用open-local
新增VG
首先新增LV
fdisk /dev/vdb
fdisk -l
pvcreate /dev/vdb1
pvscan
vgcreate -s 32M open-local-pool-1 /dev/vdb1
vgdisplay
存储初始化配置
（如果上面新建的名字就是open-local-pool-1，这个步骤可以省略）
首先kubectl get nlsc 查一下name
编辑nlsc：
kubectl edit nlsc yoda
文件内容中的open-local-pool-[0-9]+ 改为你新建的pv名字。加号+保留
创建一个Statefulset，使用已有的存储类模板
[root@laijx-k8s-test1 tmp]# cat sts-nginx.yaml apiVersion: v1kind: Servicemetadata: name: nginx-lvm labels: app: nginx-lvmspec: ports: - port: 80 name: web clusterIP: None selector: app: nginx-lvm---apiVersion: apps/v1kind: StatefulSetmetadata: name: nginx-lvmspec: selector: matchLabels: app: nginx-lvm podManagementPolicy: Parallel serviceName: "nginx-lvm" replicas: 1 volumeClaimTemplates: - metadata: name: html spec: accessModes: - ReadWriteOnce storageClassName: yoda-lvm-xfs resources: requests: storage: 5Gi template: metadata: labels: app: nginx-lvm spec: tolerations: - key: node-role.kubernetes.io/master operator: Exists effect: NoSchedule containers: - name: nginx image: 10.0.103.102/laijxtest/nginx:v1.23 imagePullPolicy: IfNotPresent volumeMounts: - mountPath: "/data" name: html command: - sh - "-c" - | while true; do echo "huizhi testing"; echo "yes ">>/data/yes.txt; sleep 120s done;
View Code
启动
Kubectl apply –f sts-nginx.yaml
现在可以看到已经创建了pv和pvc
扩容
kubectl patch pvc html-nginx-lvm-0 -p ‘{“spec”:{“resources”:{“requests”:{“storage”:”10Gi”}}}}’
将5G扩容为10G
Pvc需要等两分钟左右会显示变更为10G
进入容器查看挂载情况
删除pod，验证持久化存储
方式2、Helm安装open-local
前期准备：（具体准备过程略）
基于redhat和debian的linux发行版（本次实验使用centos7.9）
Kubernetes V1.20+ （如果有老版本的k8s，可能要先升级k8s版本）
LVM2+
Helm3.0+
集群中至少有一个空闲块设备用来测试
服务器：
Master 10.0.101.91
Node1 10.0.101.10
Node2 10.0.101.92
首先在三台服务器安装docker-ce-20.10.6（我的操作环境是内网的，有外网的话会简单很多，直接yum）
上传安装包，通过yum安装已经准备好的rpm包
yum localinstall -y ./*
编辑配置文件
Vim /etc/docker/daemon.json{"storage-driver": "overlay2","storage-opts": ["overlay2.override_kernel_check=true"],"insecure-registries" : ["10.0.103.102"],"exec-opts": ["native.cgroupdriver=systemd"],"log-driver": "json-file","log-opts": { "max-size": "100m", "max-file": "3" }}systemctl start dockersystemctl enable docker
安装k8s 1.22.4
Master：
上传rpm压缩包，解压，安装
unzip k8s-1.22.4.rpm.zip
yum localinstall -y ./*
echo “net.bridge.bridge-nf-call-iptables = 1” > /etc/sysctl.d/k8s.conf
echo “net.bridge.bridge-nf-call-ip6tables = 1” >> /etc/sysctl.d/k8s.conf
sysctl -p /etc/sysctl.d/k8s.conf
导入k8s初始化需要的镜像包，并打标签（有外网的话这步可以跳过）
unzip k8simages.zip
docker load < coredns.tar
docker image
docker load < etcd.tar
docker load < pause.tar
docker load < kube-apiserver.tar
docker load < kube-controller-manager.tar
docker load < kube-proxy.tar
docker load < kube-scheduler.tar
docker tag 8a5cc299272d k8s.gcr.io/kube-apiserver:v1.22.4
docker tag 0ce02f92d3e4 k8s.gcr.io/kube-controller-manager:v1.22.4
docker tag 721ba97f54a6 k8s.gcr.io/kube-scheduler:v1.22.4
docker tag edeff87e4802 k8s.gcr.io/kube-proxy:v1.22.4
docker tag 8d147537fb7d k8s.gcr.io/coredns/coredns:v1.8.4
docker tag 004811815584 k8s.gcr.io/etcd:3.5.0-0
docker tag ed210e3e4a5b k8s.gcr.io/pause:3.5
systemctl start kubelet
systemctl enable kubelet
开始初始化
kubeadm init –kubernetes-version=v1.22.4 –pod-network-cidr=10.244.0.0/16 –service-cidr=10.96.0.0/12
初始化完成之后执行：export KUBECONFIG=/etc/kubernetes/admin.conf
记录加入集群的命令，后面要用
kubeadm join 10.0.101.91:6443 –token ffvff0.bf7e0urhj0sys1ka –discovery-token-ca-cert-hash sha256:d7488bc4f02d96c359e322bef5097ff7d59d8ee31e6560011594f37bc0c0e29f
两台node节点操作：
上传rpm压缩包，解压，安装
unzip k8s-1.22.4.rpm.zip
yum localinstall -y ./*
echo “net.bridge.bridge-nf-call-iptables = 1” > /etc/sysctl.d/k8s.conf
echo “net.bridge.bridge-nf-call-ip6tables = 1” >> /etc/sysctl.d/k8s.conf
sysctl -p /etc/sysctl.d/k8s.conf
systemctl start kubelet
systemctl enable kubelet
加入集群
kubeadm join 10.0.101.91:6443 –token ffvff0.bf7e0urhj0sys1ka –discovery-token-ca-cert-hash sha256:d7488bc4f02d96c359e322bef5097ff7d59d8ee31e6560011594f37bc0c0e29f
安装flannel
三台服务器操作：
导入flannel镜像包
Unzip flannel.zip
cd flannel/
docker load < flannel.tar
docker load < flannel-cni-plugin.tar
Master操作：
kubectl apply -f kube-flannel.yml
[root@laijx-k8s-test1 ~]# cat k8s/kube-flannel.yml ---kind: NamespaceapiVersion: v1metadata: name: kube-flannel labels: pod-security.kubernetes.io/enforce: privileged---kind: ClusterRoleapiVersion: rbac.authorization.k8s.io/v1metadata: name: flannelrules:- apiGroups: - "" resources: - pods verbs: - get- apiGroups: - "" resources: - nodes verbs: - list - watch- apiGroups: - "" resources: - nodes/status verbs: - patch---kind: ClusterRoleBindingapiVersion: rbac.authorization.k8s.io/v1metadata: name: flannelroleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: flannelsubjects:- kind: ServiceAccount name: flannel namespace: kube-flannel---apiVersion: v1kind: ServiceAccountmetadata: name: flannel namespace: kube-flannel---kind: ConfigMapapiVersion: v1metadata: name: kube-flannel-cfg namespace: kube-flannel labels: tier: node app: flanneldata: cni-conf.json: | { "name": "cbr0", "cniVersion": "0.3.1", "plugins": [ { "type": "flannel", "delegate": { "hairpinMode": true, "isDefaultGateway": true } }, { "type": "portmap", "capabilities": { "portMappings": true } } ] } net-conf.json: | { "Network": "10.244.0.0/16", "Backend": { "Type": "vxlan" } }---apiVersion: apps/v1kind: DaemonSetmetadata: name: kube-flannel-ds namespace: kube-flannel labels: tier: node app: flannelspec: selector: matchLabels: app: flannel template: metadata: labels: tier: node app: flannel spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/os operator: In values: - linux hostNetwork: true priorityClassName: system-node-critical tolerations: - operator: Exists effect: NoSchedule serviceAccountName: flannel initContainers: - name: install-cni-plugin #image: flannelcni/flannel-cni-plugin:v1.1.0 for ppc64le and mips64le (dockerhub limitations may apply) image: docker.io/rancher/mirrored-flannelcni-flannel-cni-plugin:v1.1.0 command: - cp args: - -f - /flannel - /opt/cni/bin/flannel volumeMounts: - name: cni-plugin mountPath: /opt/cni/bin - name: install-cni #image: flannelcni/flannel:v0.19.2 for ppc64le and mips64le (dockerhub limitations may apply) image: docker.io/rancher/mirrored-flannelcni-flannel:v0.19.2 command: - cp args: - -f - /etc/kube-flannel/cni-conf.json - /etc/cni/net.d/10-flannel.conflist volumeMounts: - name: cni mountPath: /etc/cni/net.d - name: flannel-cfg mountPath: /etc/kube-flannel/ containers: - name: kube-flannel #image: flannelcni/flannel:v0.19.2 for ppc64le and mips64le (dockerhub limitations may apply) image: docker.io/rancher/mirrored-flannelcni-flannel:v0.19.2 command: - /opt/bin/flanneld args: - --ip-masq - --kube-subnet-mgr resources: requests: cpu: "100m" memory: "50Mi" limits: cpu: "100m" memory: "50Mi" securityContext: privileged: false capabilities: add: ["NET_ADMIN", "NET_RAW"] env: - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace - name: EVENT_QUEUE_DEPTH value: "5000" volumeMounts: - name: run mountPath: /run/flannel - name: flannel-cfg mountPath: /etc/kube-flannel/ - name: xtables-lock mountPath: /run/xtables.lock volumes: - name: run hostPath: path: /run/flannel - name: cni-plugin hostPath: path: /opt/cni/bin - name: cni hostPath: path: /etc/cni/net.d - name: flannel-cfg configMap: name: kube-flannel-cfg - name: xtables-lock hostPath: path: /run/xtables.lock type: FileOrCreate
View Code
正常部署flannel之后，集群node状态正常
部署个nginx测试一下
kubectl apply -f .
Pod部署正常。
部署helm3.7
tar -zxvf helm-v3.7.2-linux-amd64.tar.gz
cd linux-amd64/
mv helm /usr/bin/
helm version
Helm安装open-local
准备yaml文件
Helm安装open-local需要一些yaml文件，我已从github下载
open-local-main.zip
并且，由于我们是内网，所以需要将yaml文件中的镜像拉取策略由always改为ifnotpresent，并提前准备好镜像。
grep -C 2 -r image `find ./ -name “*.yaml”`
sed -i ‘s/imagePullPolicy:\ Always/imagePullPolicy:\ IfNotPresent/g’ ./templates/*
准备镜像
通过helm安装open-local之前，需要将用到的镜像手动导入到docker
Unzip open-local-images.zip
导入镜像
for i in `ls` ;do docker load < $i;done
开始安装open-local
cd open-local-main
helm install open-local ./helm
观察进度
kubectl get pod –A
大概几分钟之后，我们就能看到需要的open-local已经部署完成了。且使用 kubectl get sc 能够看到目前支持的存储驱动类型
Open-local使用创建样例pv和pvc，以供参考
参考open-local官方文档，创建一个StatefulSet 的同时创建pv和pvc：
[root@laijx-k8s-test1 tmp]# cat sts-nginx.yaml apiVersion: v1kind: Servicemetadata: name: nginx-lvm labels: app: nginx-lvmspec: ports: - port: 80 name: web clusterIP: None selector: app: nginx-lvm---apiVersion: apps/v1kind: StatefulSetmetadata: name: nginx-lvmspec: selector: matchLabels: app: nginx-lvm podManagementPolicy: Parallel serviceName: "nginx-lvm" replicas: 1 volumeClaimTemplates: - metadata: name: html spec: accessModes: - ReadWriteOnce storageClassName: yoda-lvm-xfs resources: requests: storage: 5Gi template: metadata: labels: app: nginx-lvm spec: tolerations: - key: node-role.kubernetes.io/master operator: Exists effect: NoSchedule containers: - name: nginx image: 10.0.103.102/laijxtest/nginx:v1.23 imagePullPolicy: IfNotPresent volumeMounts: - mountPath: "/data" name: html command: - sh - "-c" - | while true; do echo "huizhi testing"; echo "yes ">>/data/yes.txt; sleep 120s done;
View Code在已有的pv和pvc的yaml文档进行修改，创建我们自己想要的新的pv和pvc
（或者你也可以直接将这个部署单元删除，然后其他的部署单元可以使用这个样例pv和pvc了）
kubectl get pv -o yaml
新建pv文件pv01.yaml，将uid，creationTimestamp注释掉，并修改name：
[root@laijx-k8s-test1 k8s]# cat pv01.yaml apiVersion: v1items:- apiVersion: v1 kind: PersistentVolume metadata: annotations: pv.kubernetes.io/provisioned-by: local.csi.aliyun.com# creationTimestamp: "2022-10-08T01:37:50Z" finalizers: - kubernetes.io/pv-protection name: pv01 resourceVersion: "1941146"# uid: f91ffa5b-e4dc-42ed-8855-f6420b0979b4 spec: accessModes: - ReadWriteOnce capacity: storage: 5Gi claimRef:# apiVersion: v1# kind: PersistentVolumeClaim# name: pvc01# namespace: default# resourceVersion: "1941140"# uid: f8bbc82b-a781-4116-9bdd-59346b8d4240 csi: driver: local.csi.aliyun.com fsType: ext4 volumeAttributes: csi.storage.k8s.io/pv/name: pv01 csi.storage.k8s.io/pvc/name: pvc01 csi.storage.k8s.io/pvc/namespace: default storage.kubernetes.io/csiProvisionerIdentity: 1664262928261-8081-local.csi.aliyun.com vgName: open-local-pool-0 volume.kubernetes.io/selected-node: laijx-k8s-test2 volumeType: LVM volumeHandle: pv01 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - laijx-k8s-test2 persistentVolumeReclaimPolicy: Delete storageClassName: open-local-lvm volumeMode: Filesystem status: phase: Boundkind: Listmetadata: resourceVersion: "" selfLink: ""
View Code
claimRef:这个字段要注释掉，否则pv会认为已经绑定了pvc，导致报错Bound claim has lost its PersistentVolume
新建pvc文件pvc01.yaml，将uid，creationTimestamp注释掉，并修改name：
修改后的文件如下：
[root@laijx-k8s-test1 k8s]# cat pvc01.yaml apiVersion: v1items:- apiVersion: v1 kind: PersistentVolumeClaim metadata: annotations: pv.kubernetes.io/bind-completed: "yes" pv.kubernetes.io/bound-by-controller: "yes" volume.beta.kubernetes.io/storage-provisioner: local.csi.aliyun.com volume.kubernetes.io/selected-node: laijx-k8s-test2# creationTimestamp: "2022-10-08T01:37:49Z" finalizers: - kubernetes.io/pvc-protection labels: app: pv01 name: pvc01 namespace: default resourceVersion: "1941148"# uid: f8bbc82b-a781-4116-9bdd-59346b8d4240 spec: accessModes: - ReadWriteOnce resources: requests: storage: 5Gi storageClassName: open-local-lvm volumeMode: Filesystem volumeName: pv01 status: accessModes: - ReadWriteOnce capacity: storage: 5Gi phase: Boundkind: Listmetadata: resourceVersion: "" selfLink: ""
View Code创建pv、pvc
kubectl apply -f pvc01.yaml -f pv01.yaml
storageClassName若指定 open-local-lvm，则创建的文件系统为ext4,
若指定open-local-lvm-xfs，则创建的文件系统为xfs
使用pv和pvc
在部署单元中使用创建的pvc
[root@laijx-k8s-test1 k8s]# cat tomcat7-lvm.yaml apiVersion: apps/v1kind: Deploymentmetadata: name: tomcat7-2spec: replicas: 1 selector: matchLabels: app: tomcat7-2 template: metadata: labels: app: tomcat7-2 spec: containers: - name: tomcat7-2 image: 10.0.103.102/laijxtest/tomcat7:v1 ports: - containerPort: 80 command: - sh - "-c" - | while true; do echo "laijx testing"; echo "yes ">>/tmp/yes.txt; sleep 120s done; volumeMounts: - mountPath: /data/ name: tomcat7-lvm volumes: - name: tomcat7-lvm persistentVolumeClaim: claimName: pvc01---apiVersion: v1kind: Servicemetadata: name: tomcat7-2spec: ports: - name: tomcat7-2-svc port: 8080 targetPort: 8080 nodePort: 31124 selector: app: tomcat7-2 type: NodePort
View Code
启动部署单元：
kubectl apply -f tomcat7-lvm.yaml
进入容器可以看到，/data路径已经成功挂载
验证持久化：
进入容器cp /tmp/yes.txt /data/ 然后重启pod，再次观察文件是否还在：
持久化正常。
扩容pvc
kubectl patch pvc html-nginx-lvm-0 -p ‘{“spec”:{“resources”:{“requests”:{“storage”:”10Gi”}}}}’
参考文档：
https://github.com/alibaba/open-local/tree/main/docs
https://developer.aliyun.com/article/790208 Open-Local – 云原生本地磁盘管理系统
https://segmentfault.com/a/1190000042291703 如何通过 open-local 玩转容器本地存储? | 龙蜥技术
https://www.luozhiyun.com/archives/335 深入k8s：持久卷PV、PVC及其源码分析
作者注：本站文章除注明转载外，均为本站原创或编辑，欢迎大家转载，但请务必注明出处，尊重他人成果，谢谢。任何问题请联系1187616732@qq.com