系列文章

  • Cilium 系列文章

前言

在本章中，我们将直接将 Cilium 安装到 Kubernetes 集群中。

在实验中，我们用到的组件及版本为：

  • Cilium 1.13.4
  • K3s v1.26.6+k3s1
  • OS
    • Debian 10, Kernel 4.19.232, arm64
    • Ubuntu 23.04, Kernel 6.2, x86

📝Notes:

如前文所述，Cilium 对 Linux Kernel 版本要求很高。1.13.4 推荐 Kernel ≥ 5.10（使用最新 LTS 稳定版 Kernel), 最低 Linux Kernel 得是 4.19.57.
所以我们选择 2 个 OS, 一个只满足最低 Kernel 要求，一个是尽可能新的 Kernel, 看看安装和功能有哪些差别。

Cilium 安装方式

Cilium 支持 2 种安装方式：

  1. Cilium CLI
  2. Helm chart

CLI 工具能让你轻松上手 Cilium，尤其是在刚开始学习时。它直接使用 Kubernetes API 来检查与现有 kubectl 上下文相对应的集群，并为检测到的 Kubernetes 实施选择合适的安装选项。

Helm Chart 方法适用于需要对 Cilium 安装进行精细控制的高级安装和生产环境。它要求你为特定的 Kubernetes 环境手动选择最佳数据路径 (datapath) 和 IPAM 模式。

系统需求

要安装 Cilium, 最低系统需求如下：

  • 主机是 AMD64 或 AArch64（即 arm64) 架构
  • 需要开通某些防火墙规则，详见：https://docs.cilium.io/en/stable/operations/system_requirements/#firewall-rules
  • Linux Kernel >= 4.19.57（对于 RHEL8, Linux Kernel >= 4.18)
  • Linux Kernel 需要启用相关配置，具体可以参考这篇文章：玩转 PI 系列-如何在 Rockchip arm 开发板上安装 docker tailscale k3s cilium?
  • 如果需要用到 Cilium 的高级功能，则需要更高版本的内核，具体如下：
Cilium 功能最小 Kernel 版本
Bandwidth Manager>= 5.1
Egress Gateway>= 5.2
VXLAN Tunnel Endpoint (VTEP) Integration>= 5.2
WireGuard Transparent Encryption>= 5.6
Full support for Session Affinity>= 5.7
BPF-based proxy redirection>= 5.7
Socket-level LB bypass in pod netns>= 5.7
L3 devices>= 5.8
BPF-based host routing>= 5.10
IPv6 BIG TCP support>= 5.19

Cilium 安装

首次安装，我们使用 Cilium CLI 方式安装。OS 为：Debian 10, Kernel 4.19 4.19.232, arm64

安装 K3s

我们通过 K3s 安装 Kubernetes 集群。具体命令如下：

# Server Nodecurl -sfL https://rancher-mirror.rancher.cn/k3s/k3s-install.sh | INSTALL_K3S_MIRROR=cn INSTALL_K3S_EXEC='--write-kubeconfig-mode=644 --flannel-backend=none --disable-network-policy --prefer-bundled-bin' INSTALL_K3S_VERSION=v1.26.6+k3s1 sh -

📝Notes:

几个主流 Linux 发行版发布的 iptables 版本包含一个错误，该错误会导致重复规则的累积，从而对节点的性能和稳定性产生负面影响。有关如何确定你是否受此问题影响，请参阅 issue #3117。
K3s 具有一个可以正常运行的 iptables (v1.8.8) 版本。你可以通过使用 --prefer-bundled-bin 选项来启动 K3s，或从操作系统中卸载 iptables/nftables 包，从而让 K3s 使用捆绑的 iptables 版本。
版本--prefer-bundled-bin 标志从 2022-12 版本开始可用（v1.26.0+k3s1、v1.25.5+k3s1、v1.24.9+k3s1、v1.23.15+k3s1）。

验证：

$ systemctl status k3s● k3s.service - Lightweight Kubernetes   Loaded: loaded (/etc/systemd/system/k3s.service; enabled; vendor preset: enabled)   Active: active (running)
$ k3s kubectl get nodeNAME          STATUS   ROLES                  AGE    VERSIONlinaro-alip   NotReady    control-plane,master   3d1h   v1.26.6+k3s1

🐾注意，由于没有安装 flannel, 也还没开始安装 Cilium, 所以 node 状态应为：NotReady.

安装 Cilium CLI

CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/master/stable.txt)CLI_ARCH=amd64if [ "$(uname -m)" = "aarch64" ]; then CLI_ARCH=arm64; ficurl -L --fail --remote-name-all https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}sha256sum --check cilium-linux-${CLI_ARCH}.tar.gz.sha256sumsudo tar xzvfC cilium-linux-${CLI_ARCH}.tar.gz /usr/local/binrm cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}

验证：

$ cilium versioncilium-cli: v0.15.2 compiled with go1.20.4 on linux/arm64cilium image (default): v1.13.4cilium image (stable): v1.13.4cilium image (running): 1.13.4

Cilium Install

export KUBECONFIG=/etc/rancher/k3s/k3s.yamlcilium install

通过该命令，cilium 会自动进行一些环境信息的识别，以及参数的选择和判断：

🔮 Auto-detected Kubernetes kind: k3s✨ Running "k3s" validation checks✅ Detected k3s version "v1.26.6+k3s1"ℹ️ Using Cilium version 1.13.4🔮 Auto-detected cluster name: default🔮 Auto-detected datapath mode: tunnel🔮 Auto-detected kube-proxy has been installedℹ️ helm template --namespace kube-system cilium cilium/cilium --version 1.13.4 --setcluster.id=0,cluster.name=default,encryption.nodeEncryption=false,ipam.mode=kubernetes,kubeProxyReplacement=disabled,operator.replicas=1,serviceAccounts.cilium.name=cilium,serviceAccounts.operator.name=cilium-operator,tunnel=vxlanℹ️ Storing helm values file in kube-system/cilium-cli-helm-values Secret🔑 Created CA in secret cilium-ca🔑 Generating certificates for Hubble...🚀 Creating Service accounts...🚀 Creating Cluster roles...🚀 Creating ConfigMap for Cilium version 1.13.4...🚀 Creating Agent DaemonSet...🚀 Creating Operator Deployment...⌛ Waiting for Cilium to be installed and ready...

验证：

$ cilium status --wait    /¯¯\ /¯¯\__/¯¯\    Cilium:             OK \__/¯¯\__/    Operator:           OK /¯¯\__/¯¯\    Envoy DaemonSet:    disabled (using embedded mode) \__/¯¯\__/    Hubble Relay:       disabled    \__/       ClusterMesh:        disabledDaemonSet              cilium             Desired: 1, Ready: 1/1, Available: 1/1Deployment             cilium-operator    Desired: 1, Ready: 1/1, Available: 1/1Containers:            cilium             Running: 1                       cilium-operator    Running: 1Cluster Pods:          7/7 managed by CiliumHelm chart version:    1.13.4Image versions         cilium             quay.io/cilium/cilium:v1.13.4@sha256:bde8800d61aaad8b8451b10e247ac7bdeb7af187bb698f83d40ad75a38c1ee6b: 1                       cilium-operator    quay.io/cilium/operator-generic:v1.13.4@sha256:09ab77d324ef4d31f7d341f97ec5a2a4860910076046d57a2d61494d426c6301: 1

运行以下命令验证群集是否具有正确的网络连接：

$ cilium connectivity test --request-timeout 30s --connect-timeout 10sℹ️  Monitor aggregation detected, will skip some flow validation steps✨ [k8s-cluster] Creating namespace for connectivity check...(...)---------------------------------------------------------------------------------------------------------------------📋 Test Report---------------------------------------------------------------------------------------------------------------------✅ 69/69 tests successful (0 warnings)

🐾Warning:

在中国安装时，由于网络环境所限，可能部分测试会失败（如访问 1.1.1.1:443).
属于正常情况。
连接性测试需要至少两个 worker node 才能在群集中成功部署。连接性测试 pod 不会在以控制面角色运行的节点上调度。如果您没有为群集配置两个 worker node，连接性测试命令可能会在等待测试环境部署完成时停滞。

查看 Cilium Install 具体启用了哪些功能：

$ kubectl -n kube-system exec ds/cilium -- cilium statusDefaulted container "cilium-agent" out of: cilium-agent, config (init), mount-cgroup (init), apply-sysctl-overwrites (init), mount-bpf-fs (init), clean-cilium-state (init), install-cni-binaries (init)KVStore:                 Ok   DisabledKubernetes:              Ok   1.26 (v1.26.6+k3s1) [linux/arm64]Kubernetes APIs:         ["cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumEndpoint", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "core/v1::Namespace", "core/v1::Node", "core/v1::Pods", "core/v1::Service", "discovery/v1::EndpointSlice", "networking.k8s.io/v1::NetworkPolicy"]KubeProxyReplacement:    DisabledHost firewall:           DisabledCNI Chaining:            noneCNI Config file:         CNI configuration file management disabledCilium:                  Ok   1.13.4 (v1.13.4-4061cdfc)NodeMonitor:             Listening for events on 4 CPUs with 64x4096 of shared memoryCilium health daemon:    OkIPAM:                    IPv4: 9/254 allocated from 10.0.0.0/24,IPv6 BIG TCP:            DisabledBandwidthManager:        DisabledHost Routing:            LegacyMasquerading:            IPtablesController Status:       48/48 healthyProxy Status:            No managed proxy redirectGlobal Identity Range:   min 256, max 65535Hubble:                  Ok   Current/Max Flows: 4095/4095 (100.00%), Flows/s: 11.68   Metrics: DisabledEncryption:              DisabledCluster health:          1/1 reachable   (2023-07-19T12:25:40Z)

这里有几个点注意一下：

  1. datapath mode: tunnel: 因为兼容性原因，Cilium 会默认启用 tunnel（基于 vxlan) 的 datapatch 模式，也就是 overlay 网络结构。
  2. KubeProxyReplacement: Disabled Cilium 是没有完全替换掉 kube-proxy 的，后面我们会出文章介绍如何实现替换。
  3. IPv6 BIG TCP: Disabled 该功能要求 Linux Kernel >= 5.19, 所以在 Kernel 4.19.232 状态为禁用。
  4. BandwidthManager: Disabled 该功能要求 Linux Kernel >= 5.1, 所以目前是禁用的
  5. Host Routing: Legacy Legacy Host Routing 还是会用到 iptables, 性能较弱；但是 BPF-based host routing 需要 Linux Kernel >= 5.10
  6. Masquerading: IPtables IP 伪装有几种方式：基于 eBPF 的，和基于 iptables 的。默认使用基于 iptables, 推荐使用 基于 eBPF 的。
  7. Hubble Relay: disabled 默认 Hubble 也是禁用的。

Cilium 的最重要的特点就是其性能，所以只要是可以增强性能的，后续会一一介绍如何启用。

安装 Cilium Hubble

$ cilium hubble enable --ui✨ Patching ConfigMap cilium-config to enable Hubble...🚀 Creating ConfigMap for Cilium version 1.13.4...♻️ Restarted Cilium pods⌛ Waiting for Cilium to become ready before deploying other Hubble component(s)...🚀 Creating Peer Service...✨ Generating certificates...🔑 Generating certificates for Relay...✨ Deploying Relay...✨ Deploying Hubble UI and Hubble UI Backend...⌛ Waiting for Hubble to be installed...ℹ️ Storing helm values file in kube-system/cilium-cli-helm-values Secret✅ Hubble was successfully enabled!

验证：

$ cilium status    /¯¯\ /¯¯\__/¯¯\    Cilium:             OK \__/¯¯\__/    Operator:           OK /¯¯\__/¯¯\    Envoy DaemonSet:    disabled (using embedded mode) \__/¯¯\__/    Hubble Relay:       OK    \__/       ClusterMesh:        disabledDeployment             hubble-ui          Desired: 1, Ready: 1/1, Available: 1/1DaemonSet              cilium             Desired: 1, Ready: 1/1, Available: 1/1Deployment             cilium-operator    Desired: 1, Ready: 1/1, Available: 1/1Deployment             hubble-relay       Desired: 1, Ready: 1/1, Available: 1/1Containers:            hubble-ui          Running: 1                       cilium             Running: 1                       cilium-operator    Running: 1                       hubble-relay       Running: 1Cluster Pods:          9/9 managed by CiliumHelm chart version:    1.13.4Image versions         cilium             quay.io/cilium/cilium:v1.13.4@sha256:bde8800d61aaad8b8451b10e247ac7bdeb7af187bb698f83d40ad75a38c1ee6b: 1                       cilium-operator    quay.io/cilium/operator-generic:v1.13.4@sha256:09ab77d324ef4d31f7d341f97ec5a2a4860910076046d57a2d61494d426c6301: 1                       hubble-relay       quay.io/cilium/hubble-relay:v1.13.4@sha256:bac057a5130cf75adf5bc363292b1f2642c0c460ac9ff018fcae3daf64873871: 1                       hubble-ui          quay.io/cilium/hubble-ui:v0.11.0@sha256:bcb369c47cada2d4257d63d3749f7f87c91dde32e010b223597306de95d1ecc8: 1                       hubble-ui          quay.io/cilium/hubble-ui-backend:v0.11.0@sha256:14c04d11f78da5c363f88592abae8d2ecee3cbe009f443ef11df6ac5f692d839: 1

使用 Kubectl 检查集群状态

$ kubectl get nodesNAME          STATUS   ROLES                  AGE    VERSIONlinaro-alip   Ready    control-plane,master   3d1h   v1.26.6+k3s1
$ kubectl get daemonsets --all-namespacesNAMESPACE     NAME                     DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGEkube-system   cilium                   1         1         1       1            1           kubernetes.io/os=linux   28hkube-system   svclb-traefik-29b9c193   1         1         1       1            1                              2d23h
$ kubectl get deployments --all-namespacesNAMESPACE     NAME                     READY   UP-TO-DATE   AVAILABLE   AGEkube-system   local-path-provisioner   1/1     1            1           3d1hdefault       my-nginx                 2/2     2            2           32hkube-system   coredns                  1/1     1            1           3d1hkube-system   traefik                  1/1     1            1           2d23hkube-system   metrics-server           1/1     1            1           3d1hkube-system   cilium-operator          1/1     1            1           28hkube-system   hubble-relay             1/1     1            1           5m39skube-system   hubble-ui                1/1     1            1           5m38s

你应该会发现 cilium daemonset 正在集群的所有节点上运行，而 cilium-operator 部署正在单个节点上运行。

恭喜你！🎉🎉🎉你现在已经安装了 Cilium，为 Kubernetes 集群提供连接。

总结

本文我们主要介绍了 Cilium 的快速安装过程。

要安装 Cilium, 需要满足一些基本需求，其中 Cilium 对 Linux Kernel 版本的要求较高。

通过 cilium install, 在 Debian 10, Kernel 4.19.232, arm64 机器上，安装了 K3s v1.26.6+k3s1 和 Cilium 1.13.4, 启用了 Hubble, 并进行了验证。🎉🎉🎉

📚️参考文档

  • System Requirements — Cilium 1.13.4 documentation
  • Cilium Quick Installation — Cilium 1.13.4 documentation
  • Installation using Helm — Cilium 1.13.4 documentation

三人行, 必有我师; 知识共享, 天下为公. 本文由东风微鸣技术博客 EWhisper.cn 编写.