Spinnaker实战：云原生多云环境的持续部署方案

●目 录 01 声明式持续部署概述 ............................................................................................................................ 1 1.1 持续交付与持续部署 ................................................................................................................... 2 1.1.1 为什么要持续交付 ............................................................................................................ 2 1.1.2 持续交付的好处 ................................................................................................................ 3 1.1.3 保持随时可交付 ................................................................................................................ 4 1.1.4 解决问题：提高发布频率 ................................................................................................ 4 1.1.5 自动化持续部署 ................................................................................................................ 5 1.2 命令式与声明式 ........................................................................................................................... 6 1.2.1 简单易用的命令式 ............................................................................................................ 7 1.2.2 抽象和归纳的声明式 ........................................................................................................ 8 1.3 常见的声明式系统 ....................................................................................................................... 9 1.3.1 Kubernetes ......................................................................................................................... 9 1.3.2 Terraform ......................................................................................................................... 11 1.3.3 Ansible ............................................................................................................................. 12 1.4 声明式与命令式结合：声明式脚本流水线 .............................................................................. 13 1.4.1 核心思想 .......................................................................................................................... 13 1.4.2 代码即流水线 .................................................................................................................. 14 1.4.3 步骤执行 .......................................................................................................................... 15 1.5 声明式脚本流水线的意义 .......................................................................................................... 16 1.5.1 简化行为描述 .................................................................................................................. 16 1.5.2 降低学习曲线 .................................................................................................................. 17 1.5.3 落地持续部署 .................................................................................................................. 17 1.5.4 实现自动化 ...................................................................................................................... 17 1.6 本章小结 .................................................................................................................................... 18 02 管理云基础设施 ...................................................................................................................... 19 2.1 迁移至云原生与混合云的挑战 .................................................................................................. 20 2.1.1 凭据管理 .......................................................................................................................... 20 2.1.2 多云架构 .......................................................................................................................... 20 2.1.3 跨地域部署 ...................................................................................................................... 21 2.1.4 自动伸缩 .......................................................................................................................... 21 2.1.5 不可变的基础设施和部署制品 ...................................................................................... 22 2.1.6 服务发现 .......................................................................................................................... 22 2.2 组织云基础设施 ......................................................................................................................... 23 2.2.1 以应用为中心 .................................................................................................................. 23 2.2.2 抽象对云的操作 .............................................................................................................. 24 2.2.3 云模型 .............................................................................................................................. 26 2.2.4 多云配置 .......................................................................................................................... 26 2.3 流量组织形式 ............................................................................................................................. 27 2.3.1 启用/不启用 ..................................................................................................................... 27 2.3.2 启用/启用......................................................................................................................... 27 2.4 持续部署工具对比 ..................................................................................................................... 27 2.4.1 Tekton.............................................................................................................................. 28 2.4.2 Argo CD ........................................................................................................................... 31 2.5 本章小结 .................................................................................................................................... 36 03 Spinnaker 简介 ...................................................................................................................... 37 3.1 概念 ............................................................................................................................................ 38 3.2 应用管理 .................................................................................................................................... 38 3.2.1 应用................................................................................................................................. 39 3.2.2 服务器组 .......................................................................................................................... 39 3.2.3 集群................................................................................................................................. 39 3.2.4 负载均衡器 ...................................................................................................................... 41 3.2.5 防火墙 .............................................................................................................................. 41 3.3 应用程序部署 ............................................................................................................................. 42 3.3.1 流水线 .............................................................................................................................. 42 3.3.2 阶段................................................................................................................................. 43 3.3.3 任务................................................................................................................................. 43 3.3.4 部署策略 .......................................................................................................................... 43 3.4 云提供商 .................................................................................................................................... 45 3.5 Spinnaker 架构 ........................................................................................................................... 46 3.5.1 Deck ................................................................................................................................ 48 3.5.2 Gate ................................................................................................................................. 50 3.5.3 Clouddriver ...................................................................................................................... 50 3.5.4 Orca ................................................................................................................................. 51 3.5.5 Echo ................................................................................................................................ 52 3.5.6 Front50 ............................................................................................................................. 53 3.5.7 Igor .................................................................................................................................. 54 3.5.8 Fiat .................................................................................................................................. 54 3.5.9 Rosco ............................................................................................................................... 55 3.5.10 Kayenta .......................................................................................................................... 56 3.6 本章小结 .................................................................................................................................... 57 04 安装Spinnaker ....................................................................................................................... 59 4.1 环境要求 .................................................................................................................................... 59 4.1.1 Kubernetes ....................................................................................................................... 59 4.1.2 Kubectl ............................................................................................................................. 62 4.1.3 Jenkins .............................................................................................................................. 63 4.1.4 Docker Registery .............................................................................................................. 6.2 安装部署 .................................................................................................................................... 67 4.2.1 Halyard 命令行工具 ....................................................................................................... 67 4.2.2 选择云提供商 .................................................................................................................. 70 4.2.3 选择运行环境 .................................................................................................................. 71 4.2.4 选择存储方式 .................................................................................................................. 71 4.2.5 部署................................................................................................................................. 73 4.2.6 升级................................................................................................................................. 78 4.2.7 备份配置 .......................................................................................................................... 79 4.2.8 常见问题 .......................................................................................................................... 81 4.3 本章小结 .................................................................................................................................... 82 05 Spinnaker 基本工作流程：流水线 .......................................................................................... 84 5.1 管理流水线 ................................................................................................................................ 85 5.1.1 创建流水线 ...................................................................................................................... 85 5.1.2 配置流水线 ...................................................................................................................... 87 5.1.3 添加自动触发器 .............................................................................................................. 87 5.1.4 添加阶段 .......................................................................................................................... 88 5.1.5 手动运行流水线 .............................................................................................................. 89 5.1.6 禁用流水线 ...................................................................................................................... 91 5.1.7 删除流水线 ...................................................................................................................... 91 5.1.8 锁定流水线 ...................................................................................................................... 92 5.1.9 重命名流水线 .................................................................................................................. 93 5.1.10 通过JSON 编辑流水线 ................................................................................................ 93 5.1.11 流水线历史版本 ............................................................................................................ 94 5.2 部署制品 .................................................................................................................................... 95 5.2.1 在流水线中使用制品 ...................................................................................................... 98 5.2.2 自定义触发器制品 ........................................................................................................ 103 5.2.3 Kubernetes Manifest 制品 ............................................................................................ 104 5.2.4 制品类型 ........................................................................................................................ 108 5.3 启动参数 .................................................................................................................................. 108 5.4 阶段 .......................................................................................................................................... 109 5.4.1 基础设施阶段 ................................................................................................................ 110 5.4.2 集成外部系统阶段 ........................................................................................................ 112 5.4.3 测试阶段 ........................................................................................................................ 113 5.4.4 流程控制阶段 ................................................................................................................ 113 5.4.5 自定义阶段 .................................................................................................................... 114 5.5 触发器 ...................................................................................................................................... 114 5.5.1 时间型触发器 ................................................................................................................ 115 5.5.2 事件型触发器 ................................................................................................................ 115 5.6 通知 .......................................................................................................................................... 116 5.7 流水线表达式 ........................................................................................................................... 118 5.7.1 编写表达式 .................................................................................................................... 119 5.7.2 测试表达式 .................................................................................................................... 124 5.8 版本控制和审计 ....................................................................................................................... 125 5.9 动态流水线示例 ....................................................................................................................... 126 5.10 本章小结 ................................................................................................................................ 132 06 深入核心概念 ........................................................................................................................ 133 6.1 虚拟机阶段 ............................................................................................................................... 133 6.1.1 Bake ............................................................................................................................... 133 6.1.2 Tag Image ....................................................................................................................... 135 6.1.3 Find Image From Cluster ............................................................................................... 135 6.1.4 Find Image From Tags ................................................................................................... 136 6.1.5 Deploy ............................................................................................................................ 137 6.1.6 Disable Cluster ............................................................................................................... 139 6.1.7 Disable Server Group ..................................................................................................... 140 6.1.8 Enable Server Group ...................................................................................................... 141 6.1.9 Resize Server Group ...................................................................................................... 142 6.1.10 Clone Server Group ...................................................................................................... 143 6.1.11 Rollback Cluster ........................................................................................................... 144 6.1.12 Scale Down Cluster ...................................................................................................... 145 6.2 Kubernetes 阶段 ........................................................................................................................ 145 6.2.1 Bake (Manifest) .............................................................................................................. 146 6.2.2 Delete (Manifest) ........................................................................................................... 147 6.2.3 Deploy (Manifest) .......................................................................................................... 148 6.2.4 Find Artifacts From Resource（Manifest） .................................................................. 151 6.2.5 Patch (Manifest) ............................................................................................................. 152 6.2.6 Scale （Manifest） ....................................................................................................... 154 6.2.7 Undo Rollout（Manifest） ............................................................................................ 155 6.3 集成外部系统阶段 ................................................................................................................... 156 6.3.1 Jenkins ............................................................................................................................ 156 6.3.2 运行 Script 脚本 .......................................................................................................... 158 6.3.3 Travis 阶段 ..................................................................................................................... 160 6.3.4 Concourse 阶段 .............................................................................................................. 162 6.3.5 Wercker 阶段 ................................................................................................................. 163 6.3.6 Webhook 阶段 ............................................................................................................... 165 6.3.7 自定义 Webhook 阶段 .................................................................................................. 167 流程控制阶段 ........................................................................................................................... 170 .1 Wait ............................................................................................................................... 171 .2 Manual Judgment ........................................................................................................... 171 .3 Check Preconditions ....................................................................................................... 173 .4 Pipeline .......................................................................................................................... 174 6.5 其他阶段 .................................................................................................................................. 175 6.6 部署制品类型 ........................................................................................................................... 176 6.6.1 Docker 镜像 .................................................................................................................. 176 6.6.2 Base ............................................................................................................................ 178 6.6.3 AWS S3 .......................................................................................................................... 179 6. Git Repo ......................................................................................................................... 181 6.6.5 GitHub 文件 .................................................................................................................. 182 6.6.6 GitLab 文件 .................................................................................................................. 184 6.6.7 Helm ............................................................................................................................... 185 6.6.8 HTTP 文件 ..................................................................................................................... 188 6.6.9 Kubernetes 对象............................................................................................................ 189 6.6.10 Maven .......................................................................................................................... 190 6.7 配置触发器 ............................................................................................................................... 192 6.7.1 Git ................................................................................................................................. 192 6.7.2 Docker Registry ............................................................................................................. 194 6.7.3 Helm Chart ..................................................................................................................... 196 6.7.4 Artifactory ...................................................................................................................... 197 6.7.5 Webhook ........................................................................................................................ 198 6.7.6 Jenkins ............................................................................................................................ 201 6.7.7 Concourse...................................................................................................................... 202 6.7.8 Travis ............................................................................................................................. 202 6.7.9 CRON ............................................................................................................................ 203 6.7.10 Pipeline ........................................................................................................................ 204 6.7.11 Pub/Sub ........................................................................................................................ 204 6.8 使用流水线模板 ....................................................................................................................... 205 6.8.1 安装 Spin CLI ............................................................................................................... 206 6.8.2 创建流水线模板 ............................................................................................................ 209 6.8.3 渲染流水线模板 ............................................................................................................ 211 6.8.4 使用模板创建流水线 .................................................................................................... 211 6.8.5 继承模板或覆盖 ............................................................................................................ 213 6.9 消息通知 .................................................................................................................................. 213 6.9.1 Email .............................................................................................................................. 216 6.9.2 Slack ............................................................................................................................... 218 6.9.3 SMS............................................................................................................................... 220 6.9.4 企业微信机器人 ............................................................................................................ 221 6.9.5 钉钉机器人 .................................................................................................................... 223 6.10 本章小结 ................................................................................................................................ 226 07 自动金丝雀分析 .................................................................................................................... 227 7.1 Spinnaker 自动金丝雀发布 ..................................................................................................... 227 7.2 安装组件 .................................................................................................................................. 229 7.2.1 安装 Prometheus ........................................................................................................... 229 7.2.2 集成 Minio .................................................................................................................... 232 7.2.3 集成 Prometheus ........................................................................................................... 233 7.3 配置金丝雀 ............................................................................................................................... 233 7.3.1 创建一个金丝雀配置 .................................................................................................... 234 7.3.2 创建和使用选择器模板 ................................................................................................ 239 7.3.3 创建金丝雀阶段 ............................................................................................................ 240 7.4 获取金丝雀报告 ....................................................................................................................... 248 7.5 工作原理 .................................................................................................................................. 250 7.6 佳实践 .................................................................................................................................. 251 7.7 本章小结 .................................................................................................................................. 253 08 混沌工程 ............................................................................................................................... 254 8.1 理论基础 .................................................................................................................................. 254 8.1.1 概念定义 ........................................................................................................................ 254 8.1.2 发展历程 ........................................................................................................................ 255 8.2 为什么需要混沌工程 ............................................................................................................... 256 8.2.1 与测试的区别 ................................................................................................................ 256 8.2.2 与故障注入的区别 ........................................................................................................ 256 8.2.3 核心思想 ........................................................................................................................ 257 8.3 五大原则 .................................................................................................................................. 257 8.3.1 建立稳定状态的假设 .................................................................................................... 257 8.3.2 用多样的现实世界事件做验证 .................................................................................... 258 8.3.3 在生产环境中进行测试 ................................................................................................ 258 8.3.4 快速终止和小爆炸半径 ............................................................................................ 259 8.3.5 自动化实验以持续运行 ................................................................................................ 259 8.4 如何实现混沌工程 ................................................................................................................... 259 8.4.1 设计实验步骤 ................................................................................................................ 260 8.4.2 确定成熟度模型 ............................................................................................................ 260 8.4.3 确定应用度模型 ............................................................................................................ 262 8.4.4 绘制成熟度模型 ............................................................................................................ 263 8.5 在 Spinnaker 中实施混沌工程 ............................................................................................... 263 8.5.1 Gremlin .......................................................................................................................... 2 8.5.2 Chaos Mesh .................................................................................................................... 265 8.6 本章小结 .................................................................................................................... 268 09 使部署更加安全 .................................................................................................................... 269 9.1 集群部署 .................................................................................................................................. 269 9.1.1 部署策略 ........................................................................................................................ 269 9.1.2 回滚策略 ........................................................................................................................ 278 9.1.3 时间窗口 ........................................................................................................................ 283 9.2 流水线执行 ............................................................................................................................... 285 9.2.1 并发............................................................................................................................... 285 9.2.2 锁定............................................................................................................................... 286 9.2.3 禁用............................................................................................................................... 287 9.2.4 阶段条件判断 ................................................................................................................ 288 9.2.5 人工确认 ........................................................................................................................ 288 9.3 自动验证阶段 ........................................................................................................................... 295 9.4 审计和可追溯 ........................................................................................................................... 299 9.4.1 消息通知 ........................................................................................................................ 299 9.4.2 流水线变更历史 ............................................................................................................ 300 9.4.3 事件流记录 .................................................................................................................... 301 9.5 本章小结 .................................................................................................................................. 302 10 佳实践 ............................................................................................................................... 303 10.1 南北流量自动灰度发布：Kubernetes + Nginx Ingress ......................................................... 304 10.1.1 环境准备 ...................................................................................................................... 304 10.1.2 部署 Nginx Ingress ..................................................................................................... 305 10.1.3 初始化环境 .................................................................................................................. 308 10.1.4 创建流水线 .................................................................................................................. 309 10.1.5 运行流水线 .................................................................................................................. 311 10.1.6 原理分析 ...................................................................................................................... 317 10.1.7 生产建议 ...................................................................................................................... 319 10.2 东西流量自动灰度发布：Kubernetes + Service Mesh .......................................................... 319 10.2.1 环境准备 ...................................................................................................................... 320 10.2.2 安装 Istio..................................................................................................................... 321 10.2.3 Bookinfo 应用 ............................................................................................................. 322 10.2.4 初始化环境 .................................................................................................................. 324 10.2.5 创建流水线 .................................................................................................................. 326 10.2.6 运行流水线 .................................................................................................................. 328 10.2.7 原理分析 ...................................................................................................................... 332 10.3 本章小结 ................................................................................................................................ 334 11 生产建议 ............................................................................................................................... 336 11.1 SSL ......................................................................................................................................... 336 11.2 认证 ........................................................................................................................................ 341 11.2.1 SAML ........................................................................................................................... 342 11.2.2 OAuth ........................................................................................................................... 345 11.2.3 LDAP ........................................................................................................................... 349 11.2.4 x509 .............................................................................................................................. 350 11.3 授权 ........................................................................................................................................ 351 11.3.1 YAML .......................................................................................................................... 353 11.3.2 SAML ........................................................................................................................... 354 11.3.3 LDAP ........................................................................................................................... 354 11.3.4 GitHub .......................................................................................................................... 355 11.3.5 Service Account ............................................................................................................ 356 11.3.6 流水线权限 .................................................................................................................. 358 11.4 Redis 配置优化 ....................................................................................................................... 359 11.5 横向扩容 ................................................................................................................................ 360 11.6 使用MySQL 作为存储系统 .................................................................................................. 363 11.6.1 Front50 ......................................................................................................................... 366 11.6.2 Clouddriver .................................................................................................................. 367 11.6.3 Orca .............................................................................................................................. 369 11.7 监控 ........................................................................................................................................ 372 11.7.1 Prometheus ................................................................................................................... 373 11.7.2 Grafana ......................................................................................................................... 378 11.8 本章小结 ................................................................................................................................ 382 12 扩展 Spinnaker .................................................................................................................... 383 12.1 配置开发环境 ......................................................................................................................... 383 12.1.1 Kork ............................................................................................................................. 383 12.1.2 组件概述 ...................................................................................................................... 384 12.1.3 环境配置 ...................................................................................................................... 385 12.2 编写新阶段 ............................................................................................................................. 386 12.3 本章小结 ................................................................................................................................ 394 13 迁移到Spinnaker ................................................................................................................. 395 13.1 如何说服团队 ......................................................................................................................... 395 13.2 迁移原则 ................................................................................................................................ 396 13.2.1 小化变更工作流 ...................................................................................................... 396 13.2.2 利用已有设施 .............................................................................................................. 397 13.2.3 组织架构不变性 .......................................................................................................... 397 13.3 本章小结 ................................................................................................................................ 399