快速部署OpenShift应用

发布时间：2020-08-14 15:04:46 来源：网络阅读：5579 作者：川川Jason 栏目：云计算

本文介绍了使用Service Catalog和OC命令部署OpenShift应用、部署基本概念和流程、扩展存储、清理OpenShift对象等。以Angular 6集成Spring Boot 2,Spring Security,JWT和CORS中的Spring Boot和Angular项目为例，详细讲解了S2I和Pipeline两种部署方式。

OKD版本3.11，Spring Boot项目源码heroes-api，Angular项目源码heroes-web。

初识OpenShift部署

Service Catalog

快速部署OpenShift应用
OpenShift初始安装中含有一些样例APP供大家学习使用。其中有Apache HTTP Server和Apache HTTP Server(httpd)，这两者有什么区别？分别点击进入可以发现：

Apache HTTP Server使用template(template名字为httpd-example)部署方式。

Apache HTTP Server(httpd)使用builder image(image stream名字为httpd)部署方式。
Service Catalog样例使用了template和builder image(image+source)两种部署方式。进入Application Console中的openshift项目可查看template和image。
查看template，点击Resources -> Other Resources -> Template：
快速部署OpenShift应用
查看Image Stream，点击Builds -> Images：

其他部署方式
在Service Catalog中，除从Catalog直接选择Item外，还提供了其他三种方式：
快速部署OpenShift应用
Deploy Image可以直接从image或image stream部署应用：

Import YAML / JSON 用来从YAML或JSON创建资源，比如image stream、template：

Select from Project 从指定的Project中选择template来部署应用：

部署Apache HTTP Server

Apache HTTP Server的两种部署方式本质上是相同的，Build策略均为S2I(Source-to-Image)，使用S2I构建的Docker镜像来部署应用。Source均使用Apache HTTP Server (httpd) S2I Sample Application，Docker基础镜像(builder image)均使用Apache HTTP Server Container Image。httpd-example template定义了整体部署流程并实现了参数化。
以下是httpd-example template中BuildConfig部分的定义：

- apiVersion: v1
  kind: BuildConfig
  metadata:
    annotations:
      description: Defines how to build the application
      template.alpha.openshift.io/wait-for-ready: 'true'
    name: '${NAME}'
  spec:
    output:
      to:
        kind: ImageStreamTag
        name: '${NAME}:latest'
    source:
      contextDir: '${CONTEXT_DIR}'
      git:
        ref: '${SOURCE_REPOSITORY_REF}'
        uri: '${SOURCE_REPOSITORY_URL}'
      type: Git
    strategy:
      sourceStrategy:
        from:
          kind: ImageStreamTag
          name: 'httpd:2.4'
          namespace: '${NAMESPACE}'
      type: Source
    triggers:
      - type: ImageChange
      - type: ConfigChange
      - github:
          secret: '${GITHUB_WEBHOOK_SECRET}'
        type: GitHub
      - generic:
          secret: '${GENERIC_WEBHOOK_SECRET}'
        type: Generic

参数定义及默认值：

parameters:
  - description: The name assigned to all of the frontend objects defined in this template.
    displayName: Name
    name: NAME
    required: true
    value: httpd-example
  - description: The OpenShift Namespace where the ImageStream resides.
    displayName: Namespace
    name: NAMESPACE
    required: true
    value: openshift
  - description: Maximum amount of memory the container can use.
    displayName: Memory Limit
    name: MEMORY_LIMIT
    required: true
    value: 512Mi
  - description: The URL of the repository with your application source code.
    displayName: Git Repository URL
    name: SOURCE_REPOSITORY_URL
    required: true
    value: 'https://github.com/openshift/httpd-ex.git'
  - description: >-
      Set this to a branch name, tag or other ref of your repository if you are
      not using the default branch.
    displayName: Git Reference
    name: SOURCE_REPOSITORY_REF
  - description: >-
      Set this to the relative path to your project if it is not in the root of
      your repository.
    displayName: Context Directory
    name: CONTEXT_DIR
  - description: >-
      The exposed hostname that will route to the httpd service, if left blank a
      value will be defaulted.
    displayName: Application Hostname
    name: APPLICATION_DOMAIN
...

我们先使用builder image方式部署Apache，来了解一下部署的整体流程：
快速部署OpenShift应用
点击"advanced options"，可以设置git branch、context、secret，自定义Route、Build Configuration、Deployment Configuration、Resource Limits等。此处填完基本内容后直接点击Create，创建App，然后从成功页面进入Project Overview：

部署过程中自动创建Service、Route、Build、Deployment、Image。进入Application Console的Applications和Builds可以查看详细信息，其中会创建3个pod：httpd-1-build、http-1-deploy、httpd-1-xxxxx，部署完毕后http-1-deploy会自动删除。
部署成功后，测试访问Apache Server(Route定义的Hostname)，页面如下：
快速部署OpenShift应用
下面解释一下涉及到的基本概念。

基本概念

Service(Kubernetes Service)
内部load balancer，用在OpenShift内部网络中，可使用Service ClusterIP或Hostname访问。

apiVersion: v1
kind: Service
metadata:
  annotations:
    openshift.io/generated-by: OpenShiftWebConsole
  creationTimestamp: '2019-03-26T02:12:50Z'
  labels:
    app: httpd
  name: httpd
  namespace: my-project
  resourceVersion: '3004428'
  selfLink: /api/v1/namespaces/my-project/services/httpd
  uid: a81c759f-4f6c-11e9-9a7d-02fa2ffc40e6
spec:
  clusterIP: 172.30.225.159
  ports:
    - name: 8080-tcp
      port: 8080
      protocol: TCP
      targetPort: 8080
  selector:
    deploymentconfig: httpd
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}

其中，selector定义了查找container(pod)进行负载均衡的标签。
Route
定义一个hostname来公开Service，以便外部客户可以访问Service，默认hostname为：[app-name]-[project-name].[openshift_master_default_subdomain]。
Build
构建App Image，使用S2I时即从builder image和Source Code来构建App Image。默认builder image和build配置变化时会重新build。

查看Builds -> httpd -> #1 的YAML文本，可以了解Build流程为FetchInputs -> Assemble -> CommitContainer -> PushImage：

...
status:
  completionTimestamp: '2019-03-26T02:13:30Z'
  config:
    kind: BuildConfig
    name: httpd
    namespace: my-project
  duration: 40000000000
  output:
    to:
      imageDigest: 'sha256:5c1f20f20baaa796f4518d11ded13c6fac33e7a377774cfec77aa1e6e6a7cbb2'
  outputDockerImageReference: 'docker-registry.default.svc:5000/my-project/httpd:latest'
  phase: Complete
  stages:
    - durationMilliseconds: 3434
      name: FetchInputs
      startTime: '2019-03-26T02:12:56Z'
      steps:
        - durationMilliseconds: 3434
          name: FetchGitSource
          startTime: '2019-03-26T02:12:56Z'
    - durationMilliseconds: 2127
      name: CommitContainer
      startTime: '2019-03-26T02:13:11Z'
      steps:
        - durationMilliseconds: 2127
          name: CommitContainer
          startTime: '2019-03-26T02:13:11Z'
    - durationMilliseconds: 3426
      name: Assemble
      startTime: '2019-03-26T02:13:10Z'
      steps:
        - durationMilliseconds: 3426
          name: AssembleBuildScripts
          startTime: '2019-03-26T02:13:10Z'
    - durationMilliseconds: 16143
      name: PushImage
      startTime: '2019-03-26T02:13:14Z'
      steps:
        - durationMilliseconds: 16143
          name: PushImage
          startTime: '2019-03-26T02:13:14Z'
  startTimestamp: '2019-03-26T02:12:50Z'

Build Strategy
OpenShift支持Source-to-Image、Docker、Pipeline、Custom四种Build Strategy。

strategy:
  sourceStrategy:
    from:
      kind: "ImageStreamTag"
      name: "builder-image:latest"

strategy:
  dockerStrategy:
    from:
      kind: "ImageStreamTag"
      name: "debian:latest"

spec:
  source:
    git:
      uri: "https://github.com/openshift/ruby-hello-world"
  strategy:
    jenkinsPipelineStrategy:
      jenkinsfilePath: some/repo/dir/filename

strategy:
  customStrategy:
    from:
      kind: "DockerImage"
      name: "openshift/sti-image-builder"

Deployment
部署App Image，Deployment包含三种对象：DeploymentConfig、ReplicationController、Pod。DeploymentConfig包含部署策略、image配置、环境变量等，ReplicationController包含复制相关信息。App Image和deployment配置变化时会自动重新Deploy。

进入Deployments -> httpd -> #1，编辑Replicas或调节pods数可以增删pod：
快速部署OpenShift应用
Deployment Strategy
修改或升级App，即重新部署应用时的部署方式。部署配置(DeploymentConfig)支持三种策略：Rolling、Recreate、Custom。通过修改Route可以实现蓝/绿部署、A/B部署。

Rolling 默认策略，当新版本Pod状态变为Ready后才scale down老版本Pod，可能同时存在新老版本的Pod
Recreate 先终止所有Pod(Scale down the previous deployment to zero)再部署新Pod
Custom 自定义部署行为

ImageStream
OpenShift管理容器镜像的方式，其中定义了dockerImageReference，ImageStream tag定义了同docker image各版本的映射关系。Build成功后会自动创建ImageStream。

apiVersion: image.openshift.io/v1
kind: ImageStream
metadata:
  annotations:
    openshift.io/generated-by: OpenShiftWebConsole
  creationTimestamp: '2019-03-26T02:12:50Z'
  generation: 1
  labels:
    app: httpd
  name: httpd
  namespace: my-project
  resourceVersion: '3004571'
  selfLink: /apis/image.openshift.io/v1/namespaces/my-project/imagestreams/httpd
  uid: a81b14bf-4f6c-11e9-9a7d-02fa2ffc40e6
spec:
  lookupPolicy:
    local: false
status:
  dockerImageRepository: 'docker-registry.default.svc:5000/my-project/httpd'
  tags:
    - items:
        - created: '2019-03-26T02:13:30Z'
          dockerImageReference: >-
            docker-registry.default.svc:5000/my-project/httpd@sha256:5c1f20f20baaa796f4518d11ded13c6fac33e7a377774cfec77aa1e6e6a7cbb2
          generation: 1
          image: >-
            sha256:5c1f20f20baaa796f4518d11ded13c6fac33e7a377774cfec77aa1e6e6a7cbb2
      tag: latest

Template
定义整体部署流程并实现参数化，包含Service、Route、ImageStream、BuildConfig、DeploymentConfig、parameters等部分。

了解了以上基本概念就很容易理解httpd-example template了，您可以自己部署测试，此处不再赘述。

OC Tool

使用oc new-app部署应用
继续之前，先将以前创建的测试project删除或新建一个project。

$ oc delete project my-project
$ oc new-project my-project

在Service Catalog一节我们提到了部署应用的三种方式：template、builder image(image+source)、image，对应的命令如下：

$ oc new-app httpd-example -p APPLICATION_DOMAIN=httpd-example.apps.itrunner.org
$ oc new-app openshift/httpd:2.4~https://github.com/openshift/httpd-ex.git --name=httpd-ex
$ oc new-app my-project/httpd-ex --name=httpd

说明：

image+source的语法为[image]~[source]
第三种方式使用的image为第二种方式中生成的
后面两种方式不会自动创建Route，需要手工创建：

$ oc expose service httpd-ex --name httpd-ex --hostname=httpd-ex.apps.itrunner.org
$ oc expose service httpd --name httpd --hostname=httpd.apps.itrunner.org

从JSON/YAML创建资源：

$ oc create -f <filename> -n <project>

使用oc命令还可以直接从source code创建应用，可以使用本地或远程source code：

$ oc new-app /path/to/source/code
$ oc new-app https://github.com/sclorg/cakephp-ex

可以指定子目录：

$ oc new-app https://github.com/sclorg/s2i-ruby-container.git --context-dir=2.0/test/puma-test-app

可以指定branch：

$ oc new-app https://github.com/openshift/ruby-hello-world.git#beta4

OpenShift自动检测代码根目录或指定目录，如果存在Dockerfile则使用Docker build策略，如果存在Jenkinsfile则使用Pipeline build策略，否则使用Source build策略(S2I)。

下面的例子使用了Source build策略：

$ oc new-app https://github.com/sclorg/cakephp-ex

使用Source build策略时，new-app通过检测根目录或指定目录的文件来确定language builder：

Language	Files
jee	pom.xml
nodejs	app.json, package .json
perl	cpanfile, index.pl
php	composer.json, index.php
python	requirements.txt, setup.py
ruby	Gemfile, Rakefile, config.ru
scala	build.sbt
golang	Godeps, main.go

然后根据语言，从OpenShift Server或Docker Hub Registry中查找与语言匹配的image。

也可以指定策略，如下：

$ oc new-app /home/user/code/myapp --strategy=docker

查看template和image stream
查看所有template和image stream：

$ oc new-app --list

单独查看template或image stream：

$ oc get templates -n openshift
$ oc get imagestreams -n openshift

查看httpd-example template详细信息：

$ oc describe template httpd-example -n openshift

查看httpd image stream详细信息：

$ oc describe imagestream httpd -n openshift

查看httpd-example template的YAML定义：

$ oc new-app --search --template=httpd-example --output=yaml

从所有template、image stream、docker image中查找"httpd"：

$  oc new-app --search httpd

再谈Route

前面的例子Route使用的协议均为http，如何启用https呢?
使用Web Console时，编辑route启用Secure route即可：
快速部署OpenShift应用
TLS Termination有三种类型：edge、passthrough、reencrypt

edge 访问route使用https协议，route到内部网络为非加密的，如未配置证书则使用默认证书。
reencrypt 全部访问路径均是加密的
passthrough 加密通信直接发送到目标，route不需提供TLS Termination。

使用oc命令创建route：

$ oc create route edge httpd-ex --service httpd-ex --hostname httpd-ex.apps.itrunner.org --path / --insecure-policy Redirect -n my-project

S2I

Source-to-Image (S2I)是一个框架，可以容易地将应用程序源代码作为输入生成一个新的docker image。

使用S2I构建image，在装配过程中可以执行大量复杂的操作，所有操作仅创建一个新的layer，加速了处理过程。S2I使得软件开发工程师不必关心docker image的制作，仅负责编写assemble、run等脚本，也可以防止开发工程师在image构建过程中执行任意yum安装等不适宜的操作。S2I简化了docker image的制作。

S2I需要以下三个基本要素：

builder image S2I以此image为基础来构建新的image
Sources
S2I Scripts

在构建过程中，S2I先获取sources和scripts，将其打包为tar文件后放入builder image中。在执行assemble script前，S2I解压tar文件到io.openshift.s2i.destination指定的目录，默认目录为/tmp（分别解压到/tmp/src、/tmp/scripts目录）。
快速部署OpenShift应用

S2I Scripts

S2I Scripts可以位于以下位置，优先级从高到低：

在BuildConfig中指定位置

strategy:
  sourceStrategy:
    from:
      kind: "ImageStreamTag"
      name: "builder-image:latest"
    scripts: "http://somehost.com/scripts_directory"

应用程序源码的.s2i/bin目录
在builder image的Dockerfile中定义位置（在样例Apache HTTP Server中使用了这种方式）

LABEL io.openshift.s2i.scripts-url="image:///usr/libexec/s2i"

io.openshift.s2i.scripts-url和BuildConfig定义位置均可采用以下形式：

image:///path_to_scripts_dir - image的绝对路径
file:///path_to_scripts_dir - host绝对或相对路径
http(s)://path_to_scripts_dir - URL

S2I Scripts：

Script	Description
assemble (required)	获取源码、编译、打包。在增量编译时，如果定义了save-artifacts，先恢复artifact
run (required)	运行应用
save-artifacts (optional)	收集依赖以便加速后续编译，比如.m2
usage (optional)	显示image使用帮助信息
test/run (optional)	检查image能否正常工作

Example assemble script

#!/bin/bash

# restore build artifacts
if [ "$(ls /tmp/artifacts/ 2>/dev/null)" ]; then
    mv /tmp/artifacts/* $HOME/.
fi

# move the application source
mv /tmp/s2i/src $HOME/src

# build application artifacts
pushd ${HOME}
make all

# install the artifacts
make install
popd

Example run script

#!/bin/bash

# run the application
/opt/application/run.sh

Example save-artifacts script

#!/bin/bash

# Besides the tar command, all other output to standard out must 
# be surpressed.  Otherwise, the tar stream will be corrupted.
pushd ${HOME} >/dev/null
if [ -d deps ]; then
    # all deps contents to tar stream
    tar cf - deps
fi
popd >/dev/null

注意：save-artifacts只能有tar stream输出，不能含有其它任何输出。
Example usage script

#!/bin/bash

# inform the user how to use the image
cat <<EOF
This is a S2I sample builder image, to use it, install
https://github.com/openshift/source-to-image
EOF

S2I Tool

仅为学习S2I的基本知识和S2I Tool的使用，部署OpenShift应用时是不必安装的。

安装S2I Tool
下载S2I后解压安装：

# tar -xvf release.tar.gz .
# cp /path/to/s2i /usr/local/bin

S2I命令

create       创建生成builder image的基础目录结构
build         构建新的image
rebuild      重建image
usage       显示image usage信息
version     显示s2i version
completion  Generate completion for the s2i command (bash or zsh)

示例

s2i create语法：

s2i create <imageName> <destination> [flags]

$ s2i create ruby-centos7 ruby-centos7

执行以上命令生成的目录结构如下：

ruby-centos7
├── Dockerfile
├── Makefile
├── README.md
├── s2i
│   └── bin
│       ├── assemble
│       ├── run
│       ├── save-artifacts
│       └── usage
└── test
    ├── run
    └── test-app
        └── index.html

构建builder image

$ cd ruby-centos7
$ make build

构建App image

s2i build语法：

s2i build <source> <image> [<tag>] [flags]

# cd ruby-centos7
# s2i build test/test-app/ ruby-centos7 ruby-app

运行image

# docker run --rm -d -p 8080:8080 --name ruby-app ruby-app

增量构建与save-artifacts

Maven、Angular等项目编译时需要下载其他依赖，为提高编译速度，避免重复下载，S2I支持增量build。增量build依赖以前构建的image，image名字必须相同，且image中必须含有save-artifacts script。

增量build流程如下：

S2I从以前构建的app image创建一个新的docker container
S2I运行container中的save-artifacts将依赖打包为tar文件输出到stdout
S2I下载source与artifacts一起打包作为新image的输入
S2I启动新container、运行assemble ...

artifacts默认解压到/tmp/artifacts目录中。

启用增量构建

使用s2i命令时增加参数--incremental=true：

# s2i build test/test-app/ ruby-centos7 ruby-app --incremental=true

在OpenShift Web Console中修改BuildConfig，增加incremental: true，然后点击Start Build。

strategy:
  type: "Source"
  sourceStrategy:
    from:
      kind: "ImageStreamTag"
      name: "incremental-image:latest" 
    incremental: true

部署Spring Boot App

Builder Image

Dockerfile

# heroes-api-centos7
FROM centos:latest

RUN yum -y update && yum clean all

# Set the labels that are used for OpenShift to describe the builder image.
LABEL maintainer="Sun Jingchuan <jason@163.com>" \
      io.k8s.description="Heroes API" \
      io.k8s.display-name="Heroes API" \
      io.openshift.expose-services="8080:http" \
      io.openshift.tags="spring-boot,heroes-api" \
      # this label tells s2i where to find its mandatory scripts(run, assemble, save-artifacts)
      # io.openshift.s2i.scripts-url="image:///usr/libexec/s2i" \
      io.openshift.s2i.scripts-url="image:///tmp/scripts" \
      io.openshift.s2i.destination="/tmp"

ENV JAVA_HOME=/usr/lib/jdk1.8.0_202 \
    MAVEN_HOME=/usr/lib/apache-maven-3.6.0 \
    APP_ROOT=/opt/heroes
ENV PATH=${JAVA_HOME}/bin:${MAVEN_HOME}/bin:${APP_ROOT}/bin:${PATH} HOME=${APP_ROOT}

# Include jdk and maven in lib
COPY lib /usr/lib
COPY bin ${APP_ROOT}/bin
# Copy the S2I scripts to /usr/libexec/s2i
# COPY .s2i/bin /usr/libexec/s2i

RUN chmod -R u+x ${APP_ROOT}/bin && \
    chgrp -R 0 ${APP_ROOT} && \
    chmod -R g=u ${APP_ROOT} /etc/passwd

USER 10001
WORKDIR ${APP_ROOT}

ENTRYPOINT [ "uid_entrypoint" ]

EXPOSE 8080

# Inform the user how to run this image.
# CMD ["/usr/libexec/s2i/usage"]

说明：

lib目录中存放了JDK和Maven
使用源码中的S2I Scripts，builder image中不提供，但需定义io.openshift.s2i.scripts-url
默认，为了安全，OpenShift中只能使用User ID，不能使用用户名，不能使用docker默认用户root，因此要授予目录适当权限。
源码bin目录中存放了用户创建脚本uid_entrypoint，内容如下：

#!/bin/bash

if ! whoami &> /dev/null; then
  if [ -w /etc/passwd ]; then
    echo "${USER_NAME:-default}:x:$(id -u):0:${USER_NAME:-default} user:${HOME}:/sbin/nologin" >> /etc/passwd
  fi
fi

exec "$@"

编译builder image并上传到Registry

# docker build -t heroes-api-centos7:v1.0.0 .
# docker tag heroes-api-centos7:v1.0.0 registry.itrunner.org/heroes-api-centos7:v1.0.0
# docker push registry.itrunner.org/heroes-api-centos7:v1.0.0

import image

$ oc import-image heroes-api-centos7:v1.0.0 -n heroes --confirm --insecure --from='registry.itrunner.org/heroes-api-centos7:v1.0.0'

导入命令中指定了参数-n heroes，image会导入heroes项目中。成功导入后可在项目的Builds -> Images中查看image，但在Service Catalog中仍不可见，需要修改Image Stream定义，增加annotations：

apiVersion: image.openshift.io/v1
kind: ImageStream
metadata:
  annotations:
    openshift.io/image.dockerRepositoryCheck: '2019-03-27T08:40:27Z'
  creationTimestamp: '2019-03-27T08:39:50Z'
  generation: 1
  name: heroes-api-centos7
  namespace: heroes
  resourceVersion: '3337267'
  selfLink: /apis/image.openshift.io/v1/namespaces/heroes/imagestreams/heroes-api-centos7
  uid: e280929e-506b-11e9-a2ec-0288bf58ecc2
spec:
  lookupPolicy:
    local: false
  tags:
    - annotations:
        description: build heroes-api on CentOS 7
        iconClass: icon-spring
        openshift.io/display-name: Heroes API
        openshift.io/provider-display-name: itrunner
        sampleRepo: 'https://github.com/sunjc/heroes-api.git'
        supports: itrunner
        tags: 'builder,java'
        version: '1.0.0'
      from:
        kind: DockerImage
        name: 'registry.itrunner.org/heroes-api-centos7:v1.0.0'
...

为了在Service Catalog中显示，tags中必须含有“builder”。只有导入openshift项目的image才是全局可见的，否则仅在本项目Catalog可见。如未显示请刷新页面。

S2I Scripts

assemble

#!/bin/bash -e

# restore build artifacts
if [ -d /tmp/artifacts/.m2 ]; then
    echo "restore build artifacts"
    mv /tmp/artifacts/.m2 $HOME/.
fi

# move the application source
mv /tmp/src $HOME/src

# build the application artifacts
pushd $HOME/src
mvn clean package -Pdev -Dmaven.test.skip=true
popd

# move the artifacts
mv $HOME/src/target/heroes-api-1.0.0.jar $HOME/
rm -rf $HOME/src

#!/bin/bash

java -jar $HOME/heroes-api-1.0.0.jar

save-artifacts

#!/bin/bash

# Besides the tar command, all other output to standard out must be surpressed.  Otherwise, the tar stream will be corrupted.
pushd ${HOME} >/dev/null
if [ -d .m2 ]; then
    # all .m2 contents to tar stream
    tar cf - .m2
fi
popd >/dev/null

usage

#!/bin/bash

# inform the user how to use the image
cat <<EOF
This is a S2I sample builder image, to use it, install
https://github.com/openshift/source-to-image
EOF

部署

$ oc new-app heroes/heroes-api-centos7:v1.0.0~https://github.com/sunjc/heroes-api.git --name=heroes-api
$ oc expose service heroes-api --name heroes-api --hostname heroes.apps.itrunner.org --path /api -n heroes
或
$ oc create route edge heroes-api --service heroes-api --hostname heroes.apps.itrunner.org --path /api --insecure-policy Redirect -n heroes

说明：

如果使用私有git仓库，new-app需要增加参数--source-secret=yoursecret。Secret可在Resources -> Secrets中配置。
route的path配置为/api，不是根目录，这样可以防止外部用户访问api-docs、swagger-ui
为了内部用户可以访问api-docs、swagger-ui，给service增加externalIPs参数，指定一个或多个OpenShift Node地址，如下：

spec:
  clusterIP: 172.30.80.170
  externalIPs:
    - 10.188.12.116
  ports:
    - name: 8080-tcp
      port: 8080
      protocol: TCP
      targetPort: 8080

部署Angular App

Builder Image

Dockerfile

# heroes-web-centos7
FROM centos/httpd:latest

RUN yum -y update && \
    curl -sL https://rpm.nodesource.com/setup_10.x | bash - && yum -y install nodejs && \
    yum clean all && npm install -g @angular/cli@latest

# Set the labels that are used for OpenShift to describe the builder image.
LABEL maintainer="Sun Jingchuan <jason@163.com>" \
      io.k8s.description="Heroes Web" \
      io.k8s.display-name="Heroes Web" \
      io.openshift.expose-services="8080:http" \
      io.openshift.tags="angular,heroes-web" \
      # this label tells s2i where to find its mandatory scripts(run, assemble, save-artifacts)
      # io.openshift.s2i.scripts-url="image:///usr/libexec/s2i" \
      io.openshift.s2i.scripts-url="image:///tmp/scripts" \
      io.openshift.s2i.destination="/tmp"

ENV APP_ROOT=/opt/heroes
ENV PATH=${APP_ROOT}/bin:${PATH} HOME=${APP_ROOT} HTTPD_MAIN_CONF_PATH=/etc/httpd/conf

COPY bin ${APP_ROOT}/bin
# Copy the S2I scripts to /usr/libexec/s2i
# COPY .s2i/bin /usr/libexec/s2i

RUN chmod -R u+x ${APP_ROOT}/bin && \
    chgrp -R 0 ${APP_ROOT} && \
    chmod -R g=u ${APP_ROOT} /etc/passwd /var/www/html /run/httpd && \
    chown -R root:root /run/httpd /etc/httpd && \
    sed -i -e "s/^User apache/User default/" ${HTTPD_MAIN_CONF_PATH}/httpd.conf && \
    sed -i -e "s/^Group apache/Group root/" ${HTTPD_MAIN_CONF_PATH}/httpd.conf && \
    sed -i -e "s/^Listen 80/Listen 8080/" ${HTTPD_MAIN_CONF_PATH}/httpd.conf && \
    sed -ri " s!^(\s*CustomLog)\s+\S+!\1 |/usr/bin/cat!g; s!^(\s*ErrorLog)\s+\S+!\1 |/usr/bin/cat!g;" ${HTTPD_MAIN_CONF_PATH}/httpd.conf

USER 10001
WORKDIR ${APP_ROOT}

ENTRYPOINT [ "uid_entrypoint" ]

EXPOSE 8080

# Inform the user how to run this image.
# CMD ["/usr/libexec/s2i/usage"]

说明：

需要安装nodejs、angular/cli
httpd相关目录要授予适当权限，需要修改httpd用户、组、日志输出、监听端口

修改用户后，不能再使用80端口，否则会报错：permission denied: ah00072: make_sock: could not bind to address
编译builder image并上传到Registry

# docker build -t heroes-web-centos7:v1.0.0 .
# docker tag heroes-web-centos7:v1.0.0 registry.itrunner.org/heroes-web-centos7:v1.0.0
# docker push registry.itrunner.org/heroes-web-centos7:v1.0.0

import image

$ oc import-image heroes-web-centos7:v1.0.0 -n heroes --confirm --insecure --from='registry.itrunner.org/heroes-web-centos7:v1.0.0'

编辑ImageStream

$ oc edit is/heroes-web-centos7

增加如下annotations:

tags:
- annotations:
    description: build heroes-web on CentOS 7
    iconClass: icon-angularjs
    openshift.io/display-name: Heroes Web
    openshift.io/provider-display-name: itrunner
    sampleRepo: 'https://github.com/sunjc/heroes-web.git'
    supports: itrunner
    tags: builder,javascript
    version: 1.0.0
  from:
    kind: DockerImage
    name: 'registry.itrunner.org/heroes-web-centos7:v1.0.0'

S2I Scripts

assemble

#!/bin/bash -e

# move the application source
mv /tmp/src $HOME/src

# restore build artifacts
if [ "$(ls /tmp/artifacts/ 2>/dev/null)" ]; then
    mv /tmp/artifacts/* $HOME/src
fi

# build the application artifacts
pushd $HOME/src
npm install
ng build --prod --base-href=/heroes/

# Install the artifacts
mv dist /var/www/html/heroes
mv node_modules $HOME/node_modules
popd

rm -rf $HOME/src

#!/bin/bash

# run the application
exec httpd -D FOREGROUND $@

save-artifacts

#!/bin/bash

# Besides the tar command, all other output to standard out must be surpressed.  Otherwise, the tar stream will be corrupted.
pushd ${HOME} >/dev/null
if [ -d node_modules ]; then
    # all node_modules contents to tar stream
    tar cf - node_modules
fi
popd >/dev/null

usage

#!/bin/bash

# inform the user how to use the image
cat <<EOF
This is a S2I sample builder image, to use it, install
https://github.com/openshift/source-to-image
EOF

部署

$ oc new-app heroes/heroes-web-centos7:v1.0.0~https://github.com/sunjc/heroes-web.git --name=heroes-web
$ oc expose service heroes-web --name heroes-web --hostname heroes.apps.itrunner.org --path /heroes --port 8080-tcp -n heroes
或
$ oc create route edge heroes-web --service heroes-web --hostname heroes.apps.itrunner.org --path /heroes \
   --insecure-policy Redirect --port 8080-tcp -n heroes

Image管理

Internal Registry

前面的例子，我们使用了私有Docker Registry "registry.itrunner.org"，先将base image上传到私有Registry，然后再导入到OpenShift中。我们也可以直接使用OpenShift内部的Registry(安装在default项目中)，执行push、pull等操作。

为了访问Internal Registry必须先执行docker login登录Registry，需使用openshift用户名(或email)，使用有效的openshift token作为密码。

登录openshift获取token

$ oc login https://openshift.itrunner.org:8443 -u jason --certificate-authority=/path/to/cert.crt
$ oc whoami -t

登录Internal Registry

# docker login -u <user_name> -e <email_address> -p <token_value> <registry_server>:<port>

如在openshift节点内访问Internal Registry，registry_server可以使用service hostname "docker-registry.default.svc"，port为5000；如在外部访问，registry_server则需使用route hostname，比如"docker-registry-default.apps.itrunner.org" ，docker-registry route的TLS Termination需配置为Re-encrypt。

# docker login -u jason -p xxxxxxxxxxx docker-registry-default.apps.itrunner.org

Push Image

# docker tag heroes-web-centos7:v1.0.0 docker-registry-default.apps.itrunner.org/heroes/heroes-web-centos7:latest
# docker push docker-registry-default.apps.itrunner.org/heroes/heroes-web-centos7:latest

注意：tag格式必须为project/name
push成功后，openshift将自动创建image stream。

Pull Image

# docker pull docker-registry-default.apps.itrunner.org/heroes/heroes-web-centos7:latest

查看Repository

$  curl -u jason:xxxxxxxxxx -kv https://docker-registry-default.apps.itrunner.org/v2/_catalog?n=100

用户必须有list权限：

$ oc adm policy add-cluster-role-to-user registry-viewer user

查看image stream

$ oc get is -n heroes
NAME             DOCKER REPO                                                       TAGS            UPDATED
heroes-api           docker-registry.default.svc:5000/heroes/heroes-api           latest          7 days ago
heroes-api-centos7   docker-registry.default.svc:5000/heroes/heroes-api-centos7   v1.0.0          7 days ago
heroes-web           docker-registry.default.svc:5000/heroes/heroes-web           latest          7 days ago
heroes-web-centos7   docker-registry.default.svc:5000/heroes/heroes-web-centos7   latest,v1.0.0   19 hours ago

Image Stream

Image Stream是OpenShift管理容器镜像的方式，其中定义了dockerImageReference，利用tag定义了同docker image各版本的映射关系。Image Stream本身不包含image data，image stream元数据与其他集群信息一起存储在etcd实例中。
image stream更新后可以触发build或deployment，如docker image已更新，但未更新image stream则不会触发build或deployment。
image可以源自OpenShift Internal Registry、外部Registry(比如registry.redhat.io、docker.io、私有Registry)、OpenShift集群中的image stream。

apiVersion: image.openshift.io/v1
kind: ImageStream
metadata:
  annotations:
    openshift.io/image.dockerRepositoryCheck: '2019-04-09T05:59:17Z'
  creationTimestamp: '2019-04-01T08:02:37Z'
  generation: 2
  name: heroes-web-centos7
  namespace: heroes
  resourceVersion: '2145022'
  selfLink: >-
    /apis/image.openshift.io/v1/namespaces/heroes/imagestreams/heroes-web-centos7
  uid: 83deac8c-5454-11e9-af2b-02f23e935364
spec:
  lookupPolicy:
    local: false
  tags:
    - annotations:
        description: >-
          build heroes-web on CentOS 7. WARNING: By selecting this tag, your
          application will automatically update to use the latest version.
        iconClass: icon-angularjs
        openshift.io/display-name: Heroes Web (Latest)
        openshift.io/provider-display-name: itrunner
        sampleRepo: 'https://github.com/sunjc/heroes-web.git'
        supports: itrunner
        tags: 'builder,javascript'
      from:
        kind: DockerImage
        name: 'docker-registry.default.svc:5000/heroes/heroes-web-centos7:latest'
      generation: 2
      importPolicy: {}
      name: latest
      referencePolicy:
        type: Source
    - annotations:
        description: build heroes-web on CentOS 7
        iconClass: icon-angularjs
        openshift.io/display-name: Heroes Web 1.0.0
        openshift.io/provider-display-name: itrunner
        sampleRepo: 'https://github.com/sunjc/heroes-web.git'
        supports: itrunner
        tags: 'builder,javascript'
        version: 1.0.0
      from:
        kind: DockerImage
        name: 'registry.itrunner.org/heroes-web-centos7:v1.0.0'
      generation: 1
      importPolicy:
        insecure: true
      name: v1.0.0
      referencePolicy:
        type: Source
status:
  dockerImageRepository: 'docker-registry.default.svc:5000/heroes/heroes-web-centos7'
  tags:
    - items:
        - created: '2019-04-08T07:37:11Z'
          dockerImageReference: >-
            docker-registry.default.svc:5000/heroes/heroes-web-centos7@sha256:7e4126ec9ec0d4158d962936a38f255806731d33d6fe03b29d95d82759823fcd
          generation: 2
          image: >-
            sha256:7e4126ec9ec0d4158d962936a38f255806731d33d6fe03b29d95d82759823fcd
      tag: latest
    - items:
        - created: '2019-04-01T08:02:37Z'
          dockerImageReference: >-
            registry.itrunner.org/heroes-web-centos7@sha256:7e4126ec9ec0d4158d962936a38f255806731d33d6fe03b29d95d82759823fcd
          generation: 1
          image: >-
            sha256:7e4126ec9ec0d4158d962936a38f255806731d33d6fe03b29d95d82759823fcd
      tag: v1.0.0

Reference Policy
当使用从外部Registry导入的image时，"引用策略" 允许指定从何处提取image。有两个选项：Local和Source：

Local 从OpenShift Internal Registry提取image
Source 直接从外部Registry提取image

查询Image Stream信息

$ oc describe is/heroes-web-centos7
$ oc describe istag/heroes-web-centos7:latest

为外部Image添加tag

$ oc tag docker.io/openshift/base-centos7:latest base-centos7:latest

为Image Stream附加tag
给现有tag附加一个tag：

$ oc tag heroes-api-centos7:v1.0.0 heroes-api-centos7:latest

更新tag

$ oc tag heroes-api-centos7:v1.0.1 heroes-api-centos7:latest

说明：与附加tag不同，两个tag都应存在
删除tag

$ oc tag -d heroes-api-centos7:v1.0.0

或

$ oc delete istag/heroes-api-centos7:v1.0.0

删除Image Stream

$ oc delete is base-centos7

定期更新tag
可以使用--scheduled：

$ oc tag docker.io/python:3.6.0 python:3.6 --scheduled

也可以在tag定义中设置importPolicy.scheduled为true：

apiVersion: v1
kind: ImageStream
metadata:
  name: python
spec:
  tags:
  - from:
      kind: DockerImage
      name: docker.io/python:3.6.0
    name: latest
    importPolicy:
      scheduled: true

周期默认为15分钟。

Binary Build

Binary Build主要应用于测试和Jenkins pipeline场景。开发人员提交源码前如果想先测试一下，可使用本地源码来构建App Image。Binary Build不能自动触发，只能手动执行。
Binary Build使用oc start-build命令，需要提供BuildConfig或存在的build，支持从几种以下source来构建App Image：

--from-file 比如Dockerfile
--from-dir 本地目录，start-build打包此目录并上传到openshift
--from-archive tar、tar.gz、zip等
--from-repo 本地源码目录，start-build打包最近commit的代码并上传到openshift

From a directory

$ oc start-build heroes-web --from-dir="." --follow
或
$ oc start-build --from-build=heroes-web-1 --from-dir="." --follow

From a Git repository

$ git commit -m "My changes"
$ oc start-build heroes-web --from-repo="." --follow

Jenkins Pipeline

...
stage("Build Image") {
  steps {
    dir('heroes-web/dist') {
      sh 'oc start-build heroes-web --from-dir . --follow'
    }
  }
}
...

Pipeline部署

Jenkins是广泛应用的CI工具，大多数工程师都有使用经验，更习惯使用Jenkins部署应用。使用Jenkins Pipeline部署OpenShift应用，不影响原有流程，可以正常执行测试、代码质量检查、编译打包等操作，只需在部署时调用oc start-build。

安装Jenkins

OpenShift提供了两个Jenkins Template：jenkins-ephemeral、jenkins-persistent，一种使用瞬时存储，一种使用持久存储，两者均使用jenkins image stream(docker.io/openshift/jenkins-2-centos7:v3.11)。jenkins image安装了OpenShift Client、OpenShift Login、OpenShift Sync、Kubernetes、Kubernetes Credentialst等插件。安装后既可以在OpenShift中运行Job，也可以在Jenkins中运行Job。

jenkins-persistent采用动态存储配置，PVC默认名称为jenkins(JENKINS_SERVICE_NAME)：

- apiVersion: v1
  kind: PersistentVolumeClaim
  metadata:
    name: '${JENKINS_SERVICE_NAME}'
  spec:
    accessModes:
      - ReadWriteOnce
    resources:
      requests:
        storage: '${VOLUME_CAPACITY}'
    storageClassName: glusterfs-storage-block
- apiVersion: v1
  kind: DeploymentConfig
  metadata:
    annotations:
      template.alpha.openshift.io/wait-for-ready: 'true'
    name: '${JENKINS_SERVICE_NAME}'
  spec:
    ...
    template:
      metadata:
        labels:
          name: '${JENKINS_SERVICE_NAME}'
      spec:
        containers:
          - capabilities: {}
            ...
            volumeMounts:
              - mountPath: /var/lib/jenkins
                name: '${JENKINS_SERVICE_NAME}-data'
        ...
        volumes:
          - name: '${JENKINS_SERVICE_NAME}-data'
            persistentVolumeClaim:
              claimName: '${JENKINS_SERVICE_NAME}'

如使用GlusterFS需要先修改Template，指定storageClassName：

- apiVersion: v1
  kind: PersistentVolumeClaim
  metadata:
    name: '${JENKINS_SERVICE_NAME}'
  spec:
    accessModes:
      - ReadWriteOnce
    resources:
      requests:
        storage: '${VOLUME_CAPACITY}'
    storageClassName: glusterfs-storage-block

下面使用jenkins-persistent来安装，命令如下：

$ oc project heroes
$ oc new-app jenkins-persistent -p VOLUME_CAPACITY=2Gi  -p MEMORY_LIMIT=2Gi
--> Deploying template "openshift/jenkins-persistent" to project heroes

     Jenkins
     ---------
     Jenkins service, with persistent storage.

     NOTE: You must have persistent volumes available in your cluster to use this template.

     A Jenkins service has been created in your project.  Log into Jenkins with your OpenShift account.  The tutorial at https://github.com/openshift/origin/blob/master/examples/jenkins/README.md contains more information about using this template.

     * With parameters:
        * Jenkins Service Name=jenkins
        * Jenkins JNLP Service Name=jenkins-jnlp
        * Enable OAuth in Jenkins=true
        * Memory Limit=2Gi
        * Volume Capacity=2Gi
        * Jenkins ImageStream Namespace=openshift
        * Disable memory intensive administrative monitors=false
        * Jenkins ImageStreamTag=jenkins:2
        * Fatal Error Log File=false

--> Creating resources ...
    route.route.openshift.io "jenkins" created
    persistentvolumeclaim "jenkins" created
    deploymentconfig.apps.openshift.io "jenkins" created
    serviceaccount "jenkins" created
    rolebinding.authorization.openshift.io "jenkins_edit" created
    service "jenkins-jnlp" created
    service "jenkins" created
--> Success
    Access your application via route 'jenkins-heroes.apps.itrunner.org'
    Run 'oc status' to view your app.

注意：如MEMORY_LIMIT配置低，则Jenkins Master节点架构为Linux (i386)。
查看存储
安装成功后，可以从安装Jenkins的node查看存储：

$  oc get pods -o wide
NAME              READY     STATUS      RESTARTS   AGE       IP            NODE                              NOMINATED NODE
jenkins-1-hw5q5   1/1       Running     0          5m        10.131.0.26   app2.itrunner.org   <none>
$ oc get pvc
NAME          STATUS        VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS              AGE
jenkins       Bound         pvc-bf3ff63d-6f1c-11e9-9dd9-02ef509f23d0   2Gi        RWO            glusterfs-storage   5m

# mount | grep pvc-bf3ff63d-6f1c-11e9-9dd9-02ef509f23d0
10.188.12.116:vol_0e157791c95b65a94011aed789d2037b on /var/lib/origin/openshift.local.volumes/pods/c12d7625-6f1c-11e9-ad9d-02499a450338/volumes/kubernetes.io~glusterfs/63d-6f1c-11e9-9dd9-02ef509f23d0 type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)

# cd /var/lib/origin/openshift.local.volumes/pods/c12d7625-6f1c-11e9-ad9d-02499a450338/volumes/kubernetes.io~glusterfs/63d-6f1c-11e9-9dd9-02ef509f23d0

扩展存储
当原配置的存储容量不满足需求时，可以扩展存储。

StorageClass的属性allowVolumeExpansion必须设为true

$ oc edit sc/glusterfs-storage

# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  creationTimestamp: 2019-04-28T02:58:06Z
  name: glusterfs-storage
  resourceVersion: "1903911"
  selfLink: /apis/storage.k8s.io/v1/storageclasses/glusterfs-storage
  uid: 723320e6-6961-11e9-b13d-02947d98b66e
parameters:
  resturl: http://heketi-storage.app-storage.svc:8080
  restuser: admin
  secretName: heketi-storage-admin-secret
  secretNamespace: app-storage
provisioner: kubernetes.io/glusterfs
reclaimPolicy: Delete
volumeBindingMode: Immediate

查看glusterfs-storage：

$ oc describe sc glusterfs-storage
Name:                  glusterfs-storage
IsDefaultClass:        No
Annotations:           <none>
Provisioner:           kubernetes.io/glusterfs
Parameters:            resturl=http://heketi-storage.app-storage.svc:8080,restuser=admin,secretName=heketi-storage-admin-secret,secretNamespace=app-storage
AllowVolumeExpansion:  True
MountOptions:          <none>
ReclaimPolicy:         Delete
VolumeBindingMode:     Immediate
Events:                <none>

更新PVC spec.resources.requests.storage

$ oc edit pvc/jenkins

# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    openshift.io/generated-by: OpenShiftNewApp
    pv.kubernetes.io/bind-completed: "yes"
    pv.kubernetes.io/bound-by-controller: "yes"
    volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/glusterfs
  creationTimestamp: 2019-05-05T10:01:26Z
  finalizers:
  - kubernetes.io/pvc-protection
  labels:
    app: jenkins-persistent
    template: jenkins-persistent-template
  name: jenkins
  namespace: heroes
  resourceVersion: "1904277"
  selfLink: /api/v1/namespaces/heroes/persistentvolumeclaims/jenkins
  uid: bf3ff63d-6f1c-11e9-9dd9-02ef509f23d0
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 3Gi
  storageClassName: glusterfs-storage
  volumeName: pvc-bf3ff63d-6f1c-11e9-9dd9-02ef509f23d0
...

安装/更新插件
使用openshift用户登录jenkins。
快速部署OpenShift应用
进入系统管理 -> 插件管理，安装/更新插件。为了编译Angular需要安装NodeJS插件（另一种选择，可以使用node agent）。
配置全局工具

快速部署OpenShift应用

Maven

快速部署OpenShift应用

NodeJS

快速部署OpenShift应用

首次调用时，Jenkins会自动安装全局工具，pipeline配置如下：

tools {
  jdk 'jdk8'
  maven 'maven-3.6'
}

tools {
  nodejs 'nodejs-10.15'
}

创建配置文件
快速部署OpenShift应用

下面仍以部署Spring Boot和Angular工程为例介绍Pipeline的使用，开始之前请重建heroes project。

部署Spring Boot App

构建Builder Image

Jenkins负责编译、打包，因此Builder Image不再需要Maven，修改如下：
Dockerfile

# jdk8-centos7
FROM centos:latest

RUN yum -y update && yum clean all

# Set the labels that are used for OpenShift to describe the builder image.
LABEL maintainer="Sun Jingchuan <jason@163.com>" \
      io.k8s.description="Oracle JDK 1.8.0_202 based on CentOS 7" \
      io.k8s.display-name="Oracle JDK 1.8.0_202" \
      io.openshift.expose-services="8080:http" \
      io.openshift.tags="jdk8"

ENV JAVA_HOME=/usr/lib/jdk1.8.0_202 \
    APP_ROOT=/opt/app-root
ENV PATH=${JAVA_HOME}/bin:${APP_ROOT}/bin:${PATH} HOME=${APP_ROOT}

COPY lib/jdk1.8.0_202 ${JAVA_HOME}
COPY bin ${APP_ROOT}/bin

RUN chmod -R u+x ${APP_ROOT}/bin && \
    chgrp -R 0 ${APP_ROOT} && \
    chmod -R g=u ${APP_ROOT} /etc/passwd

USER 10001
WORKDIR ${APP_ROOT}

ENTRYPOINT [ "uid_entrypoint" ]

EXPOSE 8080

编译、上传builder image
这次我们上传到OpenShift Docker Registry，注意push前要先执行docker login。

# docker build -t jdk8-centos7 .
# docker tag jdk8-centos7:latest docker-registry-default.apps.itrunner.org/heroes/jdk8-centos7:latest
# docker push docker-registry-default.apps.itrunner.org/heroes/jdk8-centos7:latest

创建App Image Dockerfile

App Image以builder image为基础，仅需拷贝Jenkins编译好的jar，内容如下：
Dockerfile.app

# heroes-api
FROM heroes/jdk8-centos7:latest

COPY heroes-api-1.0.0.jar $HOME

CMD java -jar $HOME/heroes-api-1.0.0.jar

从Dockerfile创建BuildConfig

$ cat Dockerfile.app | oc new-build -D - --name heroes-api

执行后生成BuildConfig的部分内容如下：

spec:
  failedBuildsHistoryLimit: 5
  nodeSelector: null
  output:
    to:
      kind: ImageStreamTag
      name: 'heroes-api:latest'
  postCommit: {}
  resources: {}
  runPolicy: Serial
  source:
    dockerfile: "FROM heroes/jdk8-centos7:latest\r\n\r\nCOPY heroes-api-1.0.0.jar $HOME\r\n\r\nCMD java -jar $HOME/heroes-api-1.0.0.jar"
    type: Dockerfile
  strategy:
    dockerStrategy:
      from:
        kind: ImageStreamTag
        name: 'jdk8-centos7:latest'
    type: Docker

会自启动build，但此时没有jar导致build失败，pipeline调用时才会传入jar。

创建Pipeline BuildConfig

pipeline.yml

apiVersion: v1
kind: BuildConfig
metadata:
  name: heroes-api-pipeline
spec:
  strategy:
    jenkinsPipelineStrategy:
      jenkinsfile: |-
        pipeline {
          agent any
          tools {
            jdk 'jdk8'
            maven 'maven-3.6'
          }
          stages {
            stage("Clone Source") {
              steps {
                checkout([$class: 'GitSCM',
                  branches: [[name: '*/master']],
                  extensions: [
                    [$class: 'RelativeTargetDirectory', relativeTargetDir: 'heroes-api']
                  ],
                  userRemoteConfigs: [[url: 'https://github.com/sunjc/heroes-api.git']]
                ])
              }
            }
            stage("Build Backend") {
              steps {
                dir('heroes-api') {
                  sh 'mvn clean package -Pdev -Dmaven.test.skip=true'
                }
              }
            }
            stage("Build Image") {
              steps {
                dir('heroes-api/target') {
                  sh 'oc start-build heroes-api --from-dir . --follow'
                }
              }
            }
          }
        }
    type: JenkinsPipeline

Pipeline可以内嵌在BuildConfig内，也可以引用git中的Jenkinsfile（推荐）:

apiVersion: v1
kind: BuildConfig
metadata:
  name: heroes-api-pipeline
spec:
  source:
    git:
      uri: "https://github.com/sunjc/heroes-api"
  strategy:
    jenkinsPipelineStrategy:
      jenkinsfilePath: Jenkinsfile
    type: JenkinsPipeline

创建pipeline：

$ oc create -f ./pipeline.yml

如果在项目中没有安装Jenkins，创建pipeline后将自动部署jenkins-ephemeral。

启动pipeline

$ oc start-build heroes-api-pipeline

也可以在jenkins或Application Console -> Builds -> Pipelines中启动。

部署App

$ oc new-app heroes-api
$ oc create route edge heroes-api --service heroes-api --hostname heroes.apps.itrunner.org --path /api --insecure-policy Redirect

部署Angular App

构建Builder Image

Dockerfile

# httpd-centos7
FROM centos/httpd:latest

RUN yum -y update && yum clean all

LABEL maintainer="Sun Jingchuan <jason@163.com>" \
      io.k8s.description="Apache Httpd 2.4" \
      io.k8s.display-name="Apache Httpd 2.4" \
      io.openshift.expose-services="8080:http" \
      io.openshift.tags="httpd"

ENV APP_ROOT=/opt/app-root
ENV PATH=${APP_ROOT}/bin:${PATH} HOME=${APP_ROOT} HTTPD_MAIN_CONF_PATH=/etc/httpd/conf

COPY bin ${APP_ROOT}/bin
COPY .s2i/bin/run ${APP_ROOT}/bin/run

RUN chmod -R u+x ${APP_ROOT}/bin && \
    chgrp -R 0 ${APP_ROOT} && \
    chmod -R g=u ${APP_ROOT} /etc/passwd /var/www/html /run/httpd && \
    chown -R root:root /run/httpd /etc/httpd && \
    sed -i -e "s/^User apache/User default/" ${HTTPD_MAIN_CONF_PATH}/httpd.conf && \
    sed -i -e "s/^Group apache/Group root/" ${HTTPD_MAIN_CONF_PATH}/httpd.conf && \
    sed -i -e "s/^Listen 80/Listen 8080/" ${HTTPD_MAIN_CONF_PATH}/httpd.conf && \
    sed -ri " s!^(\s*CustomLog)\s+\S+!\1 |/usr/bin/cat!g; s!^(\s*ErrorLog)\s+\S+!\1 |/usr/bin/cat!g;" ${HTTPD_MAIN_CONF_PATH}/httpd.conf

USER 10001
WORKDIR ${APP_ROOT}

ENTRYPOINT [ "uid_entrypoint" ]

EXPOSE 8080

编译、上传Builder Image

# docker build -t httpd-centos7 .
# docker tag httpd-centos7:latest docker-registry-default.apps.itrunner.org/heroes/httpd-centos7:latest
# docker push docker-registry-default.apps.itrunner.org/heroes/httpd-centos7:latest

创建App Image Dockerfile

Dockerfile.app

# heroes-web
FROM heroes/httpd-centos7:latest

COPY heroes /var/www/html/heroes

CMD $HOME/bin/run

从Dockerfile创建BuildConfig

$ cat Dockerfile.app | oc new-build -D - --name heroes-web

执行后将启动build，此时没有传入内容导致build失败。

创建Pipeline BuildConfig

pipeline.yml

apiVersion: v1
kind: BuildConfig
metadata:
  name: heroes-web-pipeline
spec:
  source:
    git:
      uri: "https://github.com/sunjc/heroes-web"
  strategy:
    jenkinsPipelineStrategy:
      jenkinsfilePath: Jenkinsfile
    type: JenkinsPipeline

Jenkinsfile

pipeline {
  agent any
  tools {
    nodejs 'nodejs-10.15'
  }
  stages {
    stage("Clone Source") {
      steps {
        checkout([$class: 'GitSCM',
          branches: [[name: '*/master']],
          extensions: [
            [$class: 'RelativeTargetDirectory', relativeTargetDir: 'heroes-web']
          ],
          userRemoteConfigs: [[url: 'https://github.com/sunjc/heroes-web']]
        ])
      }
    }
    stage("Build Angular") {
      steps {
        dir('heroes-web') {
          sh 'npm install'
          sh 'ng config -g cli.warnings.versionMismatch false'
          sh 'ng build --prod --base-href=/heroes/'
        }
      }
    }
    stage("Build Image") {
      steps {
        dir('heroes-web/dist') {
          sh 'oc start-build heroes-web --from-dir . --follow'
        }
      }
    }
  }
}

创建pipeline：

$ oc create -f ./pipeline.yml

启动pipeline

$ oc start-build heroes-web-pipeline

部署App

$ oc new-app heroes-web
$  oc create route edge heroes-web --service heroes-web --hostname heroes.apps.itrunner.org --path /heroes \
   --insecure-policy Redirect --port 8080-tcp -n heroes

OpenShift DSL

OpenShift Jenkins image安装了OpenShift Jenkins Client插件，支持使用OpenShift Domain Specific Language(DSL)定义pipeline。

def templatePath = 'https://raw.githubusercontent.com/openshift/nodejs-ex/master/openshift/templates/nodejs-mongodb.json' 
def templateName = 'nodejs-mongodb-example' 
pipeline {
  agent {
    node {
      label 'nodejs' 
    }
  }
  options {
    timeout(time: 20, unit: 'MINUTES') 
  }
  stages {
    stage('preamble') {
        steps {
            script {
                openshift.withCluster() {
                    openshift.withProject() {
                        echo "Using project: ${openshift.project()}"
                    }
                }
            }
        }
    }
    stage('cleanup') {
      steps {
        script {
            openshift.withCluster() {
                openshift.withProject() {
                  openshift.selector("all", [ template : templateName ]).delete() 
                  if (openshift.selector("secrets", templateName).exists()) { 
                    openshift.selector("secrets", templateName).delete()
                  }
                }
            }
        }
      }
    }
    stage('create') {
      steps {
        script {
            openshift.withCluster() {
                openshift.withProject() {
                  openshift.newApp(templatePath) 
                }
            }
        }
      }
    }
    stage('build') {
      steps {
        script {
            openshift.withCluster() {
                openshift.withProject() {
                  def builds = openshift.selector("bc", templateName).related('builds')
                  timeout(5) { 
                    builds.untilEach(1) {
                      return (it.object().status.phase == "Complete")
                    }
                  }
                }
            }
        }
      }
    }
    stage('deploy') {
      steps {
        script {
            openshift.withCluster() {
                openshift.withProject() {
                  def rm = openshift.selector("dc", templateName).rollout().latest()
                  timeout(5) { 
                    openshift.selector("dc", templateName).related('pods').untilEach(1) {
                      return (it.object().status.phase == "Running")
                    }
                  }
                }
            }
        }
      }
    }
    stage('tag') {
      steps {
        script {
            openshift.withCluster() {
                openshift.withProject() {
                  openshift.tag("${templateName}:latest", "${templateName}-staging:latest") 
                }
            }
        }
      }
    }
  }
}

具体请参考OpenShift Pipeline Builds

清理对象

随着不断的构建、部署应用，build、deployment、image等对象会逐渐增多，管理员应定期清理不再需要的对象。
OpenShift提供了oc adm prune命令来清理对象，支持auth、groups、builds、deployments、images等类型。

$ oc adm prune <object_type> <options>

Pruning builds

$ oc adm prune builds [<options>]

Option	Description
--confirm	Indicate that pruning should occur, instead of performing a dry-run
--orphans	Prune all builds whose build config no longer exists, status is complete, failed, error, or canceled
--keep-complete=N	Per build config, keep the last N builds whose status is complete. (default 5)
--keep-failed=N	Per build config, keep the last N builds whose status is failed, error, or canceled (default 1)
--keep-younger-than=duration	Do not prune any object that is younger than duration relative to the current time. (default 60m)

$ oc adm prune builds --orphans --keep-complete=5 --keep-failed=1 --keep-younger-than=60m --confirm

Pruning deployments

$ oc adm prune deployments [<options>]

Option	Description
--confirm	Indicate that pruning should occur, instead of performing a dry-run
--orphans	Prune all deployments whose deployment config no longer exists, status is complete or failed, and replica count is zero
--keep-complete=N	Per deployment config, keep the last N deployments whose status is complete and replica count is zero. (default 5)
--keep-failed=N	Per deployment config, keep the last N deployments whose status is failed and replica count is zero. (default 1)
--keep-younger-than=duration	Do not prune any object that is younger than duration relative to the current time. (default 60m) Valid units of measurement include nanoseconds (ns), microseconds (us), milliseconds (ms), seconds (s), minutes (m), and hours (h)

$ oc adm prune deployments --orphans --keep-complete=5 --keep-failed=1 --keep-younger-than=60m --confirm

Pruning images

$ oc adm prune images [<options>]

system:admin用户不能执行清理image的操作，必须使用普通用户登录，且用户必须有system:image-pruner角色。

$ oc login https://openshift.itrunner.org:8443 --token=xxxx

Option	Description
--all	Include images that were not pushed to the registry, but have been mirrored by pullthrough. This is on by default. To limit the pruning to images that were pushed to the integrated registry, pass --all=false
--certificate-authority	The path to a certificate authority file to use when communicating with the OKD-managed registries. Defaults to the certificate authority data from the current user’s configuration file. If provided, secure connection will be initiated
--confirm	Indicate that pruning should occur, instead of performing a dry-run. This requires a valid route to the integrated container image registry. If this command is run outside of the cluster network, the route needs to be provided using --registry-url
--force-insecure	Use caution with this option. Allow an insecure connection to the Docker registry that is hosted via HTTP or has an invalid HTTPS certificate
--keep-tag-revisions=N	For each image stream, keep up to at most N image revisions per tag. (default 3)
--keep-younger-than=duration	Do not prune any image that is younger than duration relative to the current time. Do not prune any image that is referenced by any other object that is younger than duration relative to the current time. (default 60m)
--prune-over-size-limit	Prune each image that exceeds the smallest limit defined in the same project. This flag cannot be combined with --keep-tag-revisions nor --keep-younger-than
--registry-url	The address to use when contacting the registry. The command will attempt to use a cluster-internal URL determined from managed images and image streams. In case it fails (the registry cannot be resolved or reached), an alternative route that works needs to be provided using this flag. The registry host name may be prefixed by https:// or http:// which will enforce particular connection protocol
--prune-registry	In conjunction with the conditions stipulated by the other options, this option controls whether the data in the registry corresponding to the OKD Image API Objects is pruned. By default, image pruning processes both the Image API Objects and corresponding data in the registry. This options is useful when you are only concerned with removing etcd content, possibly to reduce the number of image objects, but are not concerned with cleaning up registry storage; or intend to do that separately by Hard Pruning the Registry, possibly during an appropriate maintenance window for the registry

使用--keep-younger-than清理image，不会清理以下情况的image：

创建时间在--keep-younger-than内的所有Pod
创建时间在--keep-younger-than内的所有ImageStream
正在运行的Pod
状态为pending的Pod
所有replication controllers
所有deployment configurations
所有build configurations
所有builds

使用--prune-over-size-limit清理超过指定Limit的image，不会清理以下情况的image：

正在运行的Pod
状态为pending的Pod
所有replication controllers
所有deployment configurations
所有build configurations
所有builds

示例：

$ oc adm prune images --keep-tag-revisions=3 --keep-younger-than=60m --confirm

$ oc adm prune images --prune-over-size-limit --confirm

清理image后不会更新registry cache，为了清理cache可以重新部署registry：

$ oc rollout latest dc/docker-registry -n default

docker prune

docker相关prune命令：

# docker container prune
# docker image prune
# docker volume prune

慎用，会删除openshift/origin-docker-builder、openshift/origin-deployer。
快速部署OpenShift应用

参考文档

OKD Latest Documentation
source-to-image

向AI问一下细节

快速部署OpenShift应用

初识OpenShift部署

Service Catalog

部署Apache HTTP Server

基本概念

OC Tool

再谈Route

S2I

S2I Scripts

S2I Tool

增量构建与save-artifacts

部署Spring Boot App

Builder Image

S2I Scripts

部署

部署Angular App

Builder Image

S2I Scripts

部署

Image管理

Internal Registry

Image Stream

Binary Build

Pipeline部署

安装Jenkins

部署Spring Boot App

部署Angular App

OpenShift DSL

清理对象

Pruning builds

Pruning deployments

Pruning images

docker prune

推荐文档

参考文档

猜你喜欢

快速部署OpenShift应用

初识OpenShift部署

Service Catalog

部署Apache HTTP Server

基本概念

OC Tool

再谈Route

S2I

S2I Scripts

S2I Tool

增量构建与save-artifacts

部署Spring Boot App

Builder Image

S2I Scripts

部署

部署Angular App

Builder Image

S2I Scripts

部署

Image管理

Internal Registry

Image Stream

Binary Build

Pipeline部署

安装Jenkins

部署Spring Boot App

部署Angular App

OpenShift DSL

清理对象

Pruning builds

Pruning deployments

Pruning images

docker prune

推荐文档

参考文档

猜你喜欢

最新资讯

相关推荐

相关标签