OpenShift 4.x Restricted Network (Disconnected) Install on Z

OpenShift 4.x Restricted Network (Disconnected) Install on Z

Alexandre de Oliveira Alexandre de Oliveira

Alexandre de Oliveira

Customer Engineer, Application Modernization, Google Cloud - Sales - LATAM

发布日期: 2021年2月25日
+ 关注

Overview

Prerequisites

Install a private image registry

Merge pull secrets

Mirroring the content

Modify install-config.yaml

Notes

References

Overview

Red Hat announced the availability of of OpenShift Container Platform on the IBM Z and IBM LinuxONE platform last year and since then we've noticed that most of our clients end up with disconnected/restricted network installed, given the nature of critical business running on the our platform.

The goal of this document is to go through the process for a disconnected aka restricted network install and assumed you have the pre-requisite infrastructure already in place (DNS, HAProxy etc) required for all installations and you know how to proceed with a regular install (generate manifests, artifacts etc) for s390x. [edTODO]

This task is composed of 3 steps :

  • create the private image registry for mirroring
  • merge pull secret
  • generate the install-config.yaml for disconnected install

Prerequisites

  • s390x or x86 RHEL 7/8* (lets call this "utility-vm" from now on)
  • podman
  • jq
  • oc client
  • utility-vm must have access to the Internet, directly or through a proxy
Note that with RHEL 7, you have options to use Docker or Podman but for RHEL 8, only Podman is available from the subscription repositories.

Install a private image registry

As the OCP nodes won't have access to the Internet, they need to pull the required images from some existing location that does. For this, we'll create an internal registry and mirror the images from the Red Hat registry. At the very least, utility-vm must have access to the Internet, directly or through a proxy. In the examples below,a proxy will be used.

  1. Export proxy variables which will be used by podman and later by oc commands:
$ export HTTP_PROXY=https://my-proxy.example.com:9090
$ export HTTPS_PROXY=https://my-proxy.example.com:9090
$ export NO_PROXY=.internal.example.com,.example.com
Note: this certainly will be different for you, tweak as per your needs
  1. Install the registry:
$ yum install docker-distribution
  1. Create the folders which will be used by the registry and self-signed certificate:
$ mkdir -p /etc/docker-distribution/certs
$ cd /etc/docker-distribution/certs
$ openssl req -newkey rsa:4096 -nodes -sha256 -keyout domain.key -x509 -days 365 -out domain.crt
  1. Create the folder which will be used by registry (where the images will reside):
$ mkdir /var/registry/docker-distribution
  1. Create a basic htpasswd authentication:
$ htpasswd -bBc /etc/docker-distribution/registry_passwd [user] [pass]
  1. Edit the config file /etc/docker-distribution/registry/config.yml :
version: 0.1
log:
fields:
    service: registry
    environment: development
storage:
    cache:
        layerinfo: inmemory
    filesystem:
        rootdirectory: /var/registry/docker-distribution
    delete:
        enabled: true
http:
    addr: :5000
    tls:
    certificate: /etc/docker-distribution/certs/domain.crt
    key: /etc/docker-distribution/certs/domain.key
    host: https://utility-vm.example.com:5000
    secret: testsecretrandom
    relativeurls: false
auth:
    htpasswd:
    realm: basic-realm
    path: /etc/docker-distribution/registry_passwd
*Note: replace the host and secret as per your need.
  1. Start the service:
$ systemctl start docker-distribution
  1. Validate that the registry service is listening and working:
$ netstat -tulpn |grep 5000
tcp        0      0 0.0.0.0:5000            0.0.0.0:*               LISTEN      55061/registry

$ curl -u openshift:redhat -k https://localhost:5000/v2/_catalog
{"repositories":[]}
Note: your registry is still empty, therefore no repositories yet.
  1. Trust the repositories:
$ cp /etc/docker-distribution/certs/domain.crt /etc/pki/ca-trust/source/anchors/
update-ca-trust
  1. Try to pull and push a sample image:
$ podman pull ubi7/ubi:7.7
Trying to pull registry.access.redhat.com/ubi7/ubi:7.7...
Getting image source signatures
Copying blob 1af260e02c36 done  
Copying blob eb3a55bf2d3b done  
Copying config f5c7f86b27 done  
Writing manifest to image destination
Storing signatures
f5c7f86b27eb77d40f1e30109e1d35444b6bba9aaacc8d88eaa15f7617ca4d05

$ podman images |grep ubi7
registry.access.redhat.com/ubi7/ubi                                7.7                          f5c7f86b27eb   9 months ago    221 MB

$ podman tag registry.access.redhat.com/ubi7/ubi:7.7 utility-vm.example.com:5000/ubi7:7.7

$ podman images |grep ubi7
utility-vm.example.com:5000/ubi7                                    7.7                          f5c7f86b27eb   9 months ago    221 MB
registry.access.redhat.com/ubi7/ubi                                7.7                          f5c7f86b27eb   9 months ago    221 MB

$ podman push utility-vm.example.com:5000/ubi7:7.7
Getting image source signatures
Copying blob b51142454f75 done  
Copying blob 9aaa11d10a00 done  
Copying config f5c7f86b27 done  
Writing manifest to image destination
Storing signatures

$ curl -u openshift:redhat -k https://utility-vm.example.com:5000/v2/_catalog
{"repositories":["ubi7"]}
  1. Open firewall ports
$ firewall-cmd --permanent --add-port=5000/tcp
success

$ firewall-cmd --list-ports
53/udp

$ firewall-cmd --reload
success

$ firewall-cmd --list-ports
53/udp 5000/tcp

Merge pull secrets

You can only have a single pull secret file for the install. A "pull secret" is a json file which contains the authentication keys that allows you to pull images from the RedHat Registry. As the mirroring process will pull images from RedHat Registry and your cluster will be pulling from your internal private registry, we need to merge both secrets in a single json file.

  1. Create a pull secret for your local registry and output it to local_pullsecret.json :
$ podman login -u openshift -p redhat --authfile ./local_pullsecret.json utility-vm.example.com:5000
  1. It will look like this:
{
    "auths": {
        "utility-vm.example.com:5000": {
            "auth": "c3BlNocaaWZ0OnJlZGhhdB=="
        }
    }
}
  1. Get the pull secret from OpenShift Cluster Manager. Save it to a file named ocp_pullsecret.json . It will look like this:
{"auths":{"cloud.openshift.com":{"auth":"c8BlbnNoaWZNoCaNoCaNoCatZGV2K2FsZXhvbGlfcmgxdThwbGY2cTB...==","email":"[email protected]"},"quay.io":{"auth":"c8BlbnNoaWZNoCaNoCaNoCatZGV2K2FsZXhvbGlfcmgxdThwbGY2cTB...==","email":"[email protected]"},"registry.connect.redhat.com":{"auth":"aTI1MT...ViXzVFbEwzclI4UHE1TTZYenpoWTBzSU45UlliejBNeW90WnBFVE1sdUZkTUFhOT...==","email":"[email protected]"},"registry.redhat.io":{"auth":"aTI1MT...ViXzVFbEwzclI4UHE1TTZYenpoWTBzSU...Ca==","email":"[your Open Shift Subscription Email]"}}}
Note: the real keys are much longer
  1. Merge pull secrets using jq :
$ jq -c --argjson var "$(jq .auths ./local_pullsecret.json)" '.auths += $var' ./ocp_pullsecret.json > merged_pullsecret.json
  1. Validate:
$ jq . merged_pullsecret.json
  1. It'll be similar to this, notice your local pull secret at the bottom:
{"auths":{"cloud.openshift.com":{"auth":"c8BlbnNoaWZNoCaNoCaNoCatZGV2K2FsZXhvbGlfcmgxdThwbGY2cTB...==","email":"[email protected]"},"quay.io":{"auth":"c8BlbnNoaWZNoCaNoCaNoCatZGV2K2FsZXhvbGlfcmgxdThwbGY2cTB...==","email":"[email protected]"},"registry.connect.redhat.com":{"auth":"aTI1MT...ViXzVFbEwzclI4UHE1TTZYenpoWTBzSU...==","email":"[email protected]"},"registry.redhat.io":{"auth":"aTI1MT...ViXzVFbEwzclI4UHE1TTZYenpoWTBzSU...==","email":"[email protected]"},"utility-vm.example.com:5000":{"auth":"c3BlNocaaWZ0OnJlZGhhdB=="}}}

Mirroring the content

Now that the pull-secret is ready, we're all set to mirror the images from the Red Hat repo.

  1. Setup environment variables:
$ export OCP_RELEASE=4.6.9
$ export LOCAL_REGISTRY=utility-vm.example.com:5000
$ export LOCAL_REPOSITORY=ocp4/openshift4
$ export PRODUCT_REPO=openshift-release-dev 
$ export LOCAL_SECRET_JSON=/var/registry/oc4.6/secrets/merged_pullsecret.json 
$ export RELEASE_NAME=ocp-release 
$ export ARCHITECTURE=s390x
Note 1: Ensure the release matches the exact output from the command openshift-install version you've downloaded.
Note 2: Set the merged pull secret path as per your environment
  1. Ensure you do a dry run before the real run.
$ oc adm release mirror -a ${LOCAL_SECRET_JSON}  \
    --from=quay.io/${PRODUCT_REPO}/${RELEASE_NAME}:${OCP_RELEASE}-${ARCHITECTURE} \
    --to=${LOCAL_REGISTRY}/${LOCAL_REPOSITORY} \
    --to-release-image=${LOCAL_REGISTRY}/${LOCAL_REPOSITORY}:${OCP_RELEASE}-${ARCHITECTURE} --dry-run
  1. Export GODEBUG=x509ignoreCN=0 to prevent oc from validating the certificate against the existence Subject Alternative Name (SAN):
$ export GODEBUG=x509ignoreCN=0
Note: if you follow the self-signed certificate instructions described here, you'll get the error below. To avoid it, you can create a certificate with SAN or use the above variable described above. In references, I'll share a link in how to create a self-signed certificate with SAN. [edTODO]
error: unable to connect to utility-vm.example.com:5000/ocp4/openshift4: Get "https://utility-vm.example.com:5000/v2/": x509: certificate relies on legacy Common Name field, use SANs or temporarily enable Common Name matching with GODEBUG=x509ignoreCN=0
  1. All set, go for the real action and mirror the images:
$ oc adm release mirror -a ${LOCAL_SECRET_JSON}  \
    --from=quay.io/${PRODUCT_REPO}/${RELEASE_NAME}:${OCP_RELEASE}-${ARCHITECTURE} \
    --to=${LOCAL_REGISTRY}/${LOCAL_REPOSITORY} \
    --to-release-image=${LOCAL_REGISTRY}/${LOCAL_REPOSITORY}:${OCP_RELEASE}-${ARCHITECTURE}
  1. You will see a lot of image names on the console the important portion you need to save is the bottom of it, similar to this:
info: Mirroring completed in 1m22.15s (61.22MB/s)

Success
Update image:  utility-vm.example.com:5000/ocp4/openshift4:4.6.9
Mirror prefix: utility-vm.example.com:5000/ocp4/openshift4

To use the new mirrored repository to install, add the following section to the install-config.yaml:

imageContentSources:
- mirrors:
  - utility-vm.example.com/ocp4/openshift4
  source: quay.io/openshift-release-dev/ocp-release
- mirrors:
  - utility-vm.example.com:5000/ocp4/openshift4
  source: quay.io/openshift-release-dev/ocp-v4.0-art-dev


To use the new mirrored repository for upgrades, use the following to create an ImageContentSourcePolicy:

apiVersion: operator.openshift.io/v1alpha1
kind: ImageContentSourcePolicy
metadata:
name: example
spec:
repositoryDigestMirrors:
- mirrors:
  - utility-vm.example.com:5000/ocp4/openshift4
  source: quay.io/openshift-release-dev/ocp-release
- mirrors:
  - utility-vm.example.com:5000/ocp4/openshift4
  source: quay.io/openshift-release-dev/ocp-v4.0-art-dev
Note: although you'll use only this portion of the output, I recommend saving the whole output, including the image names. In the case that things go wrong and you need to troubleshoot, having the entire list of images mirrored will be handy.
  1. Validate that the new repository was created and you can pull an image:
$ curl -u openshift:redhat -k https://utility-vm.example.com:5000/v2/_catalog
{"repositories":["ocp4/openshift4","ubi7"]}

$ podman pull --authfile /var/registry/oc4.6/secrets/local_pullsecret.json utility-vm.example.com:5000/ocp4/openshift4:4.6.9-operator-lifecycle-manager
Trying to pull utility-vm.example.com:5000/ocp4/openshift4:4.6.9-operator-lifecycle-manager...
Getting image source signatures
Copying blob 84b951709aed skipped: already exists  
Copying blob 69c1e0128b10 skipped: already exists  
Copying blob 11606feade9d skipped: already exists  
Copying blob 66f4d5600fff skipped: already exists  
Copying blob bb78f045807f done  
Copying config 4926ac11ab done  
Writing manifest to image destination
Storing signatures
4926ac11abea54d9acffb3f8862bdb22b2aac11570c82f426038f2729f240421
Note: replace the local_pullsecret.json path as per your environment.

Modify install-config.yaml

Now, you'll generate the artifacts as per regular instructions (linked at the end). The focus of this step will be on the modifications required on install-config.yaml:

  • add the ImageContentSourcePolicy pointing to your new private image registry
  • add the certificate bundle to authentication against your private image registry
  1. Extract the openshift-install from the registry:
$ oc adm release extract -a ${LOCAL_SECRET_JSON} --command=openshift-install "${LOCAL_REGISTRY}/${LOCAL_REPOSITORY}:${OCP_RELEASE}"
Note: as long as you have connectivity to the Internet, using proxy or directly, and all environments variable are still set, the above command should work. If it doesn't, download openshift-install from here and ensure you get the exact matching version!
  1. Even though you extracted the contents from the registry or downloaded from the Red Hat OpenShift Cluster Manager, make sure you have the exact version as OCP_RELEASE variable, otherwise, you'll encounter errors during the bootstrap process where the the temporary control-plane won't be able to find the correct tags:
$ ./openshift-install version
./openshift-install 4.6.9
built from commit a48ad4a15b42102d1747d2f5f3b635deffb950b5
release image utility-vm.example.com:5000/ocp46/openshift4@sha256:3d77e9b0fd14a5c4d50995bbb17494a02f27a69f2ffa9771b29d112fe084699f
  1. Your original install-config.yaml should be similar to this:
apiVersion: v1
baseDomain: example.com
compute:
- hyperthreading: Enabled   
architecture: s390x
name: worker
replicas: 0 
controlPlane:
hyperthreading: Enabled   
architecture: s390x
name: master 
replicas: 3 
metadata:
name: zcluster-dev 
networking:
clusterNetwork:
- cidr: 10.128.0.0/14 
    hostPrefix: 23 
networkType: OpenShiftSDN
serviceNetwork: 
- 172.30.0.0/16
platform:
none: {}
fips: false
pullSecret: '{"auths":{"cloud.openshift.com":{"auth":"c8BlbnNoaWZNoCaNoCaNoCatZGV2K2FsZXhvbGlfcmgxdThwbGY2cTBsc2t5aXJxZnBqYjBjZ25weWs6WldMQlExMzlNMDgzNoCaNoCaNoCaQ1hJNEpCWVhRWExOOEc0S0o3NEhFUFRNoCaNoCaNoCaWldBBQk1RSFBXxg==","email":"[email protected]"},"quay.io":{"auth":"c8BlbnNoaWZNoCaNoCaNoCatZGV2K2FsZXhvbGlfcmgxdThwbGY2cTBsc2t5aXJxZnBqYjBjZ25weWs6WldMQlExMzlNMDgzNoCaNoCaNoCaQ1hJNEpCWVhRWExOOEc0S0o3NEhFUFRNoCaNoCaNoCaWldBBQk1RSFBXxg==","email":"[email protected]"},"registry.connect.redhat.com":{"auth":"aTI1MT...ViXzVFbEwzclI4UHE1TTZYenpoWTBzSU45UlliejBNeW90WnBFVE1sdUZkTUFhOTBMWjJZbWNxQng5RnBOR3kwRmtVelVIaTJTZG5Kdm1ZTXk4cTdMd2xaQjJ2NnllejNoCa1WmdfdHI5dnpCclZQbXc2bTAzZHdEM3h1X0Q1SHlvNzNoCa==","email":"[email protected]"},"registry.redhat.io":{"auth":"aTI1MT...ViXzVFbEwzclI4UHE1TTZYenpoWTBzSU45UlliejBNeW90WnBFVE1sdUZkTUFhOTBMWjJZbWNxQng5RnBOR3kwRmtVelVIaTJTZG5Kdm1ZTXk4cTdMd2xaQjJ2NnllejNoCa1WmdfdHI5dnpCclZQbXc2bTAzZHdEM3h1X0Q1SHlvNzNoCa==","email":"[email protected]"},"utility-vm.example.com:5000":{"auth":"c3BlNocaaWZ0OnJlZGhhdB=="}}}'
sshKey: |
  ssh-rsa AAANacoNacoNaco....NacoNacoNaco== root@utility-vm
Note 1: For further details on install-config.yaml check here .
Note 2: I'm not covering details in how to generated ssh-key which you'll need to update in install-config.yaml, this article is focusing on the requirements for disconnected install pieces only. For more details on the whole install process, check the reference at the bottom.
  1. At the bottom, add the imageContentSources you got as an output after mirroring:
...
sshKey: |
  ssh-rsa AAANacoNacoNaco....NacoNacoNaco== root@utility-vm
imageContentSources:
- mirrors:
  - utility-vm.example.com/ocp4/openshift4
  source: quay.io/openshift-release-dev/ocp-release
- mirrors:
  - utility-vm.example.com:5000/ocp4/openshift4
  source: quay.io/openshift-release-dev/ocp-v4.0-art-dev
  1. Add to the bottom the certificate bundle you've created on Step 3 of Install a private image registry :
...
sshKey: |
  ssh-rsa AAANacoNacoNaco....NacoNacoNaco== root@utility-vm
imageContentSources:
- mirrors:
  - utility-vm.example.com/ocp4/openshift4
  source: quay.io/openshift-release-dev/ocp-release
- mirrors:
  - utility-vm.example.com:5000/ocp4/openshift4
  source: quay.io/openshift-release-dev/ocp-v4.0-art-dev
additionalTrustBundle: |
  -----BEGIN CERTIFICATE-----
  MIIF9zCCA9+gAwIBAgIJAM3thbw0gvDfMA0GCSqGSIb3DQEBCwUAMIGRMQswCQYD
  VQQGEwJVUzEQMA4GA1UECAwHQXJpem9uYTEQMA4GA1UEBwwHUGhvZW5peDENMAsG
  ...<cutting some pieces off>...
  GvkXaraLH64hQC3+HqkHMAbzEPI/AS/u3CD5RugcwqtK7f1p+5vsgvRHuzWxCJaw
  8fBSNlPP2qQS1popWFNSIic2JexHVuYCuGVH1MkTW1C5zXmoFbEKXhpu/Q==
  -----END CERTIFICATE-----
Note: very important detail, this is YAML so remember the two spaces tabulation. This is very truth also for the certificate bundle at the bottom, you must tabulate each line properly as the example above.
  1. That's it for disconnected, from this point you can follow the regular instructions. I'm referencing a few other articles where this is covered in details.

Notes

  1. Even the latest release of 4.7 on Feb, 24th, most of the instructions say the same. We will update the guide shortly.
  2. This was a step by step walkthrough of the process. Certain steps can be skipped/optimized based on your environment. There are also ways to script this such as this set of scripts created and maintained by our colleague Pat Fruth. We also have supported software like IBM Cloud Infrastructure Center that can automate this process as well as automate deployment of other z/VM and KVM based workloads on the IBM Z and LinuxONE platform.

References

Red Hat OpenShift Installation Process Experiences on IBM Z/LinuxONE

https://www.openshift.com/blog/red-hat-openshift-installation-process-experiences-on-ibm-z-linuxone

Installing a cluster on IBM Z and LinuxONE in a restricted network

https://docs.openshift.com/container-platform/4.6/installing/installing_ibm_z/installing-restricted-networks-ibm-z.html#installation-initializing-manual_installing-restricted-networks-ibm-z

Creating a mirror registry for installation in a restricted network

https://docs.openshift.com/container-platform/4.6/installing/install_config/installing-restricted-networks-preparations.html#installing-restricted-networks-preparations

How do I setup / install a Docker registry?

https://access.redhat.com/solutions/705273

Know about SAN Certificate and How to Create With OpenSSL

https://geekflare.com/san-ssl-certificate/

While this tutorial focused on deploying OpenShift itself, if you would like to play around on a no-cost OpenShift on IBM Z or IBM LinuxONE environment, please head over to our LinuxONE Community Cloud. !

To learn more about why our clients choose to run OpenShift on IBM Z and IBM LinuxONE, please checkout our whitepaper.

Authors

Alexandre de Oliveira is an IT Architect at IBM with hands-on experience on Kubernetes and OpenShift on Z.

Elton De Souza leads the on-premises Client Success mission with IBM's WW clients focusing on OpenShift, Cloud Paks and IBM Hyper Protect Services on IBM Z and IBM LinuxONE.

要查看或添加评论，请登录

社区洞察

其他会员也浏览了