cluster-api-provider-openstack v0.3.3 has been released on November 25th. This article explains how to deploy kubernetes cluster using cluster-api-provider-openstack with;
Deploy kubernetes cluster
ubuntu@capi:~$ kind create cluster
Install clusterctl
ubuntu@capi:~$ curl -L https://github.com/kubernetes-sigs/cluster-api/releases/download/v0.3.11/clusterctl-linux-amd64 -o clusterctl
ubuntu@capi:~$ chmod +x ./clusterctl
ubuntu@capi:~$ sudo mv ./clusterctl /usr/local/bin/clusterctl
Initialize the management cluster
ubuntu@capi:~$ clusterctl init --infrastructure openstack
Create workload cluster
Configure environment variables
My clouds.yaml is the following.
ubuntu@capi:~$ cat clouds.yaml
clouds:
openstack:
auth:
auth_url: http://controller.hidekazuna.test:5000
auth_type: "v3applicationcredential"
application_credential_id: "49a25feadeb24bb1b8490ff1813e8265"
application_credential_secret: "cluster-api-provider-openstack"
region_name: RegionOne
ubuntu@capi:~$
cluster-api-provider-openstack provides useful script env.rc to set environment variables from clouds.yaml
ubuntu@capi:~$ wget https://raw.githubusercontent.com/kubernetes-sigs/cluster-api-provider-openstack/master/templates/env.rc -O /tmp/env.rc
ubuntu@capi:~$ source /tmp/env.rc clouds.yaml openstack
Other environment variables are needed.
ubuntu@capi:~$ export OPENSTACK_DNS_NAMESERVERS=10.0.0.11
ubuntu@capi:~$ export OPENSTACK_FAILURE_DOMAIN=nova
ubuntu@capi:~$ export OPENSTACK_CONTROL_PLANE_MACHINE_FLAVOR=small
ubuntu@capi:~$ export OPENSTACK_NODE_MACHINE_FLAVOR=small
ubuntu@capi:~$ export OPENSTACK_IMAGE_NAME=u1804-kube-v1.17.11
ubuntu@capi:~$ export OPENSTACK_SSH_KEY_NAME=mykey
The environment variables are self-explanatory. But one thing to note is that the image is need to be built using Kubernetes Image Builder for OpenStack.
Create manifest
ubuntu@capi:~$ clusterctl config cluster external --flavor external-cloud-provider --kubernetes-version v1.17.11 --control-plane-machine-count=3 --worker-machine-count=1 > external.yaml
Delete disableServerTags from the manifest
Unfortunately the template for external cloud provider has a bug. We need to delete disableServerTags from the manifest manually.
Apply the manifest
ubuntu@capi:~$ kubectl apply -f external.yaml
cluster.cluster.x-k8s.io/external unchanged
openstackcluster.infrastructure.cluster.x-k8s.io/external created
kubeadmcontrolplane.controlplane.cluster.x-k8s.io/external-control-plane created
openstackmachinetemplate.infrastructure.cluster.x-k8s.io/external-control-plane created
machinedeployment.cluster.x-k8s.io/external-md-0 created
openstackmachinetemplate.infrastructure.cluster.x-k8s.io/external-md-0 created
kubeadmconfigtemplate.bootstrap.cluster.x-k8s.io/external-md-0 created
secret/external-cloud-config created
Check if we can continue
Check if OpenStackCluster READY is true.
ubuntu@capi:~$ kubectl get openstackcluster
NAME CLUSTER READY NETWORK SUBNET BASTION
external external true 7f7cc336-4778-4732-93eb-a8ecd18b8017 f35c75ab-8761-49fd-82d2-faa47138fe42
It is OK machine PHASE is still Provisioning as follows.
ubuntu@capi:~$ kubectl get machine
NAME PROVIDERID PHASE VERSION
external-control-plane-c4dnt openstack://a112dd08-f2ff-4f71-a639-68cf6504d36a Provisioning v1.17.11
external-md-0-84b9fff89c-w8nwc openstack://437dfe44-2aa4-4337-b740-0a729709ea61 Provisioning v1.17.11
Get workload cluster kubeconfig
ubuntu@capi:~$ export CLUSTER_NAME=external
ubuntu@capi:~$ clusterctl get kubeconfig ${CLUSTER_NAME} --namespace default > ./${CLUSTER_NAME}.kubeconfig
Deploy CNI
ubuntu@capi:~$ curl https://docs.projectcalico.org/v3.16/manifests/calico.yaml | sed "s/veth_mtu:.*/veth_mtu: \"1430\"/g" | kubectl --kubeconfig=./${CLUSTER_NAME}.kubeconfig apply -f -
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 183k 100 183k 0 0 150k 0 0:00:01 0:00:01 --:--:-- 150k
configmap/calico-config created
customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created
clusterrole.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrole.rbac.authorization.k8s.io/calico-node created
clusterrolebinding.rbac.authorization.k8s.io/calico-node created
daemonset.apps/calico-node created
serviceaccount/calico-node created
deployment.apps/calico-kube-controllers created
serviceaccount/calico-kube-controllers created
ubuntu@capi:~$
Deploy External OpenStack Cloud Provider
Create secret
ubuntu@capi:~$ wget https://raw.githubusercontent.com/kubernetes-sigs/cluster-api-provider-openstack/v0.3.3/templates/create_cloud_conf.sh -O /tmp/create_cloud_conf.sh
ubuntu@capi:~$ bash /tmp/create_cloud_conf.sh clouds.yaml openstack > /tmp/cloud.conf
ubuntu@capi:~$ kubectl --kubeconfig=./${CLUSTER_NAME}.kubeconfig create secret -n kube-system generic cloud-config --from-file=/tmp/cloud.conf
secret/cloud-config created
ubuntu@capi:~$ rm /tmp/cloud.conf
Create RBAC resources and openstack-cloud-controller-manager deamonset
ubuntu@capi:~$ kubectl --kubeconfig=./${CLUSTER_NAME}.kubeconfig apply -f https://raw.githubusercontent.com/kubernetes/cloud-provider-openstack/master/cluster/addons/rbac/cloud-controller-manager-roles.yaml
clusterrole.rbac.authorization.k8s.io/system:cloud-controller-manager created
clusterrole.rbac.authorization.k8s.io/system:cloud-node-controller created
clusterrole.rbac.authorization.k8s.io/system:pvl-controller created
ubuntu@capi:~$ kubectl --kubeconfig=./${CLUSTER_NAME}.kubeconfig apply -f https://raw.githubusercontent.com/kubernetes/cloud-provider-openstack/master/cluster/addons/rbac/cloud-controller-manager-role-bindings.yaml
clusterrolebinding.rbac.authorization.k8s.io/system:cloud-node-controller created
clusterrolebinding.rbac.authorization.k8s.io/system:pvl-controller created
clusterrolebinding.rbac.authorization.k8s.io/system:cloud-controller-manager created
ubuntu@capi:~$ kubectl --kubeconfig=./${CLUSTER_NAME}.kubeconfig apply -f https://raw.githubusercontent.com/kubernetes/cloud-provider-openstack/master/manifests/controller-manager/openstack-cloud-controller-manager-ds.yaml
serviceaccount/cloud-controller-manager created
daemonset.apps/openstack-cloud-controller-manager created
ubuntu@capi:~$
Waiting for all the pods in kube-system namespace up and running
ubuntu@capi:~$ kubectl --kubeconfig=./${CLUSTER_NAME}.kubeconfig get pod -n kube-system
NAME READY STATUS RESTARTS AGE
calico-kube-controllers-544658cf79-fkv2d 1/1 Running 1 6m19s
calico-node-5r55n 1/1 Running 1 6m19s
calico-node-fld8j 1/1 Running 1 6m19s
coredns-6955765f44-4zr75 1/1 Running 1 16m
coredns-6955765f44-nqth4 1/1 Running 1 16m
etcd-external-control-plane-2cdjf 1/1 Running 1 17m
kube-apiserver-external-control-plane-2cdjf 1/1 Running 1 17m
kube-controller-manager-external-control-plane-2cdjf 1/1 Running 1 17m
kube-proxy-jn79q 1/1 Running 1 13m
kube-proxy-xxxbw 1/1 Running 1 16m
kube-scheduler-external-control-plane-2cdjf 1/1 Running 1 17m
openstack-cloud-controller-manager-q58px 1/1 Running 1 18s
$
Wait for all machines up and running
Time has passed. All machines should be running.
ubuntu@capi:~$ kubectl get machines
NAME PROVIDERID PHASE VERSION
external-control-plane-5f6tg openstack://75935c02-9d51-4745-921e-9db0fbc868c1 Running v1.17.11
external-control-plane-j6s2v openstack://8e36167d-b922-418d-9c1b-de907c9a0fc2 Running v1.17.11
external-control-plane-p4kq7 openstack://7fcb9e4b-e107-4d5c-a559-8e55e1018c2c Running v1.17.11
external-md-0-84b9fff89c-ghm5t openstack://55255fae-af71-40ba-bcdc-1a0393b2aaf0 Running v1.17.11
Congratulations! Kubernetes cluster has been deployed successfully.
Let’s go on to use Load Balancer.
Using service of LoadBalancer type
Create deployment and service
ubuntu@capi:~$ wget https://raw.githubusercontent.com/kubernetes/cloud-provider-openstack/master/examples/loadbalancers/external-http-nginx.yaml
external-http-nginx.yaml is following. Update loadbalancer.openstack.org/floating-network-id value to your network id.
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: external-http-nginx-deployment
spec:
replicas: 2
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx
ports:
- containerPort: 80
---
kind: Service
apiVersion: v1
metadata:
name: external-http-nginx-service
annotations:
service.beta.kubernetes.io/openstack-internal-load-balancer: "false"
loadbalancer.openstack.org/floating-network-id: "9be23551-38e2-4d27-b5ea-ea2ea1321bd6"
spec:
selector:
app: nginx
type: LoadBalancer
ports:
- name: http
port: 80
targetPort: 80
Apply the manifest.
ubuntu@capi:~$ kubectl --kubeconfig=./${CLUSTER_NAME}.kubeconfig apply -f external-http-nginx.yaml
deployment.apps/external-http-nginx-deployment created
service/external-http-nginx-service created
Wait for external IP
ubuntu@capi:~$ kubectl --kubeconfig=./${CLUSTER_NAME}.kubeconfig get service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 23h
external-http-nginx-service LoadBalancer 10.103.116.174 10.0.0.235 80:32044/TCP 31s
Check if external IP really works
ubuntu@capi:~$ curl http://10.0.0.235
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
$
Delete deployment and service
ubuntu@capi:~$ kubectl --kubeconfig=./${CLUSTER_NAME}.kubeconfig delete -f external-http-nginx.yaml
deployment.apps "external-http-nginx-deployment" deleted
service "external-http-nginx-service" deleted
Cinder CSI
Clone cloud-provider-openstack repository and checkout release-1.17 branch
ubuntu@capi:~$ git clone https://github.com/kubernetes/cloud-provider-openstack.git
Cloning into 'cloud-provider-openstack'...
remote: Enumerating objects: 50, done.
remote: Counting objects: 100% (50/50), done.
remote: Compressing objects: 100% (46/46), done.
remote: Total 13260 (delta 20), reused 11 (delta 2), pack-reused 13210
Receiving objects: 100% (13260/13260), 3.57 MiB | 473.00 KiB/s, done.
Resolving deltas: 100% (6921/6921), done.
ubuntu@capi:~$
ubuntu@capi:~$ cd cloud-provider-openstack
ubuntu@capi:~/cloud-provider-openstack$ git checkout -b release-1.17 origin/release-1.17
Branch 'release-1.17' set up to track remote branch 'release-1.17' from 'origin'.
Switched to a new branch 'release-1.17'
Remove manifests/cinder-csi-plugin/csi-secret-cinderplugin.yaml because cloud-config secret was already created.
buntu@capi:~/cloud-provider-openstack$ rm manifests/cinder-csi-plugin/csi-secret-cinderplugin.yaml
Apply the manifests
ubuntu@capi:~/cloud-provider-openstack$ kubectl --kubeconfig=/home/ubuntu/${CLUSTER_NAME}.kubeconfig apply -f manifests/cinder-csi-plugin/
serviceaccount/csi-cinder-controller-sa created
clusterrole.rbac.authorization.k8s.io/csi-attacher-role created
clusterrolebinding.rbac.authorization.k8s.io/csi-attacher-binding created
clusterrole.rbac.authorization.k8s.io/csi-provisioner-role created
clusterrolebinding.rbac.authorization.k8s.io/csi-provisioner-binding created
clusterrole.rbac.authorization.k8s.io/csi-snapshotter-role created
clusterrolebinding.rbac.authorization.k8s.io/csi-snapshotter-binding created
clusterrole.rbac.authorization.k8s.io/csi-resizer-role created
clusterrolebinding.rbac.authorization.k8s.io/csi-resizer-binding created
role.rbac.authorization.k8s.io/external-resizer-cfg created
rolebinding.rbac.authorization.k8s.io/csi-resizer-role-cfg created
service/csi-cinder-controller-service created
statefulset.apps/csi-cinder-controllerplugin created
serviceaccount/csi-cinder-node-sa created
clusterrole.rbac.authorization.k8s.io/csi-nodeplugin-role created
clusterrolebinding.rbac.authorization.k8s.io/csi-nodeplugin-binding created
daemonset.apps/csi-cinder-nodeplugin created
csidriver.storage.k8s.io/cinder.csi.openstack.org created
ubuntu@capi:~/cloud-provider-openstack$
Check if csi-cinder-controllerplugin and csi-cinder-nodeplugin are running.
ubuntu@capi:~/cloud-provider-openstack$ kubectl --kubeconfig=/home/ubuntu/${CLUSTER_NAME}.kubeconfig get pod -n kube-system -l 'app in (csi-cinder-controllerplugin,csi-cinder-nodeplugin)'
NAME READY STATUS RESTARTS AGE
csi-cinder-controllerplugin-0 5/5 Running 0 88s
csi-cinder-nodeplugin-gvdd5 2/2 Running 0 88s
ubuntu@capi:~/cloud-provider-openstack$
ubuntu@capi:~/cloud-provider-openstack$ kubectl --kubeconfig=./${CLUSTER_NAME}.kubeconfig get csidrivers.storage.k8s.io
NAME CREATED AT
cinder.csi.openstack.org 2020-12-23T07:10:26Z
ubuntu@capi:~/cloud-provider-openstack$
Using Cinder CSI
ubuntu@capi:~/cloud-provider-openstack$ kubectl --kubeconfig=./${CLUSTER_NAME}.kubeconfig -f examples/cinder-csi-plugin/nginx.yaml create
storageclass.storage.k8s.io/csi-sc-cinderplugin created
persistentvolumeclaim/csi-pvc-cinderplugin created
pod/nginx created
ubuntu@capi:~/cloud-provider-openstack$
ubuntu@capi:~/cloud-provider-openstack$ kubectl --kubeconfig=./${CLUSTER_NAME}.kubeconfig get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
csi-pvc-cinderplugin Bound pvc-5b359b6a-ed6b-4ccb-9a3f-9de6925e1713 1Gi RWO csi-sc-cinderplugin 4m4s
ubuntu@capi:~/cloud-provider-openstack$ kubectl --kubeconfig=./${CLUSTER_NAME}.kubeconfig exec -it nginx -- bash
root@nginx:/#
root@nginx:/#
root@nginx:/#
root@nginx:/# ls /var/lib/www/html
lost+found
root@nginx:/# touch /var/lib/www/html/index.html
root@nginx:/# exit
exit
ubuntu@capi:~/cloud-provider-openstack$
Delete created resources
ubuntu@capi:~/cloud-provider-openstack$ kubectl --kubeconfig=./${CLUSTER_NAME}.kubeconfig delete -f examples/cinder-csi-plugin/nginx.yaml
storageclass.storage.k8s.io "csi-sc-cinderplugin" deleted
persistentvolumeclaim "csi-pvc-cinderplugin" deleted
pod "nginx" deleted
ubuntu@capi:~/cloud-provider-openstack$
ubuntu@capi:~/cloud-provider-openstack$ kubectl --kubeconfig=./${CLUSTER_NAME}.kubeconfig get pvc
No resources found in default namespace.
ubuntu@capi:~/cloud-provider-openstack$