Deploying Fluxv2 in Kapsule

Today I deploy Flux v2 in my Kapsule cluster to discover and play with Gitops principles. I plan to use this to ease the migration of the workloads running on my existing cluster to a fresh new one.

What is Fluxv2?

Fluxv2 is a set of Kubernetes controllers and a command line interface that enable you to operate Kubernetes clusters following the Gitops principles.

In very very short, Gitops allows you to reconcile the workloads running inside your Kubernetes cluster with a declarative source code stored inside of git repositories.

Fluxv2 is developed by Weavework and is based on the Gitops Toolkit. This toolkit is composed of 4 components :

Source Controller : where you define in which repositories the artefacts describing your workloads reside.
Kustomize Controller : this controller applies kustomizations to your deployments.
Helm Controller : this controller applies Helm releases.
Notification Controller : this controller sends alerts to a set of providers like Slack, Teams, etc …

Step 1 - Install `flux` CLI

I chose a straigthforward method to install the CLI. I manage the software I use on a daily basis in an Ansible playbook. I added the tasks down below to fetch a specific release from the Flux release page on Github and copy the binary to my /usr/local/bin directory.

vars:
    ...
    flux_version: '0.15.3'  
    ...

  - name: Download Flux CLI version {{ flux_version }}
    become: yes
    unarchive:
      src: https://github.com/fluxcd/flux2/releases/download/v{{ flux_version }}/flux_{{ flux_version }}_linux_amd64.tar.gz
      dest: /tmp
      remote_src: yes
      mode: 0755

  - name: Copy flux binary to /usr/local/bin
    become: yes
    copy:
      src: /tmp/flux
      dest: /usr/local/bin
      mode: 0755
      remote_src: yes
  
  ...

Test the installation of the CLI.

flux --version
flux version 0.15.3

Step 2 - Bootstrap Flux from a new git repo

Flux v2 bootstraps itself from a git repository. It supports Github, Gitlab, Azure Devops Repositories or generic git repositories. I do use a personal Github repository myself.

First, create a Personal Access Token and grant it all access under the repo category.

Remark: Github organizations are also supported.

Export both your Github username and PAT as environment variables (makes it handy to use at the command line).

export GITHUB_USER=<your Github username>
export GITHUB_TOKEN=<your Github Peronal Access Token>

Check if my Kapsule cluster does satisfy the minimal requirements for Fluxv2.

# Check the Kubernetes context
kubectx
admin@scw-k8s-seblab

# This is the correct cluster now run pre-checks
flux check --pre
► checking prerequisites
✔ kubectl 1.20.4 >=1.18.0-0
✔ Kubernetes 1.20.4 >=1.16.0-0
✔ prerequisites checks passed

So my cluster is compliant. Let’s now bootstrap Flux and let it create a fresh new Github repository named scw-kapsule-infra.

flux bootstrap github \
> --owner=$GITHUB_USER \
> --repository=scw-kapsule-infra \
> --branch=main \
> --path=./clusters/seblab \
> --personal \
> --private
► connecting to github.com
✔ repository "https://github.com/pondichys/scw-kapsule-infra" created
► cloning branch "main" from Git repository "https://github.com/pondichys/scw-kapsule-infra.git"
✔ cloned repository
► generating component manifests
✔ generated component manifests
✔ committed sync manifests to "main" ("31800f2c68b1fe3d47f9a971cbe1dbd3994c51b1")
► pushing component manifests to "https://github.com/pondichys/scw-kapsule-infra.git"
► installing components in "flux-system" namespace
✔ installed components
✔ reconciled components
► determining if source secret "flux-system/flux-system" exists
► generating source secret
✔ public key: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDk8n01SgxVEOcPR3AYEoBeaceHd/uf/AcN9pB7nZ86iNk1scFgLIXvCBLfMTDVbc/lToHRtxhZ6XqExjcb6BDkxvQjt/lie8vhB/6Mj6UexZXRZcPXUvT5z0SrzIR+W8bNOhH7yhyA7LRyNDHzA9Qhysj/efTcqW92mLsPxzynVUk63hDbNfJEsRTp7eOBP9fwh+f+wU+97ULXNFV1c/bwdeMxNsZQKZQGRudzaMhRcZ7ci61xTuByPVeLEqokqSUp0hnAsGadALO6lbJ8bCLcAAVq1vc+3s2S+n0BAsRe6XnR7RgcdB5Qv29nWzWREU0m7KMUJucsalGDZqWBRmWd
✔ configured deploy key "flux-system-main-flux-system-./clusters/seblab" for "https://github.com/pondichys/scw-kapsule-infra"
► applying source secret "flux-system/flux-system"
✔ reconciled source secret
► generating sync manifests
✔ generated sync manifests
✔ committed sync manifests to "main" ("9468e91f3e198c114f5984f0ace109055508549f")
► pushing sync manifests to "https://github.com/pondichys/scw-kapsule-infra.git"
► applying sync manifests
✔ reconciled sync configuration
◎ waiting for Kustomization "flux-system/flux-system" to be reconciled
✔ Kustomization reconciled successfully
► confirming components are healthy
✔ source-controller: deployment ready
✔ kustomize-controller: deployment ready
✔ helm-controller: deployment ready
✔ notification-controller: deployment ready
✔ all components are healthy

Check all components deployed in the dedicated namespace flux-system

kubectl get all -n flux-system
NAME                                           READY   STATUS    RESTARTS   AGE
pod/helm-controller-5b96d94c7f-mqplz           1/1     Running   0          4m55s
pod/kustomize-controller-df8bb769-r56bt        1/1     Running   0          4m55s
pod/notification-controller-55f94bc746-gnwgj   1/1     Running   0          4m55s
pod/source-controller-679d5777d9-5blbz         1/1     Running   0          4m55s

NAME                              TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
service/notification-controller   ClusterIP   10.38.141.219   <none>        80/TCP    4m59s
service/source-controller         ClusterIP   10.39.242.19    <none>        80/TCP    4m59s
service/webhook-receiver          ClusterIP   10.39.112.165   <none>        80/TCP    4m58s

NAME                                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/helm-controller           1/1     1            1           4m58s
deployment.apps/kustomize-controller      1/1     1            1           4m58s
deployment.apps/notification-controller   1/1     1            1           4m58s
deployment.apps/source-controller         1/1     1            1           4m58s

NAME                                                 DESIRED   CURRENT   READY   AGE
replicaset.apps/helm-controller-5b96d94c7f           1         1         1       4m59s
replicaset.apps/kustomize-controller-df8bb769        1         1         1       4m59s
replicaset.apps/notification-controller-55f94bc746   1         1         1       4m59s
replicaset.apps/source-controller-679d5777d9         1         1         1       4m59s

The flux-system namespace is deployed with all controllers that are part of the Gitops Toolkit and the scw-kapsule-infra Github repository contains all the manifests describing this deployment.

You can also use the command flux check to verify the version of the Flux components.

flux check
► checking prerequisites
✗ flux 0.15.3 <0.16.0 (new version is available, please upgrade)
✔ kubectl 1.20.4 >=1.18.0-0
✔ Kubernetes 1.20.4 >=1.16.0-0
► checking controllers
✔ helm-controller: deployment ready
► ghcr.io/fluxcd/helm-controller:v0.11.1
✔ kustomize-controller: deployment ready
► ghcr.io/fluxcd/kustomize-controller:v0.13.0
✔ notification-controller: deployment ready
► ghcr.io/fluxcd/notification-controller:v0.15.0
✔ source-controller: deployment ready
► ghcr.io/fluxcd/source-controller:v0.15.2
✔ all checks passed

Before continuing with the next step, clone this repository locally on your workstation.

git clone [email protected]:pondichys/scw-kapsule-infra.git
Cloning into 'scw-kapsule-infra'...
remote: Enumerating objects: 13, done.
remote: Counting objects: 100% (13/13), done.
remote: Compressing objects: 100% (6/6), done.
remote: Total 13 (delta 0), reused 13 (delta 0), pack-reused 0
Receiving objects: 100% (13/13), 19.24 KiB | 4.81 MiB/s, done.

Step 3 - Define a source

Define a source that uses the repository containing the deployment files for this blog. It is also a private Github repository that has the following structure :

k8s-blog
├── README.md
└── manifests
    ├── kustomization.yaml
    ├── scwreg-secret.yaml
    ├── seblab-blog-deployment.yaml
    ├── seblab-blog-ingress.yaml
    └── seblab-blog-namespace.yaml

1 directory, 6 files

Notice that it uses a Kustomization to replace the tag of the container image to deploy by the latest commit SHA.

Use the flux CLI to create a new source and export it to a file in the local clone of the scw-kapsule-infra repository. As I choose a private repository, the procedure differs from the standard getting started guide here.

Create a secret containing your Github username and your PAT as password.

cd scw-kapsule-infra

mkdir -p clusters/seblab/secrets

flux create secret git github \
  --url=https://github.com/pondichys/k8s-blog \
  --username=$GITHUB_USER \
  --password=$GITHUB_TOKEN \
  --export > ./clusters/seblab/flux-system/secrets/github.yaml

There is one issue with that command : it creates a file that contains your Github username and PAT in clear text in your repository … even if it is a private repository, it’s not a good idea to commit credentials in clear text in it.

There are 2 possible options to solve that problem : encrypt that secret with either Mozilla SOPS or Bitnami Sealed Secrets. As I want to keep things simple and easy, let’s try this with SOPS.

How does SOPS works with Flux

SOPS allows you to encrypt values of a YAML, JSON, ENV, INI or BIN file with a public key. Those files can then be decrypted by the related private key.

The public key is published in a git repository and available for everyone to encrypt their files.

The private key is stored in the Kubernetes cluster and Flux is configured to use it to decrypt secrets.

SOPS can work with keys stored in AWS KMS, GCP KMS, Azure Key Vault and PGP.

Install SOPS on your workstation

Still using my Ansible playbook, I added SOPS to the list of tools that I install.

vars:
    ...
    sops_version: '3.7.1'  
    ...
    - name: Mozilla SOPS version {{ sops_version }}
      become: yes
      get_url:
        url: https://github.com/mozilla/sops/releases/download/v{{ sops_version }}/sops-v{{ sops_version }}.linux
        dest: /usr/local/bin/sops
        mode: 0755
    ...

Test SOPS.

sops --version
sops 3.7.1 (latest)

Create GPG key

Create a PGP private / public key pair for your cluster.

export KEY_NAME="cluster0.seblab.be"
export KEY_COMMENT="flux secrets"

gpg --batch --full-generate-key <<EOF
%no-protection
Key-Type: 1
Key-Length: 4096
Subkey-Type: 1
Subkey-Length: 4096
Expire-Date: 0
Name-Comment: ${KEY_COMMENT}
Name-Real: ${KEY_NAME}
EOF

Retrieve the key fingerprint and store it in an environment variable. It is the second line of the sec section.

gpg --list-secret-keys "${KEY_NAME}"

gpg: checking the trustdb
gpg: marginals needed: 3  completes needed: 1  trust model: pgp
gpg: depth: 0  valid:   1  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 1u
sec   rsa4096 2021-07-03 [SCEA]
      F2AF7...
uid           [ultimate] cluster0.seblab.be (flux secrets)
ssb   rsa4096 2021-07-03 [SEA]

export KEY_FP=F2AF7...

Export the private key and create a Kubernetes secret sops-gpg in namespace flux-system.

Note : this is the only action that should be done manually to your cluster after Flux is deployed. It should be part of the cluster bootstrap process.

gpg --export-secret-keys --armor "${KEY_FP}" |
kubectl create secret generic sops-gpg \
--namespace=flux-system \
--from-file=sops.asc=/dev/stdin

Now it’s time to backup the private key and delete it from your local PGP keyring.

Note: I used Bitwarden to store a copy of the private/public key pair.

# Export the private key to a clear text file.
# Store it safely (USB stick in a safe or password vault)
# Then DELETE it from your workstation
gpg --export-secret-keys --armor "${KEY_FP}" > .sops.asc

# Delete the private key from your PGP keyring
gpg --delete-secret-keys "${KEY_FP}"

Export the public key and store it in the flux bootstrap repository under ./clusters/{name of your cluster}. The public key will then be available for everyone to be able to encrypt their secrets.

# Export the public key to the git repository of your cluster
gpg --export --armor "${KEY_FP}" > ./clusters/seblab/.sops.pub.asc

# Commit and push
git add clusters/seblab/.sops.pub.asc
git commit -m "Add SOPS public key for secret encryption"
git push origin

Configure Flux to decrypt SOPS secrets

SOPS encrypted files are handled by Flux’s Kustomizations. Patch the gotk-sync.yaml manifests in the Flux bootstrap repository add decryption with provider sops and the secret sops-gpg created earlier.

Create file clusters/seblab/flux-system/gotk-patches.yaml with the following content:

apiVersion: kustomize.toolkit.fluxcd.io/v1beta1
kind: Kustomization
metadata:
  name: flux-system
  namespace: flux-system
spec:
  decryption:
    # Use the sops provider
    provider: sops
    secretRef:
      # Reference the new 'sops-gpg' secret
      name: sops-gpg

Modify the file clusters/seblab/flux-system/kustomization.yaml to add the patch.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- gotk-components.yaml
- gotk-sync.yaml
patchesStrategicMerge:
- gotk-patches.yaml

Commit to your git repository and push to Github. Then reconcile the flux-system source.

git add kustomization.yaml gotk-patches.yaml
git commit -m "Patch GOTK to add SOPS decryption"
git push

flux reconcile source git flux-system
► annotating GitRepository flux-system in flux-system namespace
✔ GitRepository annotated
◎ waiting for GitRepository reconciliation
✔ GitRepository reconciliation completed
✔ fetched revision main/dc8d50a9aa6d65046a49aedda3d5c727ceddd18c

Now the cluster is ready to decrypt secrets.

Configure git repository for encryption

Create a SOPS configuration file in the cluster directory of Flux bootstrap repository to set up default options when using SOPS.

cat <<EOF > ./clusters/seblab/.sops.yaml
creation_rules:
  - path_regex: .*.yaml
    encrypted_regex: ^(data|stringData)$
    pgp: ${KEY_FP}
EOF

Encrypt the github token secret with SOPS

Encrypt the file clusters/seblab/flux-system/secrets/github.yaml with SOPS.

sops --encrypt --in-place flux-system/secrets/github.yaml

Deploy the github secret

Add the encrypted github.yaml file to clusters/seblab/flux-system/kustomization.yaml.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- gotk-components.yaml
- gotk-sync.yaml
- secrets/github.yaml
patchesStrategicMerge:
- gotk-patches.yaml

Commit to git and push to Github. Optionally reconcile flux-system source.

git add flux-system/kustomization.yaml flux-system/secrets/
git commit -m "Add github token secret"
git push

flux reconcile source git flux-system

Now the Github token is available for components deployed by kustomization flux-system from source flux-system.

Let’s go back to the creation of the git source repository for my blog.

Create the source git repository

Use flux CLI to create a source git repository. Store the file in folder clusters/seblab/flux-system/sources/.

# Run at root of flux bootstrap repo
mkdir clusters/seblab/flux-system/sources

flux create source git seblab-blog \
  --url=https://github.com/pondichys/k8s-blog \
  --branch=master \
  --secret-ref=github \
  --interval=10m \
  --export > ./clusters/seblab/flux-system/sources/seblab-blog-source.yaml

Add this file in clusters/seblab/flux-system/kustomization.yaml, commit all modifications to your git repository and push to Github.

# Add the new file
git add clusters/seblab/flux-system/kustomization.yaml clusters/seblab/flux-system/sources/

# Commit to the local clone
git commit -m "Add seblab-blog source GitRepository"

# Push to Github
git push origin

# Reconcile flux-system source
flux reconcile source git flux-system

# Check that the new source is present
flux get source git
NAME            READY   MESSAGE                                                                 REVISION
                        SUSPENDED
flux-system     True    Fetched revision: main/35a6d765ecbfa4b73c80d88b6fa18cc49c3b855b         main/35a6d765ecbfa4b73c80d88b6fa18cc49c3b855b   False
seblab-blog     True    Fetched revision: master/34ed77b2c8cc3ea94b86c6f20080be95c96dd684       master/34ed77b2c8cc3ea94b86c6f20080be95c96dd684 False

Ok now that the source repository is available, let’s create a kustomization to tell Flux how to deploy the blog.

Step 4 - Deploy kustomization using this blog

As always, use flux CLI to generate the seblab-blog kustomization.

mkdir clusters/seblab/flux-system/kustomizations

flux create kustomization seblab-blog \
--source=seblab-blog \
--path="./manifests" \
--prune=true \
--validation=client \
--interval=5m \
--export > ./clusters/seblab/flux-system/kustomizations/seblab-blog-kusto.yaml

Add this file in clusters/seblab/flux-system/kustomization.yaml, commit all modifications to your git repository and push to Github.

# Add the new file
git add clusters/seblab/flux-system/kustomization.yaml clusters/seblab/flux-system/kustomizations/

# Commit to the local clone
git commit -m "Add seblab-blog kustomization"

# Push to Github
git push origin

# Check that the new kustomization is present
flux get kustomization
NAME            READY   MESSAGE                                                                 REVISION
                        SUSPENDED
flux-system     True    Applied revision: main/75c0a3aa778fa59e907716df444cfd9d4bbacec6         main/75c0a3aa778fa59e907716df444cfd9d4bbacec6   False
seblab-blog     True    Applied revision: master/34ed77b2c8cc3ea94b86c6f20080be95c96dd684       master/34ed77b2c8cc3ea94b86c6f20080be95c96dd684 False

# Optional, check logs of the kustomization seblab-blog
flux logs --kind=kustomization --name=seblab-blog
2021-07-04T11:06:08.740Z info Kustomization/seblab-blog.flux-system - Kustomization applied in 917.820668ms
2021-07-04T11:06:08.764Z info Kustomization/seblab-blog.flux-system - Reconciliation finished in 1.859155176s, next run in 5m0s
2021-07-04T11:06:08.742Z info Kustomization/seblab-blog.flux-system - Discarding event, no alerts found for the involved object
2021-07-04T11:06:08.765Z info Kustomization/seblab-blog.flux-system - Discarding event, no alerts found for the involved object

Step 5 : Test !

This is the final test. I will push a modification to this blog and see if it will be automatically deployed by Flux.

This the publication flow of this blog :

fig 01 - blog publication workflow

Flux will replace the last step by detecting the change in the k8s-blog private repository and deploying the new version of the blog when it happens.

Let’s check the current image version deployed in my cluster.

kubectl get deployment seblab-blog-deployment -n blog -o=jsonpath='{$.spec.template.spec.containers[:1].image}'
rg.fr-par.scw.cloud/seblabcr_priv/seblab-blog:sha-9ad39b3

It corresponds to the image tag described in my kustomization.yaml for this deployment

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: blog

commonLabels:
  app: seblab-blog

resources:
- seblab-blog-namespace.yaml
- seblab-blog-ingress.yaml
- seblab-blog-deployment.yaml

images:
- name: rg.fr-par.scw.cloud/seblabcr_priv/seblab-blog
  newTag: sha-9ad39b3

Let’s build a new container by simply committing this article still in draft status to Github.

Check the commit SHA

git log --oneline
df9ab48 (HEAD -> master, origin/master, origin/HEAD) Add option buildDrafts in README Test build for flux deployment

The Github Action workflow starts up …

fig 02 - blog github workflow

Once the Github Action workflow is finished, check the value of the image tag in file kustomization.yaml of k8s-blog repository.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: blog

commonLabels:
  app: seblab-blog

resources:
- seblab-blog-namespace.yaml
- seblab-blog-ingress.yaml
- seblab-blog-deployment.yaml

images:
- name: rg.fr-par.scw.cloud/seblabcr_priv/seblab-blog
  newTag: sha-df9ab48

The image tag is updated … now let’s wait 5 minutes and check again the image of the current deployment

kubectl get deployment seblab-blog-deployment -n blog -o=jsonpath='{$.spec.template.spec.containers[:1].image}'
rg.fr-par.scw.cloud/seblabcr_priv/seblab-blog:sha-df9ab48

The new image has been deployed by Flux !

Now I’m a very happy blogger and I won’t need that last manual step anymore to publish new blog posts.

The new post publication flow is :

fig 03 - blog publication workflow with Flux

Conclusion

It has been one long ride to reach the Flux destination but it is a succes.

I’m now working on the complete automation of my cluster because I want to move the current workload on a fresh new Kapsule cluster. I have still some choices to make about the repository structure, the number of repo to use and I need to configure Helm chart deployments as well as notifications and monitoring of Flux.

More posts to come on these topics in the near future !

Take care.