Today I deploy Flux v2 in my Kapsule cluster to discover and play with Gitops principles. I plan to use this to ease the migration of the workloads running on my existing cluster to a fresh new one.
What is Fluxv2?
Fluxv2 is a set of Kubernetes controllers and a command line interface that enable you to operate Kubernetes clusters following the Gitops principles.
In very very short, Gitops allows you to reconcile the workloads running inside your Kubernetes cluster with a declarative source code stored inside of git repositories.
Fluxv2 is developed by Weavework and is based on the Gitops Toolkit. This toolkit is composed of 4 components :
Source Controller : where you define in which repositories the artefacts describing your workloads reside.
Kustomize Controller : this controller applies kustomizations to your deployments.
Helm Controller : this controller applies Helm releases.
Notification Controller : this controller sends alerts to a set of providers like Slack, Teams, etc …
Step 1 - Install flux CLI
I chose a straigthforward method to install the CLI. I manage the software I use on a daily basis in an Ansible playbook. I added the tasks down below to fetch a specific release from the Flux release page on Github and copy the binary to my /usr/local/bin directory.
vars:...flux_version:'0.15.3'...- name:Download Flux CLI version {{ flux_version }}become:yesunarchive:src:https://github.com/fluxcd/flux2/releases/download/v{{ flux_version }}/flux_{{ flux_version }}_linux_amd64.tar.gzdest:/tmpremote_src:yesmode:0755- name:Copy flux binary to /usr/local/binbecome:yescopy:src:/tmp/fluxdest:/usr/local/binmode:0755remote_src:yes...
Test the installation of the CLI.
1
2
flux --version
flux version 0.15.3
Step 2 - Bootstrap Flux from a new git repo
Flux v2 bootstraps itself from a git repository. It supports Github, Gitlab, Azure Devops Repositories or generic git repositories. I do use a personal Github repository myself.
First, create a Personal Access Token and grant it all access under the repo category.
Remark: Github organizations are also supported.
Export both your Github username and PAT as environment variables (makes it handy to use at the command line).
kubectl get all -n flux-system
NAME READY STATUS RESTARTS AGE
pod/helm-controller-5b96d94c7f-mqplz 1/1 Running 0 4m55s
pod/kustomize-controller-df8bb769-r56bt 1/1 Running 0 4m55s
pod/notification-controller-55f94bc746-gnwgj 1/1 Running 0 4m55s
pod/source-controller-679d5777d9-5blbz 1/1 Running 0 4m55s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/notification-controller ClusterIP 10.38.141.219 <none> 80/TCP 4m59s
service/source-controller ClusterIP 10.39.242.19 <none> 80/TCP 4m59s
service/webhook-receiver ClusterIP 10.39.112.165 <none> 80/TCP 4m58s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/helm-controller 1/1 11 4m58s
deployment.apps/kustomize-controller 1/1 11 4m58s
deployment.apps/notification-controller 1/1 11 4m58s
deployment.apps/source-controller 1/1 11 4m58s
NAME DESIRED CURRENT READY AGE
replicaset.apps/helm-controller-5b96d94c7f 111 4m59s
replicaset.apps/kustomize-controller-df8bb769 111 4m59s
replicaset.apps/notification-controller-55f94bc746 111 4m59s
replicaset.apps/source-controller-679d5777d9 111 4m59s
The flux-system namespace is deployed with all controllers that are part of the Gitops Toolkit and the scw-kapsule-infraGithub repository contains all the manifests describing this deployment.
You can also use the command flux check to verify the version of the Flux components.
Define a source that uses the repository containing the deployment files for this blog. It is also a private Github repository that has the following structure :
Notice that it uses a Kustomization to replace the tag of the container image to deploy by the latest commit SHA.
Use the flux CLI to create a new source and export it to a file in the local clone of the scw-kapsule-infra repository. As I choose a private repository, the procedure differs from the standard getting started guide here.
Create a secret containing your Github username and your PAT as password.
There is one issue with that command : it creates a file that contains your Github username and PAT in clear text in your repository … even if it is a private repository, it’s not a good idea to commit credentials in clear text in it.
There are 2 possible options to solve that problem : encrypt that secret with either Mozilla SOPS or Bitnami Sealed Secrets. As I want to keep things simple and easy, let’s try this with SOPS.
How does SOPS works with Flux
SOPS allows you to encrypt values of a YAML, JSON, ENV, INI or BIN file with a public key. Those files can then be decrypted by the related private key.
The public key is published in a git repository and available for everyone to encrypt their files.
The private key is stored in the Kubernetes cluster and Flux is configured to use it to decrypt secrets.
Export the private key and create a Kubernetes secret sops-gpg in namespace flux-system.
Note : this is the only action that should be done manually to your cluster after Flux is deployed. It should be part of the cluster bootstrap process.
Now it’s time to backup the private key and delete it from your local PGP keyring.
Note: I used Bitwarden to store a copy of the private/public key pair.
1
2
3
4
5
6
7
# Export the private key to a clear text file.# Store it safely (USB stick in a safe or password vault)# Then DELETE it from your workstationgpg --export-secret-keys --armor "${KEY_FP}" > .sops.asc
# Delete the private key from your PGP keyringgpg --delete-secret-keys "${KEY_FP}"
Export the public key and store it in the flux bootstrap repository under ./clusters/{name of your cluster}. The public key will then be available for everyone to be able to encrypt their secrets.
1
2
3
4
5
6
7
# Export the public key to the git repository of your clustergpg --export --armor "${KEY_FP}" > ./clusters/seblab/.sops.pub.asc
# Commit and pushgit add clusters/seblab/.sops.pub.asc
git commit -m "Add SOPS public key for secret encryption"git push origin
Configure Flux to decrypt SOPS secrets
SOPS encrypted files are handled by Flux’s Kustomizations. Patch the gotk-sync.yaml manifests in the Flux bootstrap repository add decryption with provider sops and the secret sops-gpg created earlier.
Create file clusters/seblab/flux-system/gotk-patches.yaml with the following content:
1
2
3
4
5
6
7
8
9
10
11
12
apiVersion:kustomize.toolkit.fluxcd.io/v1beta1kind:Kustomizationmetadata:name:flux-systemnamespace:flux-systemspec:decryption:# Use the sops providerprovider:sopssecretRef:# Reference the new 'sops-gpg' secretname:sops-gpg
Modify the file clusters/seblab/flux-system/kustomization.yaml to add the patch.
# Add the new filegit add clusters/seblab/flux-system/kustomization.yaml clusters/seblab/flux-system/kustomizations/
# Commit to the local clonegit commit -m "Add seblab-blog kustomization"# Push to Githubgit push origin
# Check that the new kustomization is presentflux get kustomization
NAME READY MESSAGE REVISION
SUSPENDED
flux-system True Applied revision: main/75c0a3aa778fa59e907716df444cfd9d4bbacec6 main/75c0a3aa778fa59e907716df444cfd9d4bbacec6 False
seblab-blog True Applied revision: master/34ed77b2c8cc3ea94b86c6f20080be95c96dd684 master/34ed77b2c8cc3ea94b86c6f20080be95c96dd684 False
# Optional, check logs of the kustomization seblab-blogflux logs --kind=kustomization --name=seblab-blog
2021-07-04T11:06:08.740Z info Kustomization/seblab-blog.flux-system - Kustomization applied in 917.820668ms
2021-07-04T11:06:08.764Z info Kustomization/seblab-blog.flux-system - Reconciliation finished in 1.859155176s, next run in 5m0s
2021-07-04T11:06:08.742Z info Kustomization/seblab-blog.flux-system - Discarding event, no alerts found for the involved object
2021-07-04T11:06:08.765Z info Kustomization/seblab-blog.flux-system - Discarding event, no alerts found for the involved object
Step 5 : Test !
This is the final test. I will push a modification to this blog and see if it will be automatically deployed by Flux.
This the publication flow of this blog :
Flux will replace the last step by detecting the change in the k8s-blog private repository and deploying the new version of the blog when it happens.
Let’s check the current image version deployed in my cluster.
1
2
kubectl get deployment seblab-blog-deployment -n blog -o=jsonpath='{$.spec.template.spec.containers[:1].image}'rg.fr-par.scw.cloud/seblabcr_priv/seblab-blog:sha-9ad39b3
It corresponds to the image tag described in my kustomization.yaml for this deployment
The image tag is updated … now let’s wait 5 minutes and check again the image of the current deployment
1
2
kubectl get deployment seblab-blog-deployment -n blog -o=jsonpath='{$.spec.template.spec.containers[:1].image}'rg.fr-par.scw.cloud/seblabcr_priv/seblab-blog:sha-df9ab48
The new image has been deployed by Flux !
Now I’m a very happy blogger and I won’t need that last manual step anymore to publish new blog posts.
The new post publication flow is :
Conclusion
It has been one long ride to reach the Flux destination but it is a succes.
I’m now working on the complete automation of my cluster because I want to move the current workload on a fresh new Kapsule cluster. I have still some choices to make about the repository structure, the number of repo to use and I need to configure Helm chart deployments as well as notifications and monitoring of Flux.
More posts to come on these topics in the near future !