Deploy Gitea and Postgresql on Kapsule

Today we will learn to deploy Gitea and Postgres inside a Kapsule managed Kubernetes cluster at Scaleway.

The story

A long time ago (in a galaxy far away …) when I started to learn new technology, I decided to rent a dedicated physical server at Soyoustart. I deployed a VMware ESXi hypervisor on it and configured it to run a small ecosystem composed of pfsense (firewall / proxy / VPN), Traefik as common reverse proxy and some tools like Gitea, Graylog, InfluxDB / Grafana, Jenkins, etc …

Today, as I wanted to learn Kubernetes and not focus on the management of a server anymore, I switched off my good old dedicated server and decided to migrate some of the older stuff to Scaleway’s Kapsule service.

Note: I used this server during 5 years and I never once got an issue with it so kudos to Soyoustart for the nice job done there.

The application that I migrate today is Gitea. And as Gitea does need a database to store some of its configuration, I will also deploy a Postgres database container that shall contain all databases for Gitea and also for other future applications.

Deploy database to Kubernetes

I hesitated a long time before deciding to deploy the database to Kubernetes. In my humble opinion, I do not feel that Kubernetes is the best choice to deploy and run stateful workloads. I know it can handle it but it does not feel right to me. Personal opinion.

I was first inclined to use a managed PostgreSQL database from Scaleway. Then I checked my budget and while it is not very expensive, I decided to not engage new expenses and have a look at the option to deploy it and run it inside my small Kapsule cluster. To practice and see if my concerns about running a database inside a Kubernetes cluster were founded.

Step 1 - Check current usage of the cluster

This step will be really rough because I’ve not yet deployed a monitoring stack inside my cluster … yeah, it’s next on my to do list.

I fired K9S (super helpful tool) and took note of cluster usage.

CPU usage: around 5%
Memory usage: around 40%

Ok we have some space available, so let’s see which version is eligible.

Step 2 - Choose the version of Postgres

At the time of writing this article, the current version of Postgres available on Docker Hub is 13.2.

Gitea has no specific requirements regarding the Postgres version so I choosed the version 13.2-alpine to be up to date.

Step 3 - deploy the database inside Kapsule

Deploying a stateful workload is done by using StatefulSets. As we will only deploy one replica and not configure database replication, we could also use a deployment in that case but why derogate to good practices?

The following components are needed:

a Secret for the Postgres password
a headless Service
a StatefulSet with a Persistent Volume Claim (to store the data) using the default StorageClass configured by Scaleway in Kapsule.

Create the Kubernetes Secret

1
kubectl create secret generic postgres-secrets --from-literal=POSTGRES_PASSWORD=<your password>

Create the Headless Service

A headless service has no IP address and thus does not act as layer 4 load-balancer / reverse-proxy. It does register a DNS name for each pod being part of the StatefulSet.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
apiVersion: v1
kind: Service
metadata:
  name: postgres-svc
  labels:
    app: postgres
spec:
  ports:
  - port: 5432
    name: postgres-port
  clusterIP: None
  selector:
      app: postgres

Create the StatefulSet

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres-sfs
spec:
  selector:
    matchLabels:
      app: postgres
  serviceName: "postgres-svc"
  replicas: 1
  template:
    metadata:
      labels:
        app: postgres
    spec:
      terminationGracePeriodSeconds: 10
      containers:
      - name: postgres-db
        image: postgres:13.2-alpine
        env:
        - name: POSTGRES_PASSWORD
          valueFrom:
            secretKeyRef:
              name: postgres-secrets
              key: POSTGRES_PASSWORD
        - name: PGDATA
          value: /var/lib/postgresql/data/pgdata
        - name: POSTGRES_HOST_AUTH_METHOD
          value: scram-sha-256
        - name: POSTGRES_INITDB_ARGS
          value: --auth-host=scram-sha-256
        ports:
        - containerPort: 5432
          name: postgres-port
        volumeMounts:
        - name: data
          mountPath: /var/lib/postgresql/data
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 10Gi

There are some things to note here.

PGDATA: should point to a subdirectory of the file system mounted with the volume. It is needed to avoid a CrashLoopBackoff error at the start of the pod because Postgres does not support to create its database files at the root of a mount point.

POSTGRES_HOST_AUTH_METHOD and POSTGRES_INITDB_ARGS: these 2 environment variables configure the Postgres instance to use scram-sha-256 for password encryption instead of md5. It is a much more secure algorithm whose only caveat is that it is not supported on all version of the Postgres client. It is supported by Gitea though.

volumeClaimTemplates: the persistent volume configuration happens through volumeClaimTemplate. We cannot use PersistentVolumeClaim in a StatefulSet because it would try to mount the same volume for each pod being part of the StatefulSet. Using volumeClaimTemplates enables the creation of one Persistent Volume for each pod of the StatefulSet.

Apply the StatefulSet YAML deployment file.

After a few seconds, the StatefulSet is created and ready.

Prepare the database for Gitea

Start a bash interactive session in the pod postgres-sfs-0.

1
2
3
4
kubectl exec -it postgres-sfs-0 -- bash

# Connect to Postgres instance as user postgres
psql -U postgres

Run the preparations for Gitea.

Create gitea database user

1
CREATE ROLE gitea WITH LOGIN PASSWORD 'SuperSecretPassword';

Create giteadb database

1
CREATE DATABASE giteadb WITH OWNER gitea TEMPLATE template0 ENCODING UTF8 LC_COLLATE 'en_US.UTF-8' LC_CTYPE 'en_US.UTF-8';

Allow gitea user to access giteadb database -> Not needed as the standard configuration in file /var/lib/postgresql/data/pgdata/pg_hba.conf allows to connect to all database from all hosts using scram-sha-256.

1
2
# Last line of file pg_hba.conf
host all all all scram-sha-256

Deploy Gitea

Create gitea namespace.

1
kubectl create namespace gitea

It is possible to deploy Gitea on Kubernetes using a Helm chart available here.

This Helm chart can deploy multiple components like a Postgres database or a Memcached cache … we won’t need any of these and we will thus deploy the chart with some specific values.

Create a file gitea-values.yaml with the following content.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
image:
  tag: 1.13.6

gitea:
  admin:
    username: <Gitea admin username - cannot be admin>
    password: <Gitea admin password>
    email: <Gitea admin email address>
  
  metrics:
    enabled: true

  database:
    builtIn:
      postgresql:
        enabled: false

  config:
    database:
      DB_TYPE: postgres
      HOST: postgres-svc.postgres.svc.cluster.local:5432
      NAME: giteadb
      USER: gitea
      PASSWD: <password of gitea Postgres user>
    service:
      DISABLE_REGISTRATION: true
      SHOW_REGISTRATION_BUTTON: false
    
  cache:
    builtIn:
      enabled: false

ingress:
  enabled: true
  annotations: 
    cert-manager.io/cluster-issuer: letsencrypt-prod
  hosts:
    - git.example.com
  tls: 
    - secretName: gitea-example-tls
      hosts:
        - git.example.com

Important things to note are:

specify the Gitea image tag because the default value is not up to date.
define Gitea admin username, password and email.
enable Prometheus /metrics endpoint for later integration in Prometheus monitoring.
disable the deployment of the built-in Postgres database because we already deployed ours.
configure the database connection parameters.
disable user registration and the register button. As this instance is exposed publicly, I do not want people to register themselves in my Gitea.
disable installation of built-in Memcached cache.
enable Ingress with cert-manager annotations to generate the TLS certificate automatically. The correct DNS name must of course point to the Kapsule cluster for this to work correctly.

Install the Gitea Helm chart with the customized values.

1
2
3
4
5
# Add the chart repository to Helm
helm repo add gitea-charts https://dl.gitea.io/charts/

# Install the chart - this command can also be used to upgrade the configuration
helm upgrade --install -f gitea-values.yaml gitea gitea-charts/gitea -n gitea

After a minute (some database migrations are running the first time the application is started), the pod is running and the application is accessible through your URL (https://git.example.com).

Note: during my first deployment, I had an issue where the init container could not finish successfully. It always ended in error with a Backoff event. Unfortunately, I could not find any logs at all and the event was not enough to debug the issue. After some investigations, I noticed a typo in my gitea-values.yaml file where I forgot a d in enabled of metrics section … So if you encounter a crash without logs in the init container of your Gitea pod, check the content of your gitea-values.yaml file!

Backups !

Well it’s cool, now we have our Gitea instance using our Postgres database ready to host our repositories … but what about backups ?

Gitea offers a complete backup command with gitea dump. This command should normally be run from inside the Gitea’s container because it needs access to the app.ini file and the files of the git repositories. It generates a zip files that contains a dump of the database and all files of the git repositories. The generated zip file is stored inside the container … which is not of any use …

I could have played with a Kubernetes CronJob running kubectl exec command to run gitea dump from inside the container but then I would generate a zip file in my Gitea container. To be completely safe, we would then need to send that file to an external storage like a BLOB storage and this will need to run another cronjob to mount the volume where the file resides and send it to a BLOB storage … lots of stuff that can potentially go wrong.

~~I thus wanted to use a Kubernetes native backup solution that also integrates with Postgres database to ensure a consistent backup of the database(s). I decided to give a shot at Kasten.~~

Update: as Kasten requires you to register for a license and all that stuff, I eventually chose to implement another solution described in this post.

And this is a really good content for another post ! Stay tuned for the next episode.