Testing Calico on AKS With Azure CNI

In this article I will run some basic tests in an Azure Kubernetes Services cluster configured with Azure CNI network plugin and Calico.

Background

By default, all network traffic is allowed between pods of a Kubernetes cluster. Pods can reach services everywhere in the cluster, even in other namespace.

What I’d like to test today is the possibility to isolate namespaces so that pods cannot connect from one namespace to another. We will try to do so with Calico network policies.

Prerequisites

An Azure account and subscription.
An AKS cluster with Azure CNI and Calico deployed (this can only happen at deployment so be carefull).
kubectl and calicoctl available on your workstation.
some spare time to test :)

Create 2 namespaces

Create the development and test namespaces in your cluster.

kubectl create namespace development
kubectl create namespace test

Add some labels on the namespaces. Labels will be used as selectors in the network policies we will deploy later on.

kubectl label namespace development env=dev
kubectl label namespace test env=test

Deploy whoami container in development

Use the following manifest to deploy whoami application and service in development namespace.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: whoami-deployment
spec:
  selector:
    matchLabels:
      app: whoami
  replicas: 1
  template:
    metadata:
      labels:
        app: whoami
    spec:
      containers:
      - name: whoami-container
        image: containous/whoami
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: whoami-service
spec:
  ports:
  - name: http
    targetPort: 80
    port: 80
  selector:
    app: whoami

Deploy with kubectl in development namespace.

kubectl -n development apply -f whoami-deployment.yml

Check the status of the service.

kubectl -n development get service
NAME             TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)   AGE
whoami-service   ClusterIP   10.10.10.18   <none>        80/TCP    6m21s

Test the service / pod deployment using kubectl port-forward

kubectl port-forward -n development service/whoami-service 8000:80
Forwarding from 127.0.0.1:8000 -> 80
Forwarding from [::1]:8000 -> 80

PS: The kubectl port-forward command does not return. It must stay in executing status for the port-forwarding to continue working. Cancel it with CTRL-C.

Access service by opening a web browser to 127.0.0.1:8000

Hostname: whoami-deployment-5d4bfbc5d7-cph77
IP: 127.0.0.1
IP: 10.240.0.11
RemoteAddr: 127.0.0.1:49890
GET / HTTP/1.1
Host: 127.0.0.1:8000
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.141 Safari/537.36 Edg/87.0.664.75
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
Accept-Encoding: gzip, deflate, br
Accept-Language: en,fr;q=0.9,fr-FR;q=0.8,en-GB;q=0.7,en-US;q=0.6,nl;q=0.5,de;q=0.4
Connection: keep-alive
Sec-Fetch-Dest: document
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: none
Sec-Fetch-User: ?1
Upgrade-Insecure-Requests: 1

Now that we know the container runs correctly, let’s spawn a new container to test from inside the cluster, in the same namespace.

kubectl run --rm -it --image=alpine testtool --namespace development
wget -qO- --timeout=2 http://whoami-service.development
Hostname: whoami-deployment-5d4bfbc5d7-cph77
IP: 127.0.0.1
IP: 10.240.0.11
RemoteAddr: 10.240.0.32:39446
GET / HTTP/1.1
Host: whoami-service.development
User-Agent: Wget
Connection: close

The connection works.

Let’s try to also reach the service from the test namespace.

kubectl run --rm -it --image=alpine testtool --namespace test
wget -qO- --timeout=2 http://whoami-service.development
Hostname: whoami-deployment-5d4bfbc5d7-cph77
IP: 127.0.0.1
IP: 10.240.0.11
RemoteAddr: 10.240.0.32:39446
GET / HTTP/1.1
Host: whoami-service.development
User-Agent: Wget
Connection: close

It also works.

Our start situation is thus:

fig1 - everything is possible

Now let’s try to apply some basic Calico network policies.

About Kubernetes and Calico network policies

Disclaimer: I’m far from being a master at Kubernetes networking stuff and what follows is only my current understanding of the subject

When Calico is deployed in a Kubernetes cluster, you can define 2 kinds of network policies: Kubernetes network policies and Calico network policies.

There are some differences between the two kinds (for more info see An overview of Kubernetes and Calico network policies) but you can define and apply both types of policies together. The order of precedence to apply these policies is defined by a spec of the network policy object named order. Kubernetes network policies have a default order of 1000 and it cannot be specified / modified in the policy definition. Calico policies allow to specify an order in their definition. You should correctly define it to avoid wrong order of precedence if you combine both types of policies.

To avoid confusion, we will only work with Calico network policies in this article.

Another thing that you must have in mind is how the deny / allow traffic behaves.

If no network policies apply to a pod, then all traffic to/from that pod is allowed.
If one or more network policies apply to a pod containing ingress rules, then only the ingress traffic specifically allowed by those policies is allowed.
If one or more network policies apply to a pod containing egress rules, then only the egress traffic specifically allowed by those policies is allowed.

For compatibility with Kubernetes, Calico network policy follows the same behavior for Kubernetes pods. For other endpoint types (VMs, host interfaces), Calico network policy is default deny. We will only configure network policies on pods.

Applying and testing policies

Network policies can make things complex so we will stick to some simple rules to bring more security in our cluster without too much complexity.

A good first step would be to deny everything in the development namespace. This policy will be the last one applied.

Create a Calico policy yaml file with the following content:

# Calico network policy that denies all traffic in development namespace
# Denies even traffic between pods in the same namespace !
apiVersion: projectcalico.org/v3
kind: NetworkPolicy
metadata:
  name: default-deny
  namespace: development
spec:
  order: 5000
# Select all endpoints in the namespace
  selector: all()
  types:
  - Ingress
# Default action if not specified is deny
  - Egress
# Default action if not specified is deny

Apply the policy

calicoctl apply -f 01-deny-all.yaml

Now the test from development namespace does not work anymore.

kubectl run --rm -it --image=alpine testtool --namespace development
wget -qO- --timeout=2 http://whoami-service.development
wget: bad address 'whoami-service.development'

fig2 - deny all

This is normal because all traffic is denied by our policy, even the traffic to kube-dns for name resolution inside the cluster. To solve that, let’s create and apply a new policy that will allow UDP egress traffic to kube-dns pods in namespace kube-system.

Policy content :

apiVersion: projectcalico.org/v3
kind: NetworkPolicy
metadata:
  name: dns-allow
  namespace: development
spec:
  order: 4000
# Applies to all endpoints in the namespace
  selector: all()
  types:
  - Egress
  
  egress:
  - action: Allow
    protocol: UDP
    destination:
# I tried to use projectcalico.org/name to select the namespace name but it did not work ...
      namespaceSelector: control-plane == "true"
      selector: 'k8s-app == "kube-dns"'
      ports:
      - 53

Apply:

calicoctl apply -f 02-allow-dns-egress.yaml

Test

kubectl run --rm -it --image=alpine testtool -n development
wget -qO- --timeout=2 http://whoami-service.development
wget: download timed out

So now name resolution is ok but connectivity inside the namespace is still not ok.

Let’s fix this by creating a new rule allowing all ingress and egress traffic from pods running inside development namespace (remember that our default rule denies all traffic!).

# Calico network policy that allows ingress and egress traffic to all pods in development namespace
# from all pods in the SAME namespace
apiVersion: projectcalico.org/v3
kind: NetworkPolicy
metadata:
  name: development-ns-allow
  namespace: development
spec:
  order: 500
# Applies to all endpoints in the namespace
  selector: all()
  types:
  - Ingress
  - Egress

  ingress:  
  - action: Allow
    protocol: TCP
    source:
      namespaceSelector: env == "dev"
  
  egress:
  - action: Allow
    protocol: TCP
    destination:
      namespaceSelector: env == "dev"

Test from development namespace

kubectl run --rm -it --image=alpine testtool -n development
wget -qO- --timeout=2 http://whoami-service.development
Hostname: whoami-deployment-5d4bfbc5d7-cph77
IP: 127.0.0.1
IP: 10.240.0.11
RemoteAddr: 10.240.0.17:41100
GET / HTTP/1.1
Host: whoami-service.development
User-Agent: Wget
Connection: close

Finally it works as expected.

Now test from outside the namespace

# From test namespace
kubectl run --rm -it --image=alpine testtool -n test
wget -qO- --timeout=2 http://whoami-service.development
wget: download timed out

# From default namespace
kubectl run --rm -it --image=alpine testtool
wget -qO- --timeout=2 http://whoami-service.development
wget: download timed out

Both tests from outside are failing with timeouts which is correct due to our current configuration.

fig3 - allow DNS and same namespace network traffic

Let’s run a last test. Add a new policy to allow ingress traffic also from the test namespace.

# Calico network policy that allows ingress traffic to all pods in development namespace
# from all pods in the test namespace
apiVersion: projectcalico.org/v3
kind: NetworkPolicy
metadata:
  name: test-ns-allow
  namespace: development
spec:
  order: 600
# Applies to all endpoints in the development namespace
  selector: all()
  types:
  - Ingress

  ingress:  
  - action: Allow
    protocol: TCP
    source:
      namespaceSelector: env == "test"

Apply the policy.

calicoctl apply -f 04-allow-ingress-from-test.yaml

Test from development namespace -> Still OK.

kubectl run --rm -it --image=alpine testtool -n development
wget -qO- --timeout=2 http://whoami-service.development
Hostname: whoami-deployment-5d4bfbc5d7-cph77
IP: 127.0.0.1
IP: 10.240.0.11
RemoteAddr: 10.240.0.18:42046
GET / HTTP/1.1
Host: whoami-service.development
User-Agent: Wget
Connection: close

Test from test namespace -> Now OK too.

kubectl run --rm -it --image=alpine testtool -n test
wget -qO- --timeout=2 http://whoami-service.development
Hostname: whoami-deployment-5d4bfbc5d7-cph77
IP: 127.0.0.1
IP: 10.240.0.11
RemoteAddr: 10.240.0.18:42046
GET / HTTP/1.1
Host: whoami-service.development
User-Agent: Wget
Connection: close

Test from default namespace -> Still fails but that is expected.

kubectl run --rm -it --image=alpine testtool
wget -qO- --timeout=2 http://whoami-service.development
wget: download timed out

The final situation is:

fig4 - allow network traffic from test namespace

Check the policies existing in the namespace.

calicoctl get networkpolicies -n development
NAMESPACE     NAME                   
development   default-deny           
development   development-ns-allow   
development   dns-allow              
development   test-ns-allow

Cleanup resources and conclusion

Now that the tests are finished, delete all resources.

kubectl delete namespace test
kubectl delete namespace development

As you have read (I hope you survived …) in this rather long article, Calico network policies allow us to isolate network traffic inside a namespace and still extend to traffic coming from other namespaces.

This gives us the possibility to secure the network flows inside our Kubernetes cluster and make our security engineers happier.

See you soon.