Bootstrap a kubernetes cluster with ArgoCD and Istio

In this blog post I’m going to show how to bootstrapp a Kubernetes cluster with ArgoCD and Istio.

In order to understand what’s going on in this blog post, I assume you have a certain familiarity with these projects, as it’s not the purpose of this post explaining what they are, and what they are used for. I would rather focus on how to make these two projects play nicely together, and be able to use them for production-ready kubernetes clusters.

I will leverage the ArgoCD app-of-apps technique, and use both ArgoCD sync weaves and custom healthchecks in order to ensure that apps are installed in the right order, and that resources are ready when needed.

ArgoCD App of Apps
ArgoCD App of Apps

As support to this blog post, I’ve created a github repository called speedwing/eks-argocd-bootstrap with all the charts required to experiment with what discussed in this and other blog posts.

The moving parts

Applications that we will install on our newly created clusters are:

We will then configure an Istio Gateway so to serve ArgoCD on http://argocd.kube (I will show how to serve https in another post).

NOTE: The github project has been updated to use ArgoCD v1.5.5 and Istio 1.6.

Challenges

There are a number of challenges that we have to face when we want to bootstrap a new cluster with ArgoCD, Istio and other services that rely on Istio itself. This happens because there are hard dependencies among applications we want to install. For example the Istio configuration required to serve ArgoCD via the Istio ingress-gateway needs to be installed after the Istio Control Plane has been fully deployed and, in turn, this can only be installed after the Istio Operator CRDs are deployed on kubernetes. Additionally, because the object lifecycle of the Istio Operator is detached from the actual kubernetes resources composing the actual Istio Control Plane, we have to instruct ArgoCD on how it can understand when these resources are ready.

The order in which these manifests have to be installed is:

There are mainly three topics I want to discuss:

  1. How to leverage the ArgoCD app-of-apps and bootstrap the cluster
  2. Use ArgoCD sync weaves to ensure charts are deployed in the right order (i.e. Istio Operator then Istio Control Plane and then everything else)
  3. Implement a custom healthcheck to ensure the Istio Control Plane resources are actually deployed and healthy

How to bootstrap the cluster with the ArgoCD app-of-apps technique

If you’re not familiar with the app-of-apps technique, here below is a quick description borrowed from the ArgoCD official documentation:

You can create an app that creates other apps, which in turn can create other apps. This allows you to declaratively manage a group of app that can be deployed and configured in concert.

Leveraging this approach we could potentially install ArgoCD and a master-app together, and, in turn, the master app will install other charts.

An easy way of achieving this is leveraging kustomize and overlays. The speedwing/eks-argocd-bootstrap github repo contains an app-of-apps example along with all the charts we will use in this tutorial. Let’s look at the folder structure:

.
├── README.md
├── TODO.md
├── applications
│   ├── argocd-istio-app
│   ├── istio
│   ├── master-app
│   ├── master-app-empty
│   └── rollout-canary-app
├── argocd-bootstrap
│   ├── base
│   │   ├── argocd-cm.yaml
│   │   ├── argocd-namespace.yaml
│   │   ├── kustomization.yaml
│   │   └── master-app.yaml
│   └── argocd-istio-bootstrap
│       ├── argocd-server.yaml
│       └── kustomization.yaml
└── blueprint
    └── playground.yaml

The two folders of interests are: argocd-bootstrap and applications.

The argocd-bootstrap folder contains kustomize charts used to install ArgoCD and the master-app, plus other charts that are outside the scope of this blogpost. The applications folder contains the master-app itself plus other charts.

Let’s dive a bit deeper into the kustomize resources.

The first chart I want to discuss is the argocd-cm.yaml that should look something like:

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
data:
  repositories: |
    - url: https://github.com/helm/charts.git
    - url: https://github.com/speedwing/eks-argocd-bootstrap.git
    - url: https://github.com/argoproj/argo-rollouts.git
    - url: https://github.com/istio/istio.git
  [...]

In this ConfigMap it’s important to add the repositories you want ArgoCD to access. For the purposes of this tutorial the two relevant ones are:

More info on how to add other types of repositories, like helm repositories, or private github repos is available here here.

We are now ready to bootstrap a kubernetes cluster, to do so, simply issue from the root folder of the repo:

kubectl apply -k argocd-bootstrap/argocd-istio-bootstrap

This operation will take a few minutes, in the meantime let’s take a look at how ArgoCD waves and custom healthchecks are helping us to get the job done.

NOTE: If you get an error like: error: unable to recognize "argocd-bootstrap/argocd-istio-bootstrap": no matches for kind "Application" in version "argoproj.io/v1alpha1” just run the command again, this happens because you’ve just installed the CRD Application and trying to install an instance of it.

ArgoCD sync waves

ArgoCD offers two different approaches to re-order manifests synchronization: phases and waves.

The tl;dr about waves is that you can order how kubernetes manifests are synchronized via an annotation. Waves can assume both positive and negative values, if no wave is specified the wave zero is assigned by default.

Let’s see now how we can leverage waves to install Istio Operator first and Istio Control Plane after and then everything else.

Here is the ArgoCD Application for installing the Istio Operator chart (source: applications/master-app/templates/istio-operator.yaml):

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: istio-operator
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "-2"
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: default
  source:
    repoURL: https://github.com/istio/istio.git
    targetRevision: "1.5.1"
    path: operator/data/operator-chart

    helm:
      parameters:
        - name: "hub"
          value: "docker.io/istio"
        - name: "tag"
          value: "1.5.1"
        - name: "operatorNamespace"
          value: "istio-operator"
        - name: "istioNamespace"
          value: "istio-system"

  destination:
    namespace: istio-operator
    server: {{ .Values.spec.destination.server }}

  syncPolicy:
    automated:
      prune: true

The ArgoCD Application that installs the Istio Control Plane is located in the file applications/master-app/templates/istio.yaml, and its content is:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: istio
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "-1"
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: default
  source:
    repoURL: https://github.com/speedwing/eks-argocd-bootstrap.git
    targetRevision: HEAD
    path: applications/istio

  destination:
    namespace: istio-system
    server: {{ .Values.spec.destination.server }}

  syncPolicy:
    automated:
      prune: true

As you can notice the Istio Oprator has been assigned argocd.argoproj.io/sync-wave: "-2", while the Istio Control Plane chart has argocd.argoproj.io/sync-wave: "-1". This is as simple and as powerful as it looks. The Operator will be installed first, as it has the lowest wave value. ArgoCD will wait for it to be deployed and healthy before proceeding to the next wave, the -1, where the Istio Control Plane is bound to. Same applies to it and, once it’s ready everything else is installed.

As mentioned above thou, the lifecycle of the Istio Control Plane kubernetes object is detached from the acutal Istio resources, and hence it will appear deployed and helthy as soon as the manifest is installed on kubernetes (and not when the actual Istio resources are created!). We do need to tweak one more thing to ensure that we give time to the Istio Operator controller to detect a new Control Plane manifest was installed, and to create all the relevant resources. This can be achieved via custom healthchecks

Istio Control Plane custom healthchecking

We are at the last step of our tutorial, we have all the charts ready, the waves configured, but ArgoCD doesn’t know it has to wait for the Istio Operator controller to create all the Istio goodness. Luckily the ArgoCD team has designed a simple but yet effective way to implement custom healthchecks.

If you’ve bootstrapped your cluster with the instructions provided in this tutorial, and issue the following command:

kubectl get istiooperator -n istio-system -o yaml

you will get something like:

apiVersion: v1
items:
- apiVersion: install.istio.io/v1alpha1
  kind: IstioOperator
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"install.istio.io/v1alpha1","kind":"IstioOperator","metadata":{"annotations":{},"labels":{"app.kubernetes.io/instance":"istio"},"name":"istio-control-plane","namespace":"istio-system"},"spec":{"addonComponents":{"prometheus":{"enabled":false}},"profile":"default","values":{"global":{"k8sIngress":{"enabled":false}}}}}
    creationTimestamp: "2020-03-27T13:12:01Z"
    finalizers:
    - istio-finalizer.install.istio.io
    generation: 2
    labels:
      app.kubernetes.io/instance: istio
    name: istio-control-plane
    namespace: istio-system
    resourceVersion: "2953"
    selfLink: /apis/install.istio.io/v1alpha1/namespaces/istio-system/istiooperators/istio-control-plane
    uid: 428a7f29-540f-49e9-9089-302ca17dc9df
  spec:
    addonComponents:
      prometheus:
        enabled: false
    profile: default
    values:
      global:
        k8sIngress:
          enabled: false
  status:
    componentStatus:
      Base:
        status: HEALTHY
      IngressGateways:
        status: HEALTHY
      Pilot:
        status: HEALTHY
    status: HEALTHY
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

The section of interest is the status:

  status:
    componentStatus:
      Base:
        status: HEALTHY
      IngressGateways:
        status: HEALTHY
      Pilot:
        status: HEALTHY
    status: HEALTHY

Luckily the Istio Operator controller will update the status section of this object by adding a status field as it performs its job. The status field is, at creation time, empty and will be set to HEALTHY once the Istio Control Plane resources have been created.

We can now leverage this information and implement a custom healthcheck in the same ConfigMap we’ve seen previously.

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
data:
  repositories: |
    - url: https://github.com/helm/charts.git
    - url: https://github.com/speedwing/eks-argocd-bootstrap.git
    - url: https://github.com/argoproj/argo-rollouts.git
    - url: https://github.com/istio/istio.git
  resource.customizations: |
    admissionregistration.k8s.io/MutatingWebhookConfiguration:
      ignoreDifferences: |
        jsonPointers:
        - /webhooks/0/clientConfig/caBundle
    install.istio.io/IstioOperator:
      health.lua: |
        hs = {}
        if obj.status ~= nil then
          if obj.status.status == "HEALTHY" then
            hs.status = "Healthy"
            hs.message = "IstioOperator Ready"
            return hs
          end
        end

        hs.status = "Progressing"
        hs.message = "Waiting for IstioOperator"
        return hs

Under the resource.customizations section you can notice the install.istio.io/IstioOperator entry where a lua file has been created for the health.lua script. This is the script that is periodically ran by ArgoCD to check the health status for kubernetes objects of type: install.istio.io/IstioOperator. The script is very simple and only check for the field status.status to be present and have value HEALTHY. The object will be marked as Progressing otherwise.

Conclusion

Now that we know what’s going on, it’s time to check that what we’ve installed is actually working.

To do so, we will need to recur to a little trick so that we can easily test our solution.

Edit the /etc/hosts file so to have something like:

##
# Host Database
#
# localhost is used to configure the loopback interface
# when the system is booting.  Do not change this entry.
##

[...]
127.0.0.1 argocd.kube
[...]

Then try to access in your browser http://argocd.kube and :fingers-crossed: it should open the login page of ArgoCD

That’s it for today. Hope you found this article useful. Good luck and feel free to share or open an issue on the github repo.

comments powered by Disqus