Skip to content

Tutorial (gen1)

This assumes that you plan to deploy workload clusters on AWS cloud, with Cluster API ("CAPI") as the cluster orchestration API provider. Also ensure you have set up a workspace repository, and it is registered as a git repo in ArgoCD. The tutorial will assume the existence of these environment variables:

  • ${ARLON_REPO}: where the arlon repo is locally checked out
  • ${WORKSPACE_REPO}: where the workspace repo is locally checked out
  • ${WORKSPACE_REPO_URL}: the workspace repo's git URL. It typically looks like https://github.com/${username}/${reponame}.git
  • ${CLOUD_REGION}: the region where you want to deploy example clusters and workloads (e.g. us-west-2)
  • ${SSH_KEY_NAME}: the name of a public ssh key name registered in your cloud account, to enable ssh to your cluster nodes

Additionally, for examples assuming arlon git register, "default" and a "prod" git repo aliases will also be given.

_Note: for the best experience, make sure your workspace repo is configured to send change notifications to ArgoCD via a webhook. See the Installation section for details.

Cluster specs

We first create a few cluster specs with different combinations of API providers and cluster types (kubeadm vs EKS). One of the cluster specs is for an unconfigured API provider (Crossplane); this is for illustrative purposes, since we will not use it in this tutorial.

arlon clusterspec create capi-kubeadm-3node --api capi --cloud aws --type kubeadm --kubeversion v1.21.10 --nodecount 3 --nodetype t2.medium --tags devel,test --desc "3 node kubeadm for dev/test" --region ${CLOUD_REGION} --sshkey ${SSH_KEY_NAME}
arlon clusterspec create capi-eks --api capi --cloud aws --type eks --kubeversion v1.21.10 --nodecount 2 --nodetype t2.large --tags staging --desc "2 node eks for general purpose" --region ${CLOUD_REGION} --sshkey ${SSH_KEY_NAME}
arlon clusterspec create xplane-eks-3node --api xplane --cloud aws --type eks --kubeversion v1.21.10 --nodecount 4 --nodetype t2.small --tags experimental --desc "4 node eks managed by crossplane" --region ${CLOUD_REGION} --sshkey ${SSH_KEY_NAME}

Ensure you can now list the cluster specs:

$ arlon clusterspec list
NAME                APIPROV  CLOUDPROV  TYPE     KUBEVERSION  NODETYPE   NODECNT  MSTNODECNT  SSHKEY  CAS    CASMIN  CASMAX  TAGS          DESCRIPTION
capi-eks            capi     aws        eks      v1.21.10     t2.large   2        3           leb     false  1       9       staging       2 node eks for general purpose
capi-kubeadm-3node  capi     aws        kubeadm  v1.21.10     t2.medium  3        3           leb     false  1       9       devel,test    3 node kubeadm for dev/test
xplane-eks-3node    xplane   aws        eks      v1.21.10     t2.small   4        3           leb     false  1       9       experimental  4 node eks managed by crossplane

Bundles

First create a static bundle containing raw YAML for the guestbook sample application from this example file:

cd ${ARLON_REPO}
arlon bundle create guestbook-static --tags applications --desc "guestbook app" --from-file examples/bundles/guestbook.yaml

(Note: the YAML is simply a concatenation of the files found in the ArgoCD Example Apps repo)

To illustrate the difference between static and dynamic bundles, we create a dynamic version of the same application, this time using a reference to a git directory containing the YAML. We could point it directly to the copy in the ArgoCD Example Apps repo, but we'll want to make modifications to it, so we instead create a new directory to host our own copy in our workspace directory:

cd ${WORKSPACE_REPO}
mkdir -p bundles/guestbook
cp ${ARLON_REPO}/examples/bundles/guestbook.yaml bundles/guestbook
git add bundles/guestbook
git commit -m "add guestbook"
git push origin main

arlon bundle create guestbook-dynamic --tags applications --desc "guestbook app (dynamic)" --repo-url ${WORKSPACE_REPO_URL} --repo-path bundles/guestbook
            # OR
# using repository aliases
  # using the default alias
arlon bundle create guestbook-dynamic --tags applications --desc "guestbook app (dynamic)" --repo-path bundles/guestbook
  # using the prod alias
arlon bundle create guestbook-dynamic --tags applications --desc "guestbook app (dynamic)" --repo-path bundles/guestbook --repo-alias prod

Next, we create a static bundle for another "dummy" application, an Ubuntu pod (OS version: "Xenial") that does nothing but print the date-time in an infinite sleep loop:

cd ${ARLON_REPO}
arlon bundle create xenial-static --tags applications --desc "xenial pod" --from-file examples/bundles/xenial.yaml

Finally, we create a bundle for the Calico CNI, which provides pod networking. Some types of clusters (e.g. kubeadm) require a CNI provider to be installed onto a newly created cluster, so encapsulating the provider as a bundle will give us a flexible way to install it. We download a known copy from the authoritative source and store it the workspace repo in order to create a dynamic bundle from it:

cd ${WORKSPACE_REPO}
mkdir -p bundles/calico
curl https://docs.projectcalico.org/v3.21/manifests/calico.yaml -o bundles/calico/calico.yaml
git add bundles/calico
git commit -m "add calico"
git push origin main

arlon bundle create calico --tags networking,cni --desc "Calico CNI" --repo-url ${WORKSPACE_REPO_URL} --repo-path bundles/calico
            # OR
# using repository aliases
  # using the default alias
arlon bundle create calico --tags networking,cni --desc "Calico CNI" --repo-path bundles/calico
  # using the prod alias
arlon bundle create calico --tags networking,cni --desc "Calico CNI" --repo-path bundles/calico --repo-alias prod

List your bundles to verify they were correctly entered:

$ arlon bundle list
NAME               TYPE     TAGS                 REPO-URL                                             REPO-PATH              DESCRIPTION
calico             dynamic  networking,cni       ${WORKSPACE_REPO_URL}                                bundles/calico         Calico CNI
guestbook-dynamic  dynamic  applications         ${WORKSPACE_REPO_URL}                                bundles/guestbook      guestbook app (dynamic)
guestbook-static   static   applications         (N/A)                                                (N/A)                  guestbook app
xenial-static      static   applications         (N/A)                                                (N/A)                  ubuntu pod in infinite sleep loop

Profiles

We can now create profiles to group bundles into useful, deployable sets. First, create a static profile containing bundles xenial-static and guestbook-static:

arlon profile create static-1 --static --bundles guestbook-static,xenial-static --desc "static profile 1" --tags examples

Secondly, create a dynamic version of the same profile. We'll store the compiled form of the profile in the profiles/dynamic-1 directory of the workspace repo. We don't create it manually; instead, the arlon CLI will create it for us, and it will push the change to git:

arlon profile create dynamic-1 --repo-url ${WORKSPACE_REPO_URL} --repo-base-path profiles --bundles guestbook-static,xenial-static --desc "dynamic test 1" --tags examples
            # OR
# using repository aliases
  # using the default alias
arlon profile create dynamic-1 --repo-base-path profiles --bundles guestbook-static,xenial-static --desc "dynamic test 1" --tags examples
  # using the prod alias
arlon profile create dynamic-1 --repo-alias prod --repo-base-path profiles --bundles guestbook-static,xenial-static --desc "dynamic test 1" --tags examples

Note: the --repo-base-path profiles option tells arlon to create the profile under a base directory profiles/ (to be created if it doesn't exist). That is in fact the default value of that option, so it is not necessary to specify it in this case.

To verify that the compiled profile was created correctly:

$ cd ${WORKSPACE_REPO}
$ git pull
$ tree profiles
profiles
├── dynamic-1
│   ├── mgmt
│   │   ├── Chart.yaml
│   │   └── templates
│   │       ├── guestbook-dynamic.yaml
│   │       ├── placeholder_configmap.yaml
│   │       └── xenial.yaml
│   └── workload
│       └── xenial
│           └── xenial.yaml
[...]

Since xenial is a static bundle, a copy of its YAML was stored in workload/xenial/xenial.yaml. This is not done for guestbook-dynamic because it is dynamic.

Finally, we create another variant of the same profile, with the only difference being the addition of Calico bundle. It'll be used on clusters that need a CNI provider:

arlon profile create dynamic-2-calico --repo-url ${WORKSPACE_REPO_URL} --repo-base-path profiles --bundles calico,guestbook-dynamic,xenial-static --desc "dynamic test 1" --tags examples
            # OR
# using repository aliases
  # using the default alias
arlon profile create dynamic-2-calico --repo-base-path profiles --bundles calico,guestbook-dynamic,xenial-static --desc "dynamic test 1" --tags examples
  # using the prod alias
arlon profile create dynamic-2-calico --repo-alias prod --repo-base-path profiles --bundles calico,guestbook-dynamic,xenial-static --desc "dynamic test 1" --tags examples

Listing the profiles should show:

$ arlon profile list
NAME              TYPE     BUNDLES                                 REPO-URL               REPO-PATH                  TAGS         DESCRIPTION
dynamic-1         dynamic  guestbook-static,xenial-static          ${WORKSPACE_REPO_URL}  profiles/dynamic-1         examples     dynamic test 1
dynamic-2-calico  dynamic  calico,guestbook-static,xenial-static   ${WORKSPACE_REPO_URL}  profiles/dynamic-2-calico  examples     dynamic test 1
static-1          static   guestbook-dynamic,xenial-static         (N/A)                  (N/A)                      examples     static profile 1

Clusters (gen1)

We are now ready to deploy our first cluster. It will be of type EKS. Since EKS clusters come configured with pod networking out of the box, we choose a profile that does not include Calico: dynamic-1. When deploying a cluster, arlon creates in git a Helm chart containing the manifests for creating and bootstrapping the cluster. Arlon then creates an ArgoCD App referencing the chart, thereby relying on ArgoCD to orchestrate the whole process of deploying and configuring the cluster. The arlon deploy command accepts a git URL and path for this git location. Any git repo can be used (so long as it's registered with ArgoCD), but we'll use the workspace cluster for convenience:

arlon cluster deploy --repo-url ${WORKSPACE_REPO_URL} --cluster-name eks-1 --profile dynamic-1 --cluster-spec capi-eks
            # OR
# using repository aliases
  # using the default alias
arlon cluster deploy --cluster-name eks-1 --profile dynamic-1 --cluster-spec capi-eks
  # using the prod alias
arlon cluster deploy --repo-alias prod --cluster-name eks-1 --profile dynamic-1 --cluster-spec capi-eks

The git directory hosting the cluster Helm chart is created as a subdirectory of a base path in the repo. The base path can be specified with --base-path, but we'll leave it unspecified in order to use the default value of clusters. Consequently, this example produces the directory clusters/eks-1/ in the repo. To verify its presence:

$ cd ${WORKSPACE_REPO}
$ git pull
$ tree clusters/eks-1
clusters/eks-1
└── mgmt
    ├── charts
    │   ├── capi-aws-eks
    │   │   ├── Chart.yaml
    │   │   └── templates
    │   │       └── cluster.yaml
    │   ├── capi-aws-kubeadm
    │   │   ├── Chart.yaml
    │   │   └── templates
    │   │       └── cluster.yaml
    │   └── xplane-aws-eks
    │       ├── Chart.yaml
    │       └── templates
    │           ├── cluster.yaml
    │           └── network.yaml
    ├── Chart.yaml
    ├── templates
    │   ├── clusterregistration.yaml
    │   ├── ns.yaml
    │   ├── profile.yaml
    │   └── rbac.yaml
    └── values.yaml

The chart contains several subcharts under mgmt/charts/, one for each supported type of cluster. Only one of them will be enabled, in this case capi-aws-eks (Cluster API on AWS with type EKS).

At this point, the cluster is provisioning and can be seen in arlon and AWS EKS:

$ arlon cluster list
NAME       CLUSTERSPEC  PROFILE  
eks-1      capi-eks     dynamic-1

$ aws eks list-clusters
{
    "clusters": [
        "eks-1_eks-1-control-plane",
    ]
}

Eventually, it will also be seen as a registered cluster in argocd, but this won't be visible for a while, because the cluster is not registered until its control plane (the Kubernetes API) is ready:

$ argocd cluster list
SERVER                                                                    NAME        VERSION  STATUS      MESSAGE
https://9F07DC211252C6F7686F90FA5B8B8447.gr7.us-west-2.eks.amazonaws.com  eks-1       1.18+    Successful  
https://kubernetes.default.svc                                            in-cluster  1.20+    Successful

To monitor the progress of the cluster deployment, check the status of the ArgoCD app of the same name:

$ argocd app list
NAME                         CLUSTER                         NAMESPACE  PROJECT  STATUS  HEALTH   SYNCPOLICY  CONDITIONS  REPO                   PATH                                          TARGET
eks-1                        https://kubernetes.default.svc  default    default  Synced  Healthy  Auto-Prune  <none>      ${WORKSPACE_REPO_URL}  clusters/eks-1/mgmt                           main
eks-1-guestbook-static                                       default    default  Synced  Healthy  Auto-Prune  <none>      ${WORKSPACE_REPO_URL}  profiles/dynamic-1/workload/guestbook-static  HEAD
eks-1-profile-dynamic-1      https://kubernetes.default.svc  argocd     default  Synced  Healthy  Auto-Prune  <none>      ${WORKSPACE_REPO_URL}  profiles/dynamic-1/mgmt                       HEAD
eks-1-xenial                                                 default    default  Synced  Healthy  Auto-Prune  <none>      ${WORKSPACE_REPO_URL}  profiles/dynamic-1/workload/xenial            HEAD

The top-level app eks-1 is the root of all argocd apps that make up the cluster and its configuration contents. The next level app eks-1-profile-dynamic-1 represents the profile, and its children apps eks-1-guestbook-static and eks-1-xenial correspond to the bundles.

Note: The overall tree-like organization of the apps and their health status can be visually observed in the ArgoCD web user interface._

The cluster is fully deployed once those apps are all Synced and Healthy. An EKS cluster typically takes 10-15 minutes to finish deploying.

Behavioral differences between static and dynamic bundles & profiles

Static bundle

A change to a static bundle does not affect existing clusters using that bundle (through a profile). To illustrate this, bring up the ArgoCD UI and open the detailed view of the eks-1-guestbook-static application, which applies the guestbook-static bundle to the eks-1 cluster. Note that there is only one guestbook-ui pod.

Next, update the guestbook-static bundle to have 3 replicas of the pod:

arlon bundle update guestbook-static --from-file examples/bundles/guestbook-3replicas.yaml

Note that the UI continues to show one pod. Only new clusters consuming this bundle will have the 3 replicas.

Dynamic profile

Before discussing dynamic bundles, we take a small detour to introduce dynamic profiles, since this will help understand the relationship between profiles and bundles. To illustrate how a profile can be updated, we remove guestbook-static bundle from dynamic-1 by specifying a new bundle set:

arlon profile update dynamic-1 --bundles xenial

Since the old bundle set was guestbook-static,xenial-static, that command resulted in the removal of guestbook-static from the profile. In the UI, observe the eks-1-profile-dynamic-1 app going through Sync and Progressing phases, eventually reaching the healthy (green) state. And most importantly, the eks-1-guestbook-static app is gone. The reason this real-time change to the cluster was possible is that the dynamic-1 profile is dynamic, meaning any change to its composition results in arlon updating the corresponding compiled Helm chart in git. ArgoCD detects this git change and propagates the app / configuration updates to the cluster.

If the profile were of the static type, a change in its composition (the set of bundles) would not have affected existing clusters using that profile. It would only affect new clusters created with the profile.

Dynamic bundle

To illustrate the defining characteristic of a dynamic bundle, we first add guestbook-dynamic to dynamic-1:

arlon profile update dynamic-1 --bundles xenial,guestbook-dynamic

Observe the re-appearance of the guestbook application, which is managed by the eks-1-guestbook-dynamic ArgoCD app. A detailed view of the app shows 1 guestbook-ui pod. Remember that a dynamic bundle's manifest content is stored in git. Use these commands to change the number of pod replicas to 3:

cd ${WORKSPACE_REPO}
git pull # to get all latest changes pushed by arlon
vim bundles/guestbook/guestbook.yaml # edit to change deployment's replicas to 3
git commit -am "increase guestbook replicas"
git push origin main

Observe the number of pods increasing to 3 in the UI. Any existing cluster consuming this dynamic bundle will be updated similarly, regardless of whether the bundle is consumed via a dynamic or static profile.

Static profile

Finally, a profile can be static. It means that it has no corresponding "compiled" component (a Helm chart) living in git. When a cluster is deployed using a static profile, the set of bundles (whether static or dynamic) it receives is determined by the bundle set defined by the profile at deployment time, and will not change in the future, even if the profile is updated to a new set at a later time.

Cluster updates and upgrades

The arlon cluster update [flags] command allows you to make changes to an existing cluster. The clusterspec, profile, or both can change, provided that the following rules and guidelines are followed.

Clusterspec

There are two scenarios. In the first, the clusterspec name associated with the cluster hasn't changed, meaning the cluster is using the same clusterspec. However, some properties of the clusterspec's properties have changed since the cluster was deployed or last updated, using arlon clusterspec update Arlon supports updating the cluster to use updated values of the following properties:

  • kubernetesVersion
  • nodeCount
  • nodeType

Note: Updating the cluster is not allowed if other properties of its clusterspec (e.g. cluster orchestration API provider, cloud, cluster type, region, pod CIDR block, etc...) have changed, however new clusters can always be created/deployed using the changed clusterspec.

A change in kubernetesVersion will result in a cluster upgrade/downgrade. There are some restrictions and caveats you need to be aware of:

  • The specific Kubernetes version must be supported by the particular implementation and release of the underlying cluster orchestration API provider cloud, and cluster type.
  • In general, the control plane will be upgraded first
  • Existing nodes are not typically not upgraded to the new Kubernetes version. Only new nodes (added as part of manual nodeCount change or autoscaling)

In the second scenario, as part of an update operation, you may choose to associate the cluster with a different clusterspec altogether. The rule governing the allowed property changes remains the same: the cluster update operation is allowed if, relative to the previously associated clusterspec, the new clusterspec's properties differ only in the values listed above.

Profile

You can specify a completely different profile when updating a cluster. All bundles previously used will be removed from the cluster, and new ones specified by the new profile will be applied. This is regardless of whether the old and new profiles are static or dynamic.

Examples

These sequence of commands updates a clusterspec to a newer Kubernetes version and a higher node count, then upgrades the cluster to the newer specifications:

arlon clusterspec update capi-eks --nodecount 3 --kubeversion v1.19.15
arlon cluster update eks-1

Note that the 2nd command didn't need any flags because the clusterspec used is the same as before.

This example updates a cluster to use a new profile my-new-profile:

arlon cluster update eks-1 --profile my-new-profile

Enabling Cluster Autoscaler in the workload cluster

Bundle creation

Register a dynamic bundle pointing to the bundles/capi-cluster-autoscaler in the Arlon repo.

The capi-cluster-autoscaler bundle requires the name of the cluster, so that it knows what namespace in the management cluster to scan for CAPI resources. To enable the cluster-autoscaler bundle, add one more parameter during cluster creation: srcType. This is the ArgoCD-defined application source type (Helm, Kustomize, Directory).

This example creates a bundle pointing to the bundles/capi-cluster-autoscaler in Arlon repo

arlon bundle create cas-bundle --tags cas,devel,test --desc "CAS Bundle" --repo-url https://github.com/arlonproj/arlon.git --repo-path bundles/capi-cluster-autoscaler --srctype helm

Profile creation

Create a profile that contains this capi-cluster-autoscaler bundle.

arlon profile create dynamic-cas --repo-url ${WORKSPACE_REPO_URL} --repo-base-path profiles --bundles cas-bundle --desc "dynamic cas profile" --tags examples

Clusterspec creation

Create a clusterspec with CAPI as ApiProvider and autoscaling enabled.In addition to this, the ClusterAutoscaler(Min|Max)Nodes properties are used to set 2 annotations on MachineDeployment required by the cluster autoscaler for CAPI.

arlon clusterspec create cas-spec --api capi --cloud aws --type eks --kubeversion v1.21.10 --nodecount 2 --nodetype t2.medium --tags devel,test --desc "dev/test"  --region ${CLOUD_REGION} --sshkey ${SSH_KEY_NAME} --casenabled

Cluster creation

Deploy a cluster from this cluster spec and profile created in the previous steps.

arlon cluster deploy --repo-url ${WORKSPACE_REPO_URL} --cluster-name cas-cluster --profile dynamic-cas --cluster-spec cas-spec

Consequently, this example produces the directory clusters/cas-cluster/ in the repo. This will contain the capi-autoscaler subchart and manifests mgmt/charts/. To verify its contents:

$ cd ${WORKSPACE_REPO_URL}
$ tree clusters/cas-cluster
clusters/cas-cluster
└── mgmt
    ├── Chart.yaml
    ├── charts
    │   ├── capi-aws-eks
    │   │   ├── Chart.yaml
    │   │   └── templates
    │   │       └── cluster.yaml
    │   ├── capi-aws-kubeadm
    │   │   ├── Chart.yaml
    │   │   └── templates
    │   │       └── cluster.yaml
    │   ├── capi-cluster-autoscaler
    │   │   ├── Chart.yaml
    │   │   └── templates
    │   │       ├── callhomeconfig.yaml
    │   │       └── rbac.yaml
    │   └── xplane-aws-eks
    │       ├── Chart.yaml
    │       └── templates
    │           ├── cluster.yaml
    │           └── network.yaml
    ├── templates
    │   ├── clusterregistration.yaml
    │   ├── ns.yaml
    │   ├── profile.yaml
    │   └── rbac.yaml
    └── values.yaml

At this point, the cluster is provisioning. To monitor the progress of the cluster deployment, check the status of the ArgoCD app of the same name. Eventually, the ArgoCD apps will be synced and healthy.

$ argocd app list

NAME                           CLUSTER                         NAMESPACE  PROJECT  STATUS     HEALTH   SYNCPOLICY  CONDITIONS  REPO                                            PATH                             TARGET
cas-cluster                    https://kubernetes.default.svc  default    default Synced  Healthy  Auto-Prune  <none>  ${WORKSPACE_REPO_URL}   clusters/cas-cluster/mgmt        main
cas-cluster-cas-bundle           cas-cluster                 default    default  Synced     Healthy  Auto-Prune  <none>   https://github.com/arlonproj/arlon.git   bundles/capi-cluster-autoscaler  HEAD
cas-cluster-profile-dynamic-cas  https://kubernetes.default.svc  argocd     default  Synced     Healthy  Auto-Prune  <none>      ${WORKSPACE_REPO_URL} profiles/dynamic-cas/mgmt