Skip to content

Tutorial

This tutorial will go through the step by step process to create a Cluster Template and then deploy one or more AWS EKS Clusters using Arlon.

Arlon allows you to create one or more workload clusters from a Cluster Template that you design and provide in the form of a Cluster API manifest file stored in a git directory. The manifest typically contains multiple related resources that together define an arbitrarily complex cluster. If you make subsequent changes to the Cluster Template, workload clusters originally created from it will automatically acquire the changes.

(NOTE: In earlier versions of arlon (before v0.10), Cluster Templates were called Base Clusters. Some parts of the code still refer to base cluster manifests. This should be considered as a synonym for cluster template manifests.)

Pre-requisites

Note: The CAPA version used here is v2.0 and the manifests created here are in accordance with this version.

Refer the compatibility matrix for Cluster API provider and CAPA versions for supported versions.

Before deploying a EKS cluster, make sure to setup the AWS Environment as stated in the quickstart guide for CAPI

Please read here to understand the Cluster API concepts that we will use in rest of this Tutorial.

Creating Cluster-API Cluster Manifest

In this section, we will go through the steps to create a CAPI cluster manifest for the three different types of node pooling constructs that CAPI EKS clusters support.

MachineDeployment

Here is an example of a manifest file that we can use to create a Cluster Template. This manifest file helps in creating an EKS cluster with 'Machine Deployment' component from the Cluster API (CAPI). This file has been generated by the following command

clusterctl generate cluster capi-quickstart --flavor eks \
  --kubernetes-version v1.24.0 \
  --control-plane-machine-count=3 \
  --worker-machine-count=3 \
  > capi-quickstart.yaml

# YAML
apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: capi-quickstart
  namespace: default
spec:
  clusterNetwork:
    pods:
      cidrBlocks:
      - 192.168.0.0/16
  controlPlaneRef:
    apiVersion: controlplane.cluster.x-k8s.io/v1beta2
    kind: AWSManagedControlPlane
    name: capi-quickstart-control-plane
  infrastructureRef:
    apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
    kind: AWSManagedCluster
    name: capi-quickstart
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
kind: AWSManagedCluster
metadata:
  name: capi-quickstart
spec: {}
---
apiVersion: controlplane.cluster.x-k8s.io/v1beta2
kind: AWSManagedControlPlane
metadata:
  name: capi-quickstart-control-plane
  namespace: default
spec:
  region: {REGION}
  sshKeyName: {SSH_KEYNAME}
  version: v1.24.0
---
apiVersion: cluster.x-k8s.io/v1beta1
kind: MachineDeployment
metadata:
  name: capi-quickstart-md-0
  namespace: default
spec:
  clusterName: capi-quickstart
  replicas: 3
  selector:
    matchLabels: null
  template:
    spec:
      bootstrap:
        configRef:
          apiVersion: bootstrap.cluster.x-k8s.io/v1beta2
          kind: EKSConfigTemplate
          name: capi-quickstart-md-0
      clusterName: capi-quickstart
      infrastructureRef:
        apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
        kind: AWSMachineTemplate
        name: capi-quickstart-md-0
      version: v1.24.0
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
kind: AWSMachineTemplate
metadata:
  name: capi-quickstart-md-0
  namespace: default
spec:
  template:
    spec:
      iamInstanceProfile: nodes.cluster-api-provider-aws.sigs.k8s.io
      instanceType: {INSTANCE_TYPE}
      sshKeyName: {SSH_KEYNAME}
---
apiVersion: bootstrap.cluster.x-k8s.io/v1beta2
kind: EKSConfigTemplate
metadata:
  name: capi-quickstart-md-0
  namespace: default
spec:
  template: {}

AWSManagedMachinePool

Initialize the environment for AWSManagedMachinePool as stated here

Before deploying an EKS cluster, make sure that the MachinePool feature gate is enabled. To do so, run this command:

kubectl describe deployment capa-controller-manager -n capa-system

In the output, in the feature gates section of the deployment, MachinePool must be set to true.

> kubectl describe deployment capa-controller-manager -n capa-system
..........
..........
--featuregates=EKS=true,EKSEnableIAM=false,EKSAllowAddRoles=false,EKSFargate=true,MachinePool=true,EventBridgeInstanceState=false,
AutoControllerIdentityCreator=true,BootstrapFormatIgnition=false,ExternalResourceGC=false
..........
..........

This manifest file helps in deploying an EKS cluster with 'AWSManagedMachinePool' component from the cluster API (CAPI). This file has been generated by the following command

clusterctl generate cluster awsmanaged-cluster --kubernetes-version v1.22.0 --flavor eks-managedmachinepool > manifest.yaml

# YAML
apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: awsmanaged-cluster
  namespace: default
spec:
  clusterNetwork:
    pods:
      cidrBlocks:
      - 192.168.0.0/16
  controlPlaneRef:
    apiVersion: controlplane.cluster.x-k8s.io/v1beta2
    kind: AWSManagedControlPlane
    name: awsmanaged-cluster-control-plane
  infrastructureRef:
    apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
    kind: AWSManagedCluster
    name: awsmanaged-cluster
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
kind: AWSManagedCluster
metadata:
  name: awsmanaged-cluster
  namespace: default
spec: {}
---
apiVersion: controlplane.cluster.x-k8s.io/v1beta2
kind: AWSManagedControlPlane
metadata:
  name: awsmanaged-cluster-control-plane
  namespace: default
spec:
  region: {REGION}
  sshKeyName: {SSH_KEYNAME}
  version: v1.22.0
---
apiVersion: cluster.x-k8s.io/v1beta1
kind: MachinePool
metadata:
  name: awsmanaged-cluster-pool-0
  namespace: default
spec:
  clusterName: awsmanaged-cluster
  replicas: 1
  template:
    spec:
      bootstrap:
        dataSecretName: ""
      clusterName: awsmanaged-cluster
      infrastructureRef:
        apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
        kind: AWSManagedMachinePool
        name: awsmanaged-cluster-pool-0
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
kind: AWSManagedMachinePool
metadata:
  name: awsmanaged-cluster-pool-0
  namespace: default
spec: {}

AWSMachinePool

An AWSMachinePool corresponds to an AWS AutoScaling Groups, which provides the cloud provider specific resource for orchestrating a group of EC2 machines.

Initialize the environment for AWSMachinePool as stated here

Before deploying an EKS cluster, make sure that the AWSMachinePool feature gate is enabled. To do so, run this command:

kubectl describe deployment capa-controller-manager -n capa-system

In the output, in the feature gates section of the deployment, MachinePool must be set to true.

> kubectl describe deployment capa-controller-manager -n capa-system
..........
..........
--featuregates=EKS=true,EKSEnableIAM=false,EKSAllowAddRoles=false,EKSFargate=true,MachinePool=true,EventBridgeInstanceState=false,
AutoControllerIdentityCreator=true,BootstrapFormatIgnition=false,ExternalResourceGC=false
..........
..........

This manifest file helps in deploying an EKS cluster with 'AWSManagedMachinePool' component from the cluster API (CAPI). This file has been generated by the following command

clusterctl generate cluster awsmanaged-cluster --kubernetes-version v1.22.0 --flavor eks-machinepool > manifest.yaml

# YAML
apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: awsmanaged-cluster
  namespace: default
spec:
  clusterNetwork:
    pods:
      cidrBlocks:
      - 192.168.0.0/16
  controlPlaneRef:
    apiVersion: controlplane.cluster.x-k8s.io/v1beta2
    kind: AWSManagedControlPlane
    name: awsmanaged-cluster-control-plane
  infrastructureRef:
    apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
    kind: AWSManagedCluster
    name: awsmanaged-cluster
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
kind: AWSManagedCluster
metadata:
  name: awsmanaged-cluster
  namespace: default
spec: {}
---
apiVersion: controlplane.cluster.x-k8s.io/v1beta2
kind: AWSManagedControlPlane
metadata:
  name: awsmanaged-cluster-control-plane
  namespace: default
spec:
  region: {REGION}
  sshKeyName: {SSH_KEYNAME}
  version: v1.22.0
---
apiVersion: cluster.x-k8s.io/v1beta1
kind: MachinePool
metadata:
  name: awsmanaged-cluster-mp-0
  namespace: default
spec:
  clusterName: awsmanaged-cluster
  replicas: 1
  template:
    spec:
      bootstrap:
        configRef:
          apiVersion: bootstrap.cluster.x-k8s.io/v1beta2
          kind: EKSConfig
          name: awsmanaged-cluster-mp-0
      clusterName: awsmanaged-cluster
      infrastructureRef:
        apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
        kind: AWSMachinePool
        name: awsmanaged-cluster-mp-0
      version: v1.22.0
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
kind: AWSMachinePool
metadata:
  name: awsmanaged-cluster-mp-0
  namespace: default
spec:
  awsLaunchTemplate:
    iamInstanceProfile: nodes.cluster-api-provider-aws.sigs.k8s.io
    instanceType: {INSTANCE_TYPE}
    sshKeyName: {SSH_KEYNAME}
  maxSize: 10
  minSize: 1
---
apiVersion: bootstrap.cluster.x-k8s.io/v1beta2
kind: EKSConfig
metadata:
  name: awsmanaged-cluster-mp-0
  namespace: default
spec: {}

Prepare Manifest

This manifest file needs to be pushed to the workspace repository before the manifest directory is prepped and then validated.

Before a manifest directory can be used as a cluster template, it must first be "prepared" or "prepped" by Arlon. The "prep" phase makes minor changes to the directory and manifest to help Arlon deploy multiple copies of the cluster without naming conflicts.

manifest directory preparation

To prepare a git directory to serve as cluster template, use the clustertemplate preparegit command:

arlon clustertemplate preparegit --repo-url <repoUrl> --repo-path <pathToDirectory> [--repo-revision revision]
            # OR
# using repository aliases
  # using the default alias
arlon clustertemplate preparegit --repo-path <pathToDirectory> [--repo-revision revision]
  # using the prod alias
arlon clustertemplate preparegit --repo-alias prod --repo-path <pathToDirectory> [--repo-revision revision]

manifest directory validation

Post the successful preparation of the cluster template manifest directory using clustertemplate preparegit, the cluster template manifest directory needs to be validated before the cluster template is created.

To determine if a git directory is eligible to serve as cluster template, run the clustertemplate validategit command:

arlon clustertemplate validategit --repo-url <repoUrl> --repo-path <pathToDirectory> [--repo-revision revision]
            # OR
# using repository aliases
  # using the default alias
arlon clustertemplate validategit --repo-path <pathToDirectory> [--repo-revision revision]
  # using the prod alias
arlon clustertemplate validategit --repo-alias prod --repo-path <pathToDirectory> [--repo-revision revision]

Create Cluster

To create a workload cluster from the Cluster Template:

arlon cluster create --cluster-name <clusterName> --repo-url <repoUrl> --repo-path <pathToDirectory> [--output-yaml] [--profile <profileName>] [--repo-revision <repoRevision>]
            # OR
# using repository aliases
  # using the default alias
arlon cluster create --cluster-name <clusterName> --repo-path <pathToDirectory> [--output-yaml] [--profile <profileName>] [--repo-revision <repoRevision>]
  # using the prod alias
arlon cluster create --cluster-name <clusterName> --repo-alias prod --repo-path <pathToDirectory> [--output-yaml] [--profile <profileName>] [--repo-revision <repoRevision>]

Create Cluster with Overrides

We call the concept of constructing various clusters with patches from the same base manifest as cluster overrides. The cluster overrides feature is built on top of the existing cluster template design. So, a user can create a cluster from the base manifest using the same command specified in the section above. Now, to create a cluster with overrides in the base manifest, a user should have the corresponding patch files in a single yaml file in local. Here is an example of a patch file where we want to override replicas count to 2 and change the sshkeyname:

---
apiVersion: cluster.x-k8s.io/v1beta1
kind: MachineDeployment
metadata:
  name: capi-quickstart-eks-md-0
spec:
  replicas: 2
---
apiVersion: controlplane.cluster.x-k8s.io/v1beta1
kind: AWSManagedControlPlane
metadata:
  name: capi-quickstart-eks-control-plane
spec:
  sshKeyName: random
---

  ```

Note: The metadata field in the patch file should be same as of the metadata field in the resources file.

Refer to this [document](https://blog.scottlowe.org/2019/11/12/using-kustomize-with-cluster-api-manifests/) to know more about patch files

Command to create a workload cluster from the Cluster Template manifest with overrides to the manifest is:

```shell
arlon cluster create <cluster-name> --repo-url <repo url where cluster template is present> --repo-path <repo path to the cluster template> --overrides-path <path to the patch file> --patch-repo-url <repo url where patch file should be stored> --patch-repo-path <repo path to store the patch files>
````

Runnning the above command will create a folder with the specified cluster name in patch repo path of patch repo url which contains the patch files, kustomization.yaml and configurations.yaml which are used to create the cluster.

Note that the patch file repo url can be different or same from the cluster template repo url acoording to the requirement of the user. A user can use a different repo url for string patch files for the cluster.

## Update Cluster

To update the profiles of a workload cluster:

```shell
# To add a new profile to the existing cluster
arlon cluster ngupdate <clustername> --profile <profilename>
# To delete an existing profile from the existing cluster
arlon cluster ngupdate <clustername> --delete-profile

A cluster can be created without any profile associated with the cluster. So, the above commands can be used to add a new profile to the existing cluster which will create profile app in argocd along with bundle apps associated with the profile.

An existing profile can be deleted from the cluster as well using the above command. Executing this command will delete the profile app and all the bundles associated with the profile in argocd.

Delete Cluster

To destroy a workload cluster:

arlon cluster delete <clusterName>

Arlon creates between 2 and 3 ArgoCD application resources to compose a gen2 cluster (the 3rd application, called "profile app", is used when an optional profile is specified at cluster creation time). When you destroy a gen2 cluster, Arlon will find all related ArgoCD applications and clean them up.

If the cluster which which is being deleted is a cluster created using patch files, the controller first cleans the git repo where the respective patch files of the cluster are present and then it destroys all the related ArgoCD applications and clean them up.

Enabling Cluster Autoscaler in the workload cluster

To create a cluster with autoscaler, we need:

  • bundle pointing to the bundle/capi-cluster-autoscaler in the arlon repository.
  • dynamic profile that contains the above bundle.
  • a cluster template manifest(that makes use of MachineDeployment and not MachinePools) which has the CAPI annotations for min and max nodes set ( as a part of preparegit or manually add it ).
  • run arlon cluster create with the repo-path pointing to the cluster template manifest described in the step above, set the profile to be the one created in step 2 and pass the autoscaler flag.

Bundle creation

Register a dynamic bundle pointing to the bundles/capi-cluster-autoscaler in the Arlon repo.

To enable the cluster-autoscaler bundle, add one more parameter during cluster creation: srcType. This is the ArgoCD-defined application source type (Helm, Kustomize, Directory). In addition to this, the repo-revision parameter should also be set to a stable arlon release branch ( in this case v0.10 ).

This example creates a bundle pointing to the bundles/capi-cluster-autoscaler in Arlon repo

arlon bundle create cas-bundle --tags cas,devel,test --desc "CAS Bundle" --repo-url https://github.com/arlonproj/arlon.git --repo-path bundles/capi-cluster-autoscaler --srctype helm --repo-revision v0.10

Profile creation

Create a profile that contains this capi-cluster-autoscaler bundle.

arlon profile create dynamic-cas --repo-url <repoUrl> --repo-base-path profiles --bundles cas-bundle --desc "dynamic cas profile" --tags examples

manifest directory preparation

Two additional properties cas-max and cas-min are used to set 2 annotations for Max/Min nodes on MachineDeployment required by the cluster autoscaler for CAPI as a part of the manifest directory preparation. These are the annotations required by the MachineDeployment for autoscaling.

Note: These are the default values for the cas-min and cas-max properties

 annotations:
     cluster.x-k8s.io/cluster-api-autoscaler-node-group-min-size: '1'
     cluster.x-k8s.io/cluster-api-autoscaler-node-group-max-size: '9'

These annotations are added during the preparegit step. If these annotations are already present in the manifest file, then they will not be added again as a part of preparegit

Preparing the git directory to serve as the cluster template:

arlon clustertemplate preparegit --repo-path <pathToDirectory> --cas-min 1 --cas-max 9 --repo-url <repoUrl>

manifest directory validation

To determine if a git directory is eligible to serve as cluster template, run the clustertemplate validategit command:

arlon clustertemplate validategit --repo-path <pathToDirectory> --repo-url <repoUrl>

Cluster creation with autoscaling enabled

To add CAS support for gen2 clusters, the cluster create sub-command of the arlon CLI has a autoscaler flag which deploys the capi-cluster-autoscaler helm chart on the management cluster.

To create a gen2 workload with a cluster-autoscaler pod running, from the cluster template, run this command:

Note: Use the dynamic profile that was created in the previous steps

arlon cluster create --cluster-name <clusterName> --repo-url <repoUrl> --repo-path <pathToDirectory> --profile dynamic-cas --autoscaler

Known issues and limitations

Gen2 clusters are powerful because the cluster template can be arbitrarily complex and feature rich. Since they are fairly new and still evolving, gen2 clusters have several known limitations relative to gen1.

  • You cannot customize/override any property of the cluster template on the fly when creating a workload cluster, which is an exact clone of the cluster template except for the names of its resources and their namespace. The work-around is to make a copy of the cluster template directory, push the new directory, make the desired changes, commit & push the changes, and register the directory as a new cluster template.
  • The clusters created directly from the cluster template are completely declarative whereas the clusters which are created using override property are not completely declarative.
  • If a user passes a different repository for patch repo url from the repo where cluster template is present, argocd won't be able to detect if there are any changes in the cluster template repository but will deect all the chnages in patch repo url for the cluster.
  • If you modify and commit a change to one or more properties of the cluster template that the underlying Cluster API provider deems as "immutable", new workload clusters created from the cluster template will have the modified propert(ies), but ArgoCD will flag existing clusters as OutOfSync, since the provider will continually reject attempts to apply the new property values. The existing clusters continue to function, despite appearing unhealthy in the ArgoCD UI and CLI outputs.

Examples of mutable properties in Cluster API resources:

  • Number of replicas (modification will result in a scale-up / down)
  • Kubernetes version (modification will result in an upgrade)

Examples of immutable properties:

  • Most fields of AWSMachineTemplate (instance type, labels, etc...)

For more information

For more details on gen2 clusters, refer to the design document.

Arlon CLI Version information

The Arlon CLI has a version command which shows the current version of the CLI as shown below.

arlon version 
Arlon CLI Version: 0.10.0

The CLI can be downloaded from GitHub releases with the latest release of the CLI found here. The official releases follow a semver versioning scheme. All patches to a minor version are non-breaking changes. For instance, the transition from v0.9.9 to v0.9.10 will not introduce any breaking changes. However, a change in either minor or the major versions doesn't guarantee this.

To check if there are patches available for the current minor version of the CLI, Arlon has a version check command. If upgrade patch for a CLI minor version is available, the command will request to download the newer version. For example: 

arlon version check
Arlon CLI version 0.10.0 is outdated. New patch 0.10.1 available
Otherwise, the CLI reports that it is up-to date. 
arlon version check
"Arlon CLI version 0.10.0 is up-to-date