Skip to content

Next-Generation (gen2) Clusters - New in version 0.9.x

Gen1 clusters are limited in capability by the Helm chart used to deploy the infrastructure resources. Advanced Cluster API configurations, such as those using multiple MachinePools, each with different instance types, is not supported.

Gen2 clusters solve this problem by allowing you to create workload clusters from a cluster template that you design and provide in the form of a manifest file stored in a git directory. The manifest typically contains multiple related resources that together define an arbitrarily complex cluster. If you make subsequent changes to the cluster template, workload clusters originally created from it will automatically acquire the changes.

Earlier, cluster templates were called base clusters. Some parts of the code still refer to base cluster manifests. This should be considered as a synonym for cluster template manifests.

Creating Cluster-API cluster template

Note: The CAPA version used here is v2.0 and the manifests created here are in accordance with this version.

Refer the compatibility matrix for Cluster API provider and CAPA versions for supported versions.

Before deploying a EKS cluster, make sure to setup the AWS Environment as stated in the quickstart giude for CAPI

MachineDeployment

Here is an example of a manifest file that we can use to create a cluster template. This manifest file helps in deploying an EKS cluster with 'machine deployment' component from the cluster API (CAPI). This file has been generated by the following command

clusterctl generate cluster capi-quickstart --flavor eks \
  --kubernetes-version v1.24.0 \
  --control-plane-machine-count=3 \
  --worker-machine-count=3 \
  > capi-quickstart.yaml
# YAML
apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: capi-quickstart
  namespace: default
spec:
  clusterNetwork:
    pods:
      cidrBlocks:
      - 192.168.0.0/16
  controlPlaneRef:
    apiVersion: controlplane.cluster.x-k8s.io/v1beta2
    kind: AWSManagedControlPlane
    name: capi-quickstart-control-plane
  infrastructureRef:
    apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
    kind: AWSManagedCluster
    name: capi-quickstart
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
kind: AWSManagedCluster
metadata:
  name: capi-quickstart
spec: {}
---
apiVersion: controlplane.cluster.x-k8s.io/v1beta2
kind: AWSManagedControlPlane
metadata:
  name: capi-quickstart-control-plane
  namespace: default
spec:
  region: {REGION}
  sshKeyName: {SSH_KEYNAME}
  version: v1.24.0
---
apiVersion: cluster.x-k8s.io/v1beta1
kind: MachineDeployment
metadata:
  name: capi-quickstart-md-0
  namespace: default
spec:
  clusterName: capi-quickstart
  replicas: 3
  selector:
    matchLabels: null
  template:
    spec:
      bootstrap:
        configRef:
          apiVersion: bootstrap.cluster.x-k8s.io/v1beta2
          kind: EKSConfigTemplate
          name: capi-quickstart-md-0
      clusterName: capi-quickstart
      infrastructureRef:
        apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
        kind: AWSMachineTemplate
        name: capi-quickstart-md-0
      version: v1.24.0
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
kind: AWSMachineTemplate
metadata:
  name: capi-quickstart-md-0
  namespace: default
spec:
  template:
    spec:
      iamInstanceProfile: nodes.cluster-api-provider-aws.sigs.k8s.io
      instanceType: {INSTANCE_TYPE}
      sshKeyName: {SSH_KEYNAME}
---
apiVersion: bootstrap.cluster.x-k8s.io/v1beta2
kind: EKSConfigTemplate
metadata:
  name: capi-quickstart-md-0
  namespace: default
spec:
  template: {}

gen2 cluster creation using Arlon

This manifest file needs to be pushed to the workspace repository before the manifest directory is prepped and then validated.

Before a manifest directory can be used as a cluster manifest, it must first be "prepared" or "prepped" by Arlon. The "prep" phase makes minor changes to the directory and manifest to help Arlon deploy multiple copies of the cluster without naming conflicts.

manifest directory preparation

To prepare a git directory to serve as cluster template, use the clustertemplate preparegit command:

arlon clustertemplate preparegit --repo-url <repoUrl> --repo-path <pathToDirectory> [--repo-revision revision]
            # OR
# using repository aliases
  # using the default alias
arlon clustertemplate preparegit --repo-path <pathToDirectory> [--repo-revision revision]
  # using the prod alias
arlon clustertemplate preparegit --repo-alias prod --repo-path <pathToDirectory> [--repo-revision revision]

manifest directory validation

Post the successful preparation of the cluster template directory using clustertemplate preparegit, the cluster template directory needs to be validated before the cluster template is created.

To determine if a git directory is eligible to serve as cluster template, run the clustertemplate validategit command:

arlon clustertemplate validategit --repo-url <repoUrl> --repo-path <pathToDirectory> [--repo-revision revision]
            # OR
# using repository aliases
  # using the default alias
arlon clustertemplate validategit --repo-path <pathToDirectory> [--repo-revision revision]
  # using the prod alias
arlon clustertemplate validategit --repo-alias prod --repo-path <pathToDirectory> [--repo-revision revision]

gen2 cluster creation

Note: Cluster templates only support dynamic profiles.

To create a gen2 workload cluster from the cluster template:

arlon cluster create --cluster-name <clusterName> --repo-url <repoUrl> --repo-path <pathToDirectory> [--output-yaml] [--profile <profileName>] [--repo-revision <repoRevision>]
            # OR
# using repository aliases
  # using the default alias
arlon cluster create --cluster-name <clusterName> --repo-path <pathToDirectory> [--output-yaml] [--profile <profileName>] [--repo-revision <repoRevision>]
  # using the prod alias
arlon cluster create --cluster-name <clusterName> --repo-alias prod --repo-path <pathToDirectory> [--output-yaml] [--profile <profileName>] [--repo-revision <repoRevision>]

gen2 cluster update

To update the profiles of a gen2 workload cluster:

# To add a new profile to the existing cluster
arlon cluster ngupdate <clustername> --profile <profilename>
# To delete an existing profile from the existing cluster
arlon cluster ngupdate <clustername> --delete-profile

A gen2 cluster can be created without any profile app associated with the cluster. So, the above commands can be used to add a new profile to the existing cluster which will create profile app in argocd along with bundle apps associated with the profile.

An existing profile can be deleted from the cluster as well using the above command. Executing this command will delete the profile app and all the bundles associated with the profile in argocd.

gen2 cluster deletion

To destroy a gen2 workload cluster:

arlon cluster delete <clusterName>

Arlon creates between 2 and 3 ArgoCD application resources to compose a gen2 cluster (the 3rd application, called "profile app", is used when an optional profile is specified at cluster creation time). When you destroy a gen2 cluster, Arlon will find all related ArgoCD applications and clean them up.

Known issues and limitations

Gen2 clusters are powerful because the cluster template can be arbitrarily complex and feature rich. Since they are fairly new and still evolving, gen2 clusters have several known limitations relative to gen1.

  • You cannot customize/override any property of the cluster template on the fly when creating a workload cluster, which is an exact clone of the cluster template except for the names of its resources and their namespace. The work-around is to make a copy of the cluster template directory, push the new directory, make the desired changes, commit & push the changes, and register the directory as a new cluster template.
  • If you modify and commit a change to one or more properties of the cluster template that the underlying Cluster API provider deems as "immutable", new workload clusters created from the cluster template will have the modified propert(ies), but ArgoCD will flag existing clusters as OutOfSync, since the provider will continually reject attempts to apply the new property values. The existing clusters continue to function, despite appearing unhealthy in the ArgoCD UI and CLI outputs.

Examples of mutable properties in Cluster API resources:

  • Number of replicas (modification will result in a scale-up / down)
  • Kubernetes version (modification will result in an upgrade)

Examples of immutable properties:

  • Most fields of AWSMachineTemplate (instance type, labels, etc...)

For more information

For more details on gen2 clusters, refer to the design document.