Getting started with Cluster API

This guide is specific to macOS but most of the details can be used for getting started in other OSes also, assuming you have taken care of any OS-specific parity like installation

Note: Brew / Homebrew works on Linux too, apart from macOS

Pre-requisites

kubectl CLI tool
kind CLI tool
clusterctl CLI tool
clusterawsadm CLI tool

Installing the pre-requisites:

brew install kubectl

brew install kind

brew install clusterctl

brew install clusterawsadm

brew install helm

Follow the below steps to run a Kubernetes Cluster on top of AWS

Use clusterawsadm, short for Cluster AWS Admin, to create a CloudFormation stack in your AWS account with the correct IAM resources required for cluster creation later

export AWS_REGION=us-east-1
export AWS_ACCESS_KEY_ID="dummy-access-key-id"
export AWS_SECRET_ACCESS_KEY="dummy-secret-access-key"

clusterawsadm bootstrap iam create-cloudformation-stack

Now create a local Kubernetes Cluster using kind. kind is short for Kubernetes in Docker and get the Kubernetes cluster config (kubeconfig)

kind create cluster

kind export kubeconfig

Next, we are going to run some components in the local Kubernetes Cluster. These components will be responsible for creating the Kubernetes Cluster on AWS. These components are what make a Kubernetes Cluster a management Cluster that can manage workload clusters, in this case, workload clusters on AWS

export AWS_B64ENCODED_CREDENTIALS=$(clusterawsadm bootstrap credentials encode-as-profile)

clusterctl init --infrastructure aws

The output of clusterctl init command will look something like this -

$ clusterctl init --infrastructure aws
Fetching providers
Installing cert-manager Version="v1.12.3"
Waiting for cert-manager to be available...
Installing Provider="cluster-api" Version="v1.5.1" TargetNamespace="capi-system"
Installing Provider="bootstrap-kubeadm" Version="v1.5.1" TargetNamespace="capi-kubeadm-bootstrap-system"
Installing Provider="control-plane-kubeadm" Version="v1.5.1" TargetNamespace="capi-kubeadm-control-plane-system"
Installing Provider="infrastructure-aws" Version="v2.2.1" TargetNamespace="capa-system"

Your management cluster has been initialized successfully!

You can now create your first workload cluster by running the following:

  clusterctl generate cluster [name] --kubernetes-version [version] | kubectl apply -f -

Wait for all the pods in all namespaces to be up and running

kubectl wait --all --all-namespaces --for=jsonpath='{.status.phase}'=Running pod

Let’s create the configuration for creating the workload cluster on top of AWS

export AWS_REGION=us-east-1
export AWS_SSH_KEY_NAME=default
export AWS_CONTROL_PLANE_MACHINE_TYPE=t3.large
export AWS_NODE_MACHINE_TYPE=t3.large

clusterctl generate cluster capi-quickstart \
  --kubernetes-version v1.27.4 \
  --control-plane-machine-count=1 \
  --worker-machine-count=1 \
  > capi-quickstart.yaml

Let’s create the workload cluster now

kubectl apply -f capi-quickstart.yaml

Wait for the Kubernetes Cluster to be created. You can check the progress using the following commands

kubectl get cluster # it will show cluster is "Provisioning" and then "Provisioned"

kubectl get awscluster # it will show AWS cluster is "Provisioning" and then "Provisioned"

kubectl get machine # it will show the status of the machines / nodes being created for the cluster

kubectl get awsmachine # it will show the status of the AWS machines / nodes being created for the cluster

kubectl get events # all the events that are happening, for example creation of VPC, subnet etc will be seen here

Once the Kubernetes Cluster is created, use the following command to get the Kubernetes Cluster configuration called kubeconfig

clusterctl get kubeconfig capi-quickstart > capi-quickstart.kubeconfig

In a new terminal, try the following commands

kubectl --kubeconfig capi-quickstart.kubeconfig get nodes

kubectl --kubeconfig capi-quickstart.kubeconfig get pods --all-namespaces

# the nodes will show up as not ready, to make them ready, do the following

kubectl --kubeconfig capi-quickstart.kubeconfig \
  apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.26.1/manifests/calico.yaml

To install AWS Cloud Controller Manager. Ensure that the --cluster-cidr CIDR IP range is the same as the one of the cluster. For example, look at the API server invocation command to find this detail in one of it's flags


helm repo add aws-cloud-controller-manager https://kubernetes.github.io/cloud-provider-aws

cat > aws-cloud-controller-manager-values.yaml <<EOF
image:
  tag: v1.28.1
args:
  - --v=2
  - --cloud-provider=aws
  - --cluster-cidr=192.168.0.0/16
  - --cluster-name=capi-quickstart
  - --use-service-account-credentials=true  

clusterRoleRules:
- apiGroups:
  - ""
  resources:
  - events
  verbs:
  - create
  - patch
  - update
- apiGroups:
  - ""
  resources:
  - nodes
  verbs:
  - '*'
- apiGroups:
  - ""
  resources:
  - nodes/status
  verbs:
  - patch
- apiGroups:
  - ""
  resources:
  - services
  verbs:
  - list
  - patch
  - update
  - watch
- apiGroups:
  - ""
  resources:
  - services/status
  verbs:
  - list
  - patch
  - update
  - watch
- apiGroups:
  - ""
  resources:
  - serviceaccounts
  verbs:
  - get
  - create
- apiGroups:
  - ""
  resources:
  - persistentvolumes
  verbs:
  - get
  - list
  - update
  - watch
- apiGroups:
  - ""
  resources:
  - endpoints
  verbs:
  - create
  - get
  - list
  - watch
  - update
- apiGroups:
  - coordination.k8s.io
  resources:
  - leases
  verbs:
  - create
  - get
  - list
  - watch
  - update
- apiGroups:
  - ""
  resources:
  - serviceaccounts/token
  verbs:
  - create
EOF

helm upgrade --kubeconfig capi-quickstart.kubeconfig --atomic --install aws-cloud-controller-manager aws-cloud-controller-manager/aws-cloud-controller-manager --values aws-cloud-controller-manager-values.yaml

Check if the nodes have node.cloudprovider.kubernetes.io/uninitialized key taints, node.cluster.x-k8s.io/uninitialized key taints

kubectl --kubeconfig capi-quickstart.kubeconfig get nodes -o json | jq '.items[].spec.taints'

Ideally, there should only be one taint and the output should look like this -

null
[
  {
    "effect": "NoSchedule",
    "key": "node-role.kubernetes.io/control-plane"
  }
]

Output should NOT look like this -

[
  {
    "effect": "NoSchedule",
    "key": "node-role.kubernetes.io/control-plane"
  },
  {
    "effect": "NoSchedule",
    "key": "node.cloudprovider.kubernetes.io/uninitialized",
    "value": "true"
  }
]
[
  {
    "effect": "NoSchedule",
    "key": "node.cluster.x-k8s.io/uninitialized"
  },
  {
    "effect": "NoSchedule",
    "key": "node.cloudprovider.kubernetes.io/uninitialized",
    "value": "true"
  }
]

Create image pull secrets like this -

ACCOUNT=$(aws sts get-caller-identity --query 'Account' --output text)
REGION=us-east-1
SECRET_NAME=${REGION}-ecr-registry
EMAIL=abc@xyz.com

TOKEN=`aws ecr --region=$REGION get-authorization-token --output text --query 'authorizationData[].authorizationToken' | base64 -d | cut -d: -f2`

kubectl --kubeconfig capi-quickstart.kubeconfig delete secret --ignore-not-found $SECRET_NAME
kubectl --kubeconfig capi-quickstart.kubeconfig create secret docker-registry $SECRET_NAME \
 --docker-server=https://${ACCOUNT}.dkr.ecr.${REGION}.amazonaws.com \
 --docker-username=AWS \
 --docker-password="${TOKEN}" \
 --docker-email="${EMAIL}"

and use it in the Kubernetes resource yaml file. These image pull secrets will work for a few hours or so. Then they will expire and stop working. You gotta have some mechanism to update the secrets often - using something like CronJob or similar, which updates the image pull secrets in the cluster by running the above commands or similar set of commands

Karuppiah Natarajan's Blog

Karuppiah Natarajan's Blog

Getting started with Cluster API - A Quickstart Guide