This guide is specific to macOS but most of the details can be used for getting started in other OSes also, assuming you have taken care of any OS-specific parity like installation
Note: Brew / Homebrew works on Linux too, apart from macOS
Pre-requisites
kubectl CLI tool
kind CLI tool
clusterctl CLI tool
clusterawsadm CLI tool
Installing the pre-requisites:
brew install kubectl
brew install kind
brew install clusterctl
brew install clusterawsadm
brew install helm
Follow the below steps to run a Kubernetes Cluster on top of AWS
- Use
clusterawsadm, short for Cluster AWS Admin, to create a CloudFormation stack in your AWS account with the correct IAM resources required for cluster creation later
export AWS_REGION=us-east-1
export AWS_ACCESS_KEY_ID="dummy-access-key-id"
export AWS_SECRET_ACCESS_KEY="dummy-secret-access-key"
clusterawsadm bootstrap iam create-cloudformation-stack
- Now create a local Kubernetes Cluster using
kind. kind is short for Kubernetes in Docker and get the Kubernetes cluster config (kubeconfig)
kind create cluster
kind export kubeconfig
- Next, we are going to run some components in the local Kubernetes Cluster. These components will be responsible for creating the Kubernetes Cluster on AWS. These components are what make a Kubernetes Cluster a management Cluster that can manage workload clusters, in this case, workload clusters on AWS
export AWS_B64ENCODED_CREDENTIALS=$(clusterawsadm bootstrap credentials encode-as-profile)
clusterctl init --infrastructure aws
The output of clusterctl init command will look something like this -
$ clusterctl init --infrastructure aws
Fetching providers
Installing cert-manager Version="v1.12.3"
Waiting for cert-manager to be available...
Installing Provider="cluster-api" Version="v1.5.1" TargetNamespace="capi-system"
Installing Provider="bootstrap-kubeadm" Version="v1.5.1" TargetNamespace="capi-kubeadm-bootstrap-system"
Installing Provider="control-plane-kubeadm" Version="v1.5.1" TargetNamespace="capi-kubeadm-control-plane-system"
Installing Provider="infrastructure-aws" Version="v2.2.1" TargetNamespace="capa-system"
Your management cluster has been initialized successfully!
You can now create your first workload cluster by running the following:
clusterctl generate cluster [name] --kubernetes-version [version] | kubectl apply -f -
- Wait for all the pods in all namespaces to be up and running
kubectl wait --all --all-namespaces --for=jsonpath='{.status.phase}'=Running pod
- Let’s create the configuration for creating the workload cluster on top of AWS
export AWS_REGION=us-east-1
export AWS_SSH_KEY_NAME=default
export AWS_CONTROL_PLANE_MACHINE_TYPE=t3.large
export AWS_NODE_MACHINE_TYPE=t3.large
clusterctl generate cluster capi-quickstart \
--kubernetes-version v1.27.4 \
--control-plane-machine-count=1 \
--worker-machine-count=1 \
> capi-quickstart.yaml
- Let’s create the workload cluster now
kubectl apply -f capi-quickstart.yaml
- Wait for the Kubernetes Cluster to be created. You can check the progress using the following commands
kubectl get cluster
kubectl get awscluster
kubectl get machine
kubectl get awsmachine
kubectl get events
- Once the Kubernetes Cluster is created, use the following command to get the Kubernetes Cluster configuration called kubeconfig
clusterctl get kubeconfig capi-quickstart > capi-quickstart.kubeconfig
- In a new terminal, try the following commands
kubectl --kubeconfig capi-quickstart.kubeconfig get nodes
kubectl --kubeconfig capi-quickstart.kubeconfig get pods --all-namespaces
kubectl --kubeconfig capi-quickstart.kubeconfig \
apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.26.1/manifests/calico.yaml
- To install AWS Cloud Controller Manager. Ensure that the
--cluster-cidr CIDR IP range is the same as the one of the cluster. For example, look at the API server invocation command to find this detail in one of it's flags
helm repo add aws-cloud-controller-manager https://kubernetes.github.io/cloud-provider-aws
cat > aws-cloud-controller-manager-values.yaml <<EOF
image:
tag: v1.28.1
args:
- --v=2
- --cloud-provider=aws
- --cluster-cidr=192.168.0.0/16
- --cluster-name=capi-quickstart
- --use-service-account-credentials=true
clusterRoleRules:
- apiGroups:
- ""
resources:
- events
verbs:
- create
- patch
- update
- apiGroups:
- ""
resources:
- nodes
verbs:
- '*'
- apiGroups:
- ""
resources:
- nodes/status
verbs:
- patch
- apiGroups:
- ""
resources:
- services
verbs:
- list
- patch
- update
- watch
- apiGroups:
- ""
resources:
- services/status
verbs:
- list
- patch
- update
- watch
- apiGroups:
- ""
resources:
- serviceaccounts
verbs:
- get
- create
- apiGroups:
- ""
resources:
- persistentvolumes
verbs:
- get
- list
- update
- watch
- apiGroups:
- ""
resources:
- endpoints
verbs:
- create
- get
- list
- watch
- update
- apiGroups:
- coordination.k8s.io
resources:
- leases
verbs:
- create
- get
- list
- watch
- update
- apiGroups:
- ""
resources:
- serviceaccounts/token
verbs:
- create
EOF
helm upgrade --kubeconfig capi-quickstart.kubeconfig --atomic --install aws-cloud-controller-manager aws-cloud-controller-manager/aws-cloud-controller-manager --values aws-cloud-controller-manager-values.yaml
kubectl --kubeconfig capi-quickstart.kubeconfig get nodes -o json | jq '.items[].spec.taints'
Ideally, there should only be one taint and the output should look like this -
null
[
{
"effect": "NoSchedule",
"key": "node-role.kubernetes.io/control-plane"
}
]
Output should NOT look like this -
[
{
"effect": "NoSchedule",
"key": "node-role.kubernetes.io/control-plane"
},
{
"effect": "NoSchedule",
"key": "node.cloudprovider.kubernetes.io/uninitialized",
"value": "true"
}
]
[
{
"effect": "NoSchedule",
"key": "node.cluster.x-k8s.io/uninitialized"
},
{
"effect": "NoSchedule",
"key": "node.cloudprovider.kubernetes.io/uninitialized",
"value": "true"
}
]
- Create image pull secrets like this -
ACCOUNT=$(aws sts get-caller-identity --query 'Account' --output text)
REGION=us-east-1
SECRET_NAME=${REGION}-ecr-registry
EMAIL=abc@xyz.com
TOKEN=`aws ecr --region=$REGION get-authorization-token --output text --query 'authorizationData[].authorizationToken' | base64 -d | cut -d: -f2`
kubectl --kubeconfig capi-quickstart.kubeconfig delete secret --ignore-not-found $SECRET_NAME
kubectl --kubeconfig capi-quickstart.kubeconfig create secret docker-registry $SECRET_NAME \
--docker-server=https://${ACCOUNT}.dkr.ecr.${REGION}.amazonaws.com \
--docker-username=AWS \
--docker-password="${TOKEN}" \
--docker-email="${EMAIL}"
and use it in the Kubernetes resource yaml file. These image pull secrets will work for a few hours or so. Then they will expire and stop working. You gotta have some mechanism to update the secrets often - using something like CronJob or similar, which updates the image pull secrets in the cluster by running the above commands or similar set of commands
