Set up Amazon EKS cluster for AI/ML workloads using Terraform

Set up Amazon EKS cluster for AI/ML workloads using Terraform

[추출 작업 중]

출처: 문서

본문

Help improve this page

To contribute to this user guide, choose the Edit this page on GitHub link that is located in the right pane of every page.

Set up Amazon EKS cluster for AI/ML workloads using Terraform

Tip

     Register for upcoming Amazon EKS AI/ML workshops.

This section walks you through the steps to create the infrastructure required to run training or inference workloads on Amazon EKS by using Terraform. The steps include creating an EKS cluster, GPU-enabled nodes with EKS Auto Mode or Karpenter, a monitoring stack with Prometheus and Grafana, and Amazon S3 storage for model weights.

See the documentation for EKS Auto Mode and Karpenter for more information on how those features provision and auto-scale EC2 instances in EKS clusters.

  High-level architecture and workflow

The diagram shows the AWS high-level architecture for this section’s setup.

Prerequisites

  Important

The resources you create in this tutorial, including EKS clusters, GPU instances, Application Load Balancers, and Amazon Managed Service for Prometheus, incur charges. Delete resources when you finish to avoid ongoing charges.

Terraform >= 1.15.0. For setup instructions, see Installing Terraform.

           `kubectl` >= 1.36. For setup instructions, see Set up kubectl and eksctl.

           AWS CLI >= 2.27. For setup instructions, see Installing.

           `jq`. For setup instructions, see Download jq.

Verify your tool versions:

terraform --version
aws --version
kubectl version --client
jq --version

Step 1: Download and deploy the Terraform code

This walkthrough uses the Terraform code in the sample-eks-docs AWS Samples GitHub repository. Clone the repository into a working directory:

git clone [email protected]:aws-samples/sample-eks-docs.git
cd sample-eks-docs/ai-ml/set-up-cluster

The repository has the following structure under the ai-ml/set-up-cluster/ directory you just changed into:

set-up-cluster/
├── scripts/
│   └── cleanup.sh
└── terraform/
    ├── auto-mode/
    └── karpenter/

The repository provides two deployment paths. Choose only one and use it throughout the guide.

           EKS Auto Mode (`terraform/auto-mode/`) — In addition to the core networking, storage, and load balancing add-ons, EKS Auto Mode includes and manages the following capabilities for training and inference workloads: EKS node monitoring agent, automatic node repair, SOCI snapshotter for fast container pulls, and GPU readiness for the default NodeClass. The NVIDIA device plugin is included in the Bottlerocket accelerated AMI that EKS Auto Mode uses for GPU-enabled nodes.

           Self-managed Karpenter (`terraform/karpenter/`) — On an EKS cluster without EKS Auto Mode, the Terraform code installs and configures the components required for training and inference workloads. This includes networking add-ons (VPC CNI, CoreDNS, kube-proxy), Karpenter, the EKS node monitoring agent, the NVIDIA device plugin, and SOCI snapshotter for fast container pulls.

  Important

Pick either EKS Auto Mode or self-managed Karpenter and use it throughout the guide. Switching mid-stream requires destroying the cluster and starting over.

     EKS cluster options: EKS Auto Mode and self-managed Karpenter

  Grafana is publicly accessible over HTTP with default credentials

The Grafana ALB Ingress defaults var.my_cidr to 0.0.0.0/0, which exposes Grafana to the public internet over plain HTTP with default admin credentials. Automated scanners discover public load balancers within minutes. You must restrict access by overriding var.my_cidr with your own IP address:

export MY_CIDR="$(curl -s https://checkip.amazonaws.com)/32"
terraform apply -var "my_cidr=${MY_CIDR}"

Treat source-IP allowlisting as a minimum safeguard, not a complete one. Also change the default Grafana admin password after first login. For a stronger posture, change the alb.ingress.kubernetes.io/scheme to internal (reachable only from within your VPC or a connected VPN) and add a TLS certificate.

Deploy the cluster

Change into the directory for your chosen path, initialize Terraform, and apply:

Both variants default to the us-east-2 region. To deploy in a different region, add -var "region=region-code" to the terraform apply command in the following step, where region-code is the AWS Region you want to deploy in.

The Terraform code uses all available Availability Zones in the target region, excluding use1-az3, usw1-az2, and cac1-az3 because Amazon EKS does not support control plane placement in those zones.

Review the Terraform outputs

When the apply completes, Terraform prints the following outputs (values vary based on your configuration):

Apply complete! Resources: 74 added, 0 changed, 0 destroyed.

Outputs:

cluster_name         = "ai-eks-docs"
configure_kubectl    = "aws eks update-kubeconfig --region us-east-2 --name ai-eks-docs --alias ai-eks-docs"
configure_model_bucket = "export MODEL_BUCKET=ai-eks-docs-models-20250612abc1"
model_bucket         = "ai-eks-docs-models-20250612abc1"
node_iam_role_name   = "ai-eks-docs-eks-auto-20250612..."
region               = "us-east-2"

The configure_kubectl output is a ready-to-run command that points kubectl at the cluster. The model_bucket output contains the S3 bucket name for model weights. The node_iam_role_name output shows the IAM role that nodes use.

Configure kubectl

Point kubectl at the new cluster. The configure_kubectl output is a ready-to-run command:

eval "$(terraform output -raw configure_kubectl)"

Verify the cluster

Step 2: Create dynamic GPU NodePool

GPU NodePools are opt-in. By default, terraform apply creates the cluster and monitoring stack with no GPU capacity and no GPU billing. To provision GPU nodes, pass the nodepools variable with a strategy name.

Enable the spot-ondemand strategy, which provisions G-family GPU instances with a generation greater than 4, using Spot capacity with On-Demand as a fallback:

terraform apply -var 'nodepools={"spot-ondemand"={}}'

This command applies the NodePool and NodeClass templates from the nodepools/spot-ondemand/ directory. Both paths use the same NodePool API, but they differ in the NodeClass the NodePool references.

Both paths show 0 nodes for gpu-inf until a GPU workload is scheduled. EKS Auto Mode and Karpenter only launch nodes when pending Pods require them.

Step 3: Test with a sample Pod

Test your GPU NodePool setup with an nvidia-smi Pod:

cat Note

On a self-managed Karpenter cluster, the `Nominated` event shows `karpenter/compute` instead of `eks-auto-mode/compute`.

A NodeClaim is a request Karpenter creates to provision a specific node. It shows the instance type, capacity type, AZ, and whether the node is ready:

kubectl get nodeclaims

Expected output:

NAME TYPE CAPACITY ZONE NODE READY AGE gpu-inf-z6q75 g6.xlarge spot us-east-2a i-0eb897a8302551589 True 5m

The instance type and AZ vary. Any G-family instance with a generation greater than 4 is eligible.

      Tip

If no node appears, check for Insufficient Capacity Errors:

kubectl get events | grep InsufficientCapacityError

Karpenter caches unavailable offerings for 3 minutes. Widening the allowed instance types and AZs in your NodePool increases the chances of landing capacity.

      Note

Spot instances launched by Karpenter do not appear in the EC2 Spot Requests console. Karpenter uses the EC2 `CreateFleet` API with `type: instant`. The instances appear in the EC2 Instances console with a `spot` lifecycle.

Step 4: Add reserved capacity to the NodePool (optional)

While the GPU NodePool from Step 2 provisions Spot or On-Demand instances dynamically, some use cases require guaranteed capacity. You can create an On-Demand Capacity Reservation (ODCR) to ensure GPU capacity is available when needed.

With Terraform, a single command creates the ODCR, a custom NodeClass that references the reservation by tag, and updates the NodePool to include `reserved` as a capacity type. Terraform tags the ODCR with `nodepool=reserved-spot-ondemand` and the NodeClass selects it by that tag.

      Warning

The following command creates an ODCR that bills immediately and continues billing until you destroy it with `terraform destroy` or the cleanup script, whether or not nodes are running on it.

Use defaults (`g6e.4xlarge`, 1 instance, first cluster AZ):

terraform apply -var 'nodepools={"reserved-spot-ondemand"={reservation={}}}'

Pick the instance type, count, and AZ:

terraform apply -var 'nodepools={"reserved-spot-ondemand"={reservation={instance_type="g6e.2xlarge",instance_count=1,az="us-east-2a"}}}'

The `reservation` object supports the following fields:

               `instance_type` — The GPU instance type to reserve. Default: `g6e.4xlarge`.

               `instance_count` — The number of instances to reserve. Default: `1`.

               `az` — The Availability Zone for the reservation. Default: `""` (uses the first cluster AZ).

      Important

The `spot-ondemand` and `reserved-spot-ondemand` strategies are mutually exclusive. You can enable at most one in the `nodepools` variable. If you previously used `spot-ondemand` in Step 2, the `reserved-spot-ondemand` command replaces it because both manage the same `gpu-inf` NodePool.

If you get an `InsufficientInstanceCapacity` error, the reservation cannot be fulfilled in the specified AZ. Cancel the Terraform operation (Ctrl+C), then re-run with a different `az` value:

terraform apply -var 'nodepools={"reserved-spot-ondemand"={reservation={instance_type="g6e.4xlarge",az="us-east-2b"}}}'

After applying, Terraform updates the NodePool to include `reserved`, `spot`, and `on-demand` in the capacity-type requirements. Karpenter treats `reserved` as the most cost-efficient option and launches it first. Once the reservation is full, it falls back to Spot or On-Demand.

On the EKS Auto Mode path, Terraform creates a custom `gpu-inf` NodeClass (because the bundled `default` NodeClass is read-only) that references the ODCR by tag through `capacityReservationSelectorTerms`. On the self-managed Karpenter path, Terraform re-applies the `gpu-inf` EC2NodeClass with `capacityReservationSelectorTerms` added and updates the NodePool to include `reserved`.

Verify the ODCR was created:

aws ec2 describe-capacity-reservations
--filters "Name=state,Values=active" "Name=tag:nodepool,Values=reserved-spot-ondemand"
--query 'CapacityReservations[0].{Id:CapacityReservationId,State:State,InstanceType:InstanceType,AvailableCount:AvailableInstanceCount}'
--output table
--region $(terraform output -raw region)

Verify the NodeClass references the ODCR:

Verify the NodePool is ready:

kubectl get nodepools gpu-inf

Expected output:

NAME NODECLASS NODES READY AGE gpu-inf gpu-inf 0 True 30s

After applying the changes, validate that Karpenter prioritizes reserved capacity and falls back to Spot or On-Demand. Deploy a 2-replica Deployment that requests 1 GPU per Pod. The ODCR is for 1 instance (1 GPU), so the first Pod triggers Karpenter to launch a reserved node. The second Pod cannot fit on the reserved node and triggers Karpenter to launch another node from Spot or On-Demand capacity.

cat gpu-overflow-test-55d55ff5b9-hw4m9 1/1 Running 0 4m43s 10.0.82.49 i-0f50cdbacb2017202

Check the NodeClaims to see the capacity types:

kubectl get nodeclaims

Expected output:

NAME TYPE CAPACITY ZONE NODE READY AGE gpu-inf-vw99m g6e.4xlarge reserved us-east-2c i-0f50cdbacb2017202 True 6m gpu-inf-s65s6 g6.xlarge spot us-east-2b i-08741a36089ff2088 True 5m59s

The reserved node launched first, followed by a Spot or On-Demand node once the reservation was full.

Clean up the test deployment:

kubectl delete deployment gpu-overflow-test

Monitoring

Terraform already provisioned the full monitoring stack during `terraform apply` in Step 1. The stack includes an Amazon Managed Service for Prometheus (AMP) workspace, IAM policies and EKS Pod Identity Associations for Prometheus remote-write and Grafana query access, the kube-prometheus-stack Helm chart (Prometheus, Grafana, kube-state-metrics, node-exporter), and the NVIDIA DCGM Exporter for GPU metrics.

This section covers verification of the deployed monitoring components.

Verify monitoring pods

Wait for all monitoring pods to be ready:

kubectl wait --for=condition=Ready pod --all -n monitoring --timeout=300s kubectl get pods -n monitoring

Expected output:

NAME READY STATUS RESTARTS AGE kube-prometheus-stack-grafana-7c58f54f77-rftrj 3/3 Running 0 5m kube-prometheus-stack-kube-state-metrics-d68dcbc84-5smxq 1/1 Running 0 5m kube-prometheus-stack-operator-5895df479f-ttm47 1/1 Running 0 5m kube-prometheus-stack-prometheus-node-exporter-t9q7s 1/1 Running 0 5m kube-prometheus-stack-prometheus-node-exporter-x6vfb 1/1 Running 0 5m prometheus-kube-prometheus-stack-prometheus-0 2/2 Running 0 5m

Access Grafana

Grafana is exposed through an internet-facing AWS Application Load Balancer (ALB), restricted to the CIDR you set in `var.my_cidr`. Print the load balancer URL (allow a minute or two for the ALB to provision):

echo "http://$(kubectl get ingress kube-prometheus-stack-grafana -n monitoring -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')"

Open the URL in your browser. Log in with username `admin` and the password from the following command:

kubectl --namespace monitoring get secrets kube-prometheus-stack-grafana -o jsonpath="{.data.admin-password}" | base64 -d ; echo

Verify the metrics pipeline

To verify the metrics pipeline is working end to end:

Navigate to Connections > Data sources and confirm Amazon-Managed-Prometheus is listed as the default datasource.

                  Validate the AMP datasource in Grafana

Navigate to Drilldown > Metrics and search for the `up` metric. You should see results from your cluster’s scrape targets.

                  Validate the `up` metric in Grafana

If `up` shows results, the pipeline (cluster → Prometheus → AMP → Grafana) is working.

Validate DCGM GPU metrics

The DCGM Exporter DaemonSet runs on GPU nodes and reports GPU utilization, memory, temperature, power draw, NVLink bandwidth, and tensor activity metrics.

Verify the DCGM exporter DaemonSet:

kubectl get daemonset dcgm-exporter -n monitoring

Once a GPU node is running (from Step 2 or Step 4), you should see one or more ready Pods. To validate DCGM metrics, navigate to Drilldown > Metrics in Grafana and search for `DCGM_`.

            Validate DCGM metrics in Grafana

To view the dashboard, navigate to Dashboards > GPU Monitoring > NVIDIA DCGM Exporter Dashboard.

            NVIDIA DCGM Exporter Dashboard in Grafana

Model weights S3 bucket

Terraform already created an Amazon S3 bucket for storing model weights, a `model-storage-sa` ServiceAccount in the `default` namespace, an IAM policy scoped to the bucket, and an EKS Pod Identity Association that links them. Workload Pods that set `serviceAccountName: model-storage-sa` can read from and write to the bucket.

Verify the bucket

Retrieve the bucket name from Terraform outputs:

MODEL_BUCKET=$(terraform output -raw model_bucket) echo ${MODEL_BUCKET}

Verify the bucket exists:

aws s3api head-bucket --bucket ${MODEL_BUCKET}

Expected output:

{ "BucketArn": "arn:aws:s3:::ai-eks-docs-models-20250612abc1", "BucketRegion": "us-east-2", "AccessPointAlias": false }

Run a one-off Pod with the AWS CLI image, using the `model-storage-sa` ServiceAccount, to confirm EKS Pod Identity is wired up and S3 access works:

cat

Next steps

With your cluster ready, you can proceed to Load & Serve Model to deploy a large language model and interact with the inference endpoint.

Cleanup

  Tip

If you plan to continue with the next sections of this guide, skip the full cleanup. Only run it when you are done.

Delete the test workloads so no Pods are holding GPU nodes:

kubectl delete pod nvidia-smi --ignore-not-found
kubectl delete deployment gpu-overflow-test --ignore-not-found

If you only want to release the ODCR and fall back to Spot and On-Demand capacity, switch the nodepools variable back to the spot-ondemand strategy:

terraform apply -var 'nodepools={"spot-ondemand"={}}'

This drops reserved from the NodePool capacity-type requirements and destroys the ODCR, and leaves the cluster, monitoring stack, and S3 bucket in place.

Important

Cancelling a reservation does not terminate instances already running on it. Those instances keep running at standard On-Demand rates until they are terminated. Delete the GPU workloads first, as shown previously, so the reserved node drains before the reservation is released.

Drain the Karpenter-managed nodes before destroying, so no in-flight node lifecycle blocks the destroy. Delete any PodDisruptionBudgets that would prevent a drain, then delete the NodeClaims:

kubectl delete pdb -A --all --ignore-not-found
kubectl delete nodeclaim --all --wait=true --timeout=900s

Then destroy everything Terraform created, including the EKS cluster, the VPC, the monitoring stack, the NodePools and NodeClasses, the S3 model bucket, and any ODCR:

terraform destroy

Warning

The model weights S3 bucket is created with force_destroy = true, so terraform destroy deletes the bucket along with any model weights you uploaded to it. Copy anything you want to keep to another location first.

Note

The repository also ships a scripts/cleanup.sh helper that runs the preceding drain and destroy steps and then sweeps any orphaned EBS volumes tagged with the cluster name. Run it from inside the terraform// directory you applied from, and pass --auto-approve to skip the Terraform confirmation prompt.

Confirm no active Capacity Reservation remains for the cluster:

aws ec2 describe-capacity-reservations \
  --filters "Name=state,Values=active" "Name=tag:nodepool,Values=reserved-spot-ondemand" \
  --query 'CapacityReservations[].CapacityReservationId' \
  --output text

An empty result means no reservation is active and no further charges apply.

Javascript is disabled or is unavailable in your browser.

To use the Amazon Web Services Documentation, Javascript must be enabled. Please refer to your browser's Help pages for instructions.

Document Conventions Command-line interface (CLI) Inference

Did this page help you? - Yes

Thanks for letting us know we're doing a good job!

If you've got a moment, please tell us what we did right so we can do more of it.

Did this page help you? - No

Thanks for letting us know this page needs work. We're sorry we let you down.

If you've got a moment, please tell us how we can make the documentation better.

더 알아보기 (Learn more)

• 공식 문서