Identify scaling metric thresholds for AI inference
Identify scaling metric thresholds for AI inference
[추출 작업 중]
출처: 문서
본문
Help improve this page
To contribute to this user guide, choose the Edit this page on GitHub link that is located in the right pane of every page.
Identify scaling metric thresholds for AI inference
Tip
Register for upcoming Amazon EKS AI/ML workshops.
This section shows how to load test a vLLM inference server on Amazon EKS to identify when it becomes saturated. Use the load test results to configure the queue depth and latency thresholds for horizontal autoscaling in the subsequent Autoscale AI inference with HPA and KEDA section.
The walkthrough uses the following tools:
k6 (Grafana k6) is an open-source load-testing tool, run as the `grafana/k6` container image, that sends inference requests to the replica at controlled request rates.
Ministral-3-8B-Instruct-2512 model on a `g6e.4xlarge` instance. You can apply the same methodology to any open source model and GPU type.
Prerequisites
This section builds on two earlier sections in the documentation. Complete both before you start:
Cluster Setup
provides an EKS cluster with a GPU NodePool, the kube-prometheus-stack Monitoring stack (Prometheus and Grafana), and an Amazon S3 bucket for model weights.
Load & Serve Models
deploys the vLLM inference server and creates the vllm-inference-app ServiceMonitor.
Confirm each item below before you start, and complete the linked section first if a check fails.
If you opened a new terminal, set the cluster name and region you used earlier:
export CLUSTER_NAME=ai-eks-docs
export AWS_REGION=us-east-2
Capture the name of the Amazon S3 model bucket that was created in the Cluster Setup steps. The load generator passes this value to vLLM.
MODEL_BUCKET=$(aws s3api list-buckets \
--query "Buckets[?starts_with(Name, '${CLUSTER_NAME}-models-')].Name | [0]" \
--output text)
echo "Model bucket: ${MODEL_BUCKET}"
Confirm the monitoring stack is running
The kube-prometheus-stack from the Monitoring setup runs in the monitoring namespace.
kubectl get pods -n monitoring
The Prometheus and Grafana pods should be Running:
NAME READY STATUS RESTARTS AGE
kube-prometheus-stack-grafana-6f9c8b7d5c-2xk9p 3/3 Running 0 3h
kube-prometheus-stack-operator-7b8c9d6f4-q4m7n 1/1 Running 0 3h
prometheus-kube-prometheus-stack-prometheus-0 2/2 Running 0 3h
If these are missing, complete the Monitoring setup before continuing.
Confirm the vLLM model is running
The vllm-inference-app Deployment and vllm-inference-svc Service from Load & Serve Models run in the default namespace.
kubectl get deployment vllm-inference-app
The Deployment should report 1/1 ready:
NAME READY UP-TO-DATE AVAILABLE AGE
vllm-inference-app 1/1 1 1 3h
If the Deployment is missing or not ready, complete Load & Serve Models before continuing.
Access Grafana
You watch the load test in the pre-loaded vLLM dashboard, so make sure the Grafana load balancer you set up in the Access Grafana section is reachable. Print its URL:
echo "http://$(kubectl get ingress kube-prometheus-stack-grafana -n monitoring -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')"
Open the URL in your browser and log in with username admin and the password from the following command:
kubectl --namespace monitoring get secrets kube-prometheus-stack-grafana -o jsonpath="{.data.admin-password}" | base64 -d ; echo
Confirm the vLLM ServiceMonitor exists
The vllm-inference-app ServiceMonitor, created in the Monitor vLLM step, runs in the default namespace and tells Prometheus where to scrape vLLM metrics.
kubectl get servicemonitor vllm-inference-app
Expected output:
NAME AGE
vllm-inference-app 3h
If it is missing, complete the Monitor vLLM step.
Step 1: Warm up the GPU
Even though the model is loaded into GPU memory and the replica reports Ready, the first requests take longer than later ones because vLLM and the GPU finish initializing on the first inferences. vLLM captures CUDA graphs, compiles and autotunes GPU kernels, and allocates its KV cache memory pools, while the GPU raises its clock speeds from idle to their boost range.
These are one-time costs, so measuring thresholds before warm-up records startup work instead of steady-state serving. The following warm-up sends 100 sequential requests to complete this initialization, so the load test that follows measures steady-state capacity.
First, create the ConfigMap that holds the warm-up script (warmup.js). The script reads the target model from the MODEL environment variable, which the Job sets from the MODEL_BUCKET value you captured earlier:
Example k6 warm-up ConfigMap
cat Example Warm-up Job
cat Access Grafana for details) and navigate to Dashboards > GPU Monitoring > Performance Testing - vLLM Load Analysis. The vLLM Request Rate panel shows the burst of warm-up traffic. The vLLM Average Latency panel shows that the first request is slower, which is the one-time GPU warm-up cost.
vLLM dashboard showing the GPU warm-up effect on latency
Then delete the warm-up Job:
kubectl delete job vllm-warmup
Step 2: Send load at increasing request rates
In this step, you run a load test that sends requests to the single vLLM replica at increasing rates: 10, 20, 30, 40, 50, 60, and 70 requests per second. This simulates increasing user demand. The test uses k6 in a constant-arrival-rate model, where the TARGET_RPS environment variable sets the target request rate. Each rate runs for 60 seconds to allow metrics to stabilize, followed by a 30-second pause before the next rate increase.
The goal of this test is to identify the saturation point, which is the request rate at which a single replica can no longer keep up with incoming requests. You monitor queue depth (vllm:num_requests_waiting) and end-to-end latency (vllm:e2e_request_latency_seconds). Increasing queue depth and latency indicate that the replica is approaching saturation.
First, create the ConfigMap that holds the k6 load-test script (script.js). The script reads the target model from the MODEL environment variable, which each pod sets from the MODEL_BUCKET value you captured earlier:
Example k6 load test ConfigMap
cat Example Load test loop
for RPS in 10 20 30 40 50 60 70; do echo "=== Testing at ${RPS} req/s ===" cat
Queue depth (`vllm:num_requests_waiting`) — how many requests are waiting to be processed.
p95 end-to-end latency (`vllm:e2e_request_latency_seconds`) — the 95th percentile response time.
View the metrics in Grafana
Open Grafana (open the load balancer hostname and log in as admin; see Access Grafana for details) and navigate to Dashboards > GPU Monitoring > Performance Testing - vLLM Load Analysis. This dashboard reads from Amazon Managed Service for Prometheus, so panels can lag the live state by up to a minute because metrics are remote-written in batches.
Performance Testing - vLLM Load Analysis dashboard during the load test
Based on the load test results, queue depth stayed near zero through 50 concurrent requests. It then increased sharply at 60 concurrent requests, peaking between approximately 150 and 350 queued requests. At the same time, p95 end-to-end latency rose from a steady baseline of about 2.5 seconds to 6–10 seconds. This behavior indicates the onset of sustained overload and suggests that autoscaling should begin before queue depth reaches levels that cause latency to approach 10 seconds. A practical starting point is to trigger scaling when queue depth exceeds 25 requests for 30–60 seconds. As a secondary signal, you can also trigger scaling when p95 end-to-end latency exceeds 5 seconds over the same interval. This helps account for workload patterns where latency increases before queue depth builds up.
Step 4: Choose the metrics and thresholds to scale on
Using the metrics observed at the saturation point, you determine the thresholds used for configuring autoscaling in the next section.
Queue depth (`vllm:num_requests_waiting`) — the number of requests waiting to be processed once a replica becomes saturated. Set the threshold above the point where the queue first remains positive, so transient spikes do not trigger scaling. For the load test in this section, queue depth remained near zero through 50 concurrent requests and increased sharply to approximately 150–350 queued requests at 60 concurrent requests. A practical starting point is to trigger scaling when queue depth exceeds 25 requests for 30–60 seconds.
Latency (`vllm:e2e_request_latency_seconds`) — the end-to-end response time as a replica approaches its capacity limit. Set the threshold below your latency SLO, so the autoscaler scales out before user-visible latency becomes unacceptable. For the load test in this section, p95 end-to-end latency remained stable at approximately 2.5 seconds and increased to 6–10 seconds at saturation. As a secondary signal, a practical starting point is to trigger scaling when p95 end-to-end latency exceeds 5 seconds for 30–60 seconds. This helps account for workload patterns where latency rises before queue depth builds up or when latency increases without queueing.
Scale-down — the rate at which the autoscaler removes replicas after demand drops. Scale down more slowly than you scale up. For this example, scale down only when queue depth is `0` and p95 end-to-end latency is below 3 seconds for 5 minutes. This prevents the autoscaler from removing replicas during short pauses in traffic and helps avoid oscillation.
These values provide a starting point for configuring autoscaling in the next section. Repeat this process for your own model, GPU type, and request patterns to determine the appropriate thresholds for your deployment.
Javascript is disabled or is unavailable in your browser.
To use the Amazon Web Services Documentation, Javascript must be enabled. Please refer to your browser's Help pages for instructions.
Document Conventions Autoscaling HPA and KEDA
Did this page help you? - Yes
Thanks for letting us know we're doing a good job!
If you've got a moment, please tell us what we did right so we can do more of it.
Did this page help you? - No
Thanks for letting us know this page needs work. We're sorry we let you down.
If you've got a moment, please tell us how we can make the documentation better.