FAQs

General

How does a provisioner decide to manage a particular node?

See Configuring provisioners for information on how Karpenter provisions and manages nodes.

What cloud providers are supported?

AWS is the first cloud provider supported by Karpenter, although it is designed to be used with other cloud providers as well. See Cloud provider for details.

Can I write my own cloud provider for Karpenter?

Yes, but there is no documentation yet for it. Start with Karpenter’s GitHub cloudprovider documentation to see how the AWS provider is built, but there are other sections of the code that will require changes too.

What operating system nodes does Karpenter deploy?

By default, Karpenter uses Amazon Linux 2 images.

Can I provide my own custom operating system images?

Karpenter allows you to create your own AWS AMIs using custom launch templates. See Launch Templates and Custom Images for details.

Can Karpenter deal with workloads for mixed architecture cluster (arm vs. amd)?

Yes. Build and prepare custom arm images as described in Launch Templates and Custom Images. Specify the desired architecture when you deploy workloads.

What RBAC access is required?

All of the required RBAC rules can be found in the helm chart template. See clusterrolebinding.yaml, clusterrole.yaml, rolebinding.yaml, and role.yaml files for details.

Can I run Karpenter outside of a Kubernetes cluster?

Yes, as long as the controller has network and IAM/RBAC access to the Kubernetes API and your provider API.

Compatibility

Which versions of Kubernetes does Karpenter support?

Karpenter is tested with Kubernetes v1.19 and later.

What Kubernetes distributions are supported?

Karpenter documents integration with a fresh install of the latest AWS Elastic Kubernetes Service (EKS). Existing EKS distributions can be used, but this use case has not yet been documented. Other Kubernetes distributions (KOPs, etc.) can be used, but setting up cloud provider permissions for those distributions has not been documented.

How does Karpenter interact with AWS node group features?

Provisioners are designed to work alongside static capacity management solutions like EKS Managed Node Groups and EC2 Auto Scaling Groups. You can manage all capacity using provisioners, use a mixed model with dynamic and statically managed capacity, or use a fully static approach. We expect most users will use a mixed approach in the near term and provisioner-managed in the long term.

How does Karpenter interact with Kubernetes features?

• Kubernetes Cluster Autoscaler: Karpenter can work alongside cluster autoscaler. See Kubernetes cluster autoscaler for details.
• Kubernetes Scheduler: Karpenter focuses on scheduling pods that the Kubernetes scheduler has marked as unschedulable. See Scheduling for details on how Karpenter interacts with the Kubernetes scheduler.

Provisioning

What features does the Karpenter provisioner support?

See Provisioner API for provisioner examples and descriptions of features.

Can I create multiple (team-based) provisioners on a cluster?

Yes, provisioners can identify multiple teams based on labels. See Provisioner API for details.

If multiple provisioners are defined, which will my pod use?

Pending pods will be handled by any Provisioner that matches the requirements of the pod. There is no ordering guarantee if multiple provisioners match pod requirements. We recommend that Provisioners are setup to be mutually exclusive. Read more about this recommendation in the EKS Best Practices Guide for Karpenter. To select a specific provisioner, use the node selector karpenter.sh/provisioner-name: my-provisioner.

How can I configure Karpenter to only provision pods for a particular namespace?

There is no native support for namespaced based provisioning. Karpenter can be configured to provision a subset of pods based on a combination of taints/tolerations and node selectors. This allows Karpenter to work in concert with the kube-scheduler in that the same mechanisms that kube-scheduler uses to determine if a pod can schedule to an existing node are also used for provisioning new nodes. This avoids scenarios where pods are bound to nodes that were provisioned by Karpenter which Karpenter would not have bound itself. If this were to occur, a node could remain non-empty and have its lifetime extended due to a pod that wouldn’t have caused the node to be provisioned had the pod been unschedulable.

We recommend using Kubernetes native scheduling constraints to achieve namespace based scheduling segregation. Using native scheduling constraints ensures that Karpenter, kube-scheduler and any other scheduling or auto-provisioning mechanism all have an identical understanding of which pods can be scheduled on which nodes. This can be enforced via policy agents, an example of which can be seen here.

Can I set total limits of CPU and memory for a provisioner?

Yes, the setting is provider-specific. See examples in Accelerators, GPU Karpenter documentation.

Can I mix spot and on-demand EC2 run types?

Yes, see Example Provisioner Resource for an example.

Can I restrict EC2 instance types?

• Attribute-based requests are currently not possible.
• You can select instances with special hardware, such as gpu.

Can I use Bare Metal instance types?

Yes, Karpenter supports provisioning metal instance types when a Provisioner’s node.kubernetes.io/instance-type Requirements only include metal instance types. If other instance types fulfill pod requirements, then Karpenter will prioritize all non-metal instance types before metal ones are provisioned.

How does Karpenter dynamically select instance types?

Karpenter batches pending pods and then binpacks them based on CPU, memory, and GPUs required, taking into account node overhead, VPC CNI resources required, and daemon sets that will be packed when bringing up a new node. By default Karpenter uses all available instance types, but it can be constrained in the provisioner spec with the instance-type well-known label in the requirements section. After the pods are binpacked on the most efficient instance type (i.e. the smallest instance type that can fit the pod batch), Karpenter takes 19 other instance types that are larger than the most efficient packing, and passes all 20 instance type options to an API called Amazon EC2 Fleet. The EC2 fleet API attempts to provision the instance type based on a user-defined allocation strategy. If you are using the on-demand capacity type, then Karpenter uses the lowest-price allocation strategy. So fleet will provision the lowest price instance type it can get from the 20 Karpenter passed it. If the instance type is unavailable for some reason, then fleet will move on to the next cheapest instance type. If you are using the spot capacity type, Karpenter uses the capacity-optimized-prioritized allocation strategy which tells fleet to find the instance type that EC2 has the most capacity of which will decrease the probability of a spot interruption happening in the near term. See Choose the appropriate allocation strategy for information on fleet optimization.

What if there is no Spot capacity? Will Karpenter use On-Demand?

The best defense against running out of Spot capacity is to allow Karpenter to provision as many different instance types as possible. Even instance types that have higher specs, e.g. vCPU, memory, etc., than what you need can still be cheaper in the Spot market than using On-Demand instances. When Spot capacity is constrained, On-Demand capacity can also be constrained since Spot is fundamentally spare On-Demand capacity. Allowing Karpenter to provision nodes from a large, diverse set of instance types will help you to stay on Spot longer and lower your costs due to Spot’s discounted pricing. Moreover, if Spot capacity becomes constrained, this diversity will also increase the chances that you’ll be able to continue to launch On-Demand capacity for your workloads.

If your Karpenter Provisioner specifies flexibility to both Spot and On-Demand capacity, Karpenter will attempt to provision On-Demand capacity if there is no Spot capacity available. However, it’s strongly recommended that you specify at least 20 instance types in your Provisioner (or none and allow Karpenter to pick the best instance types) as our research indicates that this additional diversity increases the chances that your workloads will not need to launch On-Demand capacity at all. Today, Karpenter will warn you if the number of instances in your Provisioner isn’t sufficiently diverse.

Technically, Karpenter has a concept of an “offering” for each instance type, which is a combination of zone and capacity type (equivalent in the AWS cloud provider to an EC2 purchase option – Spot or On-Demand). Whenever the Fleet API returns an insufficient capacity error for Spot instances, those particular offerings are temporarily removed from consideration (across the entire provisioner) so that Karpenter can make forward progress with different options.

Workloads

How can someone deploying pods take advantage of Karpenter?

See Application developer for descriptions of how Karpenter matches nodes with pod requests.

Can I use Karpenter with EBS disks per availability zone?

Yes. See Persistent Volume Topology for details.

Can I set --max-pods on my nodes?

Not yet.

Deprovisioning

How does Karpenter deprovision nodes?

See Deprovisioning nodes for information on how Karpenter deprovisions nodes.

Upgrading

How do I upgrade Karpenter?

Karpenter is a controller that runs in your cluster, but it is not tied to a specific Kubernetes version, as the Cluster Autoscaler is. Use your existing upgrade mechanisms to upgrade your core add-ons in Kubernetes and keep Karpenter up to date on bug fixes and new features.

Karpenter requires proper permissions in the KarpenterNode IAM Role and the KarpenterController IAM Role. To upgrade Karpenter to version $VERSION, make sure that the KarpenterNode IAM Role and the KarpenterController IAM Role have the right permission described in https://karpenter.sh/$VERSION/getting-started/getting-started-with-eksctl/cloudformation.yaml. Next, locate KarpenterController IAM Role ARN (i.e., ARN of the resource created in Create the KarpenterController IAM Role) and the cluster endpoint, and pass them to the helm upgrade command

helm upgrade --install --namespace karpenter --create-namespace \
  karpenter karpenter/karpenter \
  --version ${KARPENTER_VERSION} \
  --set serviceAccount.annotations."eks\.amazonaws\.com/role-arn"=${KARPENTER_IAM_ROLE_ARN} \
  --set clusterName=${CLUSTER_NAME} \
  --set clusterEndpoint=${CLUSTER_ENDPOINT} \
  --set aws.defaultInstanceProfile=KarpenterNodeInstanceProfile-${CLUSTER_NAME} \
  --wait # for the defaulting webhook to install before creating a Provisioner

For information on upgrading Karpenter, see the Upgrade Guide.

Why do I get an unknown field "startupTaints" error when creating a provisioner with startupTaints?

error: error validating "provisioner.yaml": error validating data: ValidationError(Provisioner.spec): unknown field "startupTaints" in sh.karpenter.v1alpha5.Provisioner.spec; if you choose to ignore these errors, turn validation off with --validate=false

The startupTaints parameter was added in v0.10.0. Helm upgrades do not upgrade the CRD describing the provisioner, so it must be done manually. For specific details, see the Upgrade Guide