Installation

This guide describes how to install Kubeflow Trainer control plane on a Kubernetes cluster.

You can skip these steps if the Kubeflow platform is already deployed using manifests or package distributions, as it includes Kubeflow Trainer by default.

Prerequisites

These are the minimal requirements to install Kubeflow Trainer control plane:

  • Kubernetes >= 1.31

  • kubectl >= 1.31

Tip

If you don’t have Kubernetes cluster, you can quickly create one locally using Kind:

kind create cluster # or minikube start

Installing the Kubeflow Trainer Controller Manager

Run the following command to deploy a released version of Kubeflow Trainer control plane:

export VERSION=v2.1.0
helm install kubeflow-trainer oci://ghcr.io/kubeflow/charts/kubeflow-trainer \
    --namespace kubeflow-system \
    --create-namespace \
    --version ${VERSION#v}

For the latest changes run (where 48e7a93 is the desired commit):

helm install kubeflow-trainer oci://ghcr.io/kubeflow/charts/kubeflow-trainer \
    --namespace kubeflow-system \
    --create-namespace \
    --version 0.0.0-sha-48e7a93

Note

The Trainer CRDs (TrainJob, TrainingRuntime, ClusterTrainingRuntime) are installed by the chart by default. If you manage the CRDs out-of-band (previously via Helm’s --skip-crds flag), set --set crds.enabled=false to skip installing them with the chart.

You can enable the default ClusterTrainingRuntimes together with the control plane in a single step. A post-install Helm hook applies the runtimes once the CRDs and controller are ready:

helm install kubeflow-trainer oci://ghcr.io/kubeflow/charts/kubeflow-trainer \
    --namespace kubeflow-system \
    --create-namespace \
    --version ${VERSION#v} \
    --set runtimes.defaultEnabled=true

To enable specific runtimes instead of all of them:

helm install kubeflow-trainer oci://ghcr.io/kubeflow/charts/kubeflow-trainer \
    --namespace kubeflow-system \
    --create-namespace \
    --version ${VERSION#v} \
    --set runtimes.torchDistributed.enabled=true \
    --set runtimes.deepspeedDistributed.enabled=true

You can also enable runtimes on an existing installation with helm upgrade using the same --set flags. The hook reconciles runtimes on every upgrade: newly enabled runtimes are applied and disabled ones are removed. Disabling all runtimes removes the installer itself, so in that case delete any remaining runtimes manually or with helm uninstall.

For the available Helm values to configure runtimes, see the kubeflow-trainer Helm chart documentation.

Install with Kustomize

Run the following command to deploy Kubeflow Trainer control plane with kustomize:

kubectl apply --server-side -k "https://github.com/kubeflow/trainer.git/manifests/overlays/manager?ref=${VERSION}"

For the latest changes run:

kubectl apply --server-side -k "https://github.com/kubeflow/trainer.git/manifests/overlays/manager?ref=master"

Run the following command to deploy Kubeflow Trainer built-in runtimes

kubectl apply --server-side -k "https://github.com/kubeflow/trainer.git/manifests/overlays/runtimes?ref=master"

Verify the Control Plane

Ensure that the JobSet and Trainer controller manager pods are running:

$ kubectl get pods -n kubeflow-system

NAME                                                  READY   STATUS    RESTARTS   AGE
jobset-controller-manager-54968bd57b-88dk4            2/2     Running   0          65s
kubeflow-trainer-controller-manager-cc6468559-dblnw   1/1     Running   0          65s

Next Steps