Kubeflow: Orchestrating End-to-End ML Pipelines on Kubernetes
Building and deploying machine learning models in production can be a complex endeavor. From data preparation and model training to hyperparameter tuning, serving, and monitoring, the ML lifecycle involves numerous distinct steps, often requiring specialized tools and infrastructure. Managing this entire workflow, especially at scale, presents significant challenges for data scientists and MLOps engineers alike. How do you ensure reproducibility, track experiments, and seamlessly transition models from development to production environments?
Enter Kubeflow, an open-source project dedicated to making deployments of machine learning (ML) workflows on Kubernetes simple, portable, and scalable. By leveraging the power of Kubernetes – the de facto standard for container orchestration – Kubeflow provides a robust platform that streamlines the entire ML journey. It integrates various components like Jupyter notebooks, ML training operators, and serving engines, all designed to run natively on your Kubernetes cluster, offering a unified experience for developing, orchestrating, and scaling your ML pipelines.
This guide will walk you through setting up Kubeflow and demonstrating how to build and execute an end-to-end ML pipeline. We’ll cover everything from installation to creating a simple, reproducible workflow, ensuring you have the foundational knowledge to harness Kubeflow’s capabilities for your MLOps initiatives. Get ready to transform your ML development process with the power and flexibility of Kubernetes and Kubeflow.
TL;DR: Kubeflow for End-to-End ML Pipelines
Kubeflow simplifies the deployment and management of ML workflows on Kubernetes. It provides a platform for data scientists and MLOps engineers to build, train, and deploy models at scale. This guide covers its installation and demonstrates a basic ML pipeline.
Key Commands:
# Install kfctl (Kubeflow CLI)
wget https://github.com/kubeflow/kfctl/releases/download/v1.2.0/kfctl_v1.2.0_linux.tar.gz
tar -xvf kfctl_v1.2.0_linux.tar.gz
sudo mv kfctl /usr/local/bin/
# Set environment variables for Kubeflow deployment
export KUBEFLOW_TAG=v1.2.0
export KUBEFLOW_APP_NAME=kf-cluster
export KUBEFLOW_DIR=$(pwd)/${KUBEFLOW_APP_NAME}
mkdir -p ${KUBEFLOW_DIR}
cd ${KUBEFLOW_DIR}
# Download Kubeflow configuration
wget https://raw.githubusercontent.com/kubeflow/manifests/${KUBEFLOW_TAG}/kfctl_k8s_istio.yaml
# Deploy Kubeflow (can take 10-20 minutes)
kfctl apply -f kfctl_k8s_istio.yaml
# Access Kubeflow UI (port-forwarding example)
kubectl port-forward -n istio-system svc/istio-ingressgateway 8080:80
# Or find the external IP of istio-ingressgateway
Prerequisites
Before diving into the Kubeflow installation and pipeline creation, ensure you have the following in place:
* **Kubernetes Cluster**: A running Kubernetes cluster (version 1.16-1.20 is generally recommended for Kubeflow 1.2, but newer versions might work with appropriate manifest adjustments). This can be a local cluster (e.g., Kind, Minikube, K3s) or a managed cloud cluster (e.g., AWS EKS, GCP GKE, Azure AKS). For production, a multi-node cluster with sufficient resources (at least 4 vCPUs and 16GB RAM for a basic Kubeflow deployment) is essential.
* **`kubectl`**: The Kubernetes command-line tool, configured to connect to your cluster. Refer to the official Kubernetes documentation for installation instructions.
* **`kustomize`**: A Kubernetes native configuration management tool used by Kubeflow. It’s often bundled with `kubectl` (v1.14+). You can check its version with `kubectl kustomize version`.
* **`curl` or `wget`**: For downloading files.
* **Basic understanding of Kubernetes**: Familiarity with concepts like Pods, Deployments, Services, Namespaces, and Ingress is beneficial. For networking and security aspects, our Network Policies Security Guide can be a helpful reference.
* **Basic understanding of Machine Learning concepts**: Familiarity with ML workflows (data preprocessing, training, evaluation, serving) will help you understand the pipeline components.
* **Sufficient Cluster Resources**: Kubeflow is resource-intensive. Ensure your cluster has enough CPU, memory, and persistent storage. A typical full installation can consume several GBs of RAM and multiple CPU cores. For cost optimization on cloud providers, consider tools like Karpenter for dynamic node provisioning.
Step-by-Step Guide: Deploying Kubeflow and Running an ML Pipeline
Step 1: Install `kfctl` and Prepare Environment
`kfctl` is the command-line interface (CLI) tool for deploying Kubeflow. It simplifies the process of configuring and deploying the various Kubeflow components onto your Kubernetes cluster. We’ll download a specific version of `kfctl` and place it in our system’s PATH. We’ll also set up environment variables that define the Kubeflow version, application name, and the directory where Kubeflow’s configuration files will reside. This ensures a consistent and reproducible installation.
# 1. Download kfctl. We're using v1.2.0 as it's a stable and well-documented version for demonstrations.
# You can find other versions on the official Kubeflow releases page.
wget https://github.com/kubeflow/kfctl/releases/download/v1.2.0/kfctl_v1.2.0_linux.tar.gz
# 2. Extract the archive
tar -xvf kfctl_v1.2.0_linux.tar.gz
# 3. Move the kfctl binary to a directory in your PATH (e.g., /usr/local/bin)
sudo mv kfctl /usr/local/bin/
# 4. Verify kfctl installation
kfctl version
# 5. Set environment variables for Kubeflow deployment
export KUBEFLOW_TAG=v1.2.0
export KUBEFLOW_APP_NAME=kf-cluster
export KUBEFLOW_DIR=$(pwd)/${KUBEFLOW_APP_NAME}
# 6. Create the directory for Kubeflow configurations
mkdir -p ${KUBEFLOW_DIR}
# 7. Change to the Kubeflow directory
cd ${KUBEFLOW_DIR}
# 8. Download the Kubeflow configuration file. This YAML defines the components to be installed.
# We're using kfctl_k8s_istio.yaml for a standard Kubernetes deployment with Istio.
# Istio is a common choice for managing traffic within Kubeflow, though other options exist.
wget https://raw.githubusercontent.com/kubeflow/manifests/${KUBEFLOW_TAG}/kfctl_k8s_istio.yaml
**Verify:**
After executing the commands, you should see the `kfctl` version output, and the `kfctl_k8s_istio.yaml` file should be present in your `kf-cluster` directory.
# Expected output for kfctl version
kfctl version:
Built with kfctl v1.2.0+g06dd76a
Kubeflow version: v1.2.0
Kubernetes version: v1.19.0
# Expected output for ls -l
ls -l
total 48
-rw-r--r-- 1 user user 45242 Oct 26 10:00 kfctl_k8s_istio.yaml
Step 2: Deploy Kubeflow to Your Kubernetes Cluster
Now that `kfctl` is installed and the configuration file is downloaded, we can proceed with deploying Kubeflow. This step will apply all the necessary Kubernetes manifests defined in `kfctl_k8s_istio.yaml` to your cluster. This process involves creating multiple namespaces, Deployments, Services, Custom Resource Definitions (CRDs), and other Kubernetes objects. It’s a comprehensive deployment that includes core Kubeflow components, Istio for service mesh capabilities, and other essential services. This can take a significant amount of time, depending on your cluster’s size and internet connection.
# Deploy Kubeflow using the downloaded configuration file.
# This command will read the kfctl_k8s_istio.yaml and apply all resources to your cluster.
# Be patient, as this process can take 10-20 minutes or even longer, depending on your cluster and network.
kfctl apply -f kfctl_k8s_istio.yaml
**Verify:**
Monitor the output of the `kfctl apply` command. It will show the creation of various resources. Once it completes, you can check the status of the deployed components. All pods should eventually reach a `Running` or `Completed` state.
# Check if all Kubeflow namespaces are created
kubectl get ns | grep kubeflow
# Check pods in the 'kubeflow' namespace (core Kubeflow components)
kubectl get pods -n kubeflow
# Check pods in the 'kubeflow-pipelines' namespace (Kubeflow Pipelines components)
kubectl get pods -n kubeflow-pipelines
# Check pods in the 'istio-system' namespace (Istio components for ingress and service mesh)
kubectl get pods -n istio-system
You should see output similar to this (truncated for brevity), indicating namespaces and pods are being created:
# Example: kubectl get ns | grep kubeflow
kubeflow Active 5m
kubeflow-pipelines Active 5m
knative-serving Active 5m
# Example: kubectl get pods -n kubeflow
pod/admission-webhook-deployment-... 1/1 Running 0 5m
pod/jupyter-web-app-... 1/1 Running 0 5m
... (many more pods)
# Example: kubectl get pods -n kubeflow-pipelines
pod/ml-pipeline-... 1/1 Running 0 5m
pod/ml-pipeline-persistenceagent-... 1/1 Running 0 5m
... (many more pods)
# Example: kubectl get pods -n istio-system
pod/istio-ingressgateway-... 1/1 Running 0 5m
pod/istiod-... 1/1 Running 0 5m
Step 3: Access the Kubeflow Dashboard
After successful deployment, the next step is to access the Kubeflow Central Dashboard. This dashboard provides a unified interface for managing Jupyter notebooks, running pipelines, monitoring experiments, and deploying models. The primary way to access it is through the Istio Ingress Gateway, which exposes Kubeflow to external traffic. Depending on your Kubernetes cluster setup (local vs. cloud), you’ll either port-forward the gateway or use its external IP address.
# Option A: For local clusters (Kind, Minikube) or when an external IP is not provisioned
# This command forwards port 8080 on your local machine to port 80 of the Istio Ingress Gateway service.
kubectl port-forward -n istio-system svc/istio-ingressgateway 8080:80
# Option B: For cloud clusters (EKS, GKE, AKS) where an external IP is provisioned
# Get the external IP address of the Istio Ingress Gateway service.
# It might take a few minutes for the External IP to be assigned.
kubectl get svc -n istio-system istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}'
**Verify:**
If you used Option A, open your web browser and navigate to `http://localhost:8080`.
If you used Option B, copy the external IP address obtained from the command and paste it into your web browser.
You should be greeted by the Kubeflow Central Dashboard login page. The default credentials are typically:
* **Email:** `user@example.com`
* **Password:** `12341234`
Example of Kubeflow Central Dashboard. (Source: Kubeflow Documentation)
Step 4: Create a Jupyter Notebook Server
Jupyter notebooks are the primary interface for data scientists to interact with Kubeflow. They allow for interactive development, data exploration, and model prototyping. Kubeflow provides a Notebooks server that provisions Jupyter instances directly on your Kubernetes cluster, complete with custom images, resource requests, and persistent storage. This ensures that your development environment is consistent, reproducible, and scalable.
1. From the Kubeflow Central Dashboard, navigate to the **Notebooks** section on the left sidebar.
2. Click **New Server**.
3. Configure your notebook server:
* **Name**: `my-first-notebook`
* **Namespace**: `kubeflow-user-example-com` (default user namespace)
* **Image**: Select a pre-built image like `jupyter/tensorflow-notebook:latest` or `jupyter/scipy-notebook:latest`. You can also specify custom images.
* **CPU/Memory**: Allocate sufficient resources (e.g., 2 CPUs, 4GiB Memory).
* **Workspace Volume**: Create a new volume (e.g., `my-notebook-volume`, 10GiB size).
* **Data Volumes**: Optionally add additional data volumes.
* **Configurations**: Leave as default for now.
4. Click **Launch**.
**Verify:**
Monitor the status of your notebook server. It might take a few minutes for the pod to provision and the server to start. Once it’s `Running`, click **Connect** to open your JupyterLab interface.
# You can also verify the notebook pod creation using kubectl
kubectl get pods -n kubeflow-user-example-com | grep my-first-notebook
my-first-notebook-... 1/1 Running 0 2m
Step 5: Define and Run a Simple Kubeflow Pipeline
Kubeflow Pipelines is a platform for building and deploying portable, scalable machine learning workflows. It orchestrates multi-step ML workflows as directed acyclic graphs (DAGs), where each step runs in its own container. We’ll create a simple pipeline that demonstrates data preprocessing, model training, and evaluation. This will involve writing Python code using the Kubeflow Pipelines SDK.
1. **Open a New Notebook**: In your `my-first-notebook` JupyterLab instance, create a new Python 3 notebook (e.g., `simple_pipeline.ipynb`).
2. **Install Kubeflow Pipelines SDK**: Add the following to a cell and run it:
*Note: If you’re using a pre-built Kubeflow image, the SDK might already be installed.*
pip install kfp --user
3. **Write the Pipeline Code**: In your notebook, paste the following Python code. This pipeline consists of three simple components: `preprocess`, `train`, and `evaluate`. Each component is a Python function that will be compiled into a container image.
import kfp
from kfp import dsl
from kfp import compiler
# Define a base image for our pipeline components
BASE_IMAGE = 'python:3.9-slim-buster'
# Define pipeline components as Python functions
@dsl.component(base_image=BASE_IMAGE, packages_to_install=['scikit-learn'])
def preprocess_op(data_path: str, output_path: str):
"""
A simple preprocessing component.
Loads dummy data, performs a mock preprocessing step, and saves it.
"""
import pandas as pd
import numpy as np
import os
print(f"Preprocessing data from {data_path}")
# Simulate loading data
# In a real scenario, you'd load from a persistent storage like S3, GCS, or a PVC
dummy_data = pd.DataFrame(np.random.rand(100, 5), columns=[f'feature_{i}' for i in range(5)])
dummy_data['target'] = np.random.randint(0, 2, 100)
# Simulate a preprocessing step (e.g., scaling, feature engineering)
processed_data = dummy_data * 2
# Save processed data
os.makedirs(os.path.dirname(output_path), exist_ok=True)
processed_data.to_csv(output_path, index=False)
print(f"Processed data saved to {output_path}")
@dsl.component(base_image=BASE_IMAGE, packages_to_install=['scikit-learn'])
def train_op(processed_data_path: str, model_path: str):
"""
A simple training component.
Loads processed data, trains a dummy model, and saves it.
"""
import pandas as pd
from sklearn.linear_model import LogisticRegression
import joblib
import os
print(f"Training model with data from {processed_data_path}")
processed_data = pd.read_csv(processed_data_path)
X = processed_data.drop('target', axis=1)
y = processed_data['target']
# Simulate model training
model = LogisticRegression()
model.fit(X, y)
# Save the trained model
os.makedirs(os.path.dirname(model_path), exist_ok=True)
joblib.dump(model, model_path)
print(f"Model trained and saved to {model_path}")
@dsl.component(base_image=BASE_IMAGE, packages_to_install=['scikit-learn'])
def evaluate_op(model_path: str, processed_data_path: str, metrics_path: str):
"""
A simple evaluation component.
Loads the trained model and processed data, evaluates the model, and saves metrics.
"""
import pandas as pd
from sklearn.metrics import accuracy_score
import joblib
import json
import os
print(f"Evaluating model from {model_path} with data from {processed_data_path}")
model = joblib.load(model_path)
processed_data = pd.read_csv(processed_data_path)
X = processed_data.drop('target', axis=1)
y_true = processed_data['target']
y_pred = model.predict(X)
accuracy = accuracy_score(y_true, y_pred)
print(f"Model Accuracy: {accuracy}")
# Save metrics
metrics = {'accuracy': accuracy}
os.makedirs(os.path.dirname(metrics_path), exist_ok=True)
with open(metrics_path, 'w') as f:
json.dump(metrics, f)
print(f"Metrics saved to {metrics_path}")
# Define the Kubeflow Pipeline
@dsl.pipeline(
name='Simple ML Pipeline',
description='A basic end-to-end ML workflow.'
)
def simple_ml_pipeline(
data_path: str = '/tmp/data/raw_data.csv',
processed_data_path: str = '/tmp/data/processed_data.csv',
model_path: str = '/tmp/model/model.joblib',
metrics_path: str = '/tmp/metrics/metrics.json'
):
"""
Orchestrates the preprocessing, training, and evaluation steps.
"""
preprocess_task = preprocess_op(data_path=data_path, output_path=processed_data_path)
train_task = train_op(processed_data_path=preprocess_task.outputs['output_path'], model_path=model_path)
evaluate_task = evaluate_op(model_path=train_task.outputs['model_path'],
processed_data_path=preprocess_task.outputs['output_path'],
metrics_path=metrics_path)
# Compile the pipeline
# This generates a .yaml file that defines the pipeline's structure and components.
pipeline_filename = 'simple_ml_pipeline.yaml'
compiler.Compiler().compile(simple_ml_pipeline, pipeline_filename)
print(f"Pipeline compiled to {pipeline_filename}")
# Run the pipeline (this requires the Kubeflow Pipelines client)
# If running outside the notebook, you'd use `kfp.Client().create_run_from_pipeline_func(...)`
# For this example, we'll manually upload the compiled YAML via the UI.
4. **Run the Pipeline in Kubeflow UI**:
* After running the Python code in your notebook, the `simple_ml_pipeline.yaml` file will be created in your current directory.
* Go back to the Kubeflow Central Dashboard.
* Navigate to **Pipelines** on the left sidebar.
* Click **Upload pipeline**.
* Select the `simple_ml_pipeline.yaml` file you just generated.
* Give it a name (e.g., `Simple ML Pipeline`).
* Click **Upload**.
* Once uploaded, click on the pipeline name, then click **Create run**.
* Give the run a name (e.g., `My First ML Run`).
* Leave parameters as default.
* Click **Start**.
**Verify:**
Monitor the run in the Kubeflow Pipelines UI. You’ll see a DAG visualization of your pipeline, and each step will transition from `Pending` to `Running` to `Succeeded`. You can click on individual steps to view their logs, inputs, and outputs.
Example of a Kubeflow Pipeline Run Graph. (Source: Kubeflow Documentation)
Congratulations! You’ve successfully deployed Kubeflow and run your first end-to-end ML pipeline. This simple example lays the groundwork for more complex, real-world ML workflows, incorporating advanced features like hyperparameter tuning, model serving, and continuous integration/delivery (CI/CD) for ML.
Production Considerations
Deploying Kubeflow in a production environment requires careful planning and consideration beyond a simple development setup. Here are key aspects to address:
1. **Resource Management and Scaling**:
* **Node Sizing**: Kubeflow is resource-intensive. Ensure your Kubernetes nodes are adequately sized (CPU, Memory, Storage) to handle the various components (Istio, Minio, MySQL, Prometheus, Jupyter, Pipelines, etc.) and your ML workloads. For GPU-intensive tasks (like LLM GPU Scheduling), dedicated GPU nodes are essential.
* **Autoscaling**: Implement Kubernetes Horizontal Pod Autoscalers (HPAs) for stateless components and consider cluster autoscaling for dynamic node provisioning. Tools like Karpenter can significantly optimize costs by provisioning nodes precisely when needed.
* **Persistent Storage**: Use a robust, highly available, and performant storage solution (e.g., Persistent Volumes backed by cloud storage like AWS EBS, GCP Persistent Disk, Azure Disk, or a distributed file system like CephFS). Ensure proper backup and disaster recovery strategies for your data and models.
2. **Security and Access Control**:
* **Authentication and Authorization**: Integrate Kubeflow with your organization’s identity provider (e.g., LDAP, OAuth2/OIDC with Dex, Google Identity Platform). Implement Kubernetes RBAC to control who can access which Kubeflow resources and namespaces.
* **Network Policies**: Restrict inter-pod communication using Kubernetes Network Policies. This isolates sensitive components and prevents unauthorized access. For advanced network encryption, consider solutions like Cilium WireGuard Encryption.
* **Image Security**: Use trusted container images from secure registries. Implement image scanning for vulnerabilities. Consider using tools like Sigstore and Kyverno for ensuring the integrity and security of your container supply chain.
* **Secrets Management**: Store sensitive information (API keys, database credentials) using Kubernetes Secrets, ideally integrated with a dedicated secrets management solution (e.g., HashiCorp Vault, cloud provider secret managers).
3. **Observability and Monitoring**:
* **Logging**: Centralize logs from all Kubeflow components and ML workloads using a solution like the EFK/Loki stack.
* **Metrics**: Monitor resource utilization (CPU, memory, GPU), pipeline run status, model performance, and data drift. Prometheus and Grafana are commonly used for this. Kubeflow includes some pre-configured dashboards. For deeper insights into network and application performance, eBPF Observability with Hubble can provide granular metrics.
* **Alerting**: Set up alerts for critical events, such as pipeline failures, resource exhaustion, or model degradation.
4. **CI/CD for ML (MLOps)**:
* **Automated Pipeline Deployment**: Integrate Kubeflow Pipelines with your CI/CD system (Jenkins, GitLab CI, GitHub Actions) to automatically build, test, and deploy new pipeline versions.
* **Model Versioning and Registry**: Use a model registry (e.g., MLflow, Kubeflow’s Metadata Store) to track model versions, lineage, and associated metrics.
* **Reproducibility**: Ensure that experiments and pipelines are reproducible by versioning code, data, and environment configurations.
5. **Networking and Ingress**:
* **Istio Configuration**: Fine-tune Istio’s configuration for production traffic, including TLS/SSL termination, custom domains, and advanced routing rules. For more advanced traffic management, explore the Kubernetes Gateway API as an alternative or complementary solution.
* **Service Mesh**: Understand how Istio (or other service meshes like Istio Ambient Mesh) manages internal traffic within Kubeflow, providing features like traffic splitting, retries, and circuit breakers.
6. **Backup and Disaster Recovery**:
* Regularly back up critical Kubeflow components, especially the database used by Kubeflow Pipelines and the Minio object storage.
* Have a clear disaster recovery plan to restore Kubeflow services in case of an outage.
Troubleshooting
Kubeflow is a complex system, and issues can arise during deployment or operation. Here are some common problems and their solutions:
1. **`kfctl apply` hangs or fails with a timeout.**
* **Issue**: The deployment process gets stuck, often due to resource constraints or network issues.
* **Solution**:
* **Check Resources**: Ensure your Kubernetes cluster has enough CPU, memory, and disk space. A minimal Kubeflow installation often requires at least 4 vCPUs and 16GB RAM.
* **Increase Timeout**: `kfctl` might time out on slower clusters. You can increase the timeout with the `–timeout` flag (e.g., `kfctl apply -f kfctl_k8s_istio.yaml –timeout 30m`).
* **Check Logs**: Use `kubectl get events -A` or `kubectl describe pod
* **Network Connectivity**: Verify that your cluster nodes can pull images from Docker Hub or other registries.
2. **`ImagePullBackOff` or `ErrImagePull` for pods.**
* **Issue**: Pods cannot pull their required container images.
* **Solution**:
* **Registry Access**: Check if your cluster nodes have internet access to the image registries (e.g., `gcr.io`, `docker.io`).
* **Image Name/Tag**: Verify that the image names and tags specified in the Kubeflow manifests are correct and exist.
* **Private Registry**: If using a private registry, ensure Kubernetes has the correct `imagePullSecrets` configured.
* **`kubectl describe pod
3. **Kubeflow Dashboard is inaccessible (`localhost:8080` or external IP).**
* **Issue**: Cannot reach the Kubeflow UI after port-forwarding or using the external IP.
* **Solution**:
* **Port-Forwarding**: Ensure the `kubectl port-forward` command is still running in your terminal. If it exited, restart it.
* **Ingress Gateway Status**: Check the status of the `istio-ingressgateway` service and its pods:
kubectl get svc -n istio-system istio-ingressgateway
kubectl get pods -n istio-system -l app=istio-ingressgateway
Ensure the service has an `EXTERNAL-IP` (for cloud clusters) or the pods are `Running`.
* **Istio Components**: Verify all Istio pods in the `istio-system` namespace are running.
* **Firewall**: Check if any local or cloud provider firewalls are blocking access to the port.
4. **Jupyter Notebook server fails to start or gets stuck in `Pending` state.**
* **Issue**: Your requested notebook server doesn’t become `Running`.
* **Solution**:
* **Resource Quotas**: Check if your namespace (`kubeflow-user-example-com` by default) has resource quotas that are being exceeded.
* **Persistent Volume Claim (PVC)**: Ensure the PVC for the notebook’s workspace volume is bound and healthy.
kubectl get pvc -n kubeflow-user-example-com
kubectl describe pvc -n kubeflow-user-example-com
If the PVC is stuck in `Pending`, your cluster might not have a default StorageClass or your provisioner is not working.
* **Image**: Double-check the selected notebook image. Try a different, simpler image to rule out image-specific issues.
* **Logs**: `kubectl logs -n kubeflow-user-example-com
5. **Kubeflow Pipeline runs fail or individual steps get stuck.**
* **Issue**: Your pipeline run isn’t completing successfully.
* **Solution**:
* **Check Step Logs**: In the Kubeflow Pipelines UI, click on the failed step to view its logs. This is the most common way to diagnose pipeline failures (e.g., Python errors, missing dependencies, file not found).
* **Component Image**: Ensure the base image used for your pipeline components (`BASE_IMAGE` in our example) has all necessary dependencies or that `packages_to_install` in `@dsl.component` is correct.
* **Resource Limits**: Pipeline steps run in containers. If a step consumes too many resources, it might be OOMKilled (Out Of Memory Killed). Check the pod logs for `OOMKilled` events and adjust resource requests/limits in your component definition if needed.
* **Artifact Passing**: Verify that data is correctly passed between pipeline steps. Errors often occur when a component expects an input that wasn’t correctly produced by the previous step.
6. **Kubeflow components show `CrashLoopBackOff` status.**
* **Issue**: A pod repeatedly starts and crashes.
* **Solution**:
* **`kubectl logs
* **`kubectl describe pod