Datasets

Object-detection workflows need a persistent volume for downloaded datasets and trained weights.

The current cgcsdk package supports:

• RoboflowDataset: download a dataset version from Roboflow.
• VolumeDataset: use a dataset that is already stored on a CGC volume.

Download from Roboflow

import os
import time

from cgc.sdk import Image, Volume
from cgc.sdk.recipes.datasets import RoboflowDataset
from cgc.sdk.recipes.yolo import YoloConfig, YoloWorkflow


data = Volume.from_name(
    name="object-detection-data",
    mount_path="/root/data",
    size=50,
    storage_class="ssd-rwx-ec",
    create_if_missing=True,
)
data.ensure()
deadline = time.monotonic() + 300
while not data.ready():
    if time.monotonic() >= deadline:
        raise RuntimeError("Dataset volume did not become ready")
    time.sleep(5)

dataset = RoboflowDataset(
    workspace_id="your-workspace",
    project_id="your-project",
    version=1,
    format="yolov11",
    api_key=os.environ["ROBOFLOW_API_KEY"],
)

workflow = YoloWorkflow(
    name="yolo-training",
    image=Image.from_registry("ultralytics/ultralytics:latest"),
    volume=data,
    config=YoloConfig(),
)

download_run = workflow.download(dataset).submit()
status = download_run.wait(timeout_seconds=1800)
print("Download status:", status)
print(download_run.logs(timestamps=True))
if status != "Succeeded":
    raise RuntimeError(f"Download failed: {status}; job left for diagnosis: {download_run.name}")
download_run.delete()

Dataset arguments

Argument	Meaning
workspace_id	Roboflow workspace identifier from the dataset URL or dashboard.
project_id	Roboflow project identifier.
version	Numeric dataset version to download.
format	Roboflow export format. Use yolov11 for the shown Ultralytics workflow.
api_key	Roboflow API key. Read it from an environment variable rather than writing it directly in the script.

workflow.download(dataset) prepares a workflow job. submit() creates that job in CGC. The dataset is downloaded under the workflow volume's mount path.

Set the key before running your script:

# Linux/macOS
export ROBOFLOW_API_KEY="your-key"

# PowerShell
$env:ROBOFLOW_API_KEY="your-key"

Use a dataset already on a volume

from cgc.sdk.recipes.datasets import VolumeDataset


dataset = VolumeDataset(
    name="license-plates",
    path="datasets/license-plates",
    data_yaml="data.yaml",
)

check_run = workflow.download(dataset).submit()
status = check_run.wait()
print(status)
print(check_run.logs())
if status != "Succeeded":
    raise RuntimeError(f"Dataset check failed: {status}; job left for diagnosis: {check_run.name}")
check_run.delete()

For VolumeDataset, download() does not download data. It checks that the dataset configuration exists.

Argument	Meaning
name	Logical dataset name used by the workflow.
path	Dataset directory. A relative path is resolved under the workflow volume mount path.
data_yaml	YOLO dataset configuration file, relative to path unless absolute.
target_class	Optional class selected by higher-level application logic. It is empty by default.

The directory should contain the images, labels, and data.yaml expected by Ultralytics.

See the complete roboflow_dataset.py script on the Workflow examples page.

Wait for volume readiness before submitting the job. Deleting the job does not remove the dataset from the PVC.