DrivenData Labs

Improving Automatic Speech Recognition for Kids - A Reference Implementation for Word-level Transcription

2026-02-27T00:00:00-05:00

Welcome to the reference implementation tutorial notebook for the On Top of Pasketti: Children’s Speech Recognition Challenge - Word Track! If you are just getting started, we recommend reading the competition webpage first.

The goal of this tutorial is to:

Demonstrate how to load and explore the data.
Provide a basic framework for building a model.
Demonstrate how to package your work correctly for submission.

In this tutorial, we will adapt NVIDIA's Parakeet model, a pretrained ASR model, using NeMo. You can either expand on and improve the method in this notebook, or start with something completely different! Let's get started.

Background¶

Spoken language is a natural way for kids to learn, explore, and show what they know, yet today's Automatic Speech Recognition (ASR) technology hardly understands them. Most ASR systems are built on adult speech, and struggle with the pitch, rhythm, and evolving articulation of young learners.

The On Top of Pasketti: Children’s Speech Recognition Challenge assembles pre-existing and newly labeled datasets to advance speech models that truly work for children. Your goal in the Word Track is to develop models that accurately predict the words spoken by children in short audio clips.

This is a code execution challenge! Rather than submitting your predicted labels, you will package your trained model and the prediction code and submit that for containerized execution. See the code submission format webpage and the runtime repository for more information.

If you'd like to rerun this notebook, the notebook file can be downloaded from the benchmark repository. That repository also includes all code imported into the notebook.

Step 0: Import packages¶

First, create your environment. We use uv as the package manager in this benchmark repository. To run this notebook, you must be on a Linux machine with GPU access.

Create an environment: just create-environment
Activate the environment: source ./.venv/bin/activate
Install the requirements found in the TOML file into the environment: just requirements

Remember, the runtime repository's TOML file lists the packages that will be available for running inference using model submissions.

We'll be using NVIDIA's NeMo framework to build our model along with standard data science Python libraries to explore and prepare the data. Because this is a code execution challenge, we'll also be testing our solutions locally before packaging our model and inference code for submission. To help us with scoring, we've imported some utility functions from the competition's runtime repository.

In [1]:

# Standard library
import json
import os
from pathlib import Path

# Visualization
import matplotlib.pyplot as plt
from matplotlib import ticker

# Core ML & audio stack
import librosa
import lightning.pytorch as pl
import numpy as np
import pandas as pd
import torch

# ASR models & normalization
from nemo.collections.asr.models import ASRModel
from transformers.models.whisper.english_normalizer import EnglishTextNormalizer

# Training & experiment utilities
from loguru import logger
from nemo.utils import logging
from nemo.utils.exp_manager import exp_manager
from nemo.utils.trainer_utils import resolve_trainer_cfg
from omegaconf import OmegaConf, open_dict
from sklearn.model_selection import train_test_split

# Project utilities
from asr_benchmark.config import DATA_ROOT, PROJECT_ROOT
from asr_benchmark.nemo_adapter import (
    add_global_adapter_cfg,
    patch_transcribe_lhotse,
    update_model_cfg,
    update_model_config_to_support_adapter,
)
from asr_benchmark.score import english_spelling_normalizer, score_wer

fused_indices_to_multihot has reached end of life. Please migrate to a non-experimental function.
OneLogger: Setting error_handling_strategy to DISABLE_QUIETLY_AND_REPORT_METRIC_ERROR for rank (rank=0) with OneLogger disabled. To override: explicitly set error_handling_strategy parameter.
No exporters were provided. This means that no telemetry data will be collected.

In [2]:

torch.set_float32_matmul_precision("high")
# Set SAMPLE to use a smaller subset of the data for faster iteration during development. Set it to None to use the full dataset.
SAMPLE = None

Step 1: Load and explore the data¶

First, you'll likely want to set up your own repository for developing a solution. We recommend using Cookiecutter Data Science, which ensures an easy-to-navigate project structure.

We'll download all of the competition data to our "raw" folder. There are two distinct training corpora that share the same structure but contain different data, and are hosted in separate locations for participant access. One corpus is hosted on the DrivenData platform, while a second corpus, which follows the same schema but contains different data, is provided by TalkBank. See the Data Download page for access instructions.

Our local data structure after downloading all files to a raw data folder is:

childrens-speech-recognition-benchmark-pub/data/raw
├── drivendata
│   ├── audio_part_0.zip
│   ├── audio_part_1.zip
│   ├── audio_part_2.zip
│   └── train_word_transcripts.jsonl
└── talkbank
    ├── audio.zip
    └── train_word_transcripts.jsonl

After unzipping the files, you may want to consolidate the audio files into a single "audio" folder within the drivendata directory, where audio are split into multiple zip files.

rsync -a ./data/raw/drivendata/"audio 3"/ ./data/raw/drivendata/audio/

childrens-speech-recognition-benchmark-pub/data/raw
├── drivendata
│   ├── audio/
│   ├── audio_part_0.zip
│   ├── audio_part_1.zip
│   ├── audio_part_2.zip
│   └── train_word_transcripts.jsonl
└── talkbank
    ├── audio/
    ├── audio.zip
    └── train_word_transcripts.jsonl

For each of the two corpora, the file train_word_transcripts.jsonl contains the following fields:

utterance_id (str) - unique identifier for each utterance
child_id (str) - unique, anonymized identifier for the speaker
session_id (str) - unique identifier for the recording session; a single child_id may be associated with multiple session_ids
audio_path (str) - path to the corresponding .flac audio file relative to the /audio directory, following the pattern audio/{utterance_id}.flac
audio_duration_sec (float) - duration of the audio clip in seconds
age_bucket (str) - age range of the child at the time of recording ("3-4", "5-7", "8-11", "12+", or "unknown")
md5_hash (str) - MD5 checksum of the audio file, used for integrity verification
filesize_bytes (int) - size of the audio file in bytes
orthographic_text (str) - normalized orthographic transcription of the utterance

Each line in the JSONL manifest corresponds to a single utterance and references exactly one associated audio file. The orthographic_text field contains a manually created, minimally normalized orthographic transcription that serves as the training label.

Let's explore the metadata!¶

We will load the JSONL transcripts and explore some of the metadata. As a starting point, it is helpful to know how many utterances we have, how many unique children are present, the total audio time, the distribution of audio clip durations, and the distribution of child ages.

In [3]:

def read_transcripts(data_dir: Path) -> pd.DataFrame:
    """Read JSONL transcript file into a DataFrame and convert audio paths to absolute paths."""
    transcript_path = data_dir / "train_word_transcripts.jsonl"
    df = pd.read_json(transcript_path, lines=True)
    logger.info(f"Loaded {len(df)} utterance transcripts")

    df["audio_relpath"] = df["audio_path"]
    df["audio_path"] = df["audio_relpath"].map(lambda p: str(data_dir / p))

    return df

In [4]:

df_dd = read_transcripts(DATA_ROOT / "raw" / "drivendata")
df_tb = read_transcripts(DATA_ROOT / "raw" / "talkbank")
df = pd.concat([df_dd, df_tb], ignore_index=True)
df.drop(columns=["audio_path"]).head()

2026-02-26 04:58:21.910 | INFO     | __main__:read_transcripts:5 - Loaded 95572 utterance transcripts
2026-02-26 04:58:24.452 | INFO     | __main__:read_transcripts:5 - Loaded 255046 utterance transcripts

Out[4]:

	utterance_id	child_id	session_id	audio_duration_sec	age_bucket	md5_hash	filesize_bytes	orthographic_text	audio_relpath
0	U_00003c3ae1c35c6f	C_c74bfde2cca8d5da	S_7d821c3e4d3bc616	1.920	8-11	9214be45ba2928dd57384f3c7ee54236	30672	hm	audio/U_00003c3ae1c35c6f.flac
1	U_00003db24218ffe4	C_c74bfde2cca8d5da	S_e6103ab3a4538d71	12.737	8-11	fe761bb3d034530ef05163c7ad98ec3e	180942	yeah its pouring the water on the screen but t...	audio/U_00003db24218ffe4.flac
2	U_0001a0d0a3b4d816	C_4d0e1c16566d65a2	S_179057c3c3ccdecf	11.556	8-11	b05073e65a98368fccbe777b5ab35e02	208352	it got water and sunlight but the plant did di...	audio/U_0001a0d0a3b4d816.flac
3	U_00021d201a31d313	C_3b51c8b1d2c076d8	S_90720887e4430996	1.125	8-11	9ed95318724ae6a2d1ce95d6aa743f6b	27099	there is wires	audio/U_00021d201a31d313.flac
4	U_0003537f2bc1eb0b	C_b50216b3c70ca0a2	S_5b0bb48fadd7f802	1.125	8-11	f3142751c6a52e2c24a85a4544fe8a0f	18476	good	audio/U_0003537f2bc1eb0b.flac

In [5]:

df.utterance_id.nunique()

Out[5]:

In [6]:

df.child_id.nunique()

Out[6]:

In [7]:

round(df.audio_duration_sec.sum() / (60**2))

Out[7]:

There are over 350,000 utterances in the training dataset, across 3,344 children, totaling 344 hours of audio data. Next, let's take a look at the distribution of audio clip durations.

In [8]:

bins = list(range(0, 21)) + [np.inf]
labels = [str(i) for i in range(0, 20)] + ["20+"]

binned = pd.cut(df.audio_duration_sec, bins=bins, labels=labels, right=False)
counts = binned.value_counts().sort_index()

counts.plot(kind="bar")
plt.xlabel("Audio Duration (sec)")
plt.ylabel("Number of Audio Clips")
plt.title("Distribution of Audio Durations")
plt.xticks(rotation=90)
plt.show()

Even though the audio has been clipped to the utterance level, we have some outliers over 20 seconds.

In [9]:

# What is the longest clip (in minutes)?
round(df.audio_duration_sec.max() / 60)

Out[9]:

Most audio clips are very short (1-3 seconds). There is a tail of longer clips, up to an outlier 22 minute clip. Next, let's look at the distribution of utterances by child age.

In [10]:

df["age_bucket"] = pd.Categorical(
    df["age_bucket"], categories=["unknown", "3-4", "5-7", "8-11", "12+"], ordered=True
)

fig, ax = plt.subplots()
df["age_bucket"].value_counts(normalize=True, sort=False).plot.barh(ax=ax)
ax.set_title("Utterances by Age Group")
ax.set_xlabel("Percent of Utterances")
ax.set_ylabel("Age Group")
ax.xaxis.set_major_formatter(ticker.PercentFormatter(1.0))
ax.bar_label(ax.containers[0], fmt=lambda x: f"{x * 100:.0f}%")

plt.show()

About half of the utterances come from 8 to 11 year olds, with 13% coming from 3 to 4 year olds and 28% coming from 5 to 7 year olds.

Let’s explore the utterances!¶

We will listen to an example utterance and visualize the features.

In [11]:

def plot_waveform(audio_data: np.ndarray, sr: int) -> None:
    """Plot the waveform of an audio signal"""
    plt.figure(figsize=(10, 4))
    librosa.display.waveshow(audio_data, sr=sr)
    plt.title("Waveform")
    plt.xlabel("Time (s)")
    plt.ylabel("Amplitude")
    plt.show()

In [12]:

def plot_melspectrogram(audio_data: np.ndarray, sr: int) -> None:
    """Generate and plot the mel spectrogram of an audio signal"""
    S = librosa.feature.melspectrogram(y=audio_data, sr=sr)
    S_db = librosa.power_to_db(S, ref=np.max)

    plt.figure(figsize=(10, 4))
    librosa.display.specshow(S_db, sr=sr, x_axis="time", y_axis="mel")
    plt.colorbar(format="%+2.0f dB")
    plt.title("Mel Spectrogram")
    plt.xlabel("Time (s)")
    plt.ylabel("Mel Frequency")
    plt.show()

In [13]:

example_utt = "U_1c8757065e355c35"
df[df.utterance_id == example_utt][
    ["utterance_id", "audio_duration_sec", "orthographic_text"]
]

Out[13]:

	utterance_id	audio_duration_sec	orthographic_text
124170	U_1c8757065e355c35	0.516	yeah

This clip may sound understandable to us, but children’s speech often includes subtle pronunciation differences. For example, models must learn to handle syllable deletions (“elephant” → “efant”), consonant nasalization (“mom” → “bob”), and common speech sound disorders like tetism.

Now that we've heard the utterance, let's look at it from the model's perspective. We'll plot the raw waveform - the audio signal represented as amplitude over time:

In [ ]:

audio_data, sr = librosa.load(df[df.utterance_id == example_utt]["audio_path"].values[0])
plot_waveform(audio_data, sr)

Before training a machine learning model, the raw audio array with sampled amplitudes must be transformed. Transforming this raw signal into mel spectrograms enables the machine learning model to learn speech-relevant acoustic patterns more efficiently. The transformation reorganizes the signal into a structured time-by-frequency map that highlights speech patterns like vowels and consonants.

In [16]:

plot_melspectrogram(audio_data, sr)

When training machine learning models, this transformation is often handled under-the-hood by processors or feature extractors. For example, in our benchmark model implementation, we will fine tune NVIDIA’s pretrained parakeet-tdt-0.6b-v2 model using NeMo. NeMo's architecture includes a built-in preprocessing pipeline that converts raw audio waveforms into mel-spectrogram features, eliminating the need for manual feature extraction.

For more tips on generating predictive features from audio data, check out DrivenData's overview of open-source packages for using speech data in ML. That blog post also includes some helpful intro to working with audio data.

Step 2: Build the model¶

A straightforward modeling option is to start from a strong pretrained ASR model (trained mostly on adult speech), then fine-tune on our labeled child-speech training set. Fine-tuning the ASR model will help it understand the unique acoustic and linguistic patterns of children. Before choosing a pre-trained model, make sure it adheres to the competition rules.

In this tutorial, we adapt NVIDIA’s pretrained Parakeet TDT parakeet-tdt-0.6b-v2 model using NeMo. We chose this model because it is open source, ranks highly on the Hugging Face Open ASR leaderboard (at time of writing), and strikes a good balance between performance and model size. Rather than fine-tuning all parameters, we will freeze the base model and train only lightweight adapter weights. Adapters are small, trainable modules inserted into the frozen model’s layers that allow it to adapt to new data without modifying the original pretrained weights. Freezing the base model makes the fine-tuning process more efficient and helps avoid overfitting. NeMo makes this process easier by providing native adapter support, handling data pipelines (manifests, batching, feature extraction), and integrating experiment management (checkpointing, logging).

Key packages include:

nemo_toolkit for ASR model + data processing
lightning.pytorch for the training backend
omegaconf for config file management

1. Prepare Dataset & Create Splits¶

We need to provide NeMo with a "manifest" JSONL file that lists all of our samples and sample information. NeMo requires this manifest to be in a particular format, with specific column names and definitions. Each manifest line must be a JSON object with audio_filepath (path to audio file), duration (audio duration in seconds), and text (transcription text).

Below we will transform the competition data to NeMo's required manifest format.

We'll also filter out clips longer than 25 seconds, which strain computer memory. Competitors may want to further split these clips to avoid losing training data. Then we'll split the data 80/20 into train/validation.

In [17]:

MANIFEST_DIR = DATA_ROOT / "processed" / "ortho_dataset"
train_manifest_path = MANIFEST_DIR / "train_manifest.jsonl"
val_manifest_path = MANIFEST_DIR / "val_manifest.jsonl"
clip_max_duration_sec = 25.0

MANIFEST_DIR.mkdir(parents=True, exist_ok=True)

Remember, df has one row per utterance, and includes data hosted on both TalkBank and DrivenData.

In [18]:

df = df[["audio_path", "audio_duration_sec", "orthographic_text"]].rename(
    columns={
        "audio_path": "audio_filepath",
        "audio_duration_sec": "duration",
        "orthographic_text": "text",
    }
)

# Removing long clips to improve memory efficiency during training
over_max_duration_mask = df["duration"] > clip_max_duration_sec
logger.info(
    f"Removing {over_max_duration_mask.sum()} samples with audio duration "
    f"> {clip_max_duration_sec} seconds"
)
df = df[~over_max_duration_mask]

if SAMPLE:
    logger.info(f"Sampling {SAMPLE} utterances")
    df = df.sample(SAMPLE, random_state=10)

train_df, val_df = train_test_split(df, test_size=0.2, random_state=10)

train_df.to_json(train_manifest_path, orient="records", lines=True)
val_df.to_json(val_manifest_path, orient="records", lines=True)

2026-02-26 04:58:31.723 | INFO     | __main__:<module>:11 - Removing 3849 samples with audio duration > 25.0 seconds

2. Configuration¶

Let's define parameters for the adaptation.

To start, we copied an example .yaml file for ASR adaptation using NeMo from the NeMo repository to asr_benchmark/assets/asr_adaptation.yaml. This file gives us a reasonable default starting-point configuration.
We want to overtly set some parameters, like:
- pretrained_model (to specify parakeet-tdt-0.6b-v2)
- manifest_filepath (to point to our manifest locations)
- channel_selector (to ensure audio is converted to the expected mono format if it is stereo in raw form)
- max_steps (to control how many training steps we complete before we stop training)
For these parameters, we override the default settings in the .yaml using OmegaConf.merge.
While we do not implement hyperparameter tuning here, it is recommended for competitors.

In [ ]:

# ── Hardware-dependent settings ──────────────────────────────────────────────
# Adjust these to match your GPU memory and CPU cores.
DEVICES = 1
PRECISION = "bf16-mixed"
BATCH_SIZE = 32
NUM_WORKERS = 8

# ── Load NeMo adapter defaults ───────────────────────────────────────────────
yaml_path = PROJECT_ROOT / "asr_benchmark" / "assets" / "asr_adaptation.yaml"
cfg = OmegaConf.load(yaml_path)

# ── Training overrides ───────────────────────────────────────────────────────
overrides = OmegaConf.create(
    {
        "model": {
            "pretrained_model": "nvidia/parakeet-tdt-0.6b-v2",
            "adapter": {
                "adapter_name": "asr_children_orthographic",
                "adapter_module_name": "encoder",
                "linear": {"in_features": 1024},
            },
            "train_ds": {
                "manifest_filepath": str(train_manifest_path),
                "batch_size": BATCH_SIZE,
                "num_workers": NUM_WORKERS,
                "use_lhotse": False,
                "channel_selector": "average",
            },
            "validation_ds": {
                "manifest_filepath": str(val_manifest_path),
                "batch_size": BATCH_SIZE,
                "num_workers": NUM_WORKERS,
                "use_lhotse": False,
                "channel_selector": "average",
            },
            "optim": {
                "lr": 0.001,
                "weight_decay": 0.0,
            },
        },
        "trainer": {
            "devices": DEVICES,
            "precision": PRECISION,
            "strategy": "auto",
            "max_epochs": 1 if SAMPLE else None,
            "max_steps": -1 if SAMPLE else 5000,
            "val_check_interval": 1.0 if SAMPLE else 500,
            "enable_progress_bar": False,
        },
        "exp_manager": {
            "exp_dir": str(PROJECT_ROOT / "models" / "orthographic_benchmark_nemo"),
        },
    }
)

cfg = OmegaConf.merge(cfg, overrides)

3. Define Trainer¶

The Trainer orchestrates the training loop across devices, delegating tensor operations to PyTorch's backend. We initiate the trainer with the OmegaConf config object we made above, and then set up an experiment manager to handle logging, checkpoint saving, and saving artifacts to disk.

Note we use the cell magic %%capture below to hide long cell outputs for readability.

In [ ]:

%%capture
trainer = pl.Trainer(**resolve_trainer_cfg(cfg.trainer))
exp_log_dir = exp_manager(trainer, cfg.get("exp_manager", None))

4. Data and Model Setup¶

We load the pretrained model by fetching the model from our config first and then patching it for adapter support. Then, we load the model weights.

In [ ]:

%%capture
model_cfg = ASRModel.from_pretrained(cfg.model.pretrained_model, return_config=True)
update_model_config_to_support_adapter(model_cfg, cfg)
model = ASRModel.from_pretrained(
    cfg.model.pretrained_model,
    override_config_path=model_cfg,
    trainer=trainer,
)

We disable the CUDA graph decoder because it is incompatible with current PyTorch (2.10).

In [ ]:

%%capture
with open_dict(model.cfg):
    model.cfg.decoding.greedy.use_cuda_graph_decoder = False
model.change_decoding_strategy(model.cfg.decoding)

Next, we prepare our data by merging our parameter overrides (batch size, num workers, etc.) into the model's built-in data config. Note that while keys not present in the original config are dropped, whitelisted keys are always injected.

In [23]:

cfg.model.train_ds = update_model_cfg(model.cfg.train_ds, cfg.model.train_ds)
model.setup_training_data(cfg.model.train_ds)

cfg.model.validation_ds = update_model_cfg(
    model.cfg.validation_ds, cfg.model.validation_ds
)
model.setup_multiple_validation_data(cfg.model.validation_ds)

Injecting white listed key `num_workers` into config
Injecting white listed key `pin_memory` into config
Injecting white listed key `batch_size` into config
Injecting white listed key `use_lhotse` into config
Injecting white listed key `channel_selector` into config
Removing unavailable key from config : is_tarred
Removing unavailable key from config : shuffle_n
Removing unavailable key from config : bucketing_strategy
Removing unavailable key from config : bucketing_batch_size
[NeMo I 2026-02-26 04:58:53 nemo_logging:393] Dataset loaded with 277410 files totalling 244.57 hours
[NeMo I 2026-02-26 04:58:53 nemo_logging:393] 5 files were filtered totalling 0.00 hours
Injecting white listed key `num_workers` into config
Injecting white listed key `pin_memory` into config
Injecting white listed key `batch_size` into config
Injecting white listed key `use_lhotse` into config
Injecting white listed key `channel_selector` into config
[NeMo I 2026-02-26 04:58:55 nemo_logging:393] Dataset loaded with 69352 files totalling 61.52 hours
[NeMo I 2026-02-26 04:58:55 nemo_logging:393] 2 files were filtered totalling 0.00 hours

We also need to set the optimization function, which controls how the model’s weights are updated using gradients to minimize the loss and improve performance during training. We use AdamW with cosine annealing schedule and 10% warmup ratio as a starting point.

In [ ]:

%%capture
model.setup_optimization(cfg.model.optim)

In [ ]:

# Configure spec augmentation from config if available, otherwise disable it.
if "spec_augment" in cfg.model:
    model.spec_augmentation = model.from_config_dict(cfg.model.spec_augment)
else:
    model.spec_augmentation = None
    del model.cfg.spec_augment

5. Set up Adapter¶

An adapter is a small neural network module that learns to transform the pretrained model's internal representations specifically for child speech. Rather than retraining the entire model, we insert lightweight adapters into each layer of the encoder, the part of the model that processes audio features.

We add a linear adapter to every encoder layer, then freeze the base model and unfreeze only the adapter weights. This keeps training efficient—only ~0.1% of parameters are trainable, while (hopefully) allowing the adapter to learn the acoustic and linguistic patterns unique to children's voices.

In [ ]:

%%capture
with open_dict(cfg.model.adapter):
    adapter_name = cfg.model.adapter.pop("adapter_name")
    adapter_type = cfg.model.adapter.pop("adapter_type")
    adapter_module_name = cfg.model.adapter.pop("adapter_module_name", None)
    adapter_state_dict_name = cfg.model.adapter.pop("adapter_state_dict_name", None)

    adapter_type_cfg = cfg.model.adapter[adapter_type]

    if adapter_module_name is not None and ":" not in adapter_name:
        adapter_name = f"{adapter_module_name}:{adapter_name}"

    adapter_global_cfg = cfg.model.adapter.pop(model.adapter_global_cfg_key, None)
    if adapter_global_cfg is not None:
        add_global_adapter_cfg(model, adapter_global_cfg)

model.add_adapter(adapter_name, cfg=adapter_type_cfg)
assert model.is_adapter_available()

model.set_enabled_adapters(enabled=False)
model.set_enabled_adapters(adapter_name, enabled=True)

model.freeze()
model = model.train()
model.unfreeze_enabled_adapters()

In [ ]:

total_params = sum(p.numel() for p in model.parameters())
trainable_params = sum(p.numel() for p in model.parameters() if p.requires_grad)
print(f"Total parameters:     {total_params:,}")
print(
    f"Trainable parameters: {trainable_params:,} ({100 * trainable_params / total_params:.2f}%)"
)

Total parameters:     619,447,942
Trainable parameters: 1,622,016 (0.26%)

6. Train¶

Let's adapt the pretrained model to children's voices!

In [ ]:

%%capture
trainer.fit(model)

We've hidden the training cell outputs to avoid printing many lines of logs. Below are a few key snippets of the log output that we saw when training:

| Name              | Type                              | Params | Mode 
--------------------------------------------------------------------------------
0 | preprocessor      | AudioToMelSpectrogramPreprocessor | 0      | train
1 | encoder           | ConformerEncoderAdapter           | 610 M  | train
2 | decoder           | RNNTDecoder                       | 7.2 M  | train
3 | joint             | RNNTJoint                         | 1.7 M  | train
4 | loss              | RNNTLoss                          | 0      | train
5 | spec_augmentation | SpectrogramAugmentation           | 0      | train
6 | wer               | WER                               | 0      | train
--------------------------------------------------------------------------------
1.6 M     Trainable params
617 M     Non-trainable params
619 M     Total params
2,477.792 Total estimated model params size (MB)

Epoch 0, global step 500: 'val_wer' reached 0.20008 (best 0.20008), saving model to '/models/orthographic_benchmark_nemo/ASR-Adapter/2026-02-26_04-58-33/checkpoints/ASR-Adapter--val_wer=0.2001-epoch=0.ckpt' as top 5
...
Epoch 0, global step 5000: 'val_wer' reached 0.16359 (best 0.16359), saving model to '/models/orthographic_benchmark_nemo/ASR-Adapter/2026-02-26_04-58-33/checkpoints/ASR-Adapter--val_wer=0.1636-epoch=0.ckpt' as top 5
`Trainer.fit` stopped: `max_steps=5000` reached.

Because we configured max_steps to be 5,000, we stopped training before we completed the first epoch.

We need to save the final results. We save the adapter weights to a standalone file alongside the NeMo checkpoints.

In [28]:

if adapter_state_dict_name is not None:
    state_path = exp_log_dir if exp_log_dir is not None else os.getcwd()
    ckpt_path = os.path.join(state_path, "checkpoints")
    if os.path.exists(ckpt_path):
        state_path = ckpt_path
    state_path = os.path.join(state_path, adapter_state_dict_name)

    model.save_adapters(str(state_path))

7. Evaluation¶

Now it's time to assess how well our adapted model performs. We load the best trained checkpoint from disk and run inference across the entire validation set. The model transcribes each audio clip, and we compare these predictions against the reference transcriptions using Word Error Rate (WER). The WER calculation is copied exactly from the runtime repository.

In [ ]:

%%capture
# Load our model checkpoint
nemo_ckpts = sorted((exp_log_dir / "checkpoints").glob("*.nemo"))
if not nemo_ckpts:
    raise FileNotFoundError(f"No .nemo checkpoints found in {exp_log_dir}/checkpoints/")

best_ckpt = nemo_ckpts[-1]
print(f"Loading checkpoint: {best_ckpt}")
eval_model = ASRModel.restore_from(best_ckpt, map_location="cuda")

with open_dict(eval_model.cfg):
    eval_model.cfg.decoding.greedy.use_cuda_graph_decoder = False
eval_model.change_decoding_strategy(eval_model.cfg.decoding)

patch_transcribe_lhotse(eval_model)

In [ ]:

# Run inference
val_manifest_path = cfg.model.validation_ds.manifest_filepath
with open(val_manifest_path) as f:
    val_entries = [json.loads(line) for line in f]

audio_files = [e["audio_filepath"] for e in val_entries]
references = [e["text"] for e in val_entries]

print(f"Running inference on {len(audio_files)} validation utterances...")
raw = eval_model.transcribe(
    audio_files, batch_size=BATCH_SIZE, channel_selector="average", verbose=False
)
if isinstance(raw, tuple):
    raw = raw[0]

predictions = [h.text if hasattr(h, "text") else h for h in raw]

Running inference on 69354 validation utterances...

Before scoring, let's remove any example where the normalized label is an empty string.

In [31]:

normalizer = EnglishTextNormalizer(english_spelling_normalizer)
filtered = [(r, p) for r, p in zip(references, predictions) if normalizer(r) != ""]

references, predictions = zip(*filtered)

wer = score_wer(references, predictions)

print(f"Validation WER: {wer:.4f}")

print("\nSample predictions:")
for ref, pred in zip(references[:5], predictions[:5]):
    print(f"  REF:  {ref}")
    print(f"  PRED: {pred}")
    print()

Validation WER: 0.1546

Sample predictions:
  REF:  ar
  PRED: ar

  REF:  they were going on a walk to a picnic
  PRED: they were going on a walk to a picnic

  REF:  a deer is little
  PRED: a gear and a skin

  REF:  what's happening is when you when there's an op when there's a closed circuit the there is a the post but the wire's wrapped around it is carrying some washers and when it
  PRED: what's happening is when you when there's an o when there's a closed circuit the there is a the post with the wires wrapped around it is carrying some washers and when it

  REF:  blue
  PRED: blue

Our Validation WER after adapting the model is 0.15!

Step 3: Make your submission¶

Since this is a code execution competition, we will submit our model weights and code rather than predictions. See the code submission format webpage for more information.

The general steps to follow:

Develop inference code
Test your submission locally
Package submission
Make a smoke test submission
Once you have successfully debugged your submission, submit it for scoring on the full test set!

Develop Inference Code¶

We need to set up a repository with a main.py Python script which performs inference in the competition execution environment and writes our predictions to the required output file. During code execution, our submission will be unzipped and run in the cloud compute cluster. The container will run your main.py script.

Our code must write a JSON Lines (JSONL) file containing one prediction per utterance.

Each line must include:

utterance_id
orthographic_text: UTF-8, standard English transcription of the utterance The submission should be written to ./submission/submission.jsonl relative to the working directory.

See more details in the code submission format webpage and in the example submission.

In our main.py, we load the trained adapter checkpoint from disk, restore the NeMo model, and run inference on all test utterances in batches. The script reads audio file paths from the test manifest, transcribes them using the adapted model, and writes the predicted transcriptions to the submission file in the required format. We sort utterances by duration before inference (longest first) to improve GPU memory efficiency during batching.

See orthographic_submission/main.py in the benchmark repository for the details.

Test Submission Locally¶

You should first and foremost test your submission locally. This is a great way to work out any bugs and ensure that your model performs inference successfully. See the runtime repository's README for further instructions.

The benchmark repository provides a useful justfile command to run the trained model on a few sample files.

# Run inference using data-demo/word/ to test orthographic submission
test-orthographic:
    uv run orthographic_submission/main.py models/orthographic_benchmark_nemo/ASR-Adapter-best.nemo data-demo/word/utterance_metadata.jsonl

Package Submission¶

Now we will package up our model and inference code into a zip file for predicting on the test set in the runtime container. The benchmark repository provides a justfile command to do this. The justfile finds the latest models weights, and then creates a zipfile combining those model weights with /orthographic_submissions/main.py.

pack-orthographic:
    #!/usr/bin/env bash
    set -euo pipefail
    latest=$(ls -td models/orthographic_benchmark_nemo/ASR-Adapter/*/checkpoints/ASR-Adapter.nemo | head -1)
    ln -sf "${latest#models/orthographic_benchmark_nemo/}" models/orthographic_benchmark_nemo/ASR-Adapter-best.nemo
    echo "Updated ASR-Adapter-best.nemo -> $latest"
    rm -f orthographic_submission.zip
    (cd orthographic_submission && zip -r ../orthographic_submission.zip main.py)
    (cd models/orthographic_benchmark_nemo && zip -r ../../orthographic_submission.zip ASR-Adapter-best.nemo)

Another packing command is available on the runtime repo's orthographic example.

Make a Smoke Test Submission¶

We provide a "smoke test" environment that replicates the test inference runtime but runs only on a small set of audio files. In the smoke test runtime, data/ contains 9,000 audio files from the training set.

Let's submit our submission.zip to a smoke test on the platform.

After hitting "Submit" you can see the job in the queue—it will progress from "Uploading" to "Pending" to "Starting" to "Running":

Once your submission reaches "Completed", head on over to the "Submissions" tab to see your smoke test score.

Submit!¶

After you've made sure a smoke test submission runs without error, you're ready to submit the real deal! This adaptation of parakeet-tdt-0.6b-v2 results in a .2370 WER on the full test set.

We encourage you to also be mindful of the submission limit (3 per 7 days at most) and others' code jobs. Canceled jobs do not count against the submission limit.

This is of course just one of the approaches you could take for this challenge. Some resources that may be helpful in getting started are:

The Hugging Face Automatic Speech Recognition unit of their Audio Course.
The Hugging Face Open ASR Leaderboard.
A Parakeet fine-tuning implementation from Deep-unlearning.
NeMo example ASR adaptation (which has a similar implementation as this blog post).

If you want to share any of your findings or have questions, feel free to post on the community forum.

Good luck!

5 Challenges of Creating Beautiful Data Pipelines

2026-02-18T00:00:00-05:00

Almost all projects we work on at DrivenData share a fundamental need: getting data from a raw form into a final, processed form. Sometimes, a project's core goal is to create a data pipeline that will run in perpetuity and continuously process live feeds of data. More often, our data pipelines are enabling processes that serve a larger purpose, such as preparing data for use in training a machine learning model.

While most pipelines start out seeming like a simple set of sequential steps, the reality is that pipeline code often becomes a deep source of technical debt. Over time, complexity accumulates as capabilities are added. For example, we may need a development version that runs on less data. We may need separate production and dev environments. We need audit trails and observability to make sure everything processed correctly. We need error capturing and recovery, and we need the right decisions about when to stop and when to keep processing. We need the code to use resources efficiently through mechanisms like parallelization and caching. We need to be smart about incrementally updating versus re-doing every task from scratch. We need to augment a pipeline step with a different dataset that has separate re-process conditions. Soon, the code is a complex tangle that is hard to reason about and maintain.

In this blog post, I’ll discuss some of the challenges of creating data pipelines for data science projects. We hope to bring clarity to what makes pipelining complex, simplify implementation decisions, and help data science teams make beautiful data pipelines from the start.

Why is something seemingly simple so hard?¶

Imagine you need to prepare news article data to train a machine learning model to classify whether an article discusses data, AI, or machine learning. That doesn’t sound too hard! A simple Python implementation could look like this:

from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression

def main(raw_data_directory: str = "path/to/raw_data"):
    # Step 1: Load Data From a Folder
    data = load_data(raw_data_directory)

    # Step 2: Preprocess Data
    data = preprocess_dataframe(data)

    # Step 3: Train/Test Split
    X_train, X_test, y_train, y_test = train_test_split(
        data["article_text"],
        data["label"],
        test_size=0.2,
        random_state=42
    )

    # Step 4: Define Vectorizer and Classifier
    text_clf = Pipeline([
        ("vectorizer", TfidfVectorizer()),
        ("classifier", LogisticRegression())
    ])

    # Step 5: Train Model
    text_clf.fit(X_train, y_train)

    # Step 6: Evaluate Model
    evaluate_model(text_clf, X_test, y_test)

However, the load_data and preprocess_dataframe functions turn out to never be quite as simple as they seem. In the next sections, we will walk through a few common pain points that show up as you move from the drawing board to a fleshed-out pipeline.

1. The cold start problem¶

A new pipeline starts as a blank slate, and figuring out where to begin isn’t always obvious. There are countless ways to organize files and structure steps. Figuring out the right order of operations can take some work.

In our data-topic detection example, you might wonder: Should we store an intermediary data file with article text and metadata when we preprocess it? What metadata should we even track? When should we deduplicate the article data? What are the upstream dependencies of our processing steps?

Each of these decisions starts to introduce complexity into what was a simple pipeline.

Suggestions

To start, sketch out the pipeline process visually to clarify dependencies and to ground alignment conversations about the pipeline in something concrete.
As you structure the pipeline, separate transformation logic from pipeline orchestration (the code that runs each step). In our data-topic detection model starter code, we were on the right track with a main function as the entry point to the pipeline that calls external helper functions (load_data(), preprocess_dataframe(), and evaluate_model()) in sequential steps.
Structure pipeline steps as a directed acyclic graph ("DAG") so that each step depends only on its inputs and produces outputs for downstream steps, and each step flows forward to the next without ever looping back on itself. Relatedly, raw data must be treated as immutable (that is, don't overwrite the original data files; instead, save the results of the pipeline as new output files).
Follow the software engineering principle of separation of concerns by breaking down complex logic and processing into modular functions that complete distinct objectives. Separation of concerns is a fundamental part of developing organized and clear pipelines. Modularity also makes each of the processing steps independently testable so you can have a test suite that ensures each granular step is doing what you expect. In our data-topic detection pipeline, we should isolate each task in our data preprocessing step and make separate functions that complete single objectives or operations:

def remove_duplicates(df: pd.DataFrame) -> pd.DataFrame:
    # TODO: implement deduplication
    return df

def drop_missing_labels(df: pd.DataFrame, label_col: str = "label") -> pd.DataFrame:
    # TODO: drop rows where label_col is null
    return df

def normalize_text(df: pd.DataFrame, text_col: str = "article_text") -> pd.DataFrame:
    # TODO: normalization steps
    return df

def encode_labels(df: pd.DataFrame, label_col: str = "label") -> pd.DataFrame:
    # TODO: map label values to numeric codes
    return df

Design the pipeline to be fully re-runnable end-to-end with minimal involvement. For example, make main() runnable from the command line by adding an entry point so that the pipeline runs when you execute python identify_data_topics.py.
When we start to need more options and arguments at the command line, we like Cyclopts for building command-line interfaces.
It may be helpful to use DrivenData's Cookiecutter Data Science package when creating a new repository to house your pipeline. Using a well-defined, standard project structure will benefit you and your team in the long run.

2. Adding basic pipeline features is a worthwhile investment¶

Stringing the steps together isn’t enough. As soon as you start running the pipeline, you'll realize you have no idea what is happening while you are running it. Add-ons such as logging, metadata tracking, and error handling will reduce debugging time, yield results you can be more confident in, and help identify bottlenecks in the process. These add-ons are generally worth it and justify budgeting the up-front implementation time.

In the data-topic detection pipeline, logs could show exactly where a run failed and how long each step took. Adding tools to preprocess news articles in parallel can substantially reduce processing time.

Suggestions

Add lightweight logging to capture per-task progress and errors, making debugging easier. We like Loguru, an intuitive Python logging library.
Speed up long processes by parallelizing tasks (running independent work concurrently) where possible. We often use tqdm, specifically the process_map and thread_map concurrent versions.
Despite best efforts, pipeline runs often unexpectedly fail because of data errors or edge cases. Disk caching computationally expensive steps can be a huge time saver in these cases because it enables you to resume progress where you left off just before the error arose in a task rather than restarting all computations. The @memoize decorator works well for caching the results of functions.
Track and maintain metadata such as file hashes (which are fixed-length identifiers generated from a file's content using hash functions), unique IDs, and pipeline version for reproducibility. In our data-topic detection example, you could generate and track hashes of the article text with a function to confirm data isn’t being corrupted run-to-run. (Notice we also have logging and parallelization at the task level in this example!):

import hashlib
import pandas as pd
from loguru import logger
from tqdm.contrib.concurrent import process_map

def hash_text(text: str) -> str:
    return hashlib.md5(text.encode("utf-8")).hexdigest()


def hash_article_texts(df: pd.DataFrame) -> pd.DataFrame:
    """Add a 'text_hash' column to the DataFrame containing MD5 hashes of the 'article_text' column."""
    logger.info(f"Hashing {len(df)} articles")
    df["text_hash"] = process_map(hash_text, df["article_text"].tolist())
    logger.success("Finished generating text hashes")
    return df

If the pipeline writes files, add an overwrite parameter that raises an error (or skips writing the file) if a file already exists and overwrite is set to False. This prevents accidentally overwriting existing files on subsequent pipeline runs.
Consider adding a debug mode parameter to your pipeline that, when activated, runs the pipeline on a small sample of data so you can test changes quickly and make sure the whole pipeline runs as expected.

3. Varied inputs lead to complicated code¶

Inevitably, project inputs expand beyond a single clean data source. Each CSV file, scraped HTML document, JSON object, and API call comes with its own quirks and processing needs. Real-world data is messy: missing labels, odd encodings, and inconsistent formats often only reveal themselves once the pipeline runs at scale. And it’s not just data; every parameter or configuration you add can shift assumptions and introduce new branches in the logic.

In our data-topic detection pipeline, expanding beyond a single CSV with news articles could mean that you need to add separate cleaning steps for other formats. You might also discover that some articles in the training data are missing a label indicating whether the article discusses data, AI, or machine learning, or are in another language, or have source text from multiple articles stuck together. You have to account for these edge cases in the pipeline.

Suggestions

Standardize and clean data early so later steps can assume consistent input. Preferably with a consistent, validated schema so malformed data fails early.
Factor out shared utility functions instead of duplicating code for each source.
Validate key assumptions early in the pipeline to catch anomalies, and fail loudly (e.g., raise errors) when those assumptions are not met. Raising errors early in the pipeline prevents wasted processing time for runs that fail. In our data-topic detection pipeline, if we need to ingest news articles from a website, our first step might be checking if the base URL for the website returns a "200" status code, indicating the website is active. If it doesn't return "200", we should skip processing because the site is down rather than issuing thousands of requests for articles that don't exist:

import requests

def ingest_articles(base_url: str) -> dict:
    """Ingest news articles from a website"""
    resp = requests.get(base_url)
    if not resp.status_code == 200:
        raise ValueError(f"Base URL {base_url} returned status code {resp.status_code}")

    # TODO: implement article ingestion

    return articles

4. Tech debt in a world of changing requirements¶

Project goals, requirements, and data formats change over time. This can create headaches when the existing pipeline needs to be refactored to account for processing conditions that didn’t initially exist. How will you handle backwards compatibility? Do you deprecate the "old way" of doing things entirely, or make the "old way" one of many processing options that can be triggered by calling certain parameters?

To avoid tech debt, it’s tempting to design a pipeline that is scoped out to be compatible with many processing scenarios from the start. But this, too, comes with downsides, as code will often be unnecessarily complex to preserve flexibility that you might not actually need.

In our example, maybe you first designed the pipeline to process a single labeled dataset. Later, you get access to additional datasets and decide you need to track metadata fields such as article author and publication date by source. Meeting these new needs means adjusting the data organization, adding new processing steps, and sometimes rethinking the overall pipeline structure.

Suggestions

Changing goals, requirements, and data sources is sometimes unavoidable. Before diving into the refactor, visually map out options for updated pipelines, consider the level of effort and tradeoffs required for each option, and align with teammates on a course of action. We often make pipeline visualizations on simple annotated slides:
Sprawling if/else logic across the pipelines for different data sources is a code smell. Your goal should be to standardize what you need to at the moment the data source is first initialized and then share as much unchanged pipeline code as possible.
Tech debt comes into play for what packages and libraries you choose as well. When multiple library options are available, try to pick the most well-maintained and widely used libraries up front. Look at how long it's been since the most recent commit to the package's source code repository, and how many stars the repository has on GitHub. Well-maintained and widely used packages are less likely to introduce new bugs down the line as new versions of other dependencies are released.
Be deliberate about where flexibility is truly valuable. Communicate with your team about the tradeoffs, and make simplifying assumptions where you can.

5. Beauty takes time¶

There’s always a balance to strike between getting something working quickly and building something clean and easily maintainable. Extra beautification efforts, such as clear file names, informative comments, and tidy dataset creation, pay off in the long run. You or a teammate will inevitably return to make modifications to the pipeline, and clear, well-documented code will make it easier to understand and less prone to having bugs accidentally introduced when updating it. But, if there is a quick deadline, it may be reasonable to take some shortcuts.

In our data-topic detection example, naming an intermediate dataset processed_labeled_articles instead of test_dataset_3 makes the pipeline clearer without slowing you down. Refactoring article processing code that works, but is duplicative or inefficient, requires more effort; whether the refactor is worth it is a judgment call.

Suggestions

The level of investment in the pipeline development should be proportional to how long one run takes and how often it will be run end-to-end. If the completed pipeline will be run just once, it may be enough to make a pipeline that produces accurate results relatively fast. In cases where the pipeline will be run often, it is usually worth it to make the pipeline accurate, efficient, and refactored for clarity and maintainability. It is a good idea to align with your team on how much polish is needed before diving into any big code refactors.
Pick descriptive names for files, functions, and outputs from the start.
Include descriptive docstrings in functions. We've found that LLM coding assistants (e.g., Copilot) or tools that set up docstring structures (e.g., autoDocstring extension for VS Code) make properly documenting functions quick and easy.
Adding type hints (explicit input/output types in function signatures) is helpful for clarity and aids in error detection, especially when used alongside type-checking tools like MyPy, ty, and Pyright.
Refactor small inconsistencies as you go instead of letting them pile up. Likewise, format code as you go using automatic formatting tools, like Black or Ruff.
Pipelines are often parameterized: the code can be configured to run in different ways by changing inputs (parameters) rather than modifying the code itself. For each run of your pipeline, you should record the configuration used, environment variables, timestamps, and pointers to log files or outputs for future reference. You can, of course, do this recording manually (e.g., copying the exact command you ran, including parameters, and other pertinent information into a tracking file or notebook). But we recommend capturing and persisting this information automatically using config files/ environment variables for setting pipeline parameters and automated run tracking systems for recording run metadata.

Embracing challenging pipelines with intentional effort¶

Making data pipelines is an aspect of most data science projects. And it often seems so simple from the outside: just move data from A to B. As soon as you start building, though, the hidden complexity emerges.

There are many tools you can use to facilitate pipelines rather than creating Python pipelines from scratch (e.g., Dagster). But these tools add learning curves and complexity. It often makes sense to start an implementation from a Python script and then invest in more tooling later if needed.

With some of the best practices we discuss, we can prevent some of the pain points and common pitfalls that domain-specific tools try to solve for and make it easy to have reliable, trustworthy pipelines with minimal dependencies and the right level of investment.

For further reading about general data science best practices, DrivenData's ebook The 10 Rules of Reliable Data Science is publicly available for download.

AI Agents in Data Science Competitions: Lessons from the Leaderboard

2026-02-02T00:00:00-05:00

"Hi ChatGPT, please do data science on this dataset."

There's an odd tension at the heart of this quote. To those who are familiar with data science, it feels like a very wrong statement. To those who aren't familiar, it can feel totally fine. But AI is making that feeling of wrongness less and less relevant to actually getting results. AI agents are getting better, both at data science and at inferring motivation. So how good are they?

Over the past year or so, we've been checking in periodically on agent performance at data science tasks. We've been using this specific framing: assume you know nothing about data science and just a little about using AI agents—how high can you make it in some of our previous data science competition leaderboards?

Below we share some of the things we've seen work, not work, and some thoughts about the future.

Setup¶

We've iterated on our test setup a bit, but we've settled on simplicity. Using an agent can take many forms,¹ but here we're showing the results of using Anthropic's Claude Code (Opus 4.5) and OpenAI's Codex (GPT 5.2 Codex).² We set down a few rules:

Give each agent the same prompt and computing environment
Give each 24 hours to run
Give them moderately good hardware
Poke them when they stop working, but don't steer using any data science knowledge³
Otherwise, let them do anything they want

We wanted each agent to have all the resources a competitor would have at the beginning of a competition, so we gave it all the documentation, the data, and the benchmark notebook that we publish to get competitors started in the right direction. Here is what the prompt looks like:

I need you to create a high-performance submission file for a data science
competition. The goal is to create the highest-performing model possible.
Think as much as you possibly can about ways to make the best possible model.
You have 24 hours from the moment you start.

**I know a {metric} of less than {score} is possible. Do NOT give up until
this is achieved. Do everything you can to achieve this in under 24 hours.**

**Do NOT ask for any input from me. Run without interruption until you
achieve the goal or 24 hours have elapsed.**

<PROBLEM_DESCRIPTION>
...
</PROBLEM_DESCRIPTION>

<METRICS>
...
</METRICS>

<DIRECTORY_LAYOUT>
- `benchmark/`: the directory containing a benchmark notebook given to
  all participants
- `data/`: directory containing the data files (described below)
- `docs/`: the full competition documentation given to competition participants
- `output/`: the directory in which to save your outputs
- `src/`: the directory in which to save your code, notebooks, etc.
</DIRECTORY_LAYOUT>

<DATA_FILES>
| File name              | Description                         |
|------------------------|-------------------------------------|
...
</DATA_FILES>

<SUBMISSION_FORMAT>
...
</SUBMISSION_FORMAT>

<HARDWARE>
You have access to a machine with:
 - ...
</HARDWARE>

<SOFTWARE>
...
</SOFTWARE>

<OUTPUT>
- Write all code to `src/`
- Create submission outputs in the `output/` directory
...
</OUTPUT>

<SUCCESS_CRITERIA>
- You successfully run `make submission` to create a valid submission file.
- The submission file must be in the correct format.
- The submission file must be generated by your code.
- Your best performing submission is at `output/submission.csv`.
- Your estimate of {metric} on the test set is better than {score} OR
  24 hours have elapsed since you began.
</SUCCESS_CRITERIA>

We haven't aggressively tuned this prompt—we've only added to it enough so that agents can move past introspection of the environment and get straight to (and keep at) modeling. Agents have free reign over a Docker container with GPU access, so they can install anything they want.

Sample of results¶

We've run a lot of tests and trials in various formats, but here's a sample of where things stand today. We'll be showing results from three competitions: Conser-vision, Flu Shot Learning, and Goodnight Moon.

We want to show results from competitions with a variety of data modalities, dataset sizes, and levels of difficulty. Conser-vision is a practice competition where the goal is to classify the species of animal detected in wildlife camera trap images. Flu Shot Learning is also a practice competition using tabular data to predict whether people got H1N1 and seasonal flu vaccines. Finally, Goodnight Moon, Hello Early Literacy Screening is a prize competition that we ran recently to help score audio recordings from literacy screener exercises completed by young students.

We've split these results into two tables: a "final" results table and a "best" results table. Like some human competitors, the agents often overfit to the training data without knowing it. They then create many model versions and select a final submission with great performance on the training data but worse performance on the leaderboard than previous versions of their models. We show rankings for both these "final" submissions as well as the "best" performing version that the agent did not select as their final. These "best" rankings help characterize what agent performance might be in the absence of this overfitting.

Competition	Model	Final rank (Percentile)	Best rank (Percentile)
Conser-vision An image classification competition	Claude Opus 4.5	51 of 573 (91st)	20 of 573 (96th)
	GPT 5.2 - Codex	20 (96th)	11 (98th)
Flu Shot Learning A tabular data competition	Claude Opus 4.5	251 of 2,277 (89th)	251 of 2,277 (89th)
	GPT 5.2 - Codex	297 (87th)	297 (87th)
Goodnight Moon An audio-based competition	Claude Opus 4.5	33 of 37 (13th)	19 of 37 (51st)
	GPT 5.2 - Codex	19 (51st)	12 (70th)

Table 1: Final and best submission ranks. Best agent performance per competition in bold.

We also calculate the agent's progress toward first place. We define "progress to first" here as the improvement over the benchmark's performance normalized by the best performing model in the competition. (We provide a benchmark for each competition which is a simple solution demonstrating the end-to-end process of getting the data through submitting to the competition.) A submission that scored the same as the benchmark would have 0% and a submission that scored the same as the competition winner would have 100%. For some competitions, scores at the top of the leaderboard are quite bunched, while for others only a few participants might have had a key insight, leading to bigger jumps on the leaderboard. This metric captures how much absolute performance has been captured by the agent's best model.

Competition	Model	Final: Progress to 1st	Best: Progress to 1st
Conser-vision	Claude Opus 4.5	45%	60%
	GPT 5.2 - Codex	62%	75%
Flu Shot Learning	Claude Opus 4.5	93%	93%
	GPT 5.2 - Codex	92%	92%
Goodnight Moon	Claude Opus 4.5	-25%	42%
	GPT 5.2 - Codex	46%	52%

Table 2: Progress to 1st (final vs best). Best agent performance per competition in bold.

We'll use these results as a reference to talk about:

What works well today
Current challenges
Open questions and opportunities

What works well¶

The agents reach an 80% solution remarkably quickly, which is a great head start compared to starting from scratch (or from our provided benchmarks). This rapid baseline development could significantly reduce initial time investment for competitions, giving folks more time to work on "the other 90%."

Agents are much better at recovering from errors than they were. Only months ago, we'd see agents write code that didn't work, and then loop on trying to fix that until they ran out of time (or we ran out of money). Now, agents get to a working submission basically every time. For example, when building a submission for the Goodnight Moon competition, Codex received the error:

RuntimeError: The size of tensor a (125) must match the size of tensor b (40320) at non-singleton dimension 1

For folks who have been down in the guts of PyTorch, this will be instantly recognizable as "the way you are spending the next 4 hours." To solve it, the agent needed to know how Wav2Vec downsamples audio when building embeddings, which it simply remembered when it saw this error. Most practitioners would not simply remember this, and neither would agents from six months ago.

Agents are fast. Data science competitions are often framed as a search problem: if you wrote every possible computer program, you'd just need to hunt through your big pile of programs to find the one that wins.⁴ You can't write every possible program, but you can write a lot more programs with an agent than you can without. On that same Goodnight Moon competition, Codex generated 62 different submissions and still had hours left to spare.

Performance can be really good, and this is the worst these agents are ever going to be. Even with today's tools, our results are kind of the worst they could be. Consider the constraints we put on these experiments again:

Run in 24 hours or less
No human steering
An unoptimized prompt with no additional scaffolding
A single, mid-level consumer GPU

Each of these could be relaxed with a little bit of work. Coaxing Codex to run for 9 days on the Goodnight Moon competition instead of 24 hours resulted in its rank rising to #7 from #12 and its "progress to first" rising from 52% to 66%.

When asked to do better on a competition, agents generally do run longer and then output a better submission. They hit diminishing returns and top out prior to the top of the leaderboard, but still with very respectable scores.

Finally, actually running these agents has become so much easier. Near the beginning of these experiments we were running things like the OpenHands agent, which was great, but did require a good deal of setup and hand-holding. With the advent of agent harnesses coming from the frontier labs themselves, many of the rough edges have been made moderately smoother (though there are still a lot of rough edges as things are moving fast).

Current challenges¶

Simply getting the models to try hard enough for long enough is one of the biggest challenges both in the past and today. We found that we had to give the agents specific metric scores to beat otherwise they would train maybe a couple iterations and then stop. That can be seen in the example prompt: "I know a {metric} of less than {score} is possible. Do NOT give up until this is achieved. Do everything you can to achieve this in under 24 hours." These numbers need not be realistic—impossible scores serve just as well as realistic scores.

The agents seem very resistant to kick off long training runs. We'll often see competitors whose models need to train for multiple days to reach the top of the leaderboard. But these agents, even with scaffolding timeouts removed, will often not kick off training runs that will last more than 15 minutes. On the one hand, you can iterate a lot if none of your training runs last longer than 15 minutes, but on the other hand, you're never going to eke out that last bit of performance if all of your hyperparameters are tuned to complete training runs in relatively short periods of time.

Agents are actually overfitting more now than they have in our prior iterations of these experiments. As touched on in the Results section above, we found that very often, earlier versions of an agent's code performed significantly better than their final submissions. In Claude's attempt at the Conser-vision competition, it created validation splits and intended to perform five-way cross-validation. However, we found that it never actually executed the cross-validation and instead used a single split. We suspect this shortcut stems from the same hesitance to initiate long training runs. Better prompting strategies would likely alleviate this to a degree.

Hardware is still a significant constraint, which is not a limitation of the agents themselves, but still a limit on the overall workflow. We find that when running many of these sessions, most of the wall clock time is still spent in training compute (even despite short training runs). Significantly less time is actually spent with the model thinking or writing code. That is just a reality of data science competitions today. It seems like it'll be a reality tomorrow, too.

Non-determinism still plays a fairly large part in the final scores, though this has decreased a lot since we first started our experiments. A single session could get you a top 2% submission or a top 50% submission. Even Anthropic has referred to Claude Code as a "slot machine." An obvious solution to this is to try multiple times, but that runs into the next challenge that we saw.

Usage limits are real, especially for Claude Code. It's difficult to keep a session running for 24 hours without hitting usage limits or spending a significant amount of money. Obviously, this is great from the model provider's perspective, but it is a real constraint. Since we started running these experiments, this token consumption has gotten worse and then gotten better. Right now, the trend lines look favorable.

Sensitive data can't be tossed at random APIs. To get frontier performance right now, one needs to be using frontier models. Frontier models are not open-weight models, so they can't be run on your local hardware. That means to run an agent, you need to be shipping off some amount of competition data (or data derived from it) to another organization, e.g. OpenAI or Anthropic (or one of their partner providers). If the competition data is sensitive in some way, sharing the data isn't a good idea and competition rules often forbid it. Running agents locally is an alternative, but performance is roughly a year behind the frontier, which our experimentation suggests is pretty far, performance-wise.

It's easy to get out over your skis. We've seen this trend elsewhere, but in these experiments we assumed no expertise in data science on the part of the user. In reality, if you have little expertise in data science, it's very easy to quickly get to a 90% solution and then get completely stuck because you have no idea what's been built. That can also be the case here. Even as data scientists, we often don't have expertise in every data modality, every technique, every modeling strategy. If an agent builds you something that is far out of your wheelhouse, you have few levers to pull to meaningfully improve it.

Performance, it turns out, is not purely a function of raw intelligence of the base model. A lot of capabilities are elicited by the sophistication of the agents' scaffolding. For example, the latest Gemini models perform very well on a number of benchmarks and they find some use in our own work. But we found that the Gemini CLI is significantly less developed than the others we show here. Sessions would often end in either failure or scores significantly worse than the ones we see here.⁵

Opportunities and open questions¶

These small-scale experiments have helped us gain some perspective on big questions.

Are competitions still helpful?¶

Competitions serve various purposes beyond identifying the best model, but let's take that as the default case. From the perspective of a competition organizer seeking state-of-the-art models, little has changed. Previously, if you needed state-of-the-art, only the few top solutions mattered. Now with coding agents, if competitors only submit what their agents give them, there's little chance they'll perform better than the top-performing humans (who can also use agents). The top models in the past were created by humans with impeccable expertise and insight. The top models in the present are still created by humans with impeccable expertise and insight, they just now have more tools in their toolbox.

Do agents change who can participate?¶

We have some very technical "non-technical" folks on our team. They're terrific at reasoning about datasets, looking for potential pitfalls, finding new angles to look at a problem, etc., but knowing which versions of Python maintain ordered dictionary keys is not within their core competencies (can't blame 'em). Could they now coach an agent into developing a state-of-the-art model without needing to write code themselves (i.e. "vibe code to the top")?

Such democratization seems more likely if competitions approach the "aleatoric limit"—where data randomness fundamentally caps predictive performance. At this limit, traditional and agent-assisted approaches might converge on similar solutions or at least converge at similar levels of performance. We see that in the Flu Shot Learning results above: all the top scores are clustered closely together.

We've observed that top competitors often win multiple competitions without reusing code or architectures, suggesting some set of transferable skills. It's unclear whether these skills skew toward the highly technical or toward reasoning skills that allow for consistently finding profound insights in new datasets and domains, but we'll happily welcome changes that make for a bigger tent.

It does seem true that the skills to push modeling forward are different in an age of agents. Data science expertise is still important, but so is prompting, setting up scaffolding and tooling for the agents, and maybe even the skill of managing dozens of agents at once.

How big is the capability overhang?¶

OpenAI has defined the capability overhang as "the gap between what AI tools can do and how typical users are using them." In some real sense and in ways we've gestured at above, the models as they exist today are capable of performing better than we're eliciting here. Those capabilities are locked away for lack of better agent scaffolding and prompting. For example, we could relax our experiment rules a bit and instruct the models how to avoid overfitting. Human experts in our competitions are smart about avoiding overfitting, exploiting every small edge for hill climbing the optimization landscape, and knowing when to invest in long, compute-intensive training runs. Can we design better agent loops that truly have this kind of knowledge? How hard is that? How true is it that unlocking those hidden capabilities will translate into performance gains that matter? Questions and opportunities abound here.⁶

How are competitions going to evolve?¶

As we've mentioned above, agent performance can be improved by spending time locating weaknesses, trying different approaches, running experiments, etc. This sounds familiar. Instead of writing code that optimizes models, we're now writing code that writes code that optimizes models. This isn't AutoML; it transforms the data scientist's job rather than eliminating it. The required skills evolve, but human guidance, creativity, and oversight remain essential if the goal is to push forward the state-of-the-art.

Is 80% good enough?¶

In what contexts is the agent's "80% solution" good enough? Not every problem requires state-of-the-art performance to deliver business value. Often, a good-enough solution delivered quickly provides more organizational value than a perfect solution delivered late. If an 80% solution can be delivered in one minute (as opposed to a 100% solution in months), what does that imply about how work will change going forward?

How do we account for taste?¶

These solutions are never really delivered in one minute. We could only assess agent performance for this experiment because DrivenData and our partners have done significant groundwork: selecting appropriate problems, gathering data, choosing metrics, writing documentation, and creating solution scaffolding. A single agent can solve many different data science problems, but the bulk of the work right now is curating and assembling the many problems rather than assembling the agent. We can recognize, curate, and guide in ways that remain valuable, and as the technical details see more and more automation, those skills will only increase in value.

Conclusion¶

Agents now reach a working, competitive submission in hours instead of days. They recover from errors gracefully and iterate faster than any human could. But they resist long training runs, overfit more than they should, and plateau before the top of the leaderboard. The 80% solution is essentially solved. The last 20% is not.

Once, humans were the best at chess. Then it was humans plus computers. Then, computers alone. The game of Go followed the same arc, just later because the search space was bigger.

Data science has a bigger search space still. Right now we're somewhere in the middle: agents reach 80% fast, but the best solutions still come from humans who can find their way to a winning submission but can't fully explain every turn along the path to that solution. Competitions have always been a way of surfacing that inexplicable knowledge, but now they're also a way of measuring how much of it is left. Humans at the top are currently doing something that agents can't. Can that something survive being named?

Banner image: image by Metropolitan Transportation Authority, CC BY 2.0

There are a lot of data-science-specific agents or AutoML libraries out there: Weco AI, AutoKaggle, DS-Agent, FLAML, H2O AutoML, AutoGluon, PyCaret ↩
We have also been testing Google's Gemini CLI with Gemini 3 Pro, but its performance has been significantly lower than the others and so isn't included. ↩
Most interactions with the agents consisted of just saying "Continue". ↩
Is that program called "Claude Opus 6"? ↩
With that said it's still very early days and Google has many irons in many fires, including Antigravity and Jules. ↩
Shout out again to all those doing the work to explore this space: Weco AI, AutoKaggle, DS-Agent, FLAML, H2O AutoML, AutoGluon, PyCaret, etc. etc. ↩