DrivenData Labs

Improving Automatic Speech Recognition for Kids - A Reference Implementation for Phonetic-level Transcription

2026-03-10T00:00:00-04:00

Welcome to the reference implementation notebook for the On Top of Pasketti: Children's Speech Recognition Challenge - Phonetic Track! If you are just getting started, we recommend reading the competition webpage first.

The goal of this tutorial is to:

Demonstrate how to load and explore the data.
Provide a basic framework for building a model.
Walk through how to package your work correctly for submission.

We will be fine-tuning Wav2Vec2, a pretrained speech representation model for automatic speech recognition ("ASR"), using the Hugging Face Transformers library. Wav2Vec2 converts raw audio into contextual acoustic representations through a convolutional feature extractor followed by a Transformer encoder.

For this challenge, we adapt Wav2Vec2 to predict phonetic units (phones) represented by the International Phonetic Alphabet (IPA) rather than words or characters. While Wav2Vec2 is commonly fine-tuned for grapheme- or word-level ASR, its learned acoustic representations can also support phone-level prediction. By training the model with an IPA phone vocabulary and using Connectionist Temporal Classification (CTC), we can learn to predict phone sequences directly from audio without requiring manual time alignment.

You can either expand on and improve the method in this reference implementation, or start with something completely different! Let's get started.

Background¶

Spoken language is a natural way for kids to learn, explore, and show what they know, yet today's ASR technology hardly understands them. Most ASR systems are built on adult speech, and struggle with the pitch, rhythm, and evolving articulation of young learners.

The On Top of Pasketti: Children’s Speech Recognition Challenge assembles pre-existing and newly labeled datasets to advance speech models that truly work for children. Your goal in the Phonetic Track is to develop models that accurately predict the speech sounds, or phones, spoken by children in audio clips. Phonetic models are critical for diagnostic applications like speech pathology screening.

This is a code execution challenge! Rather than submitting your predicted labels, you will package your trained model and the prediction code and submit that for containerized execution. See the code submission format webpage and the runtime repository for more information.

If you'd like to rerun this notebook, the notebook file can be downloaded from the reference implementation repository. That repository also includes all code imported into the notebook.

Step 0: Import packages¶

First, create your environment. We use uv as the package manager in this reference implementation repository.

Create an environment: just create-environment
Activate the environment: source ./.venv/bin/activate
Install the requirements found in the TOML file into the environment: just requirements

Remember, the runtime repository's TOML file lists the packages that will be available for running inference using model submissions.

We'll be using PyTorch and Hugging Face Transformers to build our model along with standard data science Python libraries to explore and prepare the data. Because this is a code execution challenge, we'll also be testing our solutions locally before packaging our model and inference code for submission. To help us with scoring, we've imported some utility functions from the competition's runtime repository.

In [1]:

# Standard library
from dataclasses import dataclass
import json
from pathlib import Path
import random
from typing import Dict, List, Union, Optional

# Data Science & Utilities
from IPython.display import display
from loguru import logger
import numpy as np
import pandas as pd
import tqdm

# Visualization
from matplotlib import ticker
import matplotlib.pyplot as plt

# Core ML & Audio Stack
from datasets import Dataset, Audio, Features, Value, load_from_disk
import torch
from transformers import (
    Wav2Vec2CTCTokenizer,
    Wav2Vec2FeatureExtractor,
    Wav2Vec2Processor,
    Wav2Vec2ForCTC,
    TrainingArguments,
    Trainer,
)

# Project Utilities
from asr_benchmark.config import DATA_ROOT, PROJECT_ROOT
from asr_benchmark.score import VALID_IPA_CHARS, score_ipa_cer

In [2]:

pd.options.display.max_rows = 200
pd.options.display.max_colwidth = 1200

# Force 'auto' to use the standard console tqdm
tqdm.auto.tqdm = tqdm.tqdm

Step 1: Load and explore the data¶

First, you'll likely want to set up your own repository for developing a solution. We recommend using Cookiecutter Data Science, which ensures an easy-to-navigate project structure.

We'll download all of the competition data to our "raw" folder. There are two distinct training corpora that share the same structure but contain different data, and are hosted in separate locations for participant access. One corpus is hosted on the DrivenData platform, while a second corpus, which follows the same schema but contains different data, is provided by TalkBank. See the Data Download page for access instructions.

Our local data structure after downloading all files to a raw data folder is:

childrens-speech-recognition-benchmark-pub/data/raw
    ├── drivendata
    │   ├── audio.zip
    │   └── train_phon_transcripts.jsonl
    └── talkbank
        ├── audio.zip
        └── train_phon_transcripts.jsonl

After unzipping the audio, we can start exploring the data!

For each of the two corpora, the file train_phon_transcripts.jsonl contains the following fields:

utterance_id (str) - unique identifier for each utterance
child_id (str) - unique, anonymized identifier for the speaker
session_id (str) - unique identifier for the recording session; a single child_id may be associated with multiple session_ids
audio_path (str) - path to the corresponding .flac audio file relative to the /audio directory, following the pattern audio/{utterance_id}.flac
audio_duration_sec (float) - duration of the audio clip in seconds
age_bucket (str) - age range of the child at the time of recording ("3-4", "5-7", "8-11", "12+", or "unknown")
md5_hash (str) - MD5 checksum of the audio file, used for integrity verification
filesize_bytes (int) - size of the audio file in bytes
phonetic_text (str) - phonetic transcription of the utterance using the International Phonetic Alphabet (IPA)

Each line in the JSONL manifest corresponds to a single utterance and references exactly one associated audio file. The phonetic_text field contains a manually created, minimally normalized phonetic transcription that serves as the training label.

Let's explore the metadata!¶

We will load the JSONL transcripts and explore some of the metadata. As a starting point, it is helpful to know how many utterances we have, how many unique children are present, the total audio time, the distribution of audio clip durations, and the distribution of child ages.

In [3]:

def read_transcripts(data_dir: Path) -> pd.DataFrame:
    """Read JSONL transcript file into a DataFrame and convert audio paths to absolute paths."""
    transcript_path = data_dir / "train_phon_transcripts.jsonl"
    df = pd.read_json(transcript_path, lines=True)
    logger.info(f"Loaded {len(df)} utterance transcripts")

    df["audio_relpath"] = df["audio_path"]
    df["audio_path"] = df["audio_relpath"].map(lambda p: str(data_dir / p))

    return df

In [4]:

df_dd = read_transcripts(DATA_ROOT / "raw" / "drivendata")
df_tb = read_transcripts(DATA_ROOT / "raw" / "talkbank")
df = pd.concat([df_dd, df_tb], ignore_index=True)
df.drop(columns=["audio_path"]).head()

2026-03-09 23:53:06.457 | INFO     | __main__:read_transcripts:5 - Loaded 12043 utterance transcripts
2026-03-09 23:53:07.659 | INFO     | __main__:read_transcripts:5 - Loaded 141024 utterance transcripts

Out[4]:

	utterance_id	child_id	session_id	audio_duration_sec	age_bucket	md5_hash	filesize_bytes	phonetic_text	audio_relpath
0	U_0004fcba47fc2b22	C_90db3102e2ca5699	S_419bdf9a10b462ab	1.435	3-4	5895195afa942429e393e5f9ada72e77	121365	ʔə ʔæpɫ	audio/U_0004fcba47fc2b22.flac
1	U_000727b46808376d	C_1c2507f6a16497c4	S_bae094ac403ace8c	1.430	5-7	4323bbdc2503cd0e63431a328243ac44	39763	hjuːdə	audio/U_000727b46808376d.flac
2	U_0012a1c1c3646a51	C_0efba146ed1d6bd1	S_80c5357ec852f45e	0.692	3-4	fb204762a593add9edf112321d3f5220	57768	ʔɛfɹi	audio/U_0012a1c1c3646a51.flac
3	U_00142fc9f1318b66	C_d4df815189b10ac9	S_26c9e4507452cad3	1.685	3-4	20a33e749593a0471122d4671c5e9aca	111562	sɑnd tɔiz	audio/U_00142fc9f1318b66.flac
4	U_00181407f98d6b68	C_63baf19fc3f58441	S_8f2a5a16e483fe0f	2.083	3-4	39828bb94a994b7a55790596ce870b16	175378	æn nʌɾɚ wʌn ɪn hɪɚ	audio/U_00181407f98d6b68.flac

In [5]:

df.utterance_id.nunique()

Out[5]:

In [6]:

df.child_id.nunique()

Out[6]:

In [7]:

round(df.audio_duration_sec.sum() / (60**2))

Out[7]:

There are over 153,000 utterances in the training dataset, across 1,003 children, totaling 85 hours of audio data. Next, let's take a look at the distribution of audio clip durations.

In [8]:

bins = list(range(0, 21)) + [np.inf]
labels = [str(i) for i in range(0, 20)] + ["20+"]

binned = pd.cut(df.audio_duration_sec, bins=bins, labels=labels, right=False)
counts = binned.value_counts().sort_index()

counts.plot(kind="bar")
plt.xlabel("Audio Duration (sec)")
plt.ylabel("Number of Audio Clips")
plt.title("Distribution of Audio Durations")
plt.xticks(rotation=90)
plt.show()

Most audio clips are very short (1-3 seconds). Even though the audio has been clipped to the utterance level, we have some outliers over 20 seconds. Next, let's look at the distribution of utterances by child age.

In [9]:

df["age_bucket"] = pd.Categorical(
    df["age_bucket"], categories=["unknown", "3-4", "5-7", "8-11", "12+"], ordered=True
)

fig, ax = plt.subplots()
df["age_bucket"].value_counts(normalize=True, sort=False).plot.barh(ax=ax)
ax.set_title("Utterances by Age Group")
ax.set_xlabel("Percent of Utterances")
ax.set_ylabel("Age Group")
ax.xaxis.set_major_formatter(ticker.PercentFormatter(1.0))
ax.bar_label(ax.containers[0], fmt=lambda x: f"{x * 100:.0f}%")

plt.show()

All age buckets are well represented:

24% of the utterances come from 3 to 4 year olds
16% of the utterances come from 5 to 7 year olds
38% of the utterances come from 8 to 11 year olds
18% of the utterances come from 12 year olds and older

Let's explore the utterances!¶

We will listen to an example utterance and explore its phonetic transcription.

Children’s speech often includes subtle pronunciation differences when compared to adult speech. In the Phonetic Track, models must learn to map pronunciations that vary by age, development, and region to target labels that reflect the phones each child actually produced.

In [10]:

df[df.utterance_id == "U_1c8757065e355c35"][
    ["utterance_id", "audio_duration_sec", "phonetic_text"]
]

Out[10]:

	utterance_id	audio_duration_sec	phonetic_text
27819	U_1c8757065e355c35	0.516	jæ

The ground truth phonetic_text labels are normalized phonetic transcriptions of individual utterances using the International Phonetic Alphabet (IPA), with a one-to-one mapping between Unicode characters and phones. Each transcription captures the full sequence of speech sounds in the corresponding audio clip and may include substitutions, omissions, or non-standard productions that are typically ignored in word-level ASR.

All phonetic labels are restricted to the predefined IPA character set used during phonetic transcription. This set is provided in the scoring script in the runtime repository for local validation of predictions.

Step 2: Build the Model¶

A straightforward modeling option is to start from a strong pretrained ASR model, then fine-tune on our labeled phonetic child-speech training set. In this tutorial, we fine-tune Facebook's pretrained Wav2Vec2-base model using Hugging Face Transformers.

Wav2Vec2 is relatively simple and efficient to fine-tune, making it a reasonable starting point for this challenge. It uses a convolutional neural network ("CNN") feature extractor followed by a transformer encoder to learn audio representations.

A CNN is a neural network that learns useful features from data by applying small pattern detectors called filters. These filters scan across the input (such as an image or audio signal) and learn to recognize important patterns.
Wav2Vec2 is pretrained on unlabeled speech audio and later fine-tuned for ASR using a defined output vocabulary, typically consisting of characters. For this challenge, we instead define a vocabulary of IPA phone symbols, mapping each phonetic symbol to a unique integer ID.

We will freeze the feature extractor to preserve the robust pre-trained audio processing capabilities, then fine-tune the transformer encoder and a newly initialized CTC head configured for our phonetic character vocabulary. Hugging Face makes this process easier by providing model architectures, data processing utilities (tokenizers, feature extractors, data collators), and integrated training pipelines.

Key packages include:

transformers for Wav2Vec2 model + training utilities
datasets for data loading and preprocessing
torch for the training backend

1. Prepare Dataset¶

We need to process our dataframe containing the DrivenData and TalkBank datasets. We filter out clips longer than 25 seconds, which strain computer memory. Competitors may want to further split these clips to avoid losing training data. We remove one corrupted file before limiting the data to just the phonetic transcription and the audio filepath.

In [11]:

# Filter down to audio less than 25 seconds to reduce strain on memory
df = df[(df.audio_duration_sec <= 25)]

# Filter out corrupted file
df = df[df.utterance_id != "U_b8a4e8220e65219b"]

# For now, we only need the transcript and the audio path
df = df[["phonetic_text", "audio_path"]]

We then create a Hugging Face Dataset from our dataframe, which gives us a standard format that works cleanly with dataset transforms and the training pipeline. We also cast the audio column to 16 kHz so clips are decoded and resampled to the sampling rate expected by Wav2Vec2.

In [12]:

# Audio sampling rate
SR = 16000

# Enforce string types so that datasets can consume them properly
schema = Features(
    {
        "phonetic_text": Value("string"),
        "audio_path": Value("string"),
    }
)

dataset = Dataset.from_pandas(df.reset_index(drop=True), features=schema)
dataset = dataset.cast_column("audio_path", Audio(sampling_rate=SR))

2. Build Vocabulary and Tokenizer¶

Our model treats each IPA character as one token, so it is important that every valid character has a consistent ID. To create a character to ID mapping, we take the imported VALID_IPA_CHARS from the scoring script and map each IPA character to an index for the tokenizer.

Special tokens are added to the set of IPA characters:

| replaces spaces as the word delimiter
[UNK] is a fallback token that maps any character not in VALID_IPA_CHARS to a single token ID
[PAD] is the padding token used to pad sequences to equal length in batches

This mapping is saved so we can reuse the same token IDs when we initialize the tokenizer now and when we load the model for inference later.

In [13]:

# VALID_IPA_CHARS contains the following IPA characters:
print(*VALID_IPA_CHARS)

æ o b c ɑ   ʔ ɛ s j ɹ f ʊ ə ʁ i ɔ v ɟ x ʌ ŋ χ d h ʝ k n l ɐ ʤ ɪ ɚ ç z ː m t ɫ ʃ ʧ w r p ɾ ɬ ʒ θ u g ð e

In [14]:

unk_tok = "[UNK]"
pad_tok = "[PAD]"
space_tok = "|"

all_toks = sorted([char for char in VALID_IPA_CHARS if char != " "]) + [
    unk_tok,
    pad_tok,
    space_tok,
]

vocab_dict = {char: idx for idx, char in enumerate(all_toks)}

vocab_path = DATA_ROOT / "vocab/phonetic_vocab.json"
vocab_path.parent.mkdir(parents=True, exist_ok=True)
with vocab_path.open("w") as f:
    json.dump(vocab_dict, f)

Next, we initialize a tokenizer to convert text labels to token IDs. The tokenizer reads the vocabulary mapping we just saved, so each IPA character and special token uses the same ID during training and inference. We also initialize the feature extractor, which converts raw 16 kHz waveforms into model-ready input values for Wav2Vec2.

Finally, the Wav2Vec2Processor is initialized. The processor combines the feature extractor and tokenizer in one object, so we can consistently preprocess audio and encode/decode labels.

In [15]:

tokenizer = Wav2Vec2CTCTokenizer(
    str(vocab_path), unk_token=unk_tok, pad_token=pad_tok, word_delimiter_token=space_tok
)

# Create Wav2Vec2 Feature Extractor
feature_extractor = Wav2Vec2FeatureExtractor(
    feature_size=1,
    sampling_rate=SR,
    padding_value=0.0,
    do_normalize=True,
    return_attention_mask=False,
)

# Create processor (combines tokenizer and feature extractor)
processor = Wav2Vec2Processor(feature_extractor=feature_extractor, tokenizer=tokenizer)

3. Create Data Collator¶

Data collators prepare training batches from variable-length sequences. In our model, since audio clips have different lengths, the data collator adds padding to shorter clips so all clips in a batch have the same length. It marks padded positions in the labels with -100, which tells the training algorithm to ignore those positions when calculating loss.

In [16]:

@dataclass
class DataCollatorCTCWithPadding:
    """
    Data collator that will dynamically pad the inputs received.
    """

    processor: Wav2Vec2Processor
    padding: Union[bool, str] = True
    max_length: Optional[int] = None
    max_length_labels: Optional[int] = None
    pad_to_multiple_of: Optional[int] = None
    pad_to_multiple_of_labels: Optional[int] = None

    def __call__(
        self, features: List[Dict[str, Union[List[int], torch.Tensor]]]
    ) -> Dict[str, torch.Tensor]:
        input_features = [
            {"input_values": feature["input_values"]} for feature in features
        ]
        label_features = [{"input_ids": feature["labels"]} for feature in features]

        batch = self.processor.pad(
            input_features,
            padding=self.padding,
            max_length=self.max_length,
            pad_to_multiple_of=self.pad_to_multiple_of,
            return_tensors="pt",
        )
        labels_batch = self.processor.tokenizer.pad(
            label_features,
            padding=self.padding,
            max_length=self.max_length_labels,
            pad_to_multiple_of=self.pad_to_multiple_of_labels,
            return_tensors="pt",
        )

        # replace padding with -100 to ignore loss correctly
        labels = labels_batch["input_ids"].masked_fill(
            labels_batch.attention_mask.ne(1), -100
        )

        batch["labels"] = labels

        return batch


# Initialize data collator
data_collator = DataCollatorCTCWithPadding(processor=processor, padding=True)

4. Preprocess Audio¶

We define a preprocessing function that extracts features and tokenizes the phonetic text labels using the Wav2Vec2Processor object we already created. The function replaces spaces with word delimiters before tokenization to prepare labels for training.

In [17]:

def preprocess_batch(examples):
    # Start by loading the audio and processing with the feature extractor
    processed_batch = {
        "input_values": [
            processor(item["array"], sampling_rate=SR).input_values[0]
            for item in examples["audio_path"]
        ]
    }

    # Replace spaces with word delimiter and tokenize for CTC
    processed_batch["labels"] = [
        processor(text=ex.replace(" ", "|")).input_ids for ex in examples["phonetic_text"]
    ]

    return processed_batch

We apply this preprocessing across the dataset in parallel and save the results to disk for faster development iteration.

In [18]:

PROCESSED_DATASET_DIR = DATA_ROOT / "processed" / "phonetic_dataset"

if PROCESSED_DATASET_DIR.exists():
    processed_dataset = load_from_disk(str(PROCESSED_DATASET_DIR))
    print(
        f"Loaded preprocessed dataset from {PROCESSED_DATASET_DIR.relative_to(PROJECT_ROOT)} ({len(processed_dataset)} examples)"
    )
else:
    processed_dataset = dataset.map(preprocess_batch, batched=True, num_proc=4)
    processed_dataset.save_to_disk(str(PROCESSED_DATASET_DIR))
    print(
        f"Preprocessed and saved dataset to {PROCESSED_DATASET_DIR.relative_to(PROJECT_ROOT)}"
    )

Preprocessed and saved dataset to data/processed/phonetic_dataset

5. Model Configuration and Training Setup¶

We load the pretrained Wav2Vec2-base model with a CTC architecture and initialize a new CTC head configured for our phonetic vocabulary. CTC enables direct audio-to-phone prediction without requiring explicit alignment between audio frames and individual characters. This simplifies our setup because we don't have to label exactly when each phone occurs in the audio, rather we just have to specify the sequence of phones.

The feature extractor is frozen to preserve audio processing learned from the model pretraining; we lock those weights and don't update them during training. We will only fine-tune the transformer encoder (which learns acoustic patterns) and the CTC head (which converts these patterns into phone predictions).

In [19]:

# Load pretrained Wav2Vec2 model
model = Wav2Vec2ForCTC.from_pretrained(
    "facebook/wav2vec2-base",
    ctc_loss_reduction="mean",
    ctc_zero_infinity=True,  # Replace inf CTC loss with 0 to prevent NaN gradients
    pad_token_id=processor.tokenizer.pad_token_id,
    ignore_mismatched_sizes=True,
    vocab_size=len(processor.tokenizer),
)

# Freeze feature extractor layers
model.freeze_feature_encoder()

/.venv/lib/python3.11/site-packages/transformers/configuration_utils.py:309: UserWarning: Passing `gradient_checkpointing` to a config initialization is deprecated and will be removed in v5 Transformers. Using `model.gradient_checkpointing_enable()` instead, or if you are using the `Trainer` API, pass `gradient_checkpointing=True` in your `TrainingArguments`.
  warnings.warn(
Some weights of Wav2Vec2ForCTC were not initialized from the model checkpoint at facebook/wav2vec2-base and are newly initialized: ['lm_head.bias', 'lm_head.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.

We then filter out a few utterances that violate CTC's length constraints. This violation occurred because, for some utterances, there were too many tokens for the audio time. Wav2Vec2 downsamples audio by 320x, so the output sequence length must exceed the label length to avoid infinite loss values.

In [20]:

# Filter out samples that violate the CTC constraint:
# Wav2Vec2 downsamples audio by 320x, so input_length // 320 must be > label_length.
# Samples violating this produce inf CTC loss -> NaN gradients.
WAV2VEC2_DOWNSAMPLE = 320
before_filter = len(processed_dataset)


def is_valid_ctc_sample(example):
    input_len = len(example["input_values"])
    label_len = len(example["labels"])
    # CTC requires: output_timesteps > label_length (including blanks)
    output_timesteps = input_len // WAV2VEC2_DOWNSAMPLE
    return output_timesteps > label_len and label_len > 0 and input_len > 0


processed_dataset = processed_dataset.filter(is_valid_ctc_sample, num_proc=4)
print(
    f"CTC filter: {before_filter} -> {len(processed_dataset)} samples ({before_filter - len(processed_dataset)} removed)"
)

CTC filter: 152997 -> 152989 samples (8 removed)

The Trainer expects separate training and validation datasets, so we can simply split our processed_dataset such that 90% goes to training and 10% goes to validation.

In [21]:

# Split dataset into train and validation
dataset_split = processed_dataset.train_test_split(test_size=0.1, shuffle=True, seed=42)

train_dataset = dataset_split["train"]
eval_dataset = dataset_split["test"]

While the loss function drives the actual training and weight updates, the final metric upon which model inferences will be evaluated is Character Error Rate ("CER"). CER measures the edit distance between predicted and reference phonetic sequences at the character level. In training, whenever we compute validation loss, we also calculate the CER so that we can monitor training progress on a human-interpretable metric and select the best model checkpoint.

To compute CER, we run the model on the validation dataset to generate phone predictions, then compare them to the ground truth labels. The score_ipa_cer function is taken directly from the runtime repository. Please note that the score_ipa_cer function normalizes the prediction and reference text before computing the CER.

In [22]:

def compute_metrics(pred):
    """Compute Character Error Rate (CER) for phonetic transcription."""
    pred_logits = pred.predictions
    pred_ids = np.argmax(pred_logits, axis=-1)

    # Replace -100 with pad_token_id for decoding
    pred.label_ids[pred.label_ids == -100] = processor.tokenizer.pad_token_id

    # Decode predictions and labels
    pred_str = processor.batch_decode(pred_ids)
    # Don't group tokens when computing metrics (important for CTC)
    label_str = processor.batch_decode(pred.label_ids, group_tokens=False)

    return {"cer": score_ipa_cer(label_str, pred_str)}

Finally, the training hyperparameters are configured with reasonable starting points, such as:

learning rate of 5e-5
batch size of 27 with gradient accumulation
20 epochs with a linear warmup and decay learning rate schedule
evaluation every 1000 steps

The Hugging Face Trainer handles the training loop, checkpointing, and evaluation.

In [23]:

# Define training arguments
output_dir = str(PROJECT_ROOT / "models" / "wav2vec2-phonetic")

training_args = TrainingArguments(
    output_dir=output_dir,
    group_by_length=False,
    per_device_train_batch_size=27,
    per_device_eval_batch_size=27,
    gradient_accumulation_steps=2,
    max_grad_norm=1.0,
    learning_rate=5e-5,
    num_train_epochs=20,
    weight_decay=0.01,
    eval_strategy="steps",
    eval_steps=1000,
    save_steps=1000,
    logging_steps=100,
    warmup_steps=500,  # Shorter warmup for fewer epochs
    lr_scheduler_type="linear",
    bf16=True,
    fp16=False,
    gradient_checkpointing=False,
    dataloader_num_workers=8,
    dataloader_pin_memory=True,
    save_total_limit=2,
    metric_for_best_model="cer",
    greater_is_better=False,
    load_best_model_at_end=True,
    report_to="none",
)

# Initialize trainer
trainer = Trainer(
    model=model,
    data_collator=data_collator,
    args=training_args,
    compute_metrics=compute_metrics,
    train_dataset=train_dataset,
    eval_dataset=eval_dataset,
    processing_class=processor,
)

6. Train the Model¶

Let's fine-tune the pre-trained model to predict phones from child speech!

In [24]:

trainer.train()

/.venv/lib/python3.11/site-packages/torch/utils/checkpoint.py:232: UserWarning: None of the inputs have requires_grad=True. Gradients will be None
  check_backward_validity(args)

[51000/51000 5:31:03, Epoch 20/20]

Step	Training Loss	Validation Loss	Cer
1000	2.737000	2.372736	0.804864
2000	1.511900	1.412020	0.437561
3000	1.329400	1.259690	0.411161
4000	1.217900	1.158037	0.405470
5000	1.177500	1.124078	0.385218
6000	1.097700	1.099532	0.380330
7000	1.069700	1.064977	0.374003
8000	1.022000	1.048235	0.368162
9000	0.995600	1.019231	0.369283
10000	1.001500	1.003653	0.362655
11000	0.951600	1.001552	0.361395
12000	0.947500	0.978547	0.357539
13000	0.891000	0.972168	0.356887
14000	0.902900	0.972453	0.354248
15000	0.865300	0.975841	0.353684
16000	0.842200	0.970502	0.351386
17000	0.845300	0.938102	0.350180
18000	0.807200	0.918144	0.346744
19000	0.815000	0.922606	0.348579
20000	0.822800	0.914684	0.344350
21000	0.772600	0.922295	0.344311
22000	0.789100	0.913038	0.344261
23000	0.772700	0.913228	0.340529
24000	0.731100	0.916719	0.341589
25000	0.730100	0.891391	0.340897
26000	0.719800	0.907422	0.339106
27000	0.710800	0.889043	0.338860
28000	0.714000	0.894029	0.337538
29000	0.681100	0.899334	0.336400
30000	0.680500	0.898606	0.336394
31000	0.669900	0.906611	0.335474
32000	0.653300	0.897395	0.336311
33000	0.652500	0.891148	0.333599
34000	0.630100	0.912029	0.335535
35000	0.640200	0.897706	0.332863
36000	0.606400	0.906883	0.334392
37000	0.602900	0.896555	0.332149
38000	0.609200	0.902984	0.332338
39000	0.580700	0.912111	0.332071
40000	0.567600	0.907288	0.331847
41000	0.563100	0.914157	0.331440
42000	0.582300	0.919074	0.330508
43000	0.557400	0.915264	0.331312
44000	0.545300	0.923404	0.330492
45000	0.533200	0.920011	0.330525
46000	0.518600	0.917619	0.328974
47000	0.519500	0.916954	0.329331
48000	0.510400	0.925846	0.328996
49000	0.501500	0.929667	0.329320
50000	0.517700	0.925447	0.328879
51000	0.520300	0.924588	0.328924

Out[24]:

TrainOutput(global_step=51000, training_loss=1.2747246808070762, metrics={'train_runtime': 19869.636, 'train_samples_per_second': 138.593, 'train_steps_per_second': 2.567, 'total_flos': 2.43363172711743e+20, 'train_loss': 1.2747246808070762, 'epoch': 20.0})

7. Evaluate and Test Inference¶

Now it's time to assess how well our final fine-tuned model performs. We evaluate on the validation set to compute the overall CER, then run inference on random samples to inspect individual predictions.

In [25]:

# Evaluate on validation set
eval_results = trainer.evaluate()

print("Evaluation Results:")
print(f"  CER: {eval_results['eval_cer']:.4f}")
print(f"  Loss: {eval_results['eval_loss']:.4f}")

[567/567 00:44]

Evaluation Results:
  CER: 0.3289
  Loss: 0.9254

Our model results in a CER of .33 on the validation set.

In [26]:

# Run inference on a few samples

model.eval()
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model.to(device)

# Get a few random samples
num_samples = 5
sample_indices = random.sample(range(len(eval_dataset)), num_samples)

print("Sample predictions:")
print("=" * 80)

for idx in sample_indices:
    sample = eval_dataset[idx]

    # Prepare input
    input_values = torch.tensor(sample["input_values"]).unsqueeze(0).to(device)

    # Get prediction
    with torch.no_grad():
        logits = model(input_values).logits

    pred_ids = torch.argmax(logits, dim=-1)

    # Decode
    pred_str = processor.decode(pred_ids[0])
    label_str = processor.decode(sample["labels"], group_tokens=False)

    print(f"\nSample {idx}:")
    print(f"  Ground truth: {label_str}")
    print(f"  Prediction:   {pred_str}")

    # Calculate CER for this sample
    cer = score_ipa_cer([label_str], [pred_str])
    print(f"  CER: {cer:.4f}")

print("\n" + "=" * 80)

Sample predictions:
================================================================================

Sample 10476:
  Ground truth: dei ɑksd dʌ dɑːktɚ tu hæv gɪv sʌm fɑiv sents
  Prediction:   ðei ækst ðə dɑktɚ tu hæv gɪv sʌm fɑiv sɛnts
  CER: 0.1591

Sample 1824:
  Ground truth: nʊs
  Prediction:   nəs
  CER: 0.3333

Sample 409:
  Ground truth: ʧɹɑieŋgɫ
  Prediction:   ʧɹɑieŋgɫ
  CER: 0.0000

Sample 12149:
  Ground truth: ɔw
  Prediction:   ɔə
  CER: 0.5000

Sample 4506:
  Ground truth: ʃʌvɫ
  Prediction:   ʃʌvʊɫ
  CER: 0.2500

================================================================================

Finally, we save the model, processor, and training configuration to disk.

In [27]:

# Save model + processor for reuse on another machine
save_dir = PROJECT_ROOT / "models" / "wav2vec2-phonetic-final"
save_dir.mkdir(parents=True, exist_ok=True)

# Trainer saves model + config + tokenizer config if provided
trainer.save_model(str(save_dir))

# Save processor artifacts (tokenizer + feature extractor)
processor.save_pretrained(str(save_dir))

processor.feature_extractor.save_pretrained(str(save_dir))

torch.save(training_args, save_dir / "training_args.pt")

Step 3: Make your submission¶

Since this is a code execution competition, we will submit our model weights and inference code rather than predictions. The platform runs your main.py in a container, which must load your model and output the file submission/submission.jsonl. See the code submission format webpage for more information.

The general steps to follow:

Develop inference code
Test your submission locally
Package submission
Make a smoke test submission
Once you have successfully debugged your submission, submit it for scoring on the full test set!

Develop Inference Code¶

We need to set up a repository with a main.py Python script which performs inference in the competition execution environment and writes our predictions to the required output file. During code execution, our submission will be unzipped and run in the cloud compute cluster. The container will run your main.py script.

Our code must write a JSON Lines (JSONL) file containing one prediction per utterance.

Each line must include:

utterance_id
phonetic_text: UTF-8, International Phonetic Alphabet (IPA) transcription of the utterance.

The submission should be written to ./submission/submission.jsonl relative to the working directory.

See more details in the code submission format webpage and in the example submission.

In our main.py, we load the fine-tuned Wav2Vec2 model and processor from the saved checkpoint directory, then run inference on all test utterances in batches. The script reads audio file paths from the test manifest, processes the audio through the model, decodes the model outputs to phonetic predictions, and writes the predicted phonetic transcriptions to the submission file in the required format. We batch utterances to improve GPU memory efficiency during processing.

See phonetic_submission/main.py for the details.

Test Submission Locally¶

You should first and foremost test your submission locally. This is a great way to work out any bugs and ensure that your model performs inference successfully. See the runtime repository's README for further instructions.

This repository provides a useful justfile command to run the trained model on a few sample files.

test-phonetic:
    uv run phonetic_submission/main.py models/wav2vec2-phonetic-final/ data-demo/phonetic/utterance_metadata.jsonl

Package Submission¶

Now we will package up our model and inference code into a zip file for predicting on the test set in the runtime container. This repository provides a justfile command to do this. The command creates a zip file combining the trained Wav2Vec2 model with /phonetic_submission/main.py.

pack-phonetic:
    rm -f phonetic_submission.zip && \
    (cd phonetic_submission && zip -r ../phonetic_submission.zip main.py) && \
    (cd models && zip -r ../phonetic_submission.zip wav2vec2-phonetic-final/)

Make a Smoke Test Submission¶

We provide a "smoke test" environment that replicates the test inference runtime but runs only on a small set of audio files. In the smoke test runtime, data/ contains 3,000 audio files from the training set.

Let's submit our submission.zip to a smoke test on the platform.

After hitting "Submit" you can see the job in the queue—it will progress from "Uploading" to "Pending" to "Starting" to "Running" to "Scoring":

Once your submission reaches "Completed", head on over to the "Submissions" tab to see your smoke test score.

Submit!¶

After you've made sure a smoke test submission runs without error, you're ready to submit the real deal! This fine-tuned Wav2Vec2 model results in a .3460 CER on the full test set.

We encourage you to also be mindful of the submission limit (3 per 7 days at most) and others' code jobs. Canceled jobs do not count against the submission limit.

If you want to share any of your findings or have questions, feel free to post on the community forum.

Good luck!

Improving Automatic Speech Recognition for Kids - A Reference Implementation for Word-level Transcription

2026-02-27T00:00:00-05:00

Welcome to the reference implementation tutorial notebook for the On Top of Pasketti: Children’s Speech Recognition Challenge - Word Track! If you are just getting started, we recommend reading the competition webpage first.

The goal of this tutorial is to:

Demonstrate how to load and explore the data.
Provide a basic framework for building a model.
Demonstrate how to package your work correctly for submission.

In this tutorial, we will adapt NVIDIA's Parakeet model, a pretrained ASR model, using NeMo. You can either expand on and improve the method in this notebook, or start with something completely different! Let's get started.

Background¶

Spoken language is a natural way for kids to learn, explore, and show what they know, yet today's Automatic Speech Recognition (ASR) technology hardly understands them. Most ASR systems are built on adult speech, and struggle with the pitch, rhythm, and evolving articulation of young learners.

The On Top of Pasketti: Children’s Speech Recognition Challenge assembles pre-existing and newly labeled datasets to advance speech models that truly work for children. Your goal in the Word Track is to develop models that accurately predict the words spoken by children in short audio clips.

If you'd like to rerun this notebook, the notebook file can be downloaded from the reference implementation repository. That repository also includes all code imported into the notebook.

Step 0: Import packages¶

First, create your environment. We use uv as the package manager in this benchmark repository. To run this notebook, you must be on a Linux machine with GPU access.

Create an environment: just create-environment
Activate the environment: source ./.venv/bin/activate
Install the requirements found in the TOML file into the environment: just requirements

Remember, the runtime repository's TOML file lists the packages that will be available for running inference using model submissions.

We'll be using NVIDIA's NeMo framework to build our model along with standard data science Python libraries to explore and prepare the data. Because this is a code execution challenge, we'll also be testing our solutions locally before packaging our model and inference code for submission. To help us with scoring, we've imported some utility functions from the competition's runtime repository.

In [1]:

# Standard library
import json
import os
from pathlib import Path

# Visualization
import matplotlib.pyplot as plt
from matplotlib import ticker

# Core ML & audio stack
import librosa
import lightning.pytorch as pl
import numpy as np
import pandas as pd
import torch

# ASR models & normalization
from nemo.collections.asr.models import ASRModel
from transformers.models.whisper.english_normalizer import EnglishTextNormalizer

# Training & experiment utilities
from loguru import logger
from nemo.utils import logging
from nemo.utils.exp_manager import exp_manager
from nemo.utils.trainer_utils import resolve_trainer_cfg
from omegaconf import OmegaConf, open_dict
from sklearn.model_selection import train_test_split

# Project utilities
from asr_benchmark.config import DATA_ROOT, PROJECT_ROOT
from asr_benchmark.nemo_adapter import (
    add_global_adapter_cfg,
    patch_transcribe_lhotse,
    update_model_cfg,
    update_model_config_to_support_adapter,
)
from asr_benchmark.score import english_spelling_normalizer, score_wer

fused_indices_to_multihot has reached end of life. Please migrate to a non-experimental function.
OneLogger: Setting error_handling_strategy to DISABLE_QUIETLY_AND_REPORT_METRIC_ERROR for rank (rank=0) with OneLogger disabled. To override: explicitly set error_handling_strategy parameter.
No exporters were provided. This means that no telemetry data will be collected.

In [2]:

torch.set_float32_matmul_precision("high")
# Set SAMPLE to use a smaller subset of the data for faster iteration during development. Set it to None to use the full dataset.
SAMPLE = None

Step 1: Load and explore the data¶

First, you'll likely want to set up your own repository for developing a solution. We recommend using Cookiecutter Data Science, which ensures an easy-to-navigate project structure.

Our local data structure after downloading all files to a raw data folder is:

childrens-speech-recognition-benchmark-pub/data/raw
├── drivendata
│   ├── audio_part_0.zip
│   ├── audio_part_1.zip
│   ├── audio_part_2.zip
│   └── train_word_transcripts.jsonl
└── talkbank
    ├── audio.zip
    └── train_word_transcripts.jsonl

After unzipping the files, you may want to consolidate the audio files into a single "audio" folder within the drivendata directory, where audio are split into multiple zip files.

rsync -a ./data/raw/drivendata/"audio 3"/ ./data/raw/drivendata/audio/

childrens-speech-recognition-benchmark-pub/data/raw
├── drivendata
│   ├── audio/
│   ├── audio_part_0.zip
│   ├── audio_part_1.zip
│   ├── audio_part_2.zip
│   └── train_word_transcripts.jsonl
└── talkbank
    ├── audio/
    ├── audio.zip
    └── train_word_transcripts.jsonl

For each of the two corpora, the file train_word_transcripts.jsonl contains the following fields:

utterance_id (str) - unique identifier for each utterance
child_id (str) - unique, anonymized identifier for the speaker
session_id (str) - unique identifier for the recording session; a single child_id may be associated with multiple session_ids
audio_path (str) - path to the corresponding .flac audio file relative to the /audio directory, following the pattern audio/{utterance_id}.flac
audio_duration_sec (float) - duration of the audio clip in seconds
age_bucket (str) - age range of the child at the time of recording ("3-4", "5-7", "8-11", "12+", or "unknown")
md5_hash (str) - MD5 checksum of the audio file, used for integrity verification
filesize_bytes (int) - size of the audio file in bytes
orthographic_text (str) - normalized orthographic transcription of the utterance

Each line in the JSONL manifest corresponds to a single utterance and references exactly one associated audio file. The orthographic_text field contains a manually created, minimally normalized orthographic transcription that serves as the training label.

Let's explore the metadata!¶

In [3]:

def read_transcripts(data_dir: Path) -> pd.DataFrame:
    """Read JSONL transcript file into a DataFrame and convert audio paths to absolute paths."""
    transcript_path = data_dir / "train_word_transcripts.jsonl"
    df = pd.read_json(transcript_path, lines=True)
    logger.info(f"Loaded {len(df)} utterance transcripts")

    df["audio_relpath"] = df["audio_path"]
    df["audio_path"] = df["audio_relpath"].map(lambda p: str(data_dir / p))

    return df

In [4]:

df_dd = read_transcripts(DATA_ROOT / "raw" / "drivendata")
df_tb = read_transcripts(DATA_ROOT / "raw" / "talkbank")
df = pd.concat([df_dd, df_tb], ignore_index=True)
df.drop(columns=["audio_path"]).head()

2026-02-26 04:58:21.910 | INFO     | __main__:read_transcripts:5 - Loaded 95572 utterance transcripts
2026-02-26 04:58:24.452 | INFO     | __main__:read_transcripts:5 - Loaded 255046 utterance transcripts

Out[4]:

	utterance_id	child_id	session_id	audio_duration_sec	age_bucket	md5_hash	filesize_bytes	orthographic_text	audio_relpath
0	U_00003c3ae1c35c6f	C_c74bfde2cca8d5da	S_7d821c3e4d3bc616	1.920	8-11	9214be45ba2928dd57384f3c7ee54236	30672	hm	audio/U_00003c3ae1c35c6f.flac
1	U_00003db24218ffe4	C_c74bfde2cca8d5da	S_e6103ab3a4538d71	12.737	8-11	fe761bb3d034530ef05163c7ad98ec3e	180942	yeah its pouring the water on the screen but t...	audio/U_00003db24218ffe4.flac
2	U_0001a0d0a3b4d816	C_4d0e1c16566d65a2	S_179057c3c3ccdecf	11.556	8-11	b05073e65a98368fccbe777b5ab35e02	208352	it got water and sunlight but the plant did di...	audio/U_0001a0d0a3b4d816.flac
3	U_00021d201a31d313	C_3b51c8b1d2c076d8	S_90720887e4430996	1.125	8-11	9ed95318724ae6a2d1ce95d6aa743f6b	27099	there is wires	audio/U_00021d201a31d313.flac
4	U_0003537f2bc1eb0b	C_b50216b3c70ca0a2	S_5b0bb48fadd7f802	1.125	8-11	f3142751c6a52e2c24a85a4544fe8a0f	18476	good	audio/U_0003537f2bc1eb0b.flac

In [5]:

df.utterance_id.nunique()

Out[5]:

In [6]:

df.child_id.nunique()

Out[6]:

In [7]:

round(df.audio_duration_sec.sum() / (60**2))

Out[7]:

There are over 350,000 utterances in the training dataset, across 3,344 children, totaling 344 hours of audio data. Next, let's take a look at the distribution of audio clip durations.

In [8]:

bins = list(range(0, 21)) + [np.inf]
labels = [str(i) for i in range(0, 20)] + ["20+"]

binned = pd.cut(df.audio_duration_sec, bins=bins, labels=labels, right=False)
counts = binned.value_counts().sort_index()

counts.plot(kind="bar")
plt.xlabel("Audio Duration (sec)")
plt.ylabel("Number of Audio Clips")
plt.title("Distribution of Audio Durations")
plt.xticks(rotation=90)
plt.show()

Even though the audio has been clipped to the utterance level, we have some outliers over 20 seconds.

In [9]:

# What is the longest clip (in minutes)?
round(df.audio_duration_sec.max() / 60)

Out[9]:

Most audio clips are very short (1-3 seconds). There is a tail of longer clips, up to an outlier 22 minute clip. Next, let's look at the distribution of utterances by child age.

In [10]:

df["age_bucket"] = pd.Categorical(
    df["age_bucket"], categories=["unknown", "3-4", "5-7", "8-11", "12+"], ordered=True
)

fig, ax = plt.subplots()
df["age_bucket"].value_counts(normalize=True, sort=False).plot.barh(ax=ax)
ax.set_title("Utterances by Age Group")
ax.set_xlabel("Percent of Utterances")
ax.set_ylabel("Age Group")
ax.xaxis.set_major_formatter(ticker.PercentFormatter(1.0))
ax.bar_label(ax.containers[0], fmt=lambda x: f"{x * 100:.0f}%")

plt.show()

About half of the utterances come from 8 to 11 year olds, with 13% coming from 3 to 4 year olds and 28% coming from 5 to 7 year olds.

Let’s explore the utterances!¶

We will listen to an example utterance and visualize the features.

In [11]:

def plot_waveform(audio_data: np.ndarray, sr: int) -> None:
    """Plot the waveform of an audio signal"""
    plt.figure(figsize=(10, 4))
    librosa.display.waveshow(audio_data, sr=sr)
    plt.title("Waveform")
    plt.xlabel("Time (s)")
    plt.ylabel("Amplitude")
    plt.show()

In [12]:

def plot_melspectrogram(audio_data: np.ndarray, sr: int) -> None:
    """Generate and plot the mel spectrogram of an audio signal"""
    S = librosa.feature.melspectrogram(y=audio_data, sr=sr)
    S_db = librosa.power_to_db(S, ref=np.max)

    plt.figure(figsize=(10, 4))
    librosa.display.specshow(S_db, sr=sr, x_axis="time", y_axis="mel")
    plt.colorbar(format="%+2.0f dB")
    plt.title("Mel Spectrogram")
    plt.xlabel("Time (s)")
    plt.ylabel("Mel Frequency")
    plt.show()

In [13]:

example_utt = "U_1c8757065e355c35"
df[df.utterance_id == example_utt][
    ["utterance_id", "audio_duration_sec", "orthographic_text"]
]

Out[13]:

	utterance_id	audio_duration_sec	orthographic_text
124170	U_1c8757065e355c35	0.516	yeah

This clip may sound understandable to us, but children’s speech often includes subtle pronunciation differences. For example, models must learn to handle syllable deletions (“elephant” → “efant”), consonant nasalization (“mom” → “bob”), and common speech sound disorders like tetism.

Now that we've heard the utterance, let's look at it from the model's perspective. We'll plot the raw waveform - the audio signal represented as amplitude over time:

In [ ]:

audio_data, sr = librosa.load(df[df.utterance_id == example_utt]["audio_path"].values[0])
plot_waveform(audio_data, sr)

Before training a machine learning model, the raw audio array with sampled amplitudes must be transformed. Transforming this raw signal into mel spectrograms enables the machine learning model to learn speech-relevant acoustic patterns more efficiently. The transformation reorganizes the signal into a structured time-by-frequency map that highlights speech patterns like vowels and consonants.

In [16]:

plot_melspectrogram(audio_data, sr)

When training machine learning models, this transformation is often handled under-the-hood by processors or feature extractors. For example, in our benchmark model implementation, we will fine tune NVIDIA’s pretrained parakeet-tdt-0.6b-v2 model using NeMo. NeMo's architecture includes a built-in preprocessing pipeline that converts raw audio waveforms into mel-spectrogram features, eliminating the need for manual feature extraction.

For more tips on generating predictive features from audio data, check out DrivenData's overview of open-source packages for using speech data in ML. That blog post also includes some helpful intro to working with audio data.

Step 2: Build the model¶

A straightforward modeling option is to start from a strong pretrained ASR model (trained mostly on adult speech), then fine-tune on our labeled child-speech training set. Fine-tuning the ASR model will help it understand the unique acoustic and linguistic patterns of children. Before choosing a pre-trained model, make sure it adheres to the competition rules.

In this tutorial, we adapt NVIDIA’s pretrained Parakeet TDT parakeet-tdt-0.6b-v2 model using NeMo. We chose this model because it is open source, ranks highly on the Hugging Face Open ASR leaderboard (at time of writing), and strikes a good balance between performance and model size. Rather than fine-tuning all parameters, we will freeze the base model and train only lightweight adapter weights. Adapters are small, trainable modules inserted into the frozen model’s layers that allow it to adapt to new data without modifying the original pretrained weights. Freezing the base model makes the fine-tuning process more efficient and helps avoid overfitting. NeMo makes this process easier by providing native adapter support, handling data pipelines (manifests, batching, feature extraction), and integrating experiment management (checkpointing, logging).

Key packages include:

nemo_toolkit for ASR model + data processing
lightning.pytorch for the training backend
omegaconf for config file management

1. Prepare Dataset & Create Splits¶

We need to provide NeMo with a "manifest" JSONL file that lists all of our samples and sample information. NeMo requires this manifest to be in a particular format, with specific column names and definitions. Each manifest line must be a JSON object with audio_filepath (path to audio file), duration (audio duration in seconds), and text (transcription text).

Below we will transform the competition data to NeMo's required manifest format.

We'll also filter out clips longer than 25 seconds, which strain computer memory. Competitors may want to further split these clips to avoid losing training data. Then we'll split the data 80/20 into train/validation.

In [17]:

MANIFEST_DIR = DATA_ROOT / "processed" / "ortho_dataset"
train_manifest_path = MANIFEST_DIR / "train_manifest.jsonl"
val_manifest_path = MANIFEST_DIR / "val_manifest.jsonl"
clip_max_duration_sec = 25.0

MANIFEST_DIR.mkdir(parents=True, exist_ok=True)

Remember, df has one row per utterance, and includes data hosted on both TalkBank and DrivenData.

In [18]:

df = df[["audio_path", "audio_duration_sec", "orthographic_text"]].rename(
    columns={
        "audio_path": "audio_filepath",
        "audio_duration_sec": "duration",
        "orthographic_text": "text",
    }
)

# Removing long clips to improve memory efficiency during training
over_max_duration_mask = df["duration"] > clip_max_duration_sec
logger.info(
    f"Removing {over_max_duration_mask.sum()} samples with audio duration "
    f"> {clip_max_duration_sec} seconds"
)
df = df[~over_max_duration_mask]

if SAMPLE:
    logger.info(f"Sampling {SAMPLE} utterances")
    df = df.sample(SAMPLE, random_state=10)

train_df, val_df = train_test_split(df, test_size=0.2, random_state=10)

train_df.to_json(train_manifest_path, orient="records", lines=True)
val_df.to_json(val_manifest_path, orient="records", lines=True)

2026-02-26 04:58:31.723 | INFO     | __main__:<module>:11 - Removing 3849 samples with audio duration > 25.0 seconds

2. Configuration¶

Let's define parameters for the adaptation.

To start, we copied an example .yaml file for ASR adaptation using NeMo from the NeMo repository to asr_benchmark/assets/asr_adaptation.yaml. This file gives us a reasonable default starting-point configuration.
We want to overtly set some parameters, like:
- pretrained_model (to specify parakeet-tdt-0.6b-v2)
- manifest_filepath (to point to our manifest locations)
- channel_selector (to ensure audio is converted to the expected mono format if it is stereo in raw form)
- max_steps (to control how many training steps we complete before we stop training)
For these parameters, we override the default settings in the .yaml using OmegaConf.merge.
While we do not implement hyperparameter tuning here, it is recommended for competitors.

In [ ]:

# ── Hardware-dependent settings ──────────────────────────────────────────────
# Adjust these to match your GPU memory and CPU cores.
DEVICES = 1
PRECISION = "bf16-mixed"
BATCH_SIZE = 32
NUM_WORKERS = 8

# ── Load NeMo adapter defaults ───────────────────────────────────────────────
yaml_path = PROJECT_ROOT / "asr_benchmark" / "assets" / "asr_adaptation.yaml"
cfg = OmegaConf.load(yaml_path)

# ── Training overrides ───────────────────────────────────────────────────────
overrides = OmegaConf.create(
    {
        "model": {
            "pretrained_model": "nvidia/parakeet-tdt-0.6b-v2",
            "adapter": {
                "adapter_name": "asr_children_orthographic",
                "adapter_module_name": "encoder",
                "linear": {"in_features": 1024},
            },
            "train_ds": {
                "manifest_filepath": str(train_manifest_path),
                "batch_size": BATCH_SIZE,
                "num_workers": NUM_WORKERS,
                "use_lhotse": False,
                "channel_selector": "average",
            },
            "validation_ds": {
                "manifest_filepath": str(val_manifest_path),
                "batch_size": BATCH_SIZE,
                "num_workers": NUM_WORKERS,
                "use_lhotse": False,
                "channel_selector": "average",
            },
            "optim": {
                "lr": 0.001,
                "weight_decay": 0.0,
            },
        },
        "trainer": {
            "devices": DEVICES,
            "precision": PRECISION,
            "strategy": "auto",
            "max_epochs": 1 if SAMPLE else None,
            "max_steps": -1 if SAMPLE else 5000,
            "val_check_interval": 1.0 if SAMPLE else 500,
            "enable_progress_bar": False,
        },
        "exp_manager": {
            "exp_dir": str(PROJECT_ROOT / "models" / "orthographic_benchmark_nemo"),
        },
    }
)

cfg = OmegaConf.merge(cfg, overrides)

3. Define Trainer¶

The Trainer orchestrates the training loop across devices, delegating tensor operations to PyTorch's backend. We initiate the trainer with the OmegaConf config object we made above, and then set up an experiment manager to handle logging, checkpoint saving, and saving artifacts to disk.

Note we use the cell magic %%capture below to hide long cell outputs for readability.

In [ ]:

%%capture
trainer = pl.Trainer(**resolve_trainer_cfg(cfg.trainer))
exp_log_dir = exp_manager(trainer, cfg.get("exp_manager", None))

4. Data and Model Setup¶

We load the pretrained model by fetching the model from our config first and then patching it for adapter support. Then, we load the model weights.

In [ ]:

%%capture
model_cfg = ASRModel.from_pretrained(cfg.model.pretrained_model, return_config=True)
update_model_config_to_support_adapter(model_cfg, cfg)
model = ASRModel.from_pretrained(
    cfg.model.pretrained_model,
    override_config_path=model_cfg,
    trainer=trainer,
)

We disable the CUDA graph decoder because it is incompatible with current PyTorch (2.10).

In [ ]:

%%capture
with open_dict(model.cfg):
    model.cfg.decoding.greedy.use_cuda_graph_decoder = False
model.change_decoding_strategy(model.cfg.decoding)

Next, we prepare our data by merging our parameter overrides (batch size, num workers, etc.) into the model's built-in data config. Note that while keys not present in the original config are dropped, whitelisted keys are always injected.

In [23]:

cfg.model.train_ds = update_model_cfg(model.cfg.train_ds, cfg.model.train_ds)
model.setup_training_data(cfg.model.train_ds)

cfg.model.validation_ds = update_model_cfg(
    model.cfg.validation_ds, cfg.model.validation_ds
)
model.setup_multiple_validation_data(cfg.model.validation_ds)

Injecting white listed key `num_workers` into config
Injecting white listed key `pin_memory` into config
Injecting white listed key `batch_size` into config
Injecting white listed key `use_lhotse` into config
Injecting white listed key `channel_selector` into config
Removing unavailable key from config : is_tarred
Removing unavailable key from config : shuffle_n
Removing unavailable key from config : bucketing_strategy
Removing unavailable key from config : bucketing_batch_size
[NeMo I 2026-02-26 04:58:53 nemo_logging:393] Dataset loaded with 277410 files totalling 244.57 hours
[NeMo I 2026-02-26 04:58:53 nemo_logging:393] 5 files were filtered totalling 0.00 hours
Injecting white listed key `num_workers` into config
Injecting white listed key `pin_memory` into config
Injecting white listed key `batch_size` into config
Injecting white listed key `use_lhotse` into config
Injecting white listed key `channel_selector` into config
[NeMo I 2026-02-26 04:58:55 nemo_logging:393] Dataset loaded with 69352 files totalling 61.52 hours
[NeMo I 2026-02-26 04:58:55 nemo_logging:393] 2 files were filtered totalling 0.00 hours

We also need to set the optimization function, which controls how the model’s weights are updated using gradients to minimize the loss and improve performance during training. We use AdamW with cosine annealing schedule and 10% warmup ratio as a starting point.

In [ ]:

%%capture
model.setup_optimization(cfg.model.optim)

In [ ]:

# Configure spec augmentation from config if available, otherwise disable it.
if "spec_augment" in cfg.model:
    model.spec_augmentation = model.from_config_dict(cfg.model.spec_augment)
else:
    model.spec_augmentation = None
    del model.cfg.spec_augment

5. Set up Adapter¶

An adapter is a small neural network module that learns to transform the pretrained model's internal representations specifically for child speech. Rather than retraining the entire model, we insert lightweight adapters into each layer of the encoder, the part of the model that processes audio features.

We add a linear adapter to every encoder layer, then freeze the base model and unfreeze only the adapter weights. This keeps training efficient—only ~0.1% of parameters are trainable, while (hopefully) allowing the adapter to learn the acoustic and linguistic patterns unique to children's voices.

In [ ]:

%%capture
with open_dict(cfg.model.adapter):
    adapter_name = cfg.model.adapter.pop("adapter_name")
    adapter_type = cfg.model.adapter.pop("adapter_type")
    adapter_module_name = cfg.model.adapter.pop("adapter_module_name", None)
    adapter_state_dict_name = cfg.model.adapter.pop("adapter_state_dict_name", None)

    adapter_type_cfg = cfg.model.adapter[adapter_type]

    if adapter_module_name is not None and ":" not in adapter_name:
        adapter_name = f"{adapter_module_name}:{adapter_name}"

    adapter_global_cfg = cfg.model.adapter.pop(model.adapter_global_cfg_key, None)
    if adapter_global_cfg is not None:
        add_global_adapter_cfg(model, adapter_global_cfg)

model.add_adapter(adapter_name, cfg=adapter_type_cfg)
assert model.is_adapter_available()

model.set_enabled_adapters(enabled=False)
model.set_enabled_adapters(adapter_name, enabled=True)

model.freeze()
model = model.train()
model.unfreeze_enabled_adapters()

In [ ]:

total_params = sum(p.numel() for p in model.parameters())
trainable_params = sum(p.numel() for p in model.parameters() if p.requires_grad)
print(f"Total parameters:     {total_params:,}")
print(
    f"Trainable parameters: {trainable_params:,} ({100 * trainable_params / total_params:.2f}%)"
)

Total parameters:     619,447,942
Trainable parameters: 1,622,016 (0.26%)

6. Train¶

Let's adapt the pretrained model to children's voices!

In [ ]:

%%capture
trainer.fit(model)

We've hidden the training cell outputs to avoid printing many lines of logs. Below are a few key snippets of the log output that we saw when training:

| Name              | Type                              | Params | Mode 
--------------------------------------------------------------------------------
0 | preprocessor      | AudioToMelSpectrogramPreprocessor | 0      | train
1 | encoder           | ConformerEncoderAdapter           | 610 M  | train
2 | decoder           | RNNTDecoder                       | 7.2 M  | train
3 | joint             | RNNTJoint                         | 1.7 M  | train
4 | loss              | RNNTLoss                          | 0      | train
5 | spec_augmentation | SpectrogramAugmentation           | 0      | train
6 | wer               | WER                               | 0      | train
--------------------------------------------------------------------------------
1.6 M     Trainable params
617 M     Non-trainable params
619 M     Total params
2,477.792 Total estimated model params size (MB)

Epoch 0, global step 500: 'val_wer' reached 0.20008 (best 0.20008), saving model to '/models/orthographic_benchmark_nemo/ASR-Adapter/2026-02-26_04-58-33/checkpoints/ASR-Adapter--val_wer=0.2001-epoch=0.ckpt' as top 5
...
Epoch 0, global step 5000: 'val_wer' reached 0.16359 (best 0.16359), saving model to '/models/orthographic_benchmark_nemo/ASR-Adapter/2026-02-26_04-58-33/checkpoints/ASR-Adapter--val_wer=0.1636-epoch=0.ckpt' as top 5
`Trainer.fit` stopped: `max_steps=5000` reached.

Because we configured max_steps to be 5,000, we stopped training before we completed the first epoch.

We need to save the final results. We save the adapter weights to a standalone file alongside the NeMo checkpoints.

In [28]:

if adapter_state_dict_name is not None:
    state_path = exp_log_dir if exp_log_dir is not None else os.getcwd()
    ckpt_path = os.path.join(state_path, "checkpoints")
    if os.path.exists(ckpt_path):
        state_path = ckpt_path
    state_path = os.path.join(state_path, adapter_state_dict_name)

    model.save_adapters(str(state_path))

7. Evaluation¶

Now it's time to assess how well our adapted model performs. We load the best trained checkpoint from disk and run inference across the entire validation set. The model transcribes each audio clip, and we compare these predictions against the reference transcriptions using Word Error Rate (WER). The WER calculation is copied exactly from the runtime repository.

In [ ]:

%%capture
# Load our model checkpoint
nemo_ckpts = sorted((exp_log_dir / "checkpoints").glob("*.nemo"))
if not nemo_ckpts:
    raise FileNotFoundError(f"No .nemo checkpoints found in {exp_log_dir}/checkpoints/")

best_ckpt = nemo_ckpts[-1]
print(f"Loading checkpoint: {best_ckpt}")
eval_model = ASRModel.restore_from(best_ckpt, map_location="cuda")

with open_dict(eval_model.cfg):
    eval_model.cfg.decoding.greedy.use_cuda_graph_decoder = False
eval_model.change_decoding_strategy(eval_model.cfg.decoding)

patch_transcribe_lhotse(eval_model)

In [ ]:

# Run inference
val_manifest_path = cfg.model.validation_ds.manifest_filepath
with open(val_manifest_path) as f:
    val_entries = [json.loads(line) for line in f]

audio_files = [e["audio_filepath"] for e in val_entries]
references = [e["text"] for e in val_entries]

print(f"Running inference on {len(audio_files)} validation utterances...")
raw = eval_model.transcribe(
    audio_files, batch_size=BATCH_SIZE, channel_selector="average", verbose=False
)
if isinstance(raw, tuple):
    raw = raw[0]

predictions = [h.text if hasattr(h, "text") else h for h in raw]

Running inference on 69354 validation utterances...

Before scoring, let's remove any example where the normalized label is an empty string.

In [31]:

normalizer = EnglishTextNormalizer(english_spelling_normalizer)
filtered = [(r, p) for r, p in zip(references, predictions) if normalizer(r) != ""]

references, predictions = zip(*filtered)

wer = score_wer(references, predictions)

print(f"Validation WER: {wer:.4f}")

print("\nSample predictions:")
for ref, pred in zip(references[:5], predictions[:5]):
    print(f"  REF:  {ref}")
    print(f"  PRED: {pred}")
    print()

Validation WER: 0.1546

Sample predictions:
  REF:  ar
  PRED: ar

  REF:  they were going on a walk to a picnic
  PRED: they were going on a walk to a picnic

  REF:  a deer is little
  PRED: a gear and a skin

  REF:  what's happening is when you when there's an op when there's a closed circuit the there is a the post but the wire's wrapped around it is carrying some washers and when it
  PRED: what's happening is when you when there's an o when there's a closed circuit the there is a the post with the wires wrapped around it is carrying some washers and when it

  REF:  blue
  PRED: blue

Our Validation WER after adapting the model is 0.15!

Step 3: Make your submission¶

Since this is a code execution competition, we will submit our model weights and code rather than predictions. See the code submission format webpage for more information.

The general steps to follow:

Develop inference code
Test your submission locally
Package submission
Make a smoke test submission
Once you have successfully debugged your submission, submit it for scoring on the full test set!

Develop Inference Code¶

Our code must write a JSON Lines (JSONL) file containing one prediction per utterance.

Each line must include:

utterance_id
orthographic_text: UTF-8, standard English transcription of the utterance The submission should be written to ./submission/submission.jsonl relative to the working directory.

See more details in the code submission format webpage and in the example submission.

In our main.py, we load the trained adapter checkpoint from disk, restore the NeMo model, and run inference on all test utterances in batches. The script reads audio file paths from the test manifest, transcribes them using the adapted model, and writes the predicted transcriptions to the submission file in the required format. We sort utterances by duration before inference (longest first) to improve GPU memory efficiency during batching.

See orthographic_submission/main.py in the benchmark repository for the details.

Test Submission Locally¶

The benchmark repository provides a useful justfile command to run the trained model on a few sample files.

# Run inference using data-demo/word/ to test orthographic submission
test-orthographic:
    uv run orthographic_submission/main.py models/orthographic_benchmark_nemo/ASR-Adapter-best.nemo data-demo/word/utterance_metadata.jsonl

Package Submission¶

Now we will package up our model and inference code into a zip file for predicting on the test set in the runtime container. The benchmark repository provides a justfile command to do this. The justfile finds the latest models weights, and then creates a zipfile combining those model weights with /orthographic_submissions/main.py.

pack-orthographic:
    #!/usr/bin/env bash
    set -euo pipefail
    latest=$(ls -td models/orthographic_benchmark_nemo/ASR-Adapter/*/checkpoints/ASR-Adapter.nemo | head -1)
    ln -sf "${latest#models/orthographic_benchmark_nemo/}" models/orthographic_benchmark_nemo/ASR-Adapter-best.nemo
    echo "Updated ASR-Adapter-best.nemo -> $latest"
    rm -f orthographic_submission.zip
    (cd orthographic_submission && zip -r ../orthographic_submission.zip main.py)
    (cd models/orthographic_benchmark_nemo && zip -r ../../orthographic_submission.zip ASR-Adapter-best.nemo)

Another packing command is available on the runtime repo's orthographic example.

Make a Smoke Test Submission¶

We provide a "smoke test" environment that replicates the test inference runtime but runs only on a small set of audio files. In the smoke test runtime, data/ contains 9,000 audio files from the training set.

Let's submit our submission.zip to a smoke test on the platform.

After hitting "Submit" you can see the job in the queue—it will progress from "Uploading" to "Pending" to "Starting" to "Running":

Once your submission reaches "Completed", head on over to the "Submissions" tab to see your smoke test score.

Submit!¶

After you've made sure a smoke test submission runs without error, you're ready to submit the real deal! This adaptation of parakeet-tdt-0.6b-v2 results in a .2370 WER on the full test set.

We encourage you to also be mindful of the submission limit (3 per 7 days at most) and others' code jobs. Canceled jobs do not count against the submission limit.

This is of course just one of the approaches you could take for this challenge. Some resources that may be helpful in getting started are:

The Hugging Face Automatic Speech Recognition unit of their Audio Course.
The Hugging Face Open ASR Leaderboard.
A Parakeet fine-tuning implementation from Deep-unlearning.
NeMo example ASR adaptation (which has a similar implementation as this blog post).

If you want to share any of your findings or have questions, feel free to post on the community forum.

Good luck!

5 Challenges of Creating Beautiful Data Pipelines

2026-02-18T00:00:00-05:00

Almost all projects we work on at DrivenData share a fundamental need: getting data from a raw form into a final, processed form. Sometimes, a project's core goal is to create a data pipeline that will run in perpetuity and continuously process live feeds of data. More often, our data pipelines are enabling processes that serve a larger purpose, such as preparing data for use in training a machine learning model.

While most pipelines start out seeming like a simple set of sequential steps, the reality is that pipeline code often becomes a deep source of technical debt. Over time, complexity accumulates as capabilities are added. For example, we may need a development version that runs on less data. We may need separate production and dev environments. We need audit trails and observability to make sure everything processed correctly. We need error capturing and recovery, and we need the right decisions about when to stop and when to keep processing. We need the code to use resources efficiently through mechanisms like parallelization and caching. We need to be smart about incrementally updating versus re-doing every task from scratch. We need to augment a pipeline step with a different dataset that has separate re-process conditions. Soon, the code is a complex tangle that is hard to reason about and maintain.

In this blog post, I’ll discuss some of the challenges of creating data pipelines for data science projects. We hope to bring clarity to what makes pipelining complex, simplify implementation decisions, and help data science teams make beautiful data pipelines from the start.

Why is something seemingly simple so hard?¶

Imagine you need to prepare news article data to train a machine learning model to classify whether an article discusses data, AI, or machine learning. That doesn’t sound too hard! A simple Python implementation could look like this:

from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression

def main(raw_data_directory: str = "path/to/raw_data"):
    # Step 1: Load Data From a Folder
    data = load_data(raw_data_directory)

    # Step 2: Preprocess Data
    data = preprocess_dataframe(data)

    # Step 3: Train/Test Split
    X_train, X_test, y_train, y_test = train_test_split(
        data["article_text"],
        data["label"],
        test_size=0.2,
        random_state=42
    )

    # Step 4: Define Vectorizer and Classifier
    text_clf = Pipeline([
        ("vectorizer", TfidfVectorizer()),
        ("classifier", LogisticRegression())
    ])

    # Step 5: Train Model
    text_clf.fit(X_train, y_train)

    # Step 6: Evaluate Model
    evaluate_model(text_clf, X_test, y_test)

However, the load_data and preprocess_dataframe functions turn out to never be quite as simple as they seem. In the next sections, we will walk through a few common pain points that show up as you move from the drawing board to a fleshed-out pipeline.

1. The cold start problem¶

A new pipeline starts as a blank slate, and figuring out where to begin isn’t always obvious. There are countless ways to organize files and structure steps. Figuring out the right order of operations can take some work.

In our data-topic detection example, you might wonder: Should we store an intermediary data file with article text and metadata when we preprocess it? What metadata should we even track? When should we deduplicate the article data? What are the upstream dependencies of our processing steps?

Each of these decisions starts to introduce complexity into what was a simple pipeline.

Suggestions

To start, sketch out the pipeline process visually to clarify dependencies and to ground alignment conversations about the pipeline in something concrete.
As you structure the pipeline, separate transformation logic from pipeline orchestration (the code that runs each step). In our data-topic detection model starter code, we were on the right track with a main function as the entry point to the pipeline that calls external helper functions (load_data(), preprocess_dataframe(), and evaluate_model()) in sequential steps.
Structure pipeline steps as a directed acyclic graph ("DAG") so that each step depends only on its inputs and produces outputs for downstream steps, and each step flows forward to the next without ever looping back on itself. Relatedly, raw data must be treated as immutable (that is, don't overwrite the original data files; instead, save the results of the pipeline as new output files).
Follow the software engineering principle of separation of concerns by breaking down complex logic and processing into modular functions that complete distinct objectives. Separation of concerns is a fundamental part of developing organized and clear pipelines. Modularity also makes each of the processing steps independently testable so you can have a test suite that ensures each granular step is doing what you expect. In our data-topic detection pipeline, we should isolate each task in our data preprocessing step and make separate functions that complete single objectives or operations:

def remove_duplicates(df: pd.DataFrame) -> pd.DataFrame:
    # TODO: implement deduplication
    return df

def drop_missing_labels(df: pd.DataFrame, label_col: str = "label") -> pd.DataFrame:
    # TODO: drop rows where label_col is null
    return df

def normalize_text(df: pd.DataFrame, text_col: str = "article_text") -> pd.DataFrame:
    # TODO: normalization steps
    return df

def encode_labels(df: pd.DataFrame, label_col: str = "label") -> pd.DataFrame:
    # TODO: map label values to numeric codes
    return df

Design the pipeline to be fully re-runnable end-to-end with minimal involvement. For example, make main() runnable from the command line by adding an entry point so that the pipeline runs when you execute python identify_data_topics.py.
When we start to need more options and arguments at the command line, we like Cyclopts for building command-line interfaces.
It may be helpful to use DrivenData's Cookiecutter Data Science package when creating a new repository to house your pipeline. Using a well-defined, standard project structure will benefit you and your team in the long run.

2. Adding basic pipeline features is a worthwhile investment¶

Stringing the steps together isn’t enough. As soon as you start running the pipeline, you'll realize you have no idea what is happening while you are running it. Add-ons such as logging, metadata tracking, and error handling will reduce debugging time, yield results you can be more confident in, and help identify bottlenecks in the process. These add-ons are generally worth it and justify budgeting the up-front implementation time.

In the data-topic detection pipeline, logs could show exactly where a run failed and how long each step took. Adding tools to preprocess news articles in parallel can substantially reduce processing time.

Suggestions

Add lightweight logging to capture per-task progress and errors, making debugging easier. We like Loguru, an intuitive Python logging library.
Speed up long processes by parallelizing tasks (running independent work concurrently) where possible. We often use tqdm, specifically the process_map and thread_map concurrent versions.
Despite best efforts, pipeline runs often unexpectedly fail because of data errors or edge cases. Disk caching computationally expensive steps can be a huge time saver in these cases because it enables you to resume progress where you left off just before the error arose in a task rather than restarting all computations. The @memoize decorator works well for caching the results of functions.
Track and maintain metadata such as file hashes (which are fixed-length identifiers generated from a file's content using hash functions), unique IDs, and pipeline version for reproducibility. In our data-topic detection example, you could generate and track hashes of the article text with a function to confirm data isn’t being corrupted run-to-run. (Notice we also have logging and parallelization at the task level in this example!):

import hashlib
import pandas as pd
from loguru import logger
from tqdm.contrib.concurrent import process_map

def hash_text(text: str) -> str:
    return hashlib.md5(text.encode("utf-8")).hexdigest()


def hash_article_texts(df: pd.DataFrame) -> pd.DataFrame:
    """Add a 'text_hash' column to the DataFrame containing MD5 hashes of the 'article_text' column."""
    logger.info(f"Hashing {len(df)} articles")
    df["text_hash"] = process_map(hash_text, df["article_text"].tolist())
    logger.success("Finished generating text hashes")
    return df

If the pipeline writes files, add an overwrite parameter that raises an error (or skips writing the file) if a file already exists and overwrite is set to False. This prevents accidentally overwriting existing files on subsequent pipeline runs.
Consider adding a debug mode parameter to your pipeline that, when activated, runs the pipeline on a small sample of data so you can test changes quickly and make sure the whole pipeline runs as expected.

3. Varied inputs lead to complicated code¶

Inevitably, project inputs expand beyond a single clean data source. Each CSV file, scraped HTML document, JSON object, and API call comes with its own quirks and processing needs. Real-world data is messy: missing labels, odd encodings, and inconsistent formats often only reveal themselves once the pipeline runs at scale. And it’s not just data; every parameter or configuration you add can shift assumptions and introduce new branches in the logic.

In our data-topic detection pipeline, expanding beyond a single CSV with news articles could mean that you need to add separate cleaning steps for other formats. You might also discover that some articles in the training data are missing a label indicating whether the article discusses data, AI, or machine learning, or are in another language, or have source text from multiple articles stuck together. You have to account for these edge cases in the pipeline.

Suggestions

Standardize and clean data early so later steps can assume consistent input. Preferably with a consistent, validated schema so malformed data fails early.
Factor out shared utility functions instead of duplicating code for each source.
Validate key assumptions early in the pipeline to catch anomalies, and fail loudly (e.g., raise errors) when those assumptions are not met. Raising errors early in the pipeline prevents wasted processing time for runs that fail. In our data-topic detection pipeline, if we need to ingest news articles from a website, our first step might be checking if the base URL for the website returns a "200" status code, indicating the website is active. If it doesn't return "200", we should skip processing because the site is down rather than issuing thousands of requests for articles that don't exist:

import requests

def ingest_articles(base_url: str) -> dict:
    """Ingest news articles from a website"""
    resp = requests.get(base_url)
    if not resp.status_code == 200:
        raise ValueError(f"Base URL {base_url} returned status code {resp.status_code}")

    # TODO: implement article ingestion

    return articles

4. Tech debt in a world of changing requirements¶

Project goals, requirements, and data formats change over time. This can create headaches when the existing pipeline needs to be refactored to account for processing conditions that didn’t initially exist. How will you handle backwards compatibility? Do you deprecate the "old way" of doing things entirely, or make the "old way" one of many processing options that can be triggered by calling certain parameters?

To avoid tech debt, it’s tempting to design a pipeline that is scoped out to be compatible with many processing scenarios from the start. But this, too, comes with downsides, as code will often be unnecessarily complex to preserve flexibility that you might not actually need.

In our example, maybe you first designed the pipeline to process a single labeled dataset. Later, you get access to additional datasets and decide you need to track metadata fields such as article author and publication date by source. Meeting these new needs means adjusting the data organization, adding new processing steps, and sometimes rethinking the overall pipeline structure.

Suggestions

Changing goals, requirements, and data sources is sometimes unavoidable. Before diving into the refactor, visually map out options for updated pipelines, consider the level of effort and tradeoffs required for each option, and align with teammates on a course of action. We often make pipeline visualizations on simple annotated slides:
Sprawling if/else logic across the pipelines for different data sources is a code smell. Your goal should be to standardize what you need to at the moment the data source is first initialized and then share as much unchanged pipeline code as possible.
Tech debt comes into play for what packages and libraries you choose as well. When multiple library options are available, try to pick the most well-maintained and widely used libraries up front. Look at how long it's been since the most recent commit to the package's source code repository, and how many stars the repository has on GitHub. Well-maintained and widely used packages are less likely to introduce new bugs down the line as new versions of other dependencies are released.
Be deliberate about where flexibility is truly valuable. Communicate with your team about the tradeoffs, and make simplifying assumptions where you can.

5. Beauty takes time¶

There’s always a balance to strike between getting something working quickly and building something clean and easily maintainable. Extra beautification efforts, such as clear file names, informative comments, and tidy dataset creation, pay off in the long run. You or a teammate will inevitably return to make modifications to the pipeline, and clear, well-documented code will make it easier to understand and less prone to having bugs accidentally introduced when updating it. But, if there is a quick deadline, it may be reasonable to take some shortcuts.

In our data-topic detection example, naming an intermediate dataset processed_labeled_articles instead of test_dataset_3 makes the pipeline clearer without slowing you down. Refactoring article processing code that works, but is duplicative or inefficient, requires more effort; whether the refactor is worth it is a judgment call.

Suggestions

The level of investment in the pipeline development should be proportional to how long one run takes and how often it will be run end-to-end. If the completed pipeline will be run just once, it may be enough to make a pipeline that produces accurate results relatively fast. In cases where the pipeline will be run often, it is usually worth it to make the pipeline accurate, efficient, and refactored for clarity and maintainability. It is a good idea to align with your team on how much polish is needed before diving into any big code refactors.
Pick descriptive names for files, functions, and outputs from the start.
Include descriptive docstrings in functions. We've found that LLM coding assistants (e.g., Copilot) or tools that set up docstring structures (e.g., autoDocstring extension for VS Code) make properly documenting functions quick and easy.
Adding type hints (explicit input/output types in function signatures) is helpful for clarity and aids in error detection, especially when used alongside type-checking tools like MyPy, ty, and Pyright.
Refactor small inconsistencies as you go instead of letting them pile up. Likewise, format code as you go using automatic formatting tools, like Black or Ruff.
Pipelines are often parameterized: the code can be configured to run in different ways by changing inputs (parameters) rather than modifying the code itself. For each run of your pipeline, you should record the configuration used, environment variables, timestamps, and pointers to log files or outputs for future reference. You can, of course, do this recording manually (e.g., copying the exact command you ran, including parameters, and other pertinent information into a tracking file or notebook). But we recommend capturing and persisting this information automatically using config files/ environment variables for setting pipeline parameters and automated run tracking systems for recording run metadata.

Embracing challenging pipelines with intentional effort¶

Making data pipelines is an aspect of most data science projects. And it often seems so simple from the outside: just move data from A to B. As soon as you start building, though, the hidden complexity emerges.

There are many tools you can use to facilitate pipelines rather than creating Python pipelines from scratch (e.g., Dagster). But these tools add learning curves and complexity. It often makes sense to start an implementation from a Python script and then invest in more tooling later if needed.

With some of the best practices we discuss, we can prevent some of the pain points and common pitfalls that domain-specific tools try to solve for and make it easy to have reliable, trustworthy pipelines with minimal dependencies and the right level of investment.

For further reading about general data science best practices, DrivenData's ebook The 10 Rules of Reliable Data Science is publicly available for download.