education

Goodnight Moon, Hello Early Literacy Screening

Help educators provide effective early literacy intervention by scoring audio clips from literacy screeners. #education

$30,000 in prizes
Completed jan 2025
399 joined

Help educators provide effective early literacy intervention by scoring audio clips from literacy screeners.

Quick Facts

Participants
399
No. of Entries
727
Prize
$30,000
Winner

sheep

1st Place

Navigation

Code submission format

This is a code execution challenge! Rather than submitting your predicted labels, package everything needed to perform inference and submit that for containerized execution. The runtime repository contains the complete specification for the runtime. The README includes detailed instructions for testing your submission and requesting additional packages.

The general process for making a submission is:

  1. Create a properly formatted submission.zip
  2. Test your submission locally using Docker
  3. Make a smoke test submission
  4. Once you have succesfully debugged your submission with steps 2 and 3, submit for prediction on the full test set!

Tracking a submission:

  • "Code jobs" tab: This tracks the execution status of your submissions.
  • "Submissions" tab: This tracks the scores of your predictions. Errors that happen during scoring will also appear here.

What to submit

All submissions for inference must run on Python 3.12. No other languages or versions of Python are supported.

Your final submission should be a zip archive named with the extension .zip (for example, submission.zip). The root level of the submission.zip file must contain a main.py which performs inference on the test set audio clips and writes the predictions to a file named submission.csv in the same directory as main.py. You can see an example of this submission setup in the runtime repository.

Here's an example of the unpacked contents of a zipped submission:

submission root directory
├── assets/
│   └── ... (all assets needed for inference, e.g., model weights)
└── main.py         # Inference script

Be sure that when you unzip the submission, main.py exists in the folder where you unzip. This file must be present at the root level of the zip archive. There should be no folder that contains it.

During code execution, your submission will be unzipped and run in our cloud compute cluster. The script will have access to the following directory structure:

submission root directory
├── data
│   ├── submission_format.csv
│   ├── test_metadata.csv
│   ├── ...
│   ├── <all of the test audio clips as wav files>
│   ├── ...
│   ├── afncjc.wav
│   └── dtttxd.wav
├── main.py
└── <additional assets included in the submission archive>

submission.csv format

Your predictions must be in a CSV file with an index column called filename and a column called score. Your predictions must be floating point values between 0.0 and 1.0. The predicted values are probabilities the audio clip is correct, where 1.0 means 100% likelihood of being correct. Your submission.csv must match this format exactly.

Here is an example of what the first six rows of your submission.csv file might look like:

filename,score
dyqmme.wav,0.3745
abtqqc.wav,0.9507
btvwdx.wav,0.732
bnycms.wav,0.5987
adrulf.wav,0.156

You can find an example submission_format.csv on the data download page.

At the end of inference, your submission root directory should look like this:

submission root directory
├── data/
│   ├── submission_format.csv
│   ├── test_metadata.csv
│   └── <all of the test audio clips as wav files>
├── main.py
├── submission.csv
└── <additional assets included in the submission archive>

Testing your submission

Before you make a full submission, you should first test locally, and then submit a smoke test submission. Smoke tests have logging enabled and run on a small sample of audio clips. Make sure to cancel any jobs that will run longer than the allowed time. Smoke tests, cancelled jobs, and failed jobs won't count against your submission limit.

Testing your submission locally

You should first and foremost test your submission locally using the make test-submission command in the runtime container repository. This is a great way to work out any bugs locally and ensure that your model performs inference successfully. Be sure to set up your local data directory using the smoke test data from the data download page.

Smoke tests

For additional debugging, we provide a "smoke test" environment that replicates the test inference runtime but runs only on a small set of audio clips. In the smoke test runtime, data/ contains eight audio clips from the training set. These are the same files that are available in the smoke.tar.gz archive on the data download page. Smoke tests are not considered for prize evaluation and are intended to let you test your code for correctness. Logging is enabled for smoke tests and disabled for full submissions.

Submission checklist

  • Submission includes main.py in the root directory of the zip. There can be extra files with more code that is called (see assets folder in the transcription example).
  • Submission contains any model weights that need to be loaded. There will be no network access.
  • Script loads the data for inference from the data folder in the root directory. All audio files for inference are in the root level of the data/ folder. This folder is read-only.
  • Script writes submission.csv to the root directory when inference is finished. The format of this file must match the submission format exactly.
  • Use the versions of submission_format.csv and test_metadata.csv from the data folder provided to the container. Do not include these files with your submission, and do not read them from other locations. We need to be able to run your submission on other audio files by replacing these metadata files. See the problem description for details on the test set metadata.

Runtime

Your code is executed within a container that is defined in our runtime repository. The limits are as follows:

  • Your submission must be written in Python and run Python 3.12 using the packages defined in the runtime repository.
  • The submission must complete execution in 4 hours or less. We expect most submissions will complete much more quickly and computation time per participant will be monitored to prevent abuse. If you find yourself requiring more time than this limit allows, open a GitHub issue in the repository to let us know.
  • The container runtime has access to a single GPU. All of your code should run within the GPU environment in the container, even if actual computation happens on the CPU. (CPU environments are provided within the container for local debugging only.)
  • The container has access to 16 vCPUs and 110GB RAM.
  • The container has a single NVIDIA T4 GPU with 16GB VRAM.
  • The container will not have network access. All necessary files (code and model assets) must be included in your submission.
  • The container execution will not have root access to the filesystem.

Requesting package installations

Since the docker container will not have network access, all packages must be pre-installed. We are happy to add packages as long as they do not conflict and can build successfully. Packages should be available through conda for Python 3.12. To request an additional package be added to the docker image, follow the instructions in the runtime repository.

Note: The runtime environment currently supports PyTorch but not TensorFlow. We strongly encourage you to work within the PyTorch ecosystem. There will be a high bar for adding TensorFlow support.

Happy building! Once again, if you have any questions or issues you can always head on over the user forum!

On this page