Water Supply Forecast Rodeo: Forecast Stage

[submissions closed] Water managers in the Western U.S. rely on accurate water supply forecasts to better operate facilities and mitigate drought. Help the Bureau of Reclamation improve seasonal water supply estimates in this probabilistic forecasting challenge! [Forecast Stage] #climate

$50,000 in prizes
3 months left
47 joined

Overview

Accurate seasonal water supply forecasts are crucial for effective water resources management in the Western United States. This region faces dry conditions and high demand for water, and these forecasts are essential for making informed decisions. They guide everything from water supply management and flood control to hydropower generation and environmental objectives.

Yet, hydrological modeling is a complex task that depends on natural processes marked by inherent uncertainties, such as antecedent streamflow, snowpack accumulation, soil moisture dynamics, and rainfall patterns. To maximize the utility of these forecasts, it's essential to provide not just accurate predictions, but also comprehensive ranges of values that effectively convey the inherent uncertainties in the predictions.

Task

The goal of this challenge is to develop probabilistic forecast models that predict seasonal water supply at the 0.10, 0.50, and 0.90 quantiles. The seasonal water supply refers to the naturalized cumulative streamflow volume over specified seasonal periods. The challenge is occuring over multiple stages: the Hindcast Stage evaluated models on historical data simulating real-time forecasting, the Forecast Stage will run in real-time during the 2024 season, and a final prize stage will ask solvers to submit additional materials for the overall and bonus prizes, like model reports and cross-validation predictions, which will be judged by a panel of technical experts.

By improving the accuracy, explainability, and uncertainty characterization of seasonal streamflow forecasts, water resources managers will be better equipped to operate facilities during high flows, mitigate impacts of drought, improve hydropower generation, and meet environmental targets.

Challenge overview

Timeline Overview

The challenge is occurring over multiple stages, each with its own prizes:

  1. Hindcast Stage: Models were evaluated on historical ground truth data to simulate forecasts that would have been made in the past.
  2. Forecast Stage: Models are be run four times each month from January through July 2024 to issue real-time forecasts for the 2024 season.
  3. Final Prize Stage: In a final stage, solvers will submit updated models and additional materials to compete for Overall Prizes and for the Explainability and Communication Bonus Track prizes.

Prize Overview

Category Prize Pool
Overall $325,000
Hindcast Stage $50,000
Forecast Stage $50,000
Explainability and Communication Bonus Track $75,000
Total $500,000

Forecast Stage Details

You are currently in the Forecast Stage for the challenge. In the Forecast Stage, you will submit models to forecast the seasonal water supply for the 2024 season. Models will be run automatically at scheduled times through July 2024.

Having your code submission successfully run during the Forecast evaluation period is required to be eligible for the Final Prize Stage (Overall prizes and Explainability bonus prizes).

Forecast Stage Key Dates

Forecast Stage opens, additional training data available December 22, 2023
    First scheduled trial issue date January 1, 2024
    Data and runtime change request deadline January 5, 2024
    Second scheduled trial issue date January 8, 2024
Deadline for code submission January 11, 2024 at 11:59 pm UTC
Evaluation period begins January 15, 2024
Forecasts evaluated against ground truth August 14, 2024

Prizes Breakdown

Place Prize Amount
1st $100,000
2nd $75,000
3rd $50,000
4th $30,000
5th $20,000
Regional and Lead Time Bonus Prizes $50,000

Overall Prizes

Overall prizes will be awarded based on an evaluation by a panel of technical experts. The evaluation will consider the overall performance across all stages of the challenge as well as a final model report detailing solutions.

Bonus prizes will also be awarded for best performance in forecast subcategories, including regional and long lead time. Additional details about the subcategories will be shared later in the challenge.

You must participate in both Hindcast and Forecast Stages by submitting successfully executed code to be eligible for overall prizes.

Place Prize Amount
1st $25,000
2nd $15,000
3rd $10,000

Hindcast Stage Prizes

October–December 2023

Test your model against historical ground truth data! Train your models and submit code. Your code will be executed to perform inference on a held-out test set. Prizes will be awarded based on a combination of your leaderboard performance and an evaluation of your model report by a panel of judges.

Place Prize Amount
1st $25,000
2nd $15,000
3rd $10,000

Forecast Stage Prizes

December 2023–July 2024

Find out how your model performs with live forecasts for the 2024 season! DrivenData will execute your model at a regular cadence from January through July 2024. After July, forecasts will be evaluated against the true water supply measurements and top leaderboard performers will win Forecast Stage prizes.

You must participate in the Hindcast Stage by submitting successfully executed code in order to be registered for the Forecast Stage.

Place Prize Amount
1st $25,000
2nd $20,000
3rd $15,000
4th $10,000
5th $5,000

Forecast Explainability and Communication Prizes

Augment your model to produce forecast explanation outputs alongside its normal predictions. Understanding how predictors drive forecasts and changes in forecasts is important to operational decision makers. Submissions will be judged by a panel of technical experts to select the winners.

You must participate in both Hindcast and Forecast Stages and submit a final model report to be eligible.

How to compete

  1. Only teams that made successful code submissions in the Hindcast Stage Evaluation Arena are eligible to participate in the Forecast Stage. Eligible teams are automatically registered. If you need to make changes to your team, please contact info@drivendata.org.
  2. Review the problem description page and read the submission requirements on the code submission format page.
  3. Update your model with the newly released training data on the data download page.
  4. Bundle your trained model and prediction code in a ZIP archive for submission.
  5. Click "Submissions" in the sidebar, and then "Make new submission" during the open submission period. You can make as many submissions as you like (subject to a daily limit). Your latest submission at the deadline will be locked in as your official submission.
  6. Keep an eye out for email notifications of failed jobs during the evaluation period. Submit a fix if you have a failed job.

The challenge rules are in place to promote fair competition and useful solutions. If you are ever unsure whether your solution meets the rules, ask the challenge organizers in the competition forum or send an email to info@drivendata.org.

Note on prize eligibility: The term Competition Sponsor in the rules includes the Bureau of Reclamation as well as all federal employees acting within the scope of their employment and federally-funded researchers acting within the scope of their funding. These parties are not eligible to win a prize in this challenge.


Sponsors

This challenge is sponsored by the Bureau of Reclamation


With support from NASA


And with collaborators from the USDA Natural Resources Conservation Service and the U.S. Army Corps of Engineers