climate

Water Supply Forecast Rodeo: Final Prize Stage

Water managers in the Western U.S. rely on accurate water supply forecasts to better operate facilities and mitigate drought. Help the Bureau of Reclamation improve seasonal water supply estimates in this probabilistic forecasting challenge! [Final Prize Stage] #climate

$400,000 in prizes
Completed jul 2024
34 joined

Water managers in the Western U.S. rely on accurate water supply forecasts to better operate facilities and mitigate drought. Help the Bureau of Reclamation improve seasonal water supply estimates in this probabilistic forecasting challenge! [Final Prize Stage]

Quick Facts

Participants
34
No. of Entries
82
Prize
$400,000
Winner

rasyidstat

1st Place

Navigation

Quick access

Explainability and Communication Bonus Track Submission Format

Water managers can make better decisions when forecasts are accompanied with additional information that supports and explains the forecast. In the Explainability and Communication Bonus Track, you will produce example forecast summaries to communicate your model's forecasts to water resource managers.

Explainability Bonus Track submissions are due Thursday, May 16, 2024 at 11:59 PM UTC.

Communicating forecasts

Water forecasts published by operational forecast agencies and water management agencies typically include supporting information that communicates and explains the forecasts to water resources managers, reservoir operators, water users, and the public. Such supporting information can include—but is not limited to—graphical, tabular, or narrative information explaining which inputs, relationships, or processes most strongly impacted a forecast for a given location and issue date.

In this track, you will be producing four example forecast summaries that are representative of the publications issued by operational forecast agencies. Each forecast summary will be a short document that succinctly summarizes and explains forecast conditions for a given forecast location and issue date.

Common questions that decisionmakers want to understand include:

  • Context:
    • How do forecast conditions compare to the range of observed historical conditions at that site?
    • How have the forecast and uncertainty bounds evolved over the forecast season?
  • Explanation:
    • Why is the forecast higher or lower than average? Which predictors or relationships between predictors most strongly influenced the forecast?
    • Why is the forecast uncertainty range especially high or low? Which predictors or relationships between predictors most strongly influenced the uncertainty?
    • Why did the forecast go up or down since the previous forecast? What change in predictors or relationships between predictors most strongly influenced this change?

To answer "explanation" questions like the ones posed above, part of the task will be to produce explainability metrics for your forecast. These should be some set of quantitative information that help readers understand why your model made this forecast.

Here are some examples of real forecast publications. Note that these are much more in-depth than the one- or two-page forecast summaries we are asking for in this challenge. These examples also may contain information about many sites, while your forecast summaries will be for a single site. You should use these as inspiration and not as a direct template for your forecast summaries.

Submissions to this track will be evaluated by a judging panel of experts, consisting of both operational water supply forecast issuers and water resource managers who use forecasts to make decisions.

Submission requirements

You are required to submit one ZIP archive containing the following five files:

  • Four forecast summaries:
    • forecast-owyhee-2023-03-15.pdf
    • forecast-owyhee-2023-05-15.pdf
    • forecast-pueblo-2023-03-15.pdf
    • forecast-pueblo-2023-05-15.pdf
  • A technical report:
    • report.pdf

See further below for detailed requirements for each file. Explainability Bonus Track submissions are due Thursday, May 16, 2024 at 11:59 PM UTC.

To make your submission, go to the "Explainability submissions" menu option in the left-hand navigation menu.

Forecast summaries

You are required to submit forecast summaries for the following two sites and two issue dates, for four forecast summaries in total:

  1. owyhee_r_bl_owyhee_dam, 2023-03-15
  2. owyhee_r_bl_owyhee_dam, 2023-05-15
  3. pueblo_reservoir_inflow, 2023-03-15
  4. pueblo_reservoir_inflow, 2023-05-15

Each forecast summary should be separate and independent and contain the following content:

  • Forecast communication output(s):
    • Presentation of the predicted water supply value (0.50 quantile) and uncertainty bounds (0.10 and 0.90 quantiles) from your model for the given site and issue date, along with any other contextual data.
    • Predictions should be generated from the version of your model from the Final Prize Stage, where the model weights are from the cross-validation iteration which uses 2023 as a test year.
    • Can be graphical, tabular, or other display(s) of your choice.
  • Explainability communication output(s):
    • Presentation of the explainability metrics for the forecast along with any other contextual data.
    • Can be graphical, tabular, or other display(s) of your choice.
  • Narrative analysis:
    • A brief written summary describing the forecast and explaining the forecast behavior. A paragraph in length is sufficient.

The set of forecast and explainability communication outputs are not required to be identical between individual forecast summaries, in the case that you find it helpful to communicate different information depending on the site or time of season. You should clearly justify any differences in your report.

All communication outputs across your forecast summaries must be generated automatically and reproducibly using code. The narrative analysis may be written by you, and the narrative analysis and overall summary document itself does not need to be generated with code.

Basic requirements

  • Up to 2 pages each, including figures
  • 8.5x11 inch paper size with minimum margin of 1 inch
  • Narrative text:
    • Minimum font size of 11
    • Minimum single-line spacing
  • Graphical display text (axis labels, table text, etc.):
    • Minimum font size of 10
  • PDF file format

Suggested format

There is no strict requirement for the format of the forecast summary. Creating an effective and easy-to-understand document is part of your task. Here is a suggested format outline:

  • Title
    • Issue date and site that this summary is for.
  • Forecast communication outputs
    • The predicted water supply and uncertainty values shown in one or more tables or graphs.
  • Explainability communication outputs
    • One or more tables or graphs presenting key explainability metrics.
  • Narrative analysis
    • A paragraph briefly summarizing the forecast and the key takeaways from the communication outputs

Report

You must submit a report about your explainability metrics and communication outputs. The report should include:

  • A brief description of what data or metrics are shown in each communication output.
  • A discussion of how each output supports communicating and explaining the forecasted conditions.
  • A detailed description of the data and methods used to calculate all explainability metrics, including an explanation of what the metrics represent and how they should be interpreted.
  • A discussion of the rationale for why each communication output is presented in that way.

To the extent that it improves the clarity of your report, you can briefly summarize your forecast model methodology as appropriate. However, you can consider this report to be an addendum to the final model report and refer to specific sections of your final model report for full details instead of duplicating an in-depth explanation of your model.

Basic requirements

  • Up to 5 pages of content, not including references
  • 8.5x11 inch paper size with minimum margin of 1 inch
  • Minimum font size of 11
  • Minimum single-line spacing
  • PDF file format

Required sections

  • Title
  • Abstract
    • Provide a concise summary of what is included in your model forecast.
  • Technical Approach
    • Forecast and uncertainty communication
      • Describe how you are presenting your model's forecast predictions (0.10, 0.50, and 0.90 quantile predictions) in your forecast summaries.
      • If showing any additional contextual data or metrics, what is included in the additional context?
      • Why have you displayed this information in this way?
    • Explainability metrics and communication
      • What are the methodology, techniques, and data used to calculate your explainability metrics?
      • What do the metrics represent? How should they be interpreted? Is there any physical explanation or intuition?
      • Why did you choose these metrics?
      • How well do the techniques generalize beyond the specific instance of the forecast, e.g., across years?
      • Could the explainability techniques be applied effectively across sites, times of season, or potentially other models or feature variables?
      • Describe how you are presenting the explainability metrics in your forecast summaries.
      • Why have you displayed this information in this way?
  • References

Code

You must write code that extends the forecast inference code for the model that you submitted cross-validation predictions for to the Final Prize Stage. This code should be able to take site_id and issue_date as input parameters and generate all communication outputs (table, graphical, or other displays of quantitative information) used in a forecast summary for that site and issue date. As documented in the "Forecast summaries" section, all communication outputs from your four forecast summaries must be automatically and reproducibly generated by this code.

Your code does not need to be submitted with your forecast summaries or reports by the submission deadline. Only finalists selected by the judging panel will be asked to submit their code.

Evaluation criteria

Usefulness (40%)

  • Is useful contextual information presented with the forecast and uncertainty bounds?
  • Do the chosen explainability metrics effectively help readers understand why the model produced the forecast and uncertainty bounds?
  • Does the narrative analysis effectively summarize and support the forecast and explainability communication outputs?

Clarity (20%)

  • Do the communication outputs clearly and effectively present forecast conditions, explainability metrics, and/or contextual information?
  • Are the communication outputs easy to interpret by a non-technical audience?

Rigor (20%)

  • Do the explainability metrics provide a meaningful and unbiased quantification of the influence of features and relationships between features on the forecast (0.50 quantile), uncertainty bounds (0.10 and 0.90 quantiles), and changes in forecast and uncertainty bounds between issue dates over the course of the forecast period?
  • Do the explainability techniques used generalize beyond the specific forecast instances, e.g., across years?

Innovation (20%)

  • Do the communication outputs incorporate a novel and innovative approach to displaying forecast information and/or explainability metrics?
  • Does the solution incorporate novel or innovative model explainability techniques or explainability metrics?
  • Do the explainability techniques used generalize across sites, times of season, modeling algorithms, or feature variables?

Good luck! If you have any questions, please reach out on the challenge forum.

On this page