Snowcast Showdown: Development Stage Hosted By Bureau of Reclamation

competition
complete
$500,000

Model report template

Each report will consist of up to 8 pages, excluding visualizations and figures, submitted in PDF format through the Model Report Submission. Incorporating graphics is encouraged and does not count against the page limit. Winners will also be responsible for submitting any accompanying code (e.g., notebooks used to visualize data).

The template below provides guiding questions as you put together your report. You do not necessarily need to answer every question, though your report should sufficiently address each of the topics described below. These are the topics that judges will consider when evaluating reports.

Template

Summary

Provide a concise summary of your solution, highlighting the key features and algorithm(s) underlying your solution, previous work that your solution builds on, and any novel or unique aspects of your solution.

Model interpretability

Algorithm selection

  • Describe the model(s) you used in your solution and why you selected them.
  • What other models did you consider and/or test? Why did you choose not to use them?
  • Does your solution incorporate any modifications or additions to a “standard” or “out of the box” model?
  • Is there anything novel about how the selected algorithm was applied? If yes, please describe.

Data sources

  • What (raw) data sources does your solution use?
  • Why did you select these particular data sources?
  • Were other data sets considered and rejected? If so, please describe.

Feature engineering

  • What preprocessing steps did you take to develop features from raw data? This includes steps taken to clean and prepare raw data (e.g. reformatting raw data, filling missing values, removing outlier values, encoding categorial values, etc.), as well as steps taken to combine or transform raw or cleaned/processed data into features.
  • Are there preprocessing steps you tested but did not end up using? If so, what were they and why were they ultimately not used?
  • What features did you select for use in your model? What criteria or methods did you use to select these features?
  • Are there any learnings, key insights, and/or visualizations worth sharing from your exploratory data analysis?

Explainability

  • What are the most important features? Do the features it considers important make scientific sense? What insights does this provide?
  • Explain how your model makes its SWE estimates in a way that would be useful to a scientist. Please provide associated visualizations and metrics to support your explanation.

Model robustness

Training, validation, and testing

  • How were available observations divided for training, validation, and testing (e.g. random sampling, stratified sampling by metadata like location, elevation, snow conditions)?
  • What approach was used to validate your model (e.g. holdout set performance, k-fold cross-validation)?
  • What metric(s) were used to evaluate model performance against validation and testing datasets?
  • Summarize validation and testing results. Tables and/or visualizations summarizing model performance are strongly encouraged.

Performance across conditions

  • How does your model performance vary by the following variables? Please provide associated visualizations and metrics to justify your explanations.
    • SWE levels? (e.g. Is it more accurate for higher values? Are errors randomly distributed?)
    • Time? (e.g. how does performance differ between peak snow and melt season)
    • Location?
    • Conditions? (e.g. elevations, snow climate zones (coastal, intermountain, continental), seasonal climate conditions (dry, normal, wet years))
  • What robustness checks were part of your training and validation process (e.g. checking for disparate error rates)?
  • Describe under which conditions your model is the most accurate and under which conditions your model is the least accurate.

Edge cases

  • How robust is your model to missing data and how is missing data handled (e.g. imputation)?
  • Are there particular failure modes for your model (e.g. does high cloud cover get interpreted as snow)?

Considerations for use

Machine specifications

  • Please describe the machine specs that were used to train your model (e.g. number of GPUs, GPU memory, CPU memory, RAM).
  • What simplifications could be made to run your solution faster without sacrificing significant accuracy?

Recommendations

  • If you were to continue working on this problem for the next year, what methods or techniques might you try in order to build on your work so far? Are there other fields or features you felt would have been very helpful to have?
  • Feel free to provide any additional notes or recommendations regarding your approach.