health

PREPARE Challenge - Phase 2: Report Arena

Advance algorithms and analytic approaches for early prediction of Alzheimer's disease and related dementias, with an emphasis on explainability of predictions. [Report Arena] #health

$250,000 in prizes
Completed jan 2025
69 joined

Advance algorithms and analytic approaches for early prediction of Alzheimer's disease and related dementias, with an emphasis on explainability of predictions. [Report Arena]

Quick Facts

Participants
69
No. of Entries
0
Prize
$250,000
Winner

sheep and cecilia

1st Place Team

Navigation

Quick access

Problem description

The aim of the PREPARE challenge is to advance accurate, innovative, and representative early prediction of Alzheimer's Disease and Alzheimer's Disease Related Dementias (AD/ADRD). Phase 2 is focused on advancing algorithms and analytic approaches for early prediction of AD/ADRD, with an emphasis on explainability of predictions. In an initial stage, participants developed models and submitted predictions in two Model Arenas (Social Determinants Track + Acoustic Track).

In this second stage, the top 15 leaderboard finalists from the Model Arenas submit reports describing their approach. To assist with the reports, we are releasing test set labels for both Model Arenas as well as additional metadata for the Acoustic Track. We encourage you to use this data to conduct additional bias and performance analyses.

Evaluation

Model Performance and Methodology (40%) How sound is the solution's methodology, including feature selection, validation techniques, and strategies to prevent data leakage and overfitting? How well does the model perform overall?

Insights and Innovation (20%) What is the depth and relevance of insights gained from the modeling process (e.g., based on how predictions are made, feature importance)? To what extent does the approach demonstrate innovation or apply novel techniques?

Bias Exploration and Mitigation (20%) How thoroughly does the solution explore and address potential biases in the data and model (e.g., systematic measurement error in important features)? How effective are the proposed strategies for mitigating identified biases?

Generalizability (10%) How likely is the model to produce valid predictions on new data and in applied settings?

Clarity & Communication (10%) How clearly and effectively are the findings communicated, both in text and visuals?

Model report content and format

The purpose of the model report is to describe your model methodology and present evidence that it advances accurate, innovative, and representative early prediction of AD/ADRD.

You are not allowed to change the model that you submitted to a model arena. See the home page for details.

Required report content:

  • Full model methodology (preprocessing, architecture, features, training)
  • Performance metrics and evaluation approach, including generalizability
  • Bias assessment of the data and model
  • Model explainability/interpretability (i.e., how the model as a whole generates predictions)
  • Theoretical or methodological insights about early, inclusive prediction of AD/ADRD

Key questions to address in the model report:

  • In what ways does your model advance accurate, innovative, and representative early prediction of AD/ADRD?
  • What clinical or research contexts could your model be feasibly and responsibly used in? Is there evidence your model would perform well for populations disproportionately affected by AD/ADRD?
  • How well does your model perform in general and across different subpopulations, key feature values, and outcome values? How well would your model perform on new data, and in an applied context?
  • How does your model make predictions, and are there insights about AD/ADRD prediction or treatment that can be drawn from it (e.g., a new kind of acoustic biomarker; a new protective social/environmental factor)?
  • How does your model enable early prediction of cognitive status and/or decline? How early can your model make predictions?
  • How could you further demonstrate the use and generalizability of your approaches, including on other datasets, in and beyond Phase 3?

Technical requirements

  • Format: PDF
  • Length: Maximum 7 pages, including figures and tables but not references
  • Page Size: 8.5x11" with 1" margins
  • Font: Minimum 11pt for main text, 10pt font for figures and tables

Report structure

1. Abstract

  • Brief overview of approach and description of model performance
  • Brief summary of how the solution advances accurate, innovative, and representative early prediction of AD/ADRD

2. Methodology

  • Model architecture, key parameters, and training process
  • Feature engineering and data preprocessing steps
  • Justification for modeling choices
  • Generalizability and robustness analysis
  • Expected limitations on generalizability

3. Performance Analysis

  • Overall model performance estimate and method
  • Justification for choice of metrics
  • Performance on primary outcome and decline (if applicable)
  • Error analysis

4. Bias and Ethics Analysis

  • Dataset bias analysis
  • Model bias analysis (see example here)
  • Subpopulation error analysis
  • Bias mitigation strategies implemented
  • Potential negative impacts and mitigation strategies
  • Privacy considerations

5. Future Directions

  • Directions for addressing model limitations or improving performance in and beyond Phase 3
  • Opportunities to demonstrate your approach on additional datasets, or in new contexts in and beyond Phase 3

6. References

  • Citations for any referenced packages, books, or papers

Additional tips and resources

Strong model reports will:

  • Incorporate visualizations, charts, and tables where they can effectively communicate findings
  • Prioritize clarity in your writing and visualizations
  • Consider both the immediate implications and potential long-term impacts of your model when considering bias and ethics

To learn more about model fairness and bias, see:

Updates to your model

You are not allowed to make substantive updates to the model that you previously submitted to a model arena.

Additional data

Test data labels and additional metadata are being released for the Report Arena. We encourage you to use this data to conduct additional bias and performance analyses.

Overview of the new data files provided:

  • Acoustic Track
    • Test set labels (acoustic_test_labels.csv)
    • Additional metadata (acoustic_additional_metadata.csv)
  • Social Determinant Track
    • Test set labels (sdoh_test_labels.csv)

Information about how to access all additional data files can be found on the data download page.

Acoustic Track: Additional metadata

acoustic_additional_metadata.csv provides additional information about each individual in the dataset, and includes both the train and test split. It includes the following columns:

  • uid (str): A unique identifier for the individual. Each row is a unique individual.
  • diagnosis (str): The specific diagnosis that the patient received, with values "Control", "ProbableAD", "MCI", "AD", and "PPA" (primary progressive aphasia). Note that this differs from the model arena labels because in the labels, probable AD, AD, and PPA are grouped into one advanced diagnosis category.
  • race (str): Patient race, with values "african american", "asian", "white", and "other".
  • language (str): Audio language, with values "english", "mandarin", and "spanish".
  • handedness (str): Patient handedness, with values "right", "left", and "ambidextrous".
  • corpus (str): Name or abbreviated name of the corpus that the patient record came from, with values "baycrest", "delaware", "depaul", "hopkins", "ivanova", "lu", "pitt", "vas", "wls", and "ye".
  • education (str): Patient education information, with values "0", "10", "11", "12", "13", "14", "15", "16", "17", "18", "19", "2", "20", "3", "4", "6", "7", "8", "9", "associates'","bachelor","college","doctorate","high school","master's","professional", and "technical". Note that some education values are provided as integers representing years of education, and others are provided as strings. This is because different corpuses recorded education values differently.

Acoustic Track: Test labels

The test labels (the cognitive status of each individual in the test set) are provided in the same format as in the Model Arena. For more details, see the Model Arena Problem Description.

acoustic_test_labels.csv includes the following columns:

  • uid (str): Unique identifier for the individual. Each row is one individual.
  • diagnosis_control (float, 0.0 or 1.0): Whether the individual is a healthy control.
  • diagnosis_mci (float, 0.0 or 1.0): Whether the individual was diagnosed with mild cognitive impairment.
  • diagnosis_adrd (float, 0.0 or 1.0): Whether the individual was diagnosed with advanced decline (primary progressive aphasia, probable AD, or AD).

Social Determinants Track: Test labels

The test labels (the cognitive status of each individual in the test set) are provided in the same format as in the Model Arena. For more details, see the Model Arena Problem Description.

sdoh_test_labels.csv includes the following columns. Each row is a unique combination of uid and year.

Special recognition prizes

Top 15 finalists on the Model Arena leaderboard who (1) submit a model report and (2) do not receive an Overall Prize.

Up to five total Special Recognition prizes of $10,000 each will be selected by Judges recognize excellence in addressing core challenge goals. Possible categories include: (1) most innovative methodology, (2) best early prediction, (3) best consideration of bias, (4) potential to generalize to populations disproportionately impacted by AD/ADRD, (5) for the Social Determinants Track only, best approach to predicting decline between 2016 and 2021.

Explainability bonus prizes

Top 15 finalists on the Model Arena leaderboard are eligible to submit submissions for the Explainability Bonus Track, where up to 4 prizes of $10,000 each will be offered. See the Explainability bonus track page for more details.

Good luck

Good luck! If you have any questions, you can always visit the competition forum!

On this page