Youth Mental Health Narratives: Automated Abstraction

Apply machine learning techniques to automate the process of data abstraction from youth suicide narratives. #health

$45,000 in prizes
Completed nov 2024
588 joined

Problem description

The objective of Youth Mental Health Narratives: Automated Abstraction challenge is to apply machine learning techniques to automate the population of standard variables from the narrative text in the National Violent Death Reporting System (NVDRS). The results of this competition will help to streamline this time-intensive, manual abstraction process and perform better data quality control, ultimately enabling the researchers who rely on these standardized variables to more effectively study youth suicides on a national scale.

Not sure where to start? Follow the steps in the "How to compete" section of the homepage. The benchmark blog post is also a good place to start.

Data


The National Violent Death Reporting System (NVDRS) collects information about violent deaths including homicides, suicides, and deaths caused by law enforcement acting in the line of duty. The sample of NVDRS data provided in this competition includes de-identified information on completed suicides from 2017-2021 among youth aged 5-24.

Information in the NVDRS is primarily derived from law enforcement reports and coroner/medical examiner reports. Data abstractors generate summary narratives from these sources, and extract information into several standardized fields that are useful for researchers. See the About page for more details of the data creation process.

Although the data is de-identified, this sample contains confidential details. As such, participants are not allowed to share the data or use it for any purpose other than this competition. Participants cannot send the data to any 3rd-party API. For example, participants cannot submit data to OpenAI's ChatGPT, Google's Gemini, etc. For more details, see the External Data and Models section and the full competition rules.

Features

Participants in this challenge are provided with narratives written by state-level abstractors that summarize key information found in law enforcement reports (LE) and coroner/medical examiner reports (CME) for completed youth suicides. Each row represents one individual.

train_features.csv on the data download page includes three columns:

  • uid (str): Unique identifier for a single case
  • NarrativeLE (str): A summary of the information in the law enforcement (LE) report
  • NarrativeCME (str): A summary of the information in the coroner/medical examiner (CME) report

Narrative are meant to briefly summarize the incident, record information that cannot be captured elsewhere, and help to facilitate quality control checks. Abstractors follow specific instructions from the NVDRS coding manual to generate the narratives (page 24). The coding manual specifies what should be included, what shouldn't be included, and how to handle multiple sources.

The data for this competition includes detailed descriptions of youth suicides. These narratives can be upsetting and difficult to read. We encourage you to prioritize your own mental health when deciding whether to work on this challenge and while engaging with the data.

If you or someone you know is struggling with mental health, you can call or text the 988 Suicide & Crisis Lifeline for 24/7, free, and confidential support. The 988 Lifeline website has additional advice and links to specialized resources.

Labels

The labels are a set of binary and categorical variables for each case. Right now, these "standard variables" are manually extracted from the narratives. This process is time consuming and prone to human error.

The NVDRS Coding Manual has more detailed definitions of each standard variable. uid is a string value. All other columns are integers. InjuryLocationType and WeaponType1 are integer encodings of categorical strings. All other fields are binary variables with a value of 0 (absent) or 1 (present). There are 24 variables total.

Participants can find the labels under train_labels.csv on the data download page. In your submission.csv, the column order and data types must exactly match the below, also specified in the submission format. The indices must exactly match the test features.

Individual identifier:

  • uid (str): The individual's ID

Mental health history and current state:

  • DepressedMood (int, 0 or 1): The person was perceived to be depressed at the time
  • MentalIllnessTreatmentCurrnt (int, 0 or 1): Currently in treatment for a mental health or substance abuse problem
  • HistoryMentalIllnessTreatmnt (int, 0 or 1): History of ever being treated for a mental health or substance abuse problem
  • SuicideAttemptHistory (int, 0 or 1): History of attempting suicide previously
  • SuicideThoughtHistory (int, 0 or 1): History of suicidal thoughts or plans
  • SubstanceAbuseProblem (int, 0 or 1): The person struggled with a substance abuse problem. This combines AlcoholProblem and SubstanceAbuseOther from the coding manual
  • MentalHealthProblem (int, 0 or 1): The person had a mental health condition at the time

Specific mental health diagnoses: Variables indicating whether specific mental illness diagnoses applied. These are based on MentalHealthDiagnosis1/2 and MentalHealthDiagnosisOther in the coding manual (5.3.3). Only the most common diagnoses are included.

  • DiagnosisAnxiety (int, 0 or 1)
  • DiagnosisDepressionDysthymia (int, 0 or 1)
  • DiagnosisBipolar (int, 0 or 1)
  • DiagnosisAdhd (int, 0 or 1)

Contributing factors:

  • IntimatePartnerProblem (int, 0 or 1): Problems with a current or former intimate partner appear to have contributed
  • FamilyRelationship (int, 0 or 1): Relationship problems with a family member (other than an intimate partner) appear to have contributed
  • Argument (int, 0 or 1): An argument or conflict appears to have contributed
  • SchoolProblem (int, 0 or 1): Problems at or related to school appear to have contributed
  • RecentCriminalLegalProblem (int, 0 or 1): Criminal legal problem(s) appear to have contributed

Disclosure of intent:

  • SuicideNote (int, 0 or 1): The person left a suicide note
  • SuicideIntentDisclosed (int, 0 or 1): The person disclosed their thoughts and/or plans to die by suicide to someone else within the last month
  • DisclosedToIntimatePartner (int, 0 or 1): Intent was disclosed to a previous or current intimate partner
  • DisclosedToOtherFamilyMember (int, 0 or 1): Intent was disclosed to another family member
  • DisclosedToFriend (int, 0 or 1): Intent was disclosed to a friend

Incident details:

  • InjuryLocationType (int, categorical): The type of place where the suicide took place. This will be one of the options specified in the coding manual (4.3.3), encoded as an integer. The order of integer encodings is solely alphabetical. Note that multiple uncommon categories from the coding manual are combined into "Other". Possible values are:

    • 1: House, apartment
    • 2: Motor vehicle (excluding school bus and public transportation)
    • 3: Natural area (e.g., field, river, beaches, woods)
    • 4: Park, playground, public use area
    • 5: Street/road, sidewalk, alley
    • 6: Other
  • WeaponType1 (int, categorical): Type of weapon used. This will be one of the options specified in the coding manual (6.1), encoded as an integer. The order of integer encodings is solely alphabetical. Possible values are:

    • 1: Blunt instrument
    • 2: Drowning
    • 3: Fall
    • 4: Fire or burns
    • 5: Firearm
    • 6: Hanging, strangulation, suffocation
    • 7: Motor vehicle including buses, motorcycles
    • 8: Other transport vehicle, eg, trains, planes, boats
    • 9: Poisoning
    • 10: Sharp instrument
    • 11: Other (e.g. taser, electrocution, nail gun)
    • 12: Unknown

External data and models

Use of external data and models is allowed in this competition provided they are freely and publicly available to all participants under a permissive open source license.

However, participants may not upload competition data to any third party services or APIs that retain the data. For example, participants cannot submit data to ChatGPT or Gemini. Participants can use external models by loading open-source model weights into an environment that they can wipe the data from afterwards, such as their local machine or a cloud compute environment.

Performance metric


Performance is evaluated according to a multi-variable average F1-score. In the event of a tie, winners will be decided by inference time (shortest is best). F1 score is a harmonized mean of precision and recall. F1 score will be calculated separately for each variable in the submission, and then averaged across all target variables. Binary F1 score will be used for binary variables. For categorical variables, micro-averaged F1 score will be used. A higher score is better.

$$ \text{Multi-variable F1}= \frac{1}{N_{binary}+N_{categorical}} \cdot \left( \sum_{n=1}^{N_{binary}} F1_{binary}(n) + \sum_{n=1}^{N_{categorical}} F1_{micro}(n) \right) $$

  • Nbinary is the number of binary target variables. F1binary(n) is the binary F1 score on the nth binary variable.
  • Ncategorical is the number of categorical target variables. F1micro(n) is the micro-averaged F1 score on the nth categorical variable.

$$ F1_{binary} = 2\frac{Precision \cdot Recall}{Precision + Recall} = \frac{2 \cdot TP}{2 \cdot TP + FP + FN} $$

$$ F1_{micro} = \frac{2 \cdot \sum_{k=1}^{K} TP_k}{ \sum_{k=1}^{K} (2 \cdot TP_k + FN_k + FP_k)} $$

K is the total number of possible classes for a given multi-class variable. TPk is the number of true positives for class k.

F1 score can be calculated using sklearn.metrics.f1_score. A full scoring script is avaiable as part of the competition runtime repository.

Submission format


This is a code execution challenge! Rather than submitting your predicted labels, you'll package everything needed to do inference and submit that for containerized execution.

Your code submission must contain a main.py script that reads in the narratives for the test set, generates predicted target variable values for each case, and outputs a single submission.csv.

See details on the submission format and process here.

Good luck!


Good luck and enjoy this problem! If you have any questions you can always visit the competition forum!