health

PREPARE Challenge - Phase 2: Model Arena (Acoustic Track)

Advance algorithms and analytic approaches for early prediction of Alzheimer's disease and related dementias based on voice recordings, with an emphasis on explainability of predictions. [Model Arena - Acoustic Track] #health

$130,000 in prizes
Completed dec 2024
401 joined

Advance algorithms and analytic approaches for early prediction of Alzheimer's disease and related dementias based on voice recordings, with an emphasis on explainability of predictions. [Model Arena - Acoustic Track]

Quick Facts

Participants
401
No. of Entries
1,366
Prize
$130,000
Winner

NiuBi Team

1st Place Team

Navigation

Quick access

Community Code

Early Detection of MCI Using Acoustic and Linguistic Features with XGBoost: Model Insights, Metrics, and Explainability

⸱
4mo 2w ago
⸱
80 downloads

Download this code
Code requirements (Download)
Early Detection of MCI Using Acoustic and Linguistic-121724.txt:
Early Detection of MCI Using Acoustic and Linguistic Features with XGBoost: Model Insights, Metrics, and Explainability
Melinda B. Chu, M.D. M.B.A.
melinda.chu@eternalmemo.ai

Note: Code requirements at the end of document:
Purpose of the Model
The model aims to predict Mild Cognitive Impairment (MCI) status using audio recordings and their transcriptions. MCI detection is essential for early intervention, as speech patterns often show subtle changes indicative of cognitive decline.
Tools & Methods
We used the following tools and approaches to build the model:
1.	Feature Extraction: Linguistic (e.g., pronoun usage, lexical diversity), Acoustic (e.g., mean pitch, RMS energy), and Prosodic features (e.g., pause durations).
2.	Model Selection: A range of classifiers including XGBoost, LightGBM, and Voting Ensembles were evaluated.
3.	Metrics Evaluated:
○	Logarithmic Loss (primary competition metric)
○	Accuracy, Precision, Recall, F1-Score, and ROC-AUC (for broader performance evaluation).
Rationale and Achievements:
●	Preserved raw diagnostic features (e.g., pauses, pitch irregularities) by avoiding aggressive preprocessing.
●	Demonstrated strong model performance across key metrics.
Community Code Criteria
1. Getting Started and EDA
●	Described the dataset: 1646 recordings converted to .wav format for OpenSMILE analysis.
●	Highlighted data distribution: Balanced vs. imbalanced labels (MCI vs. Healthy).
●	Feature exploration included statistical analysis, box plots, and coefficient of variation to identify variability.
2. Bias Measurement and Mitigation
●	Applied SMOTE to address any class imbalance during training, ensuring fairness in MCI detection.
●	Avoided normalization of pitch or speech to preserve natural variability in diverse populations.
3. Feature Selection and Importance
●	Combined linguistic, acoustic, and prosodic features with statistical analysis to identify the most impactful predictors.
●	Included XGBoost Feature Importance to rank features like mean pitch, word count, and pronoun usage.
4. Model Interpretability and Explainability
●	Metrics provide explainability for model performance:
○	Logarithmic Loss (LogLoss): Achieved a score of 0.229 on optimized XGBoost.
○	Additional metrics for clinical clarity: Precision (0.939), Recall (0.946), and ROC-AUC (0.991).
●	Emphasized feature preservation: Pauses, word repetitions, and speech energy irregularities are clinically relevant for diagnosing MCI.
________________________________________
Key Results Table
Model Name	Accuracy	Precision	Recall	F1-Score	ROC-AUC
XGBoost	0.943	0.939	0.946	0.943	0.991
LightGBM	0.945	0.939	0.950	0.945	0.990
Voting Ensemble	0.945	0.929	0.962	0.945	0.989
Random Forest	0.932	0.903	0.965	0.933	0.986
Gradient Boosting	0.907	0.889	0.927	0.908	0.974
Logistic Regression	0.823	0.787	0.881	0.831	0.889
Summary / Clinical Relevance:
This model is clinically useful because it was specifically designed to detect Mild Cognitive Impairment (MCI) by analyzing natural speech patterns. The model combines linguistic, acoustic, and prosodic features—such as word count, syntactic complexity, pitch variability, and pause durations—which are known indicators of early cognitive decline. Importantly, we preserved the raw data without excessive preprocessing to ensure that diagnostically significant features, like speech hesitations or natural pitch variations, were not lost.
The model achieved strong performance metrics, including a ROC-AUC of 0.991 and high accuracy, precision, and recall scores, validating its reliability in distinguishing between MCI and healthy controls. The rationale for selecting these features was grounded in their clinical relevance, capturing subtle changes that align with early signs of cognitive decline often missed by traditional diagnostic tools. This approach ensures the model not only performs well statistically but also aligns with real-world diagnostic needs, making it a valuable, non-invasive tool for clinicians to detect MCI early and improve patient outcomes.
Code Requirements:
Below is the list of required modules and tools:
Core Python Libraries
pandas – For data manipulation, cleaning, and managing CSV files.
pip install pandas

numpy – For numerical operations and handling arrays.
pip install numpy

scikit-learn – For model training, evaluation metrics, and preprocessing.
pip install scikit-learn

matplotlib and seaborn – For visualizations during Exploratory Data Analysis (EDA).
pip install matplotlib seaborn

imblearn – For SMOTE (Synthetic Minority Oversampling Technique) to handle class imbalance.
pip install imbalanced-learn

Machine Learning Libraries
XGBoost – The core library for the XGBoost model and hyperparameter optimization.
pip install xgboost

LightGBM – Lightweight Gradient Boosting for the LightGBM model.
pip install lightgbm

RandomForestClassifier and GradientBoostingClassifier – From scikit-learn for ensemble learning. (No separate installation needed; part of scikit-learn)

LogisticRegression – For baseline model comparison (included in scikit-learn).

Feature Extraction and Audio Processing
OpenSMILE – A tool for acoustic and prosodic feature extraction (pitch, energy, pauses).
○	Installation: Follow OpenSMILE installation instructions.
○	Used to extract acoustic features from .wav files.
Parselmouth – Python wrapper for Praat, used for advanced speech and prosody analysis.
pip install praat-parselmouth

Optimization Tools
Optuna or Bayesian Optimization (if included for hyperparameter tuning).
Optuna:
pip install optuna

Additional Tools
Jupyter Notebook (Optional) – For interactive development and visualization.
pip install notebook
Python Version: Ensure you are using Python 3.8+ to maintain compatibility with the libraries
Tool Summary
●	Programming Language: Python
●	Machine Learning Models: XGBoost, LightGBM, RandomForest, Gradient Boosting, Logistic Regression
●	Feature Extraction: OpenSMILE, Parselmouth
●	Preprocessing: SMOTE (imbalanced-learn)
●	Optimization: Bayesian Optimization (optional)
●	Evaluation Metrics: Log Loss, Accuracy, Precision, Recall, F1-Score, ROC-AUC
Code version history (1)
v1
⸱
4mo 2w ago
⸱
Latest

Tags

#python #Modeling #Dataset #Tools #Explainability #feature-selection #prepare_acoustic

License

MIT open source