Run-way Functions: Predict Reconfigurations at US Airports (Prescreened Arena) Hosted By NASA

competition
complete
$40,000

alt-text


This University Challenge was a great opportunity for us to work with teams that we otherwise wouldn't have been able to. By crowdsourcing this problem to the university community, we've been able to learn about a variety of innovative approaches, while also providing an open dataset that anyone can use in their research.

Tim Scott, Research Computer Scientist, NASA Langley Research Center

Why

Coordinating our nation’s airways is the role of the National Airspace System (NAS). The NAS is arguably the most complex transportation system in the world. Operational changes can save or cost airlines, taxpayers, consumers, and the economy at large thousands to millions of dollars on a regular basis. It is critical that decisions to change procedures are done with as much lead time and certainty as possible.

An important part of this equation is airport configuration, the combination of runways used for arrivals and departures and the flow direction on those runways. For example, one configuration may use a set of runways in a north-to-south flow (or just "south flow") while another uses south-to-north flow ("north flow"). Air traffic officials may change an airport configuration depending on weather, traffic, or other inputs.

These changes can result in delays to flights, which may have to alter their flight paths well ahead of reaching the airport to get into the correct alignment or enter holding patterns in the air as the flows are altered. The decisions to change the airport configuration are driven by data and observations, meaning it is possible to predict these changes in advance and give flight operators time to adjust schedules in order to reduce delays and wasted fuel.

The Solution

The goal of this challenge was to automatically predict airport configuration changes from real-time data sources including air traffic and weather. Better algorithms for predicting future airport configurations can support critical decisions, reduce costs, conserve energy, and mitigate delays across the national airspace network.

The challenge involved three phases. In the Open Arena, anyone could explore the data and submit predictions to see how they fared against others on the open leaderboard. The Prescreened Arena, available to U.S. university-affiliated participants, offered participants an opportunity to further develop their models and package their code to submit for the final evaluation. In the final evaluation phase, we ran the participants' code on a brand new dataset to see how well their models generalized to data that was not available during training.

The Results

Over the course of the challenge, participants tested over 350 solutions. In the final evaluation dataset, the top model achieved a mean aggregated log loss of 0.074, a significant improvement over the no-change benchmark. In the difficult situations where a configuration did change, this model was able to predict the change 41% of the time when looking ahead two hours in advance, with a precision of 49%.

Recall and precision for timepoints when the configuration changes.

Recall and precision metrics for the 2-hour lookahead with k of 1 and 3.

The winning solutions employed a range of techniques. Some trends were the use of tree-based models such as CatBoost and XGBoost and careful engineering of the feature timecourses to generate useful predictors of airport configuration.

See the results announcement for more information on the winning approaches. All of the prize-winning solutions from this competition are linked below and available for anyone to use and learn from.


RESULTS ANNOUNCEMENT + MEET THE WINNERS

WINNING MODELS + MODEL REPORTS ON GITHUB