disasters

Open Cities AI Challenge: Segmenting Buildings for Disaster Resilience

Can you map building footprints from drone imagery? This semantic segmentation challenge leverages computer vision and data from OpenStreetMap to support disaster risk management efforts in cities across Africa. #disasters

$15,000 in prizes
mar 2020
1,099 joined

Can you map building footprints from drone imagery? This semantic segmentation challenge leverages computer vision and data from OpenStreetMap to support disaster risk management efforts in cities across Africa.

Quick Facts

Participants
1,099
No. of Entries
2,127
Prize
$15,000
Winner

qubvel

1st Place

Navigation

Responsible AI

With great computational power comes great responsibility. As ML experts who directly develop and apply algorithmic systems, this Responsible AI track presents an opportunity to examine the practical ethics and appropriate use of our work applied to the field of disaster risk management.

Machine learning holds the potential to help with many applications in disaster risk management (DRM), especially when coupled with computer vision and geospatial technologies, by providing more accurate, faster, or lower-cost approaches to assessing risk. The potential benefits are significant. At the same time, we urgently need to develop a better understanding of the potential for negative or unintended consequences of their use. With growing attention given to questions of appropriate and ethical ML use for facial recognition, criminal justice, healthcare, and other domains, we have an immediate responsibility to elevate these questions in DRM.

Examples of potential harm that ML technologies present in this space include, but are not limited to:

  • Perpetuating and aggravating societal inequalities through the presence of biases throughout the machine learning development pipeline.
  • Aggravating privacy and security concerns in Fragility, Conflict and Violence (FCV) settings through combination of previously distinct datasets.
  • Limiting opportunities for public participation in disaster risk management due to increased complexity of data products.
  • Reducing the role of expert judgement in data and modeling tasks and in turn increasing probability of error or misuse.
  • Inadequately communicating methods, results, or degrees of uncertainty, which increases the chance of misuse.

This submission track is open to all. You do not have to participate in the Segmentation track of this competition to participate in the Responsible AI track. Segmentation track participants must submit at least once to this track to qualify for the $12,000 in segmentation performance prizes.

Your task

In this part of the challenge, you are asked to engage with potential machine learning pipelines for improved mapping (i.e. via semantic segmentation of buildings) for DRM applications. Your job is to bring an applied ethics lens to the design and use of AI.

Scenario description

Below is some brief background to get you started. You are free to make additional assumptions regarding data collection, management, analysis, model development, evaluation, communication, and deployment as you navigate the ethical considerations of this problem. Keep in mind that potential sources of harm can arise at different points in an ML pipeline.

Data collection and annotation

After assessing goals, scope, and resources with local government, community, and technical partners and upholding OpenDRI’s 9 principles for disaster risk data and open data projects, overhead imagery is collected via consumer drones or satellites over designated areas of interest (AOIs).

This up-to-date high-resolution imagery is manually inspected by mappers (often members of local OpenStreetMap communities) who digitally trace or update building footprint outlines to line up with what is visible on imagery. These digitized maps and drone imagery are published to OpenStreetMap and OpenAerialMap respectively where they serve as data public goods that can be used and improved by all.

For more background on data collection see the About the Project competition page and the Open Cities process diagram below.

Open Cities process diagram Source: opencitiesproject.org/about/

Data curation and management

To curate a challenge-ready dataset, the publicly available drone imagery and manually-traced building footprints are reviewed for labeling quality and assigned to training set tier 1 (higher quality) or tier 2 (lower quality). Image scenes and accompanying labels are exported, processed into standardized formats (Cloud Optimized GeoTIFF and geoJSON), and stored as SpatioTemporal Asset Catalogs (STACs) to facilitate their querying and access.

Test imagery and labels are curated and managed in a similar fashion, except that imagery locations are removed and labels are withheld for model evaluation purposes.

More info about the challenge data is available in the Problem Description.

Model development and evaluation

The task at hand is semantic segmentation of buildings using drone imagery. Semantic segmentation is a pixel-based classification task; in this case, each pixel is labeled either as a building or not a building.

Segmentation performance is often evaluated and communicated through metrics like accuracy, intersection-over-union (IoU), precision, recall, or F1 scores. In this challenge, we use a pixel-based intersection over union (Jaccard index) metric.

Applications

Building segmentation is a first step toward many important interventions that help mitigate disaster risk and enable more effective disaster response. Often times, building footprints are an input to downstream uses, such as:

  • Monitoring urban growth in order to manage new construction, especially in hazard zones like floodplains.
  • Identifying informal settlements in order to represent the most vulnerable people in society and deliver aid effectively.
  • Classifying and detecting fine-grained building attributes like roof material, construction style, large first floor openings (“soft story” buildings) in order to identify high-risk buildings for retrofitting.
  • Modeling hazard-induced building damage in order to estimate relative disaster risk and prioritize in-person inspections and interventions.

Creating your submission

In your submission, examine the ethical considerations that arise in designing and using ML/AI for disaster risk management. How might we improve the creation and application of ML solutions to mitigate biases, promote fair and ethical use, communicate insights clearly, and make safeguards to protect users and end-beneficiaries?

Your submission can focus on any (or all) of the following areas:

  • Framework: What approach or principles would you use to examine the ethical considerations of using ML in this scenario? How does this framework enable you to spot ethical issues or potential harms? How are decisions made? Who is engaged in this process? Feel free to design your own framework, or adapt or critique an existing one.
  • Identification: What are the potential harms at play in this scenario? Where are the ethical concerns most salient? How do these manifest differently in the data collection, curation, analysis, modeling, evaluation, communication, and deployment phases?
  • Mitigation: How can these ethical issues be mitigated? What technical approaches or tools would you use? When faced with trade-offs, what decisions would you make?

You may choose to touch on all of these areas (e.g., design a framework, use it to spot ethical issues, and explain how these could be addressed), dive deeply into one area (e.g., a detailed examination of sources of bias introduced during data collection, or opportunities for participatory machine learning), or something in between.

If it’s helpful to narrow in on use case, you may examine the ethical implications of AI within a specific use case outlined above or imagine your own DRM use case. For additional inspiration here, see the case studies in chapter 6 of the Machine Learning for Disaster Risk Management report.

The submission format is flexible. You can submit Jupyter notebooks, slides, blogs, essays, demos, product mockups, speculative fiction, art work, synthesis of research papers or original research, or whatever other format best suits you.

Submit!

Upload a link to your submission here. You will need to have entered the competition to access the submissions page.

Segmentation track participants must submit at least once to this track to qualify for the cash prizes. Follow the instructions on the submissions page to provide a link to your work. Beyond those instructions, there is no minimum requirement for what qualifies as a submission; the point is to engage with the ethical implications of your work and the work of others in this domain.

Rubric

Your submission will be evaluated by a panel of judges based on the following rubric. We encourage you to bring your unique perspective as a data scientist to this problem. Direct subject matter expertise in disaster risk management is not required.

  • Thoughtfulness (40%):
    • Depth of inquiry goes further than a superficial level, i.e. into second order consequences
    • Synthesizes multiple, oft-competing ideas and principles
    • Acknowledges contradictions and tradeoffs
  • Relevance (20%):
    • Applies an ethical lens specifically to a disaster risk management use case
    • Data sources considered include OpenStreetMap building labels and overhead imagery from drones and satellites
  • Innovation (20%):
    • Goes beyond well-established methods or uses them in novel ways to tackle the question
    • Takeaways are insightful, thought-provoking, and actionable
  • Clarity (20%):
    • Clearly communicates the problem(s), approach, or issues explored in the submission
    • Understandable to the non-technical layperson

Additional resources

Still not sure how to get started? Check out these resources:

On this page