TissueNet: Detect Lesions in Cervical Biopsies Hosted By French Society of Pathology
Woohoo! This competition has come to a close!
Many thanks to the participants for all of their hard work and commitment to using data for good!
Whole slide images are digital formats that allow glass slides to be viewed, managed, shared, and analyzed on a computer monitor. These extremely high resolution images require special software to be able to read and manipulate in memory. A few different packages exist to do this in Python, and we will cover the main ones here. Each WSI image in the train set is also available as a pyramidal TIF, so we'll cover some helpful tips for that format too. All test images are pyramidal TIFs.
Native whole slide images
The native whole slide image (WSI) formats present in our dataset are:
- .mrxs (MIRAX)
- .svs (Aperio)
- .ndpi (Hamamatsu)
These formats are related to the scanner used to create the images. These are provided in the native resolution and magnification. You can determine the exact resolution and magnification by referring to
test_metadata.csv. While the pyramidal TIF format has standardized, precomputed zoom levels, the available levels for WSI can vary. You can determine the zoom levels for each image using OpenSlide.
We recommend working from the pyramidal TIF versions of the slides in your machine learning workflow. The software tools for this format are better maintained and less prone to error than those available for WSIs.
Pyramidal TIFs are a multi-resolution, tiled format. Each resolution is stored as a separate layer or "page" within the TIF. Page 0 contains the image at the full resolution of the WSI. Subsequent pages contain lower resolution images―each layer is the previous layer downsampled by a factor of two, until the largest dimension of the downsampled image is less than 512 pixels. In other words, relative to the full resolution, the pages are downsampled 2x, 4x, 8x, 16x, and so on. Each layer is stored as individual 512x512 "tiles" where each tile is compressed using JPEG with quality (Q) 75.
Using the full-resolution provided in the metadata csv, you can calculate the resolution of each layer as
full_image_resolution * (2 ** page). The resolution reported in the metadata CSVs is in the units microns/pixel, so larger numbers are more zoomed out.
Given the enormous size of the full-resolution images, which makes it impossible to load even a single image into memory, you will need to get creative in your use of multiple scales! For example, one of the first things you will notice browsing the images is that much of the slide is blank space with no tissue whatsoever. It would be a waste of your limited compute time to analyze those parts at full resolution. One simple solution is to segment the image into tissue/non-tissue at a lower resolution and analyze only the tissue regions at higher resolutions.
OpenSlide supports all native whole slide image formats, including:
- .mrxs (MIRAX)
- .svs (Aperio)
- .ndpi (Hamamatsu)
Here's a code snippet to perform some basic image operations using OpenSlide:
import openslide # load the slide slide = openslide.OpenSlide("slide1.ndpi") # number of levels print(slide.level_count) >> 8 # dimensions for each level in the image print(slide.level_dimensions) >> ((67456, 84224), (33728, 42112), (16864, 21056), (8432, 10528), (4216, 5264), (2108, 2632), (1054, 1316), (527, 658)) # magnification for each level print(slide.level_downsamples) # highest resolution available in micrometers per pixel print(slide.properties["openslide.mpp-x"], slide.properties["openslide.mpp-y"]) >> 0.22717462913741793 0.22717462913741793 # generate thumbnail thumbnail = slide.get_thumbnail((output_height, output_width)) # read an image region in as a PIL Image x, y = 100, 200 level = 2 region_width = 500 region_height = 500 region = slide.read_region((x, y), level, (region_width, region_height)) # convert the PIL Image to a numpy array array = np.array(region) # red, green, blue, alpha channels print(array.shape) >> (500, 500, 4)
OpenSlide Python is unable to read in the pyramidal TIF formats. To do that, you should use PyVips.
PyVips is a Python binding for libvips, a low-level library for working with large images. PyVips can be used to read and manipulate the pyramidal TIF formats. Here's a code snippet to perform some basic image operations using PyVips:
import pyvips # load the full-resolution image path = "slide1.tif" slide = pyvips.Image.new_from_file(path, page=0) print(slide.width, slide.height) >> 67456 84224 # load the 16x downsampled image from page 4 (2 ** 4 = 16) slide = pyvips.Image.new_from_file(path, page=4) print(slide.width, slide.height) >> 4216 5264 # read an image region in as a PIL Image x, y = 100, 200 level = 2 region_width = 500 region_height = 500 region = slide.crop(x, y, region_width, region_height) # convert the PIL Image to a numpy array import numpy array = np.ndarray( buffer=region.write_to_memory(), dtype=np.uint8, shape=(region.height, region.width, region.bands) ) # red, green, blue channels print(array.shape) >> (500, 500, 3) # also look into pyvips.Region.fetch for faster region read region = pyvips.Region.new(slide).fetch(x, y, region_width, region_height) bands = 3 array = np.ndarray( buffer=region, dtype=np.uint8, shape=(region_height, region_width, bands) ) print(array.shape) >> (500, 500, 3)
If OpenSlide is installed on your machine, PyVips can also read in native WSI formats. In fact, this is how we converted WSIs to pyramidal TIFs for the competition. An example of the conversion is below:
import pyvips native_path = "image_001.svs" # path to the native WSI file native_image = pyvips.Image.openslideload(native_path) native_image.tiffsave( "image_001.tif", # path to the output pyramidal TIF file compression="jpeg", Q=75, tile=True, tile_width=512, tile_height=512, pyramid=True, )
Deep Zoom Viewer
OpenSlide Python provides support for Deep Zoom. This allows you to display images in a web browser without converting slides to the Deep Zoom format. Deep Zoom is useful if you want to efficiently view, pan around, and zoom whole slide images. Read more on the OpenSlide docs.
Cytomine allows you to display and explore native whole slide images and pyramidal TIF formats in a web browser. It also supports adding annotations and executing scripts from inside Cytomine or from any computing server using the dedicated Cytomine Python client. Cytomine can be installed locally or on any Linux server. The Cytomine GitHub repository includes examples of Python scripts demonstrating how to interact with your Cytomine instance, as well as examples of ready-to-use machine learning scripts (all
S_ prefixed repos, such as
Tips and tricks
Here are a few tricks we discovered while working with the data:
- A critical part of the challenge of this competition is how to efficiently process these enormous images. Due to the time limit on submissions, you will not have time to process every part of the image at the highest resolution available. Instead, you might consider methods to first predict which parts of the image are most important for an accurate diagnosis. The pyramidal TIF format contains multiple zoom levels that you can access with the
pagekeyword argument to
pyvips.Image.new_from_file. You might try a "multiscale" approach―detect the most important parts of the image from lower resolutions versions (
pyvips.Image.new_from_file(path, page=5)) and then analyze just those parts at higher resolutions.
pyvips.Image.cropto read in small image regions. According to a PyVips developer and our own tests,
pyvips.Region.new(image).fetch(...)is much faster than
pyvips.Image.crop(...). (Requires libvips version 0.8.6 or greater.)
- Avoid OpenSlide-based data loaders (
openslide.OpenSlide) in your machine learning pipelines. OpenSlide is no longer under active development, and several issues have arisen that have not been addressed. For example, we ran into errors reading certain SVS files (likely related to this open issue) in which parts of the image were blank. Even installing
setuptools. We found that reading the pyramidal TIFs using PyVips was a reliable way to work with the images in our machine learning pipelines. (OpenSlide Python's Deep Zoom viewer is still helpful for manually browsing the slides.)
Additional reading and research
Here are a few papers and tutorials that talk about machine learning with WSI that you may find helpful:
- Whole slide image preprocessing in Python
- Assessment of Machine Learning of Breast Pathology Structures for Automated Differentiation of Breast Cancer and High-Risk Proliferative Lesions - PubMed
- Using deep convolutional neural networks to identify and classify tumor-associated stroma in diagnostic breast biopsies
- Assessment of Machine Learning of Breast Pathology Structures for Automated Differentiation of Breast Cancer and High-Risk Proliferative Lesions
- Histologic tissue components provide major cues for machine learning-based prostate cancer detection and grading on prostatectomy specimens
- Assessment of Machine Learning Detection of Environmental Enteropathy and Celiac Disease in Children