Nuclei segmentation using deep learning: Methodology essentials

24 Jan, 2022 | Blog posts, IKOSA AI

State-of-the-art nuclei segmentation techniques using deep learning technology  solve many of the problems related to extracting reliable quantitative information on the cell nucleus-level.

This article throws light on recent DL-based methods used for nuclei segmentation tasks. Learn about the most common challenges researchers face during tissue analysis such as over- and under-segmentationoverlapping or touching nuclei and domain shift

We introduce you to a number of pre-processing, annotation and labeling best practices, intended to help you yield optimal results when implementing nucleus segmentation in digital image analysis workflows.

Find out about the various applications of algorithm training for nuclear segmentation tasks in the field of histopathology.

How the automated analysis method can help you advance your nucleus segmentation project

Advanced methods for nuclei segmentation using deep learning (DL) have risen in popularity in recent years. These approaches aim at detecting and measuring properties of nuclei in tissue sections based on automated image recognition algorithms.

Being able to correctly detect and segment cell nuclei in microscopy images is an essential task throughout various disciplines such as pathology and histology. Changes in the morphological characteristics of the nuclei in tissue samples are often considered an indication of pathological processes.

The benefits of deep learning approaches for nuclei segmentation
The benefits of deep learning approaches for nuclei segmentation.
Place your alt text here
The benefits of deep learning approaches for nuclei segmentation.

We have summed up the core benefits of using a DL-based method in nuclei segmentation tasks for you:

  • Facilitating a faster and more reliable morphological assessment
  • Improving analysis objectivity 
  • Improving results reproducibility
  • Increasing efficiency in research and diagnostics

Yet, the choice of the right DL-architecture for your research project should depend primarily on its performance in terms of processing efficiency and segmentation accuracy.

The cell nuclei segmentation workflow

The automated segmentation of cell nuclei involves a number of stages: preprocessing, segmentation, postprocessing and evaluation.

  • During the preprocessing stage the quality of input images is improved to ensure optimal segmentation results.
  • The objective of the actual segmentation stage is to extract cell nuclei present in the foreground of the image.
  • During the post processing stage the segmentation results are optimized. This involves sorting out false findings i.e. removing objects that are not actual nuclei, like image artifacts, from the segmentation results by adding additional code for morphological filtering. Post-processing is also applied to refine nuclear boundaries.
  • At the evaluation stage the researcher assesses the quality of the quantitative segmentation results based on benchmark metrics. The most common metrics used at this stage are precision, recall and F1-measure (Win et al., 2018).
The cell nucleus segmentation deep learning workflow
The cell nucleus segmentation deep learning workflow.
Place your alt text here
The cell nucleus segmentation deep learning workflow.

Precision is a measure of prediction accuracy. Recall is a metric related to the reproducibility of results. The F1-measure is a metric related to the object detection capabilities of the network. All above-mentioned metrics can take an expression from 0 to 1. The closer to 1 the value is, the greater the predictive power of the model (Fujita & Han, 2020).

Deep learning methods for nuclei segmentation: analysis frameworks, challenges and opportunities

A frequent and central goal of tissue analysis is determining the population of individual cell nuclei in microscopy images. Based on the task, an object detection algorithm may be sufficient to localize nuclei instances.

Morphological properties of the nucleus such as shape and texture allow observers to obtain quantitative details at the individual object level. This data may subsequently be utilized to visualize size distributions and detect abnormal patterns.

An essential step in this analysis workflow is the accurate splitting of the image into semantic classes. These classes represent coherent regions such as nuclei (on the intracellular level), connective tissue, epithelial and stromal areas within a histology dataset.

Each semantic class needs to be further processed by identifying the boundaries of every single nucleus i.e. performing boundary detection. Thus, three classes of pixel types are identified during the nuclear segmentation process: background, nuclei interior and nuclear boundaries (Caicedo et al., 2019).

Tips and tricks

Using state-of-the art deep learning techniques allows you to detect individual instances of a particular type of nuclei.

Traditional nucleus segmentation methods

The most common traditional approaches for nuclei segmentation analysis are thresholding and seeded watershed.

Thresholding involves the use of threshold intensity values in grayscale histograms of microscopy images in order to filter objects of interest. Intensity thresholding is thus well-suited for extracting nuclei from the foreground (Win et al., 2018; Caicedo et al., 2019).

The classical seeded watershed method partitions microscope images using watershed lines alongside the boundaries of morphological structures such as nuclei. By doing so, the locations of the nuclei – referred to as seeds – are identified. The algorithm further relies on region growing in order to expand each location until the boundaries of the nucleus are reached (Atta-Fosu et al., 2016).

However, accurate and robust nuclear instance segmentation using traditional methods can be challenging due to morphological variations in the nuclei among different organs and tissue types. Moreover, spatial configurations (e.g. clusters, touching nuclei), tissue preparation specifics or complex background containing imaging artefacts (e.g. folds, out-of-focus regions) are significantly limiting the applicability of such methods (Zhou et al., 2019).

At the same time, innovative deep learning-based methods have been on the rise in recent years and have proven to be more efficient and accurate and robust in segmenting cell nuclei across various tissue types (Jang et al., 2019). We provide a brief overview of the most common DL-based approaches used for nuclei segmentation.

Deep learning model architectures for nucleus segmentation

Deep neural network architectures apply semantic and instance segmentation in order to identify and locate nuclei in microscopy images (Naylor et al., 2017).

Most common DL-based approaches for nuclei segmentation:

Semantic segmentation app training with IKOSA AI
Associates every pixel of an image with a class label.
Instance segmentation app training with IKOSA AI
Treats multiple objects of the same class as distinct individual instance.

Semantic segmentation involves assigning areas of the image to different semantic classes being nuclei area, cytoplasm area, nuclei edges or boundaries area and background area. The task of semantic segmentation DL-algorithms is to track the presence of nuclei in the foreground of the digital slide and to later detect the boundary of each nucleus by segmenting the connected foreground area (Cui et al., 2019; Kowal et al., 2020).

Instance segmentation involves in two stages: semantic segmentation and object identification.

First, the algorithm detects objects of interest and creates bounding boxes around them. Second, the model produces a pixel-wise mask for each individual object, i.e. it performs semantic segmentation on each bounding box. Instance segmentation offers the advantage of providing additional information on rich morphological features of cellular structures (Zhou et al., 2019).

DL-algorithms classify and segment objects in the images by learning features from large amounts of representative input data (Hayakawa et al., 2019). Various types of DL-architectures have been used for segmenting biomedical images, yet they all differ in layer configuration and model depth (Kowal et al., 2020). 

Other DL-architectures used for nuclei segmentation: 

Mask R CNN

Mask R CNN is an example of a domain adaptation network particularly effective for the purposes of instance segmentation. Mask R CNN enables the simultaneous detection and segmentation of cell images by adding branching subnets to estimate the segmentation mask for each region of interest (ROI) (Allehaibi et al., 2019; Fujita & Han, 2020). The model first detects bounding boxes of nuclei and subsequently performs segmentation on the nuclei within the boxes (Vuola et al., 2019).

When comparing histopathological image inputs with the nucleus segmentation mask produced by DL-algorithms certain discrepancies in terms of objects being assigned to different classes can be observed. Due to such image-level and feature-level domain bias a phenomenon referred to as “domain shift” occurs. Domain adaptation deep learning methods tackle this issue by adding benchmark datasets as a source in order to adapt to a variety of biomedical images (Lui et al., 2020; Hsu et al., 2021).

Mask R CNN has been reported to perform outstandingly in nucleus detection tasks, while its segmentation capabilities still need improvement. This particular deep learning model yields less oversegmentation cases as compared to U-Net and can better detect clumped or grouped nuclei. Still, Mask R CNN tends to oversegment large nuclei, while it yields better performance for small and middle sized nuclei (Vuola et al., 2019).

Apparently each type of deep learning network has its advantages and is particularly efficient at different nucleus segmentation tasks. Choosing the right network for your research project or adapting a deep-learning algorithm to match your requirements requires expert knowledge in computer vision and programming skills. All this speaks in favor of a model that is robust and flexible enough to adapt to different samples, research designs and segmentation tasks.


U-Net is a U-shaped multi-scale stacked neural network based on FCN architecture. A U-Net relies on the stacking of encoder and decoder layers. The stacking of layers allows the algorithm to learn more complex features as the network depth increases (Vuola et al., 2019).

Encoder layers create representation-encoding by reducing input images by their spatial resolution through multiple convolutional layers. Decoder layers are used to create feature maps in order to predict segmentation masks.

Similarly to the FCN a typical U-Net introduces shortcut connections between the encoder and decoder layers. These skip connectors allow the network to retrieve spatial information lost during the process of pooling and to preserve features at different resolutions (Ronneberger et al., 2015; Kang et al., 2019; Vuola et al., 2019; Ibtehaz & Rahman, 2020).

U-Net has been reported to yield excellent results in nucleus segmentation tasks especially in cases of large nuclei, elliptical nuclei shapes and single nuclei. At the same time some pitfalls of U-Net as compared to other networks can still be observed with regards to nucleus detection tasks and clumping multiple nuclei into one big nucleus (Vuoloa et al., 2019).

Fully convolutional network (FCN)

The fully convolutional network (FCN) approach relies on the original CNN-architecture, but uses fully connected upsampling and skip layers.

FCNs operate on image input of arbitrary size and instead of single pixel classifications, generate entire images as output (semantic maps). The spatial precision of the output is more refined and the network is more sensitive to variations in input image resolution. This makes FCN flexible enough not to require extensive pre- and post-processing and to learn high-level features (Zhang et al., 2017; Naylor et al., 2017; Long et al., 2015).

FCN has been reported to be particularly precise in capturing deep features of cell nuclei, but not as effective in localizing nucleus boundaries. This type of architecture has been quoted for the capability to segment “normal” and “abnormal” nucleus masks in cytology studies (Zhang et al., 2017).

Convolutional neural networks (CNN)

Convolutional neural networks (CNN) are widely used for the automatic segmentation of biomedical image data for the purposes of feature extraction. CNNs are multilayer neural networks that learn an abstract representation of the input image on the basis of hierarchical layers of convolutional filters (Kowal et al., 2020).

Like other segmentation approaches convolutional neural networks classify single pixels of the image into different semantic categories. However, they have been reported to provide superior performance in nuclei segmentation, classification and detection tasks as compared to traditional methods (Alom et al., 2018).

This approach applies three types of layers in order to process image data: convolution layers, pooling layers and activation or fully connected layers.

Convolution layers are used to compress certain features in areas of interest in the input images, while pooling layers are used to create feature maps from the data obtained during convolution. The latter represent semantic maps of connected objects, which are later to be segmented into singular objects. Last but not least, a fully connected layeconnected to all activations in the previous layers is applied at the end of the CNN. The task of the network at this stage is to capture relationships between high-level features and output labels (Wang et al., 2014; Hayakawa et al., 2019; Caicedo et al., 2019; Kowal et al., 2020).

CNNs have been reported to be more effective in detecting nuclei and cytoplasm areas than traditional computer vision methods, even in cases when the staining and the background are heterogeneous. Problems may still occur in cases of overlapping nuclei, where no clear nucleus boundaries can be detected. Yet, existing research points at the superior segmentation performance of CNNs as compared to intensity-based thresholding methods in such cases. However, the separation of overlapping areas still requires a two-stage method combining convolutional neural networks with seeded watershed techniques (Kowal et al., 2020).

Several types of deep learning architectures based on CNN have been discussed in existing literature and each brings along specific advantages during segmentation tasks.

Tips and tricks

When selecting the right model for your nucleus segmentations project you should choose one that is robust and flexible enough to adapt to different samples, research designs and segmentation tasks.

Preprocessing, annotation and labeling strategies

Prior to conducting the actual quantitative segmentation analysis, a stage of preparation is necessary in order to get the image data ready for the algorithm training. Different strategies for preprocessing, annotation and labeling of the image dataset have been proposed in existing literature in order to extract the most accurate nucleus segmentation data with trained algorithms.

We present to you several proven strategies for the preprocessingannotation and labeling of the image dataset that can help you extract the most accurate nucleus segmentation data.

7 Training Techniques for Nuclei Segmentation prepared by KML Vision
Data preparation techniques for nuclei segmentation tasks.
Place your alt text here
Data preparation techniques for nuclei segmentation tasks.

Training data selection and preparation

DL-algorithms tend to be effective in segmenting nuclei even when trained on a small dataset, when the task permits it. Yet, providing a variety of training images improves the predictive power of the algorithm. DL-algorithms can be used across different experimental settings as long as variations of the morphological structures of interest have been priorly introduced to the algorithm (Caicedo et al., 2019). Otherwise, a “domain shift” (check the definition in Mask R CNN) issue can occur, where the model was trained on a specific setting, but has to perform in a different setting, limits the applicability (Liu et al., 2020; Hsu et al., 2021).

The careful selection of input images is also recommended in order to be able to balance the dataset and reduce the amount of images containing background only or images of poor quality  (Araujo et al., 2019).

Image data augmentation

Data transformation techniques in the spatial or color domain may effectively be used as part of a data augmentation pipeline. Augmentation refers to the artificial extension of a dataset to increase the amount and variance of training data so that it best represents the expected target domain. Data augmentation is amongst the most important steps in preprocessing and building a robust model for production use.

Common augmentation strategies involve random or parameterized spatial transformations such as flipping, rotation or deformation, frequently combined with adding noise or color transformations (Allehaibi et al., 2019). A nuclei segmentation model generally benefits from these operations, because when using meaningful data augmentation, the total amount of manually annotated and labeled data required for training can be kept at minimum, facilitating faster model development.

Color processing and normalization

During the segmentation process of stained virtual slides color information serves as an important indicator for detecting objects of interest. However, differences in the staining or acquisition devices of the whole-slide images may pose a difficulty for DL-algorithms to properly process the data.

Different types of stains are used to mark various objects of interest within histopathological images. Stains such as hematoxylin and eosin are used to mark those objects with a particular color. Thus hematoxylin is often used to mark cell nuclei and paint them blue. Eosin stains are normally absorbed in the cytoplasm marking it reddish in color.

Certain color variations in the markings may occur as a result of using different staining protocols, stain brands or microscope and scanning devices. This may affect the performance of the deep-learning network, especially if the color hues training input images differ from those  images to be processed  (Kowal et al., 2020).

Therefore, you should aim to ensure the homogeneity of input data in terms of color variation. You can do this by performing color transformation into an illumination resistant color space and also normalization of the individual color space components. The following color normalization techniques will help you achieve the desired outcome: histogram matching, color transfer and spectral matching.  (Hayakawa et al., 2019; Cui et al., 2019; Mahmood et al., 2019; Kowal et al., 2020).

Annotation and labeling strategies

The terms “annotation” and “labeling” are often used interchangeably and usually mean the same: adding a semantic meaning to a specific, spatially constrained image region. With annotating we refer to the process of marking image regions manually using geometric shapes, for instance a polygon, and labeling the process of adding a specific term or meaning to a respective geometry.

Annotations for bioimage segmentation
The broad choice of annotation tool shapes available in the ​IKOSA Platform is there to help you outline even the most intricate morphological features

End-to-end deep-learning-based segmentation models essentially learn to create semantic masks in the image, where each pixel is assigned to a specific semantic class. Hence, a pixel mask is also needed as a learning target for the system. While a pixel-wise annotation or “dense annotation” method requires a lot of manual effort (Qu et al., 2020) for the accurate outlining of individual objects, there are a number of annotation techniques discussed in literature to make the algorithm training process more time-efficient.

Partial annotation, or “weak annotation”, is a technique that relies on annotating only a small number of objects in the training dataset. This strategy is also referred to as “sparse labeling.” Usually, this technique does not directly result in the desired masks and requires additional preprocessing steps to subsequently generate pixel-level annotations.

While the generated masks may not be perfectly delineating the nuclei, this approach can help you save time, especially when you are creating the initial basic draft of your algorithm. Moreover, algorithms trained on partially annotated image datasets have reportedly shown similar performance results when compared to approaches involving a dense pixel-level annotation (Qu et al., 2020; Bruch et al., 2020; Ho et al., 2021).

Point annotations are probably the most time-saving method of manual image annotation. Recent work has shown that training nuclei segmentation algorithms only by marking a single location of the nucleus without actually delineating the exact boundaries is possible, which significantly reduces the effort of annotation (Yoo et al., 2019).

While fine-tuning your algorithm, carefully made annotations will definitely help you get more accurate predictions. Level up your annotating skills with our special article on how to make annotations in IKOSA.

Tips and tricks

Another smart way to save time is to annotate certain regions-of-interest (ROIs) instead of the entire images. Learn how to solve ROI-drawing tasks in IKOSA.

Dealing with touching and overlapping nuclei during algorithm development

One object class of particular interest in the analysis of histological images is the nucleus of cells. The detection of overlapping, touching and heavily clustered nucleiis one of the most challenging issues when training automated nuclei segmentation models. The presence of multiple layers of cells e.g. in a pap smear sample often results in an upper layer of cells covering and obscuring the underlying cell layers on the microscopy slide image.

Challenges during tissue analysis:

  • ​Over- and under-segmentation
  • ​Overlapping nuclei
  • Touching nuclei
  • ​Domain shift

Overlapping cell nuclei have intersecting boundaries at points of concavity (Cloppet & Boucher, 2008; Ishrad et al., 2013). Thus, extracting reliable and accurate nuclear boundaries in regions of overlap may be problematic with the majority of solutions currently available on the market.

Overlaps and touching regions in the whole-slide images may result in the inaccurate identification of nuclear boundaries and may cause either under-segmentation, i.e. the segmenting overlapping nuclei as one, or over-segmentation, i.e. segmenting one single nucleus as multiple ones (Mahmood et al.,  2019;  Zhou et al.,  2019).

Nuclei segmentation overlapping nuclei
Pap smear image ©MIS. See how cell and nuclei boundaries touch and overlap.

Several studies have attempted to address this issue using non-DL methods watershed transform algorithms (Cloppet & Boucher, 2008) or active shape models (ASM) (Plissiti & Nikou, 2012). DL-based models for splitting touching and overlapping nuclei areas are more scarce and often involve complementing the DL-algorithm with traditional techniques like seeded watershed (Kowal et al., 2020).

Yet, most DL-algorithms for splitting overlapping nuclei used in literature are not completely technically mature and still require additional postprocessing steps like binary map transformation and thresholding (Cui et al., 2019).

Recent advances in the field provide a solution to this common problem researchers face. Instance segmentation is often pointed out as a proven approach to split touching or overlapping nuclei. Instance segmentation is an especially robust method in such cases as it is a contour-aware network and particularly sensitive to the spatial and the textural dependencies between cell nuclei (Zhou et al., 2019).

Tips and tricks

Use the instance segmentation method to avoid the issue of over-or under-segmentation of the nucleus instances in your dataset.

Summing up here is a brief checklist to help you improve the outcomes of your analysis:

  • Provide a variety of training images
  • Select representative input images carefully
  • Augment the input images to obtain the optimal view
  • Perform color transformation to avoid staining-related variations
  • Consider partial annotations or annotating regions-of-interest (ROIs) to save time
  • Consider instance segmentation to split touching or heavily clustered nuclei

The use of deep-learning nuclei segmentation techniques in histopathology image analysis

We take a closer look at the uses of DL-based nucleus segmentation in the field of histopathology. In histopathology studies nuclear segmentation is an essential task, which enables nuclei morphology analysis, cell type classification as well as cancer detection and grading.

One of the biggest challenges that DL-based methods face during the analysis of histopathology image data is the segmenting of abnormal cell structures. Abnormalities in the morphological characteristics of the nuclei such as size, shape, texture, color, volume, improper distribution of nuclear to cytoplasmic ratio serve as indicators of pathological processes detected in histology images (Ishrad et al., 2013; Rączkowski et al.,  2019).

A large body of histopathology nuclei segmentation research work sets a thematic focus on two types of nuclei: lymphocyte nuclei and epithelial nuclei, since these two types are often associated with inflammatory processes on a cellular level (Ishrad et al., 2013). Nuclei segmentation studies using histopathological images have already been performed on different types of tissues, the most common ones being breast, cervix and prostate tissue.

Cell nuclei segmentation deep learning prostate tissue
Cell nuclei in prostate mouse tissue.

Nuclear segmentation on the basis of histopathology image data often poses a challenge to researchers and pathologists due to:

  • color and contrast variations
  • the presence of staining artifacts
  • the morphological variability of nucleus shapes
  • the occurrence of clustered structures and overlapping regions (Ishrad et al., 2013).

Furthermore, non-deep-learning computer vision approaches often fail to correctly segment the nuclei of cancerous or malignant cells, since these algorithms have been previously trained solely on a benign cell sample and lack the ability to adapt to novel data.

A DL-model can alleviate this problem as it automatically extracts features from raw data. Using a DL-based method proves to be more effective, due to the ability of DL-algorithms to autonomously and adaptively learn from novel image data. Furthermore, DL-methods are more robust and showcase a more reliable performance when it comes to object classification, detection, and segmentation. In particular, a better predictive performance can be achieved (Araujo et al., 2019; Wang et al., 2020).

This is how deep learning methods for cell nuclei segmentation work.
This is how deep learning methods for cell nuclei segmentation work.
Place your alt text here
This is how deep learning methods for cell nuclei segmentation work.

In histopathology studies the number of cells undergoing mitosis i.e. cell division is often reported as an indicator of tumor severity. That is why it is essential to precisely track instances of mitotic nuclei. Yet, the segmentation of nuclei undergoing mitosis is also a difficult task due to variability in the morphology and texture of mitotic cells (Wang et al., 2014).

So far nuclei segmentation studies in histopathology have been conducted on different tissue types: skin tissue / basal cells (Cruz-Roa et al., 2013), breast tissue (Naylor et. al., 2017), breast tissue / mitotic cells (Wang et al., 2014; Saha et al. 2018), breast tissue / lymphocyte cells (Lu et al. 2020), prostate tissue (Ren et al. 2017; Ali et al., 2020), cervix tissue (Araujo et al. 2019), multiorgan tissue (Kang et. al. 2019; Cui et al., 2019; Jung et al. 2019). 

Several histopathology studies even suggest that the same trained algorithms often yield different performance metrics for tissues from different organs (Kang et al., 2019; Cui et al., 2019). That is why there is a high demand for effective nuclei segmentation methods which can be generalized across various cell, tissue and organ types.

The over- or under-segmentation in cases of touching and overlapping nuclei, the misidentification of micronuclei and mitotic nuclei pose different types of issues. Prior research points at differences in the performance of the algorithm when running the algorithm on benign and malignant samples. For example, Ali et al. (2020) report that the algorithm achieves higher scores in precision, recall and F-measure in the case of malignant cells  as compared to benign cells.

In order to deal with these issues further post-processing and intervention are usually necessary (Mahmood et al.,  2019; Caicedo et al.,  2019). 

Nucleus segmentation in prostate tissue samples

Several studies have laid an emphasis on segmenting nuclei in prostate biopsy tissue samples (Ren et al., 2017, Ali et al., 2020). When analyzing prostate tissue glandular structures such as epithelial nuclei, stromal nuclei, epithelial cytoplasm, stromal cytoplasm and lumen are put in focus. The accurate nuclei segmentation in the histopathology analysis of prostate tissue biopsies is a decisive factor in the diagnosis and grading of prostate cancer. Epithelial nuclei concentration on the boundaries of the prostate gland indicate that the structure of the tissue is intact and benign. The spread of epithelial nuclei with irregular shapes across the stroma areas can indicate that the biopsy sample is malignant.

Prostate tissue microscopy
Prostate tissue microscopy image.

The procedure proposed by Ali et al. (2020) employs a few preprocessing steps i.e. affine transformation prior to performing a coarse segmentation of the input images. The next stage involves the extraction of glandular regions of the tissue. While doing so, benign tissue samples are distinguished from malignant ones.

More recent approaches also apply both boundary segmentation and region growing, while employing  prior knowledge on the specifics of prostate tissue structure. For the purposes of Gleason scoring, which is typically used for prostate cancer grading, morphological features like glandular growth patterns and the degree of differentiation have to be considered as seen under the microscope at low magnification.

The nuclei in neoplastic lesions are often enlarged and show prominent nucleoli, however, may also exhibit variations in size and shape, while mitotic figures are in general extremely uncommon. By using deep learning approaches, according to Bhattacharjee et al. (2019), a high accuracy for classifying input images into benign vs. malignant samples (81%), grade 3 vs. grades 4 and 5 prostate cancer samples (75%) and grade 4 vs. grade 5 prostate cancer samples  (76%) can be reported.

multistage segmentation approach proves to be more efficient as it allows for an easier differentiation between epithelial nuclei and stroma nuclei in standard H&E stained images (Ali et al.,  2020). 

Annotated mouse prostate microscopy images
Annotated mouse prostate microscopy images.

In the case of malignant prostate tissue samples, cell nuclei have a different appearance based on the pathological pattern associated with the dataset. With the help of advanced deep learning techniques you can extract reliable quantitative information on such specific morphological features of cell nuclei in prostate tissue datasets.

Mouse prostate tissue deep learning segmentation model
Mouse prostate tissue deep learning segmentation model.

Developing new nucleus segmentation algorithms based on your own image data may be time-consuming and requires programming skills. Some of the deep learning architectures we have presented above are available in open source libraries, however, adjusting them for the purposes of the research design still requires expert knowledge.

Extract reliable nuclei segmentation data with Sparkfinder

Our original bioimage analysis application Sparkfinder can help you master complex nuclei segmentation tasks with ease. Developed especially for the study of multichannel fluorescence images, Sparkfinder is an excellent aid when extracting useful quantitative details about the cell nuclei in your dataset including: 

  • color intensity
  • distance
  • and density.

On top of that Sparkfinder proves highly effective for splitting touching and overlapping nuclei thanks to its advanced instance segmentation capabilities. Untap the full potential of Sparkfinder to advance your nuclei segmentation research project.

Instance segmentation of cell nuclei with specialized software
Our Sparkfinder app is specifically designed to solve complex nuclei segmentation tasks with the help of instance segmentation.

Ali, T., Masood, K., Irfan, M., Draz, U., Nagra, A. A., Asif, M., … & Yasin, S. (2020). Multistage Segmentation of Prostate Cancer Tissues Using Sample Entropy Texture Analysis. Entropy, 22(12), 1370.

Al-Kofahi, Y., Zaltsman, A., Graves, R., Marshall, W., & Rusu, M. (2018). A deep learning-based algorithm for 2-D cell segmentation in microscopy images. BMC bioinformatics, 19(1), 1-11.

Allehaibi, K. H. S., Nugroho, L. E., Lazuardi, L., Prabuwono, A. S., & Mantoro, T. (2019). Segmentation and classification of cervical cells using deep learning. IEEE Access, 7, 116925-116941.

Alom, M. Z., Yakopcic, C., Taha, T. M., & Asari, V. K. (2018). Microscopic nuclei classification, segmentation and detection with improved Deep Convolutional Neural Network (DCNN) approaches. arXiv preprint arXiv:1811.03447

Araujo, F. H., Silva, R. R., Ushizima, D. M., Rezende, M. T., Carneiro, C. M., Bianchi, A. G. C., & Medeiros, F. N. (2019). Deep learning for cell image segmentation and ranking. Computerized Medical Imaging and Graphics, 72, 13-21.

Bhattacharjee, S., Park, H. G., Kim, C. H., Prakash, D., Madusanka, N., So, J. H., … & Choi, H. K. (2019). Quantitative analysis of benign and malignant tumors in histopathology: Predicting prostate cancer grading using SVM. Applied Sciences9(15), 2969.

Bruch, R., Rudolf, R., Mikut, R., & Reischl, M. (2020). Evaluation of semi-supervised learning using sparse labeling to segment cell nuclei. Current Directions in Biomedical Engineering, 6(3), 398-401.

Caicedo, J. C., Roth, J., Goodman, A., Becker, T., Karhohs, K. W., Broisin, M., … & Carpenter, A. E. (2019). Evaluation of deep learning strategies for nucleus segmentation in fluorescence images. Cytometry Part A, 95(9), 952-965.

Cloppet, F., & Boucher, A. (2008, December). Segmentation of overlapping/aggregating nuclei cells in biological images. In 2008 19th International Conference on Pattern Recognition (pp. 1-4). IEEE.

Cruz-Roa, A. A., Ovalle, J. E. A., Madabhushi, A., & Osorio, F. A. G. (2013, September). A deep learning architecture for image representation, visual interpretability and automated basal-cell carcinoma cancer detection. In International Conference on Medical Image Computing and Computer-Assisted Intervention (pp. 403-410). Springer, Berlin, Heidelberg.

Cui, Y., Zhang, G., Liu, Z., Xiong, Z., & Hu, J. (2019). A deep learning algorithm for one-step contour aware nuclei segmentation of histopathology images. Medical & biological engineering & computing, 57(9), 2027-2043.

Fujita, S., & Han, X. H. (2020). Cell Detection and Segmentation in Microscopy Images with Improved Mask R-CNN. In ACCV Workshops (pp. 58-70).

Hayakawa, T., Prasath, V. S., Kawanaka, H., Aronow, B. J., & Tsuruoka, S. (2019). Computational nuclei segmentation methods in digital pathology: A survey. Archives of Computational Methods in Engineering, 1-13.

Ho, D. J., Yarlagadda, D. V., D’Alfonso, T. M., Hanna, M. G., Grabenstetter, A., Ntiamoah, P., … & Fuchs, T. J. (2021). Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics, 88, 101866.

Hsu, J., Chiu, W., & Yeung, S. (2021). DARCNN: Domain Adaptive Region-based Convolutional Neural Network for Unsupervised Instance Segmentation in Biomedical Images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 1003-1012).

Ibtehaz, N., & Rahman, M. S. (2020). MultiResUNet: Rethinking the U-Net architecture for multimodal biomedical image segmentation. Neural Networks, 121, 74-87.

Irshad, H., Veillard, A., Roux, L., & Racoceanu, D. (2013). Methods for nuclei detection, segmentation, and classification in digital histopathology: a review—current status and future potential. IEEE reviews in biomedical engineering, 7, 97-114.

Jung, H., Lodhi, B., & Kang, J. (2019). An automatic nuclei segmentation method based on deep convolutional neural networks for histopathology images. BMC Biomedical Engineering, 1(1), 1-12.

Kang, Q., Lao, Q., & Fevens, T. (2019, October). Nuclei segmentation in histopathological images using two-stage learning. In International Conference on Medical Image Computing and Computer-Assisted Intervention (pp. 703-711). Springer, Cham.

Kowal, M., Żejmo, M., Skobel, M., Korbicz, J., & Monczak, R. (2020). Cell nuclei segmentation in cytological images using convolutional neural network and seeded watershed algorithm. Journal of digital imaging, 33(1), 231-242.

Lee, J., Kim, H., Cho, H., Jo, Y., Song, Y., Ahn, D., … & Ye, S. J. (2019). Deep-learning-based label-free segmentation of cell nuclei in time-lapse refractive index tomograms. Ieee Access, 7, 83449-83460.

Liu, D., Zhang, D., Song, Y., Zhang, F., O’Donnell, L., Huang, H., … & Cai, W. (2020). Unsupervised instance segmentation in microscopy images via panoptic domain adaptation and task re-weighting. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 4243-4252).

Long, J., Shelhamer, E., & Darrell, T. (2015). Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 3431-3440).

Lu, Z., Xu, S., Shao, W., Wu, Y., Zhang, J., Han, Z., … & Huang, K. (2020). Deep-learning–based characterization of tumor-infiltrating lymphocytes in breast cancers from histopathology images and multiomics data. JCO clinical cancer informatics, 4, 480-490.

Mahmood, F., Borders, D., Chen, R. J., McKay, G. N., Salimian, K. J., Baras, A., & Durr, N. J. (2019). Deep adversarial training for multi-organ nuclei segmentation in histopathology images. IEEE transactions on medical imaging, 39(11), 3257-3267.

Naylor, P., Laé, M., Reyal, F., & Walter, T. (2017, April). Nuclei segmentation in histopathology images using deep neural networks. In 2017 IEEE 14th international symposium on biomedical imaging (ISBI 2017) (pp. 933-936). IEEE.

Nguyen, K.; Jain, A.K.; Allen, R.L. Automated gland segmentation and classification for Gleason grading of prostate tissue images. In: Proceedings of the 20th International Conference on Pattern Recognition (ICPR), Istanbul, Turkey, 23–26 August 2010; Volume 1, pp. 1497–1500

Plissiti, M. E., & Nikou, C. (2012). Overlapping cell nuclei segmentation using a spatially adaptive active physical model. IEEE Transactions on Image Processing, 21(11), 4568-4580.

Qu, H., Wu, P., Huang, Q., Yi, J., Yan, Z., Li, K., … & Metaxas, D. N. (2020). Weakly supervised deep nuclei segmentation using partial points annotation in histopathology images. IEEE Transactions on Medical Imaging, 39(11), 3655-3666.

Rączkowski, Ł., Możejko, M., Zambonelli, J., & Szczurek, E. (2019). ARA: accurate, reliable and active histopathological image classification framework with Bayesian deep learning. Scientific reports, 9(1), 1-12.

Ren, J., Sadimin, E., Foran, D. J., & Qi, X. (2017, February). Computer aided analysis of prostate histopathology images to support a refined Gleason grading system. In Medical Imaging 2017: Image Processing (Vol. 10133, p. 101331V). International Society for Optics and Photonics.

Ronneberger, O., Fischer, P., & Brox, T. (2015, October). U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention (pp. 234-241). Springer, Cham.

Saha, M., Chakraborty, C., & Racoceanu, D. (2018). Efficient deep learning model for mitosis detection using breast histopathology images. Computerized Medical Imaging and Graphics, 64, 29-40.

Vuola, A. O., Akram, S. U., & Kannala, J. (2019, April). Mask-RCNN and U-net ensembled for nuclei segmentation. In 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019) (pp. 208-212). IEEE.

Wang, H., Xian, M., & Vakanski, A. (2020, April). Bending loss regularized network for nuclei segmentation in histopathology images. In 2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI) (pp. 1-5). IEEE.

Wang, H., Roa, A. C., Basavanhally, A. N., Gilmore, H. L., Shih, N., Feldman, M., … & Madabhushi, A. (2014). Mitosis detection in breast cancer pathology images by combining handcrafted and convolutional neural network features. Journal of Medical Imaging, 1(3), 034003.

Win, K. Y., Choomchuay, S., Hamamoto, K., & Raveesunthornkiat, M. (2018). Comparative study on automated cell nuclei segmentation methods for cytology pleural effusion images. Journal of healthcare engineering, 2018.

Yoo, I., Yoo, D., & Paeng, K. (2019, October). PseudoEdgeNet: Nuclei segmentation only with point annotations. In International Conference on Medical Image Computing and Computer-Assisted Intervention (pp. 731-739). Springer, Cham.

Zhang, L., Sonka, M., Lu, L., Summers, R. M., & Yao, J. (2017, April). Combining fully convolutional networks and graph-based approach for automated segmentation of cervical cell nuclei. In 2017 IEEE 14th international symposium on biomedical imaging (ISBI 2017) (pp. 406-409). IEEE.

Zhang, Z., Leong, K. W., Van Vliet, K., Barbastathis, G., & Ravasio, A. (2021). Deep learning for label-free nuclei detection from implicit phase information of mesenchymal stem cells. Biomedical Optics Express, 12(3), 1683-1706.

Zhou, Y., Onder, O. F., Dou, Q., Tsougenis, E., Chen, H., & Heng, P. A. (2019, June). Cia-net: Robust nuclei instance segmentation with contour-aware information aggregation. In International Conference on Information Processing in Medical Imaging (pp. 682-693). Springer, Cham.

Try Sparkfinder now

You can try Sparkfinder yourself in the trial version of the IKOSA Platform.


Join our newsletter