Navigating Prostate Histology With AI Technology

16 Feb, 2021 | Blog posts, IKOSA AI, Interviews

Learn about the latest developments in prostate tissue histology, a field in which the need for new automated image analysis methods is rapidly increasing. State-of-the-art image segmentation methods assist researchers in detecting pathological prostate conditions, grading/staging cancer, and assessing the effectiveness of new therapeutic agents based on changes in prostate tissue morphology.

In this article, we bring a comprehensive overview of existing segmentation methods in prostate histology research. We talked to experts from the Medical University of Vienna to provide you with valuable insights into how automated prostatic tissue analysis is put into practice.

Understanding normal prostate tissue histology

To get a better grasp of the pathological characteristics of prostate diseases, we need to first take a look at the anatomy of healthy prostate tissue.

As a part of the male reproductive system, the prostate is a gland that plays an important role in the generation of seminal fluid. The prostate is constituted of three histological zones: peripheral, transition, and central zone. Structures adjacent to the prostate are the bladder, the prostatic urethra, the prostatic ducts, and the seminal vesicles.

The human prostate gland consists of acini and ducts that are lined by three distinct types of cells: luminal, basal, and neuroendocrine. Layers of epithelial cells constitute gland duct boundaries and are involved in the generation of seminal fluid, the cavity formed is called a lumen. Those luminal cells are columnar and have round nuclei positioned near the cell base. The prostate can be seen as a set of tubulo-alveolar glands with lumina lined by an epithelium layer of variable height. (Singh et al., 2017)

Prostate zones
Anatomy and zones of the prostate. Image taken from the Creative Commons CC0 1.0 Universal Public Domain Dedication.

A healthy prostate gland does not have a fixed size or shape: it can be smaller or larger, oval, round, or branchy. Benign glands are characterized by large lumina and epithelial cells with prominent nuclei. The nuclei of benign prostatic gland tissue are uniformly dark or uniformly light throughout the areas and do not show prominent nucleoli (Nguyen et al., 2012b).

In the normal prostate, prostatic stroma components vary in each zone. Stromal components include myofibroblasts, fibroblasts, collagen fibers, and smooth muscle cells (Zhang et al., 2003). However, there are local differences in stroma morphology and function (e.g., gene expression) in the different zones of the prostate. Such differences may explain why cancer originates more commonly in the peripheral zone of the prostate and not in the transition zone where benign prostatic hyperplasia (BPH) usually develops. The transition zone accounts for only 10 % of prostate glandular tissue (Hägglöf & Bergh, 2012).

Healthy prostate tissue histology slide
Normal histology of prostate tissue. Image taken from

In cases of cancer, changes to lumen properties occur in the gland. Alterations in the lumen can be used as parameters of interest in digital image analysis. However, prostate cancer can develop in benign basal and luminal stem cells. The aberrant proliferation of basal cells in the prostate ranges from hyperplasia to carcinoma (but carcinoma from basal cells of the prostate is rare).

Standardize your image analysis processes using specialized AI-based software.

Focussing on tissue architecture in prostate cancer histology

Several pathological conditions like prostatitis, prostatic hyperplasia, nodular hyperplasia, prostatic intraepithelial neoplasia, and prostatic adenocarcinoma can occur in prostate tissue. All those conditions are characterized by specific changes in tissue morphology. We take a look at how affected prostate glandular structures appear during histological analysis. 

GradeGleason ScoreCharacteristics
16 (3+3)Individual, discrete, well-formed glands/uniform glands
27 (3+4)Well-formed glands with a small component of poorly formed/fused/cribriform/glomeruloid glands (more stroma between glands)
37 (4+3)Predominantly poorly formed/fused/cribriform/glomeruloid glands with a small component of well-formed glands (distinctly infiltrative margins)
48 (4+4, 3+5, 5+3)Predominantly poorly formed/fused/cribriform/glomeruloid glands with a small component of well-formed glands (distinctly infiltrative margins)
59, 10 (4+5, 5+4, 5+5)Lack of gland formation (+/-necrosis) with or without poorly formed/fused/cribriform glands + sheets of cells
Table 1: Grading of prostate cancer: The characteristics of prostate tissue with regards to Gleason score (adapted from Pudasaini & Subedi 2019

Pathologists use the presence of atypical gland patterns as hallmarks through which they can distinguish cancerous regions from benign ones. The most common system pathologists use to grade prostate cancer is the Gleason score (Gleason, 1966).

Gleasons pattern
The Gleason score for prostate cancer grading. This image is a work of the National Institutes of Health, part of the United States Department of Health and Human Services, taken or made as part of an employee’s official duties. As a work of the U.S. federal government, the image is in the public domain.

The assigned cancer grade shows to what extent the appearance of cancer cells deviates from that of normal healthy cells. To assign the Gleason score or Grade group to a given sample, pathologists look at biopsies taken from the prostate and grade each sample on a scale from 3 to 5. The two dominating Gleason grades are added together to calculate the overall Gleason score, which ranges from 6 to 10 (Chen & Zhou, 2016).  Read more on the histologic grading of prostate cancer with automated methods in our article on Pathologist Advice On Prostate Cancer Microscopy Analysis.

Did you know?

To detect prostate cancer with the help of digital histological images, the characteristics of certain nuclear, cytoplasmic, and intraluminal features of the glandular region need to be assessed.

Prostate Cancer Stages Infographic
Spread of cancer from the prostate to adjacent organs and tissues. Image created with BioRender.

​​​Detecting changes to nuclear features

Compared to normal prostate epithelial cells, the nuclei in cancerous prostate cells show differences in morphology and staining intensity and alterations of nuclear characteristics. Nuclear enlargement and the presence of prominent nucleoli are typical characteristics observed in cancer-affected specimens. Other peculiarities prominent in cancerous prostatic tissue are nucleus shape irregularities or nuclear dysmorphia. These changes are caused by alterations in the nuclear lamina resulting in lobes and herniations.

Cancerous prostate tissue histology slide
Cancerous prostate tissue histology slide.

Textural changes in the structure of atypical nuclei are mostly due to alterations on a DNA level, which for example can be detected during histological analysis with specific staining methods such as fluorescence in situ hybridization (FISH). (Carleton et al., 2018)

Cancer-affected regions display smaller nuclei count on the boundary of the gland as compared to non-cancerous regions. Cancerous glands tend to have only one nuclear layer on the boundary, while a normal gland contains multiple layers. Those nuclei can be characterized by a lighter blue color than the color of nuclei in healthy glands (Nguyen et al., 2012).

Furthermore, cancerous prostate tissues might display a cribriform pattern, where a number of glands are fused into one, resulting in the formation of nuclei clusters (Singh et al., 2017).

Prostate cancer with Gleason score 7 with minor component of cribriform glands
Prostate cancer with Gleason score 7 (3+4) showing minor components of cribriform glands. Image is taken from the Creative Commons Attribution 4.0 International license.

Cribriform within this context refers to a neoplastic epithelial proliferation resulting in large nests perforated by different-sized rounded spaces (Branca et al., 2017). This is an important characteristic because the presence of a cribriform growth pattern in radical prostatectomy specimens has been associated with distant metastasis and disease-specific mortality in patients with a Gleason score of 7 or higher (Kweldam et al., 2018).

Consequently, the presence of a cribriform pattern is now recognized as a clinically important, independent adverse prognostic indicator of prostate cancer (Branca et al., 2017). 

IKOSA is your research partner, ready to assist you.

Changes to lumen properties

Lumina in atypical glands tend to be smaller in size and more circular than in normal glands. In conditions such as basal cell hyperplasia blue mucin and eosinophilic secretions can be observed in lumina which is marked by a distinct coloring on histology slides. The presence of luminal mucin can be detected with special stains such as Alcian blue. (Trpkov, 2018; Nguyen et al., 2012b).

Existing studies also suggest that in cases of cancer with a higher Gleason grade the density and volume of lumen can be reduced  (Chatterjee et al., 2015; McGarry et al., 2018).

Changes to prostatic stroma

Cancerous prostatic tissue is composed of malignant epithelial cells and supportive stroma whose changes are important for the development of the tumor (Krušlin et al., 2015). Recent research suggests that a decreasing volume of stroma can be observed in cancer-affected prostate glands while the number of tumor cells increases (Chatterjee et al., 2015).

Cancerous stroma displays an increased number of fibroblasts and myofibroblasts. They play a significant role in the synthesis, deposition, and remodeling of the extracellular matrix and are in constant interaction with tumorous epithelial cells (Krušlin et al., 2015).

With the help of various molecules present in the extracellular matrix (ECM), a microenvironment suitable for cancer cell proliferation, movement, and differentiation is created promoting tumor growth. The complex interaction between cancer cells and various cells in the stroma plays a central role when it comes to the enhancement of tumor progression. This process is a key factor in stimulating angiogenesis and preserving cancer cell survival, proliferation, and invasion (Krušlin et al., 2015).

How does the automated histologic analysis of prostate tissue work?

Segmentation methodologies involve detecting and separating objects and structures of interest in prostate tissue images. With the help of specialized software applications researchers can conduct an accurate segmentation of prostate tissue. These automated software tools rely on state-of-the-art deep learning technology and facilitate significantly faster and more efficient data collection than conventional manual methods.

How to leverage  Artificial Intelligence  for prostate histology image analysis

  • Prostate cancer detection and grading 
  • Detection of atypical patterns in digital prostate slides 
  • Classification of  features of interest such as lumen and nuclei
  • Gathering quantitative information from digital prostate pathology slides

Various prostate segmentation models have been suggested in existing literature. Some of these methods rely on MR imaging data and are applied for tasks such as localizing prostate boundaries and zones, obtaining volume-related metrics, and tracking disease progression. Prostate zone segmentation is used to determine cancer lesion invasion towards adjacent structures such as the urethra and the seminal vesicles. (Litjens et al., 2014; Zhu et al., 2017)

Other methods relying on histology slide data and automated histopathological image analysis allow researchers to obtain valuable quantitative information on the structural features of prostate tissue. Different deep-learning techniques for epithelium segmentation, nucleus segmentation, stroma and gland segmentation, and lumen object segmentation have been discussed in existing literature (Nguyen et al., 2012; Carleton et al., 2018; Bulten et al., 2019).

Did you know?

Increase your knowledge about the use of deep-learning segmentation techniques in histopathology image analysis by reading our blogpost on the methodology essentials.

nuclei segmentation on mouse prostate tissue using IKOSA AI
Example of effective nuclei segmentation on mouse prostate tissue using IKOSA AI,

AI-backed methods also enable researchers to reliably classify specimens into the different stages of prostate cancer (Nguyen et al., 2012b). Yet, the varying shapes and sizes of prostatic glands often pose a major challenge to common segmentation techniques (Singh et al., 2017).

Using advanced computational models quantitative data related to prostatic tissue pathology can be collected. Such parameters include measures of count, size, shape, and texture of tissue components as well as measures of the spatial information about the cellular microenvironment (Bhargava & Madabhushi, 2016).

Selecting the right set of features to label is an essential step in the analysis of prostate tissue slides. Using automated image analysis tools you can study aspects like the pathology of cells, spatial arrangement of glands, classify different subtypes of cancer, and quantify various parameters related to morphology, texture, color, and topology. (Ayyad et al., 2021). Here is an overview of metrics collected with AI-based methods for prostate tissue segmentation:

ParameterMorphological structures
countnuclei count, epithelial cells count, lumen objects count, stromal cells count
areanuclei area, lumen area, epithelial cell area, stromal area
densitynucleus density, lumen density, epithelial cell density, stromal cell density
circularitynucleus circularity, lumen objects circularity, epithelial cell circularity, stromal cell circularity
sizenucleus size, lumen object size, epithelial cell size, stromal cell size 
volumenuclear volume, lumen space volume, epithelium space volume, stroma space volume 
Table 2: Parameters used when assessing the morphological features of prostate gland components with artificial intelligence methods.

Pathology image analysis becomes easier with assistive AI-technologies.

Experts in prostate tissue histology share their experience with the IKOSA software

We contacted Prof. Johannes Schmid and Bernhard Hochreiter, PhD, researchers at the Institute of Vascular Biology and Thrombosis Research at the Medical University of Vienna, and asked them to share their experience with the IKOSA Platform regarding the automated histologic analysis of prostate tissue.  Here is what the research team reported about their recent study on prostate cancer with the help of advanced computational methods. 

Prostate tissue histology staining
Fluorescence antibody staining of mouse prostate tissue: Nuclei(blue), IKK1 (green) and c-Myc(red). Image taken from Moser et al. (2021).

On the benefits of the IKOSA software

When asked about the benefits of the IKOSA software, Bernhard Hochreiter noted that he was particularly pleased with the many useful capabilities included in the IKOSA software. Especially, the option to view and process image files of different sizes including large images had proven to be very helpful. Transforming images taken with different microscope modalities into a uniform size was no longer necessary.

I was pleasantly surprised that images of various sizes can be easily managed on the web platform.

Prof. Johannes Schmidt, Researcher, Institute of Vascular Biology and Thrombosis Research at the Medical University of Vienna

The researcher also agrees that the entire analysis workflow has been faster and smoother since they implemented the IKOSA software in their lab. Being able to work remotely and perform the analysis of histologic data on an online platform has been an invaluable asset, especially during the COVID-19 pandemic.

The uses of IKOSA in applied histopathological prostate research

The researchers report how the use of the IKOSA platform helped them conduct the large-scale research project “FFG-BRIDGE Precision Histology”. The dataset used for the project consisted of complex microscopy data acquired with different imaging modalities. Prostate tissue images taken with multichannel fluorescence microscopes constituted the larger part of the dataset. This required an image analysis tool that supported these image formats and was flexible enough to adjust the different channels

prostate tissue histology multichannel imaging
Multichannel image of fluorescently stained mouse prostate tissue.

Yet, as Dr. Hochreiter explains, the biggest asset of the IKOSA platform turned out to be its AI-backed analysis capability. It has helped the team avoid many mistakes, which might have occurred during manual analysis tasks. Image analysis automation has proven invaluable during the segmentation of prostate cell nuclei.

The significant advantage is the support for image analysis using artificial intelligence and machine learning. The use of IKOSA can help avoid mistakes that could affect the research results.

Bernhard Hochreiter PhD, Researcher, Institute of Vascular Biology and Thrombosis Research at the Medical University of Vienna

Use existing AI apps or create your own. No coding skills are required.

The “FFG-BRIDGE Precision Histology” study

Due to the scarcity of human prostate tissue biopsy samples and ethical considerations using patient data a large body of pre-clinical research has been conducted on animal prostate tissue samples. Making use of the similarities in mammalian prostate anatomy, a significant number of articles on prostate histology rely on mouse tissue samples (Fagerland et al., 2020; Ding et al., 2021).

Similarly, the research team at the Medical University of Vienna used mouse prostate tissue histological images acquired with different microscope modalities like brightfield and fluorescence microscopy. 

The project resulted in a publication in the Molecular Cancer Journal on the effects of inflammatory kinase IKKα complexes on inflammatory transcription in prostate cancer.

The study involved analyzing the effects of IKK enzymes on the expression of the oncogene c-Myc protein in epithelial prostate cells. To obtain this information cell nuclei have been stained using fluorescent dyes and automatically segmented with the help of image analysis software. Thus, reliable quantitative information on c-Myc protein expression was collected.

The authors propose a model on how IKK enzymes interact with c-Myc within prostate cell nuclei and suggest a positive correlation between c-Myc and IKKα levels in mouse prostate epithelial cells. The article provides evidence that c-Myc phosphorylates IKKs and regulates gene expression in mouse prostate nuclei. This leads to increased transcriptional activity, higher proliferation, and decreased apoptosis (Moser et al., 2021).

Experience the power of AI-aided histological image analysis for the study of prostate tissue

If you are interested in conducting your prostate tissue analysis, you can try out the IKOSA platform yourself for free. In case you have further questions on how to implement the IKOSA software in your prostate tissue analysis project, feel free to contact us at

Full interview provided in the PDF file below. No email required.

*The following PDF interview document contains screenshots of the IKOSA platform from 2020. The actual interface of IKOSA may look different due to numerous enhancements to the platform.

Our authors:

KML Vision Team Benjamin Obexer Lead Content Writer

Benjamin Obexer

Lead content writer, life science professional, and simply a passionate person about technology in healthcare

KML Vision Team Elisa Opriessnig Content writer

Elisa Opriessnig

Content writer focused on the technological advancements in healthcare such as digital health literacy and telemedicine.

KML Vision Team Fanny Dobrenova Marketing Specialist

Fanny Dobrenova

Health communications and marketing expert dedicated to delivering the latest topics in life science technology to healthcare professionals.


Ayyad, S. M., Shehata, M., Shalaby, A., Abou El-Ghar, M., Ghazal, M., El-Melegy, M., … & El-Baz, A. (2021). Role of AI and histopathological images in detecting prostate cancer: a survey. Sensors, 21(8), 2586.

Bhargava, R., Madabhushi, A. (2016). Emerging themes in image informatics and molecular analysis for digital pathology. Annual review of biomedical engineering, 18, 387–412.    

Branca, G., Ieni, A., Barresi, V., Tuccari, G., Caruso, RA. (2017). An Updated Review of Cribriform Carcinomas with Emphasis on Histopathological Diagnosis and Prognostic Significance. Oncol Rev.,11(1),317. Published 2017 Mar 10. doi:10.4081/oncol.2017.317.

Bulten, W., Bándi, P., Hoven, J., Loo, RVD, Lotz, J., Weiss, N., … & Litjens, G. (2019). Epithelium segmentation using deep learning in H&E-stained prostate specimens with immunohistochemistry as reference standard. Scientific reports, 9(1), 1-10. 

Carleton, NM., Lee, G., Madabhushi, A., & Veltri, R. W. (2018). Advances in the computational and molecular understanding of the prostate cancer cell nucleus. Journal of cellular biochemistry, 119(9), 7127-7142. 

Chatterjee, A., Watson, G., Myint, E., Sved, P., McEntee, M., & Bourne, R. (2015). Changes in epithelium, stroma, and lumen space correlate more strongly with Gleason pattern and are stronger predictors of prostate ADC changes than cellularity metrics. Radiology, 277(3), 751-762.

Chen, N., & Zhou, Q. (2016). The evolving Gleason grading system. Chinese journal of cancer research = Chung-kuo yen cheng yen chiu, 28(1), 58–64.

Denmeade SR, Isaacs JT. (2003). Cellular Organization of the Normal Prostate. In: Kufe DW, Pollock RE, Weichselbaum RR, et al., editors. Holland-Frei Cancer Medicine. 6th edition. Hamilton (ON): BC Decker; 2003.

Ding, Y., Lee, M., Gao, Y., Bu, P., Coarfa, C., Miles, B., … & Ayala, G. (2021). Neuropeptide Y nerve paracrine regulation of prostate cancer oncogenesis and therapy resistance. The Prostate, 81(1), 58-71. 

Fagerland, S. M. T., Hill, D. K., van Wamel, A., de Lange Davies, C., & Kim, J. (2020). Ultrasound and magnetic resonance imaging for group stratification and treatment monitoring in the transgenic adenocarcinoma of the mouse prostate model. The Prostate, 80(2), 186-197.  

Gleason, HF. (1966). Classification of Prostatic Carcinoma. Cancer Chemother Rep, 50, 125-128.

Hägglöf, C., & Bergh, A. (2012). The stroma-a key regulator in prostate function and malignancy. Cancers, 4(2), 531–548. 

Krušlin, B., Ulamec, M., & Tomas, D. (2015). Prostate cancer stroma: an important factor in cancer growth and progression. Bosnian journal of basic medical sciences, 15(2), 1–8. 

Kweldam, CF., van der Kwast, T., van Leenders, GJ. (2018). On cribriform prostate cancer. Transl Androl Urol, 7(1), 145-154. doi:10.21037/tau.2017.12.33.

Litjens, G., Toth, R., van de Ven, W., Hoeks, C., Kerkstra, S., van Ginneken, B., … & Madabhushi, A. (2014). Evaluation of prostate segmentation algorithms for MRI: the PROMISE12 challenge. Medical image analysis, 18(2), 359-373.

McGarry, S. D., Hurrell, S. L., Iczkowski, K. A., Hall, W., Kaczmarowski, A. L., Banerjee, A., … & LaViolette, P. S. (2018). Radio-pathomic maps of epithelium and lumen density predict the location of high-grade prostate cancer. International Journal of Radiation Oncology* Biology* Physics, 101(5), 1179-1187. 

Moser, B., Hochreiter, B., Basílio, J., Gleitsmann, V., Panhuber, A., Pardo-Garcia, A., … & Schmid, J. A. (2021). The inflammatory kinase IKKα phosphorylates and stabilizes c-Myc and enhances its activity. Molecular cancer, 20(1), 1-17. 

Nguyen, K., Sarkar, A., & Jain, A. K. (2012). Structure and context in prostatic gland segmentation and classification. International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, Berlin, Heidelberg. 115-123.

Nguyen, K., Sabata, B., & Jain, A. K. (2012b). Prostate cancer grading: Gland segmentation and structural features. Pattern Recognition Letters, 33(7), 951-961. 

Pudasaini, S., & Subedi, N. (2019). Understanding the gleason grading system and its changes. Journal of Pathology of Nepal.

Shibuya, T., Takahashi, G., & Kan, T. (2019). Basal cell carcinoma of the prostate: A case report and review of the literature. Molecular and clinical oncology, 10(1), 101–104.

Singh, M., Kalaw, EM., Giron, DM., Chong, KT., Tan, CL., & Lee, HK. (2017). Gland segmentation in prostate histopathological images. Journal of medical imaging, 4(2), 027501.

Trpkov, K. (2018). Benign mimics of prostatic adenocarcinoma. Modern Pathology, 31, 22-46.

Zhang, Y., Nojima, S., Nakayama, H., Jin, Y., Enza, H. (2003). Characteristics of normal stromal components and their correlation with cancer occurrence in human prostate. Oncol Rep, 10(1), 207-11. PMID: 12469170.Zhu, Q., Du, B., Turkbey, B., Choyke, P. L., & Yan, P. (2017). Deeply-supervised CNN for prostate segmentation. International joint conference on neural networks (IJCNN), 178-184.


Join our newsletter