Prostate Tissue Histology with Computational Methods

16 Feb, 2021 | Blog posts, IKOSA AI, Interviews

Learn about the latest developments in prostate tissue histology, where the need for new automated image analysis methods is rapidly increasing. We dive deep into this exciting subject to keep you updated with the latest developments in the field. State-of-the-art image segmentation methods assist researchers in detecting pathological prostate conditions, grading/staging cancer and assessing the effectiveness of new therapeutic agents based on changes in prostate tissue morphology. 

In this article, we bring an extensive overview of existing segmentation methods in prostate histology research. We talked to experts in prostate histology research from the Medical University of Vienna to provide you valuable insights into how automated prostatic tissue analysis is put into practice. 

Normal prostate tissue histology

To better understand pathological changes characteristic of prostate diseases, we need to first take a look at the anatomy of healthy prostate tissue. 

As a part of the male reproductive system, the prostate is a gland that plays an important role in the generation of seminal fluid. The prostate is constituted of three histological zones: peripheral, transition and central zone. Structures adjacent to the prostate are the bladder, the prostatic urethra, the prostatic ducts and the seminal vesicles.   

Prostate gland tissue is composed of four components: nuclei, lumen, stroma and cytoplasm. Layers of epithelial cells constitute the gland boundaries and are involved in the generation of seminal fluid. Those cells are columnar in shape and have round nuclei positioned near the cell base. Lumen objects are described as empty white spaces within the glandular structure. The prostate basically is a set of tubulo-alveolar glands with lumina lined by epithelium of variable height (Singh et al., 2017).

Prostate zones
Zones of the prostate. Image taken from the Creative Commons CC0 1.0 Universal Public Domain Dedication.

The glandular tissue of the prostate includes secretory cells on the glandular facing side and basal cells on the basal side. The epithelial cells line a central lumen, which is filled with fluid produced by the epithelial cells (Denmeade & Isaacs, 2003).

A healthy prostate gland does not have a fixed size or shape: it can be smaller or larger, oval, round or branchy. Benign glands are characterized by large lumina and epithelial cells with prominent nuclei. The nuclei of benign prostatic gland tissue are uniformly dark or uniformly light throughout the areas and do not show prominent nucleoli (Nguyen et al., 2012b).                

In the normal prostate, prostatic stroma components vary in each zone. Stromal components include myofibroblasts, fibroblasts, collagen fibers and smooth muscle cells (Zhang et al., 2003). However, there are local differences in stroma morphology and function (e.g., gene expression) in the different zones of the prostate. Such differences may explain why cancer originates more commonly in the peripheral zone of the prostate and not in the transition zone where benign prostatic hyperplasia (BPH) usually develops. The transition zone accounts for only 10 % of prostate glandular tissue (Hägglöf & Bergh, 2012).

Healthy prostate tissue histology slide
Healthy prostate tissue histology slide. Image taken from

In cases of cancer, changes to lumen properties occur in the gland, which can be used as a feature in digital image analysis. However, prostate cancer can develop in benign basal and luminal stem cells. The aberrant proliferation of basal cells in the prostate ranges from hyperplasia to carcinoma (but carcinoma from basal cells of the prostate is rare).

Examine channel intensities of multichannel fluorescence images while segmenting cell nuclei and virtual cytoplasm with our Sparkfinder Application.

Focusing on the tissue architecture in prostate cancer histology

Several pathological conditions like prostatitis, prostatic hyperplasia, nodular hyperplasia, prostatic intraepithelial neoplasia and prostatic adenocarcinoma can occur in prostate tissue. All those conditions are characterized by specific changes in tissue morphology. We take a look at how prostate glandular structures affected appear during histological analysis. 

GradeGleason ScoreCharacteristics
16 (3+3)Individual, discrete, well-formed glands or uniform glands
27 (3+4)Well-formed glands with small component of poorly formed, fused, cribriform or glomeruloid glands (more stroma between glands)
37 (4+3)Predominantly poorly formed, fused, cribriform or glomeruloid glands with small component of well-formed glands (distinctly infiltrative margins)
48 (4+4, 3+5, 5+3)Predominantly poorly formed, fused, cribriform or glomeruloid glands with small component of well-formed glands (distinctly infiltrative margins)
59, 10 (4+5, 5+4, 5+5)Lack of gland formation (+/-necrosis) with or without poorly formed, fused or cribriform glands + sheets of cells
Table 1: The characteristics of prostate tissue with regards to Gleason score (adapted from Pudasaini & Subedi 2019

Pathologists use the presence of atypical gland patterns as hallmarks through which they can distinguish cancerous regions from benign ones. The most common system pathologists use to grade prostate cancer is the Gleason score (Gleason, 1966).

The assigned cancer grade shows to what extent the appearance of cancer cells deviates from that of normal healthy cells. To assign  the Gleason score or Grade group to a given sample, pathologists look at biopsies taken from the prostate and grade each sample on a scale from 3 to 5. 

The two dominating Gleason grades are added together to calculate the overall Gleason score, which ranges from 6 to 10 (Chen & Zhou, 2016). 

Read more on the histologic grading of prostate cancer with automated methods in our blogpost on Pathologist advice on Prostate Cancer Microscopy Analysis.

Prostate Cancer Stages Infographic
Spread of cancer from the prostate to adjacent organs and tissues. Image created with BioRender. 

Tips and tricks

To detect prostate cancer with the help of digital histological images the characteristics of certain nuclear, cytoplasmic and intraluminal features of the glandular region need to be assessed. 

​​​​Changes to nuclear features

Compared to normal prostate epithelial cells, the nuclei in cancerous prostate cancer cells are colored light blue and have prominent nucleoli appearing as small dark spots within the nucleus.  

Did you know?

Alterations of nuclear features such as size, shape and texture can be observed in cancerous prostate tissue. Nuclear enlargement and the presence of prominent nucleoli are characteristic of cancer affected specimens. 

Another feature prominent in prostatic tissue affected by cancer is nucleus shape irregularities or nuclear dysmorphia. These changes are caused by alterations in the nuclear lamina resulting in lobes and herniations.

Textural changes in the structure of nuclei are mostly due to alterations on the DNA level, including  gene fusions, mutations, copy number variations and translocations. These can be detected with specific staining methods such as fluorescence in situ hybridisation during histological analysis (Carleton et al., 2018). 

Further, glands in the cancer-affected region display a smaller nuclei count on the boundary of the gland as compared to non-cancerous regions. Cancerous glands tend to have only one nuclear layer on the boundary, while a normal gland contains multiple layers.Those nuclei are also characterized by a lighter blue color than those in healthy glands (Nguyen et al., 2012).

Also, cancerous prostate tissues might often display a cribriform pattern, where a number of glands are fused into one which results in the formation of nuclei clusters(Singh et al., 2017).

Prostate cancer with Gleason score 7 with minor component of cribriform glands
Prostate cancer with Gleason score 7 (3+4) with minor components of cribriform glands. Image is taken from  the Creative Commons Attribution 4.0 International license.

Cribriform refers, within this context, to a neoplastic epithelial proliferation in the form of large nests perforated by different-sized quite rounded spaces (Branca et al., 2017). This is important to note because the presence of a cribriform growth pattern in radical prostatectomy specimens has been previously associated with distant metastasis and disease-specific death from prostate cancer in patients with Gleason score 7 or higher (Kweldam et al., 2018).

Did you know?

Consequently, the presence of a cribriform pattern is now recognised as a clinically important, independent adverse prognostic indicator of prostate cancer (Branca et al., 2017).

Cancerous prostate tissue histology slide
Cancerous prostate tissue histology slide. Nuclei are prominently marked in dark blue. Image taken from

Examine channel intensities of multichannel fluorescence images while segmenting cell nuclei and virtual cytoplasm with our Sparkfinder Application.

Changes to lumen properties 

Lumen objects in atypical glands tend to be smaller in size and more circular than in normal glands. Blue stained mucin invading lumen objects might result in a distinct coloring on histology slides (Nguyen et al., 2012b).

Existing studies suggest that in cases of higher Gleason grade cancer a lesser density of lumen objects and a decreasing volume of lumen space can be observed in the affected tissue (Chatterjee et al., 2015; McGarry et al., 2018).

Gleason's pattern
Gleason score for prostate cancer grading. This image is a work of the National Institutes of Health, part of the United States Department of Health and Human Services, taken or made as part of an employee’s official duties. As a work of the U.S. federal government, the image is in the public domain.

Changes to prostatic stroma

Cancerous prostatic tissue is composed of malignant epithelial cells and supportive stroma whose changes are important for the development of the tumor (Krušlin et al., 2015).

Did you know?

Recent research suggests that a decreasing volume of stroma can be observed in cancer-affected prostate glands while the number of tumor cells increases (Chatterjee et al., 2015). 

Cancerous stroma consists of fibroblasts, myofibroblasts, endothelial cells and immune cells, but fibroblasts and myofibroblasts are the predominant cellular entities. They play a significant role in the synthesis, deposition and remodeling of the extracellular matrix (ECM) and are in constant interaction with tumorous epithelial cells (Krušlin et al., 2015). 

With the help of various molecules of the ECM a microenvironment suitable for cancer cell proliferation, movement and differentiation is created promoting tumor growth. A complex interaction between cancer cells and various cells in the stroma plays a central role when it comes to the enhancement of tumor progression. This process is a key factor in stimulating angiogenesis and preserving cancer cell survival, proliferation and invasion (Krušlin et al., 2015). 

How does the automated histologic analysis of prostate tissue work?

Segmentation methodologies involve detecting and separating objects and structuresof interest in prostate tissue images. With the help of specialized software applications researchers are able to conduct an accurate segmentation of prostate tissue. These automated applications rely on state-of-the-art deep learning technology and facilitate a significantly faster and more efficient data collection than conventional manual methods. 

Various prostate segmentation models have been suggested in existing literature. Some of these methods rely on MR imaging data and are applied for tasks such as localizing prostate boundaries and zones, obtaining volume-related metrics and tracking disease progression. Prostate zone segmentation is used to determine cancer lesion invasion towards adjacent structures such as the urethra and the seminal vesicles (Litjens et al., 2014; Zhu et al., 2017).

Other methods relying on histology slide data and automated histopathological image analysis allow researchers to obtain valuable quantitative information on the structural features of prostate tissue. Different deep learning techniques for epithelium segmentation, nucleus segmentation, stroma and gland segmentation and lumen objects segmentation have been discussed in literature (Nguyen et al., 2012; Carleton et al.,  2018; Bulten et al., 2019).

Increase your knowledge about the use of deep-learning segmentation techniques in histopathology image analysis further by reading our blogpost on the methodology essentials

AI-backed methods also enable researchers to reliably classify specimens into the different stages of prostate cancer (Nguyen et al., 2012b). Yet, the varying shapes and sizes of prostatic glands often pose a major challenge to common segmentation techniques (Singh et al., 2017). 

However, with the help of advanced computational methods quantitative data related to prostatic tissue alterations can be collected. Such parameters include measures of count, size, shape and texture of tissue components as well as measures of the spatial information about the cellular microenvironment (Bhargava & Madabhushi,  2016).

Here is an overview of metrics collected with AI-based methods for prostate tissue segmentation:

ParameterMorphological structures
countnuclei count, epithelial cells count, lumen objects count, stromal cells count
areanuclei area, lumen area, epithelial cell area, stromal area
densitynucleus density, lumen density, epithelial cell density, stromal cell density
circularitynucleus circularity, lumen objects circularity, epithelial cell circularity, stromal cell circularity
sizenucleus size, lumen object size, epithelial cell size, stromal cell size 
volumenuclear volume, lumen space volume, epithelium space volume, stroma space volume 
Table 2: Parameters used when assessing the morphological features of prostate gland components with computational methods.

Examine channel intensities of multichannel fluorescence images while segmenting cell nuclei and virtual cytoplasm with our Sparkfinder Application.

Experts in prostate tissue histology share their experience with the IKOSA software

We contacted Prof. Johannes Schmid and Bernhard Hochreiter PhD, researchers at the Institute of Vascular Biology and Thrombosis Research at the Medical University of Vienna, and asked them to share their experience with the IKOSA platform regarding the automated histologic analysis of prostate tissue.

Here is what the research team reported about their recent study on prostate cancer with the help of advanced computational methods.

Fluorescence antibody staining of mouse prostate tissue: Nuclei(blue), IKK1 (green) and c-Myc(red).
Fluorescence antibody staining of mouse prostate tissue: Nuclei(blue), IKK1 (green) and c-Myc(red). Image taken from Moser et al. (2021).

On the benefits of the IKOSA software

When asked about the benefits of the IKOSA software, Bernhard Hochreiter noted that he was particularly pleased with the many useful capabilities included in the IKOSA software. Especially, the option to view and process image files of different sizesincluding particularly large images had proven to be very helpful. Transforming images taken with different microscope modalities into an uniform size was no longer necessary.

I was pleasantly surprised that images of various sizes can be easily managed on the web platform.

Prof. Johannes Schmidt, Researcher, Institute of Vascular Biology and Thrombosis Research at the Medical University of Vienna

The researcher also agrees that the entire analysis workflow has been faster and smoother since they implemented the IKOSA software in their lab. Being able to work remotely and perform the analysis of histologic data on an online platform has been an invaluable asset, especially during the COVID-19 pandemic.

The uses of IKOSA in applied histopathological prostate research

The researchers report how the use of the IKOSA platform helped them conduct the large-scale research project “FFG-BRIDGE Precision Histology.” The dataset used for the project consisted of complex microscopy data acquired with different imaging modalities. Especially prostate tissue images taken with multichannel fluorescence microscopes constituted the larger part of the dataset. This required an image analysis tool which supported these image formats and was flexible enough to adjust the different channels.  

Multichannel image of histologically stained mouse prostate tissue
Multichannel image of histologically stained mouse prostate tissue.

Yet, as Hochreiter explains, the biggest asset of the IKOSA platform turned out to be its AI-backed analysis capability. It has helped the team avoid many mistakes, which might have occurred during manual analysis tasks. Image analysis automation has proven invaluable during the segmentation of prostate cell nuclei.

The significant advantage is the support for image analysis using artificial intelligence and machine learning. The use of IKOSA can help avoid mistakes that could affect the research results.

Bernhard Hochreiter PhD, Researcher, Institute of Vascular Biology and Thrombosis Research at the Medical University of Vienna

Examine channel intensities of multichannel fluorescence images while segmenting cell nuclei and virtual cytoplasm with our Sparkfinder Application.

The “FFG-BRIDGE Precision Histology” study

Due to the scarcity of human prostate tissue biopsy samples and ethical considerations using patient data a large body of pre-clinical research has been conducted on animal prostate tissue samples. Making use of the similarities in mammalian prostate anatomy, a significant number of articles on prostate histology relies on mouse tissue samples (Fagerland et al., 2020; Ding et al., 2021).  

Similarly, the research team at the Medical University of Vienna used mouse prostate tissue histological images acquired with different microscope modalities like brightfield and fluorescence microscopy. 

Did you know?

The project resulted in a publication in the Molecular Cancer Journal on the effects of inflammatory IKKα complexes on inflammatory transcription in prostate cancer. 

Full interview provided in the PDF file below. No email required.

*The following PDF interview document contains screenshots of the IKOSA platform from 2020. The actual interface of IKOSA may look different due to numerous enhancements to the platform.

Our authors:

KML Vision Team Benjamin Obexer Lead Content Writer

Benjamin Obexer

Lead content writer, life science professional, and simply a passionate person about technology in healthcare

KML Vision Team Elisa Opriessnig Content writer

Elisa Opriessnig

Content writer focused on the technological advancements in healthcare such as digital health literacy and telemedicine.

KML Vision Team Fanny Dobrenova Marketing Specialist

Fanny Dobrenova

Health communications and marketing expert dedicated to delivering the latest topics in life science technology to healthcare professionals.


Bhargava, R., Madabhushi, A. (2016). Emerging themes in image informatics and molecular analysis for digital pathology. Annual review of biomedical engineering, 18, 387–412.    

Branca, G., Ieni, A., Barresi, V., Tuccari, G., Caruso, RA. (2017). An Updated Review of Cribriform Carcinomas with Emphasis on Histopathological Diagnosis and Prognostic Significance. Oncol Rev.,11(1),317. Published 2017 Mar 10. doi:10.4081/oncol.2017.317.

Bulten, W., Bándi, P., Hoven, J., Loo, RVD, Lotz, J., Weiss, N., … & Litjens, G. (2019). Epithelium segmentation using deep learning in H&E-stained prostate specimens with immunohistochemistry as reference standard. Scientific reports, 9(1), 1-10. 

Carleton, NM., Lee, G., Madabhushi, A., & Veltri, R. W. (2018). Advances in the computational and molecular understanding of the prostate cancer cell nucleus. Journal of cellular biochemistry, 119(9), 7127-7142. 

Chatterjee, A., Watson, G., Myint, E., Sved, P., McEntee, M., & Bourne, R. (2015). Changes in epithelium, stroma, and lumen space correlate more strongly with Gleason pattern and are stronger predictors of prostate ADC changes than cellularity metrics. Radiology, 277(3), 751-762.

Chen, N., & Zhou, Q. (2016). The evolving Gleason grading system. Chinese journal of cancer research = Chung-kuo yen cheng yen chiu, 28(1), 58–64.

Denmeade SR, Isaacs JT. (2003). Cellular Organization of the Normal Prostate. In: Kufe DW, Pollock RE, Weichselbaum RR, et al., editors. Holland-Frei Cancer Medicine. 6th edition. Hamilton (ON): BC Decker; 2003.

Ding, Y., Lee, M., Gao, Y., Bu, P., Coarfa, C., Miles, B., … & Ayala, G. (2021). Neuropeptide Y nerve paracrine regulation of prostate cancer oncogenesis and therapy resistance. The Prostate, 81(1), 58-71. 

Fagerland, S. M. T., Hill, D. K., van Wamel, A., de Lange Davies, C., & Kim, J. (2020). Ultrasound and magnetic resonance imaging for group stratification and treatment monitoring in the transgenic adenocarcinoma of the mouse prostate model. The Prostate, 80(2), 186-197.  

Gleason, HF. (1966). Classification of Prostatic Carcinoma. Cancer Chemother Rep, 50, 125-128.

Hägglöf, C., & Bergh, A. (2012). The stroma-a key regulator in prostate function and malignancy. Cancers, 4(2), 531–548. 

Krušlin, B., Ulamec, M., & Tomas, D. (2015). Prostate cancer stroma: an important factor in cancer growth and progression. Bosnian journal of basic medical sciences, 15(2), 1–8. 

Kweldam, CF., van der Kwast, T., van Leenders, GJ. (2018). On cribriform prostate cancer. Transl Androl Urol, 7(1), 145-154. doi:10.21037/tau.2017.12.33.

Litjens, G., Toth, R., van de Ven, W., Hoeks, C., Kerkstra, S., van Ginneken, B., … & Madabhushi, A. (2014). Evaluation of prostate segmentation algorithms for MRI: the PROMISE12 challenge. Medical image analysis, 18(2), 359-373.

McGarry, S. D., Hurrell, S. L., Iczkowski, K. A., Hall, W., Kaczmarowski, A. L., Banerjee, A., … & LaViolette, P. S. (2018). Radio-pathomic maps of epithelium and lumen density predict the location of high-grade prostate cancer. International Journal of Radiation Oncology* Biology* Physics, 101(5), 1179-1187. 

Moser, B., Hochreiter, B., Basílio, J., Gleitsmann, V., Panhuber, A., Pardo-Garcia, A., … & Schmid, J. A. (2021). The inflammatory kinase IKKα phosphorylates and stabilizes c-Myc and enhances its activity. Molecular cancer, 20(1), 1-17. 

Nguyen, K., Sarkar, A., & Jain, A. K. (2012). Structure and context in prostatic gland segmentation and classification. International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, Berlin, Heidelberg. 115-123.

Nguyen, K., Sabata, B., & Jain, A. K. (2012b). Prostate cancer grading: Gland segmentation and structural features. Pattern Recognition Letters, 33(7), 951-961. 

Pudasaini, S., & Subedi, N. (2019). Understanding the gleason grading system and its changes. Journal of Pathology of Nepal.

Shibuya, T., Takahashi, G., & Kan, T. (2019). Basal cell carcinoma of the prostate: A case report and review of the literature. Molecular and clinical oncology, 10(1), 101–104.

Singh, M., Kalaw, EM., Giron, DM., Chong, KT., Tan, CL., & Lee, HK. (2017). Gland segmentation in prostate histopathological images. Journal of medical imaging, 4(2), 027501.

Zhang, Y., Nojima, S., Nakayama, H., Jin, Y., Enza, H. (2003). Characteristics of normal stromal components and their correlation with cancer occurrence in human prostate. Oncol Rep, 10(1), 207-11. PMID: 12469170.

Zhu, Q., Du, B., Turkbey, B., Choyke, P. L., & Yan, P. (2017). Deeply-supervised CNN for prostate segmentation. International joint conference on neural networks (IJCNN), 178-184.


Join our newsletter