Prostate cancer microscopy sample

Pathologist advice on prostate cancer microscopy analysis techniques

Written by:

cand. med. Elisa Opriessnig, BA BA MA

Dr. Fanny Dobrenova, MA

Prostate cancer research is one of the areas where the use of pathology image analysis is on the rise and about to become standard diagnostic practice. In this article, we guide you through the recent developments in automated prostate tissue analysis. Find out how advanced artificial intelligence methods like image segmentation and classification assist pathologists in the detection and diagnosis of prostate cancer in bioimage samples.

Types of prostate cancer

There are different types of prostate cancer. The most common type is adenocarcinoma of the prostate, which develops in the glands. Two different variants of adenocarcinoma can be distinguished: acinar adenocarcinoma and ductal adenocarcinoma (Humphrey 2017). Apart from the typical acinar type prostate cancer observed in more than 90 % of prostatic adenocarcinomas,  a spectrum of morphological variants and subtypes exists (see infographic 1). Typically two different groups can be differentiated: the variants of conventional acinar adenocarcinoma and cancers with unusual histological patterns for the prostate like ductal type prostate cancer/ductal prostate carcinoma or mucinous carcinoma among others (Mikuz 2015).

Histologic Types of Prostate Cancer

Prostate histology tissue samples

Prostate specimen collection methods

Prostate biopsy is a minimally invasive procedure and is the gold standard diagnostic technique for the detection of prostate cancer, by obtaining samples of suspicious tissue. By using a thin needle a number of tissue samples (biopsy samples) are collected from the prostate glandular tissue. This prostate biopsy technique is performed with either a transrectal or transperineal approach. These two approaches are typically based on tissue sampling with an ultrasound-guided core needle biopsy to gather core needle samples from different areas. During a transrectal prostate biopsy, a biopsy gun projects a thin needle into suspicious areas of the gland through the rectum. By doing so, small sections of tissues can be removed for analysis purposes. In transperineal biopsy, a needle is passed through the perineal skin into the prostate (Devetzis et al. 2021; Streicher et al. 2019). When prostate cancer is detected, a prostatectomy might have to be performed, which is the partial or complete removal of the prostate or prostatectomy samples for further analysis. However, it may also be performed to treat benign prostatic hyperplasia (Martini & Tewari 2019; Wilt et al. 2012).

Benign prostatic hyperplasia (BPH) is a condition in men in which the prostate gland is enlarged but not cancerous. It is considered a hyperplastic process resulting in growth of glandular-epithelial and stromal/muscle tissue, mainly in the periurethral area of the prostate. This can cause various secondary issues like urinary problems, which when severe enough, are usually treated by a transurethral resection of the prostate (TURP). During this surgery an instrument called a resectoscope is inserted through the penis and into the urethra, which is the tube that carries urine from the bladder. With this instrument excess prostate tissue, which may be blocking urine flow, can be removed (Wilt et al. 2012; Bill-Axelson et al. 2005)

Microscope techniques in prostate cancer diagnosis  

The microscopic analysis of tissue samples is the gold standard in cancer detection. The establishment of a histopathological diagnosis of prostate cancer requires light microscopic examination of the prostate tissue samples. For prostate cancer microscopy the tissue sections are hematoxylin and eosin (H&E)-stained (Humphrey 2017). H&E-stained samples are typically based on fixation, processing, acquisition of prostate biopsy glass slides and an analog microscope, which a pathologist uses. However, digitalization and the developments in artificial intelligence/machine learning allow for real-time, even remote,  access to images. Imaging modalities have changed over the last few years, and thus, pathologists are able to quickly generate images, analyze them and come to an almost real-time diagnosis of prostate cancer (Rocco et al. 2021, Streicher et al. 2019). The most commonly used histopathological evaluation techniques for prostate are:

  • Fluorescent confocal microscopy (FCM): 
  • Atomic force microscopy (AFM): 
  • Electron microscopy (EM):
  • Optical microscopy with molecular selectivity

One of the challenges in prostate cancer diagnosis is to determine ideal patient management and avoid unnecessary interventions. However, for those patients, who are diagnosed with cancer, it is important to differentiate between those with indolent disease from those with an unfavorable outcome. There is a growing body of literature highlighting the importance of immunohistochemical (IHC) methods in order to determine a prostate cancer prognosis and the diagnostic confirmation of “borderline” cases due to the presence of certain structures like basal cells. Among the most frequent prostate cancer imaging biomarkers are: ki-67, p53, PTEN, MYC and ERG (Carneiro et al. 2018; Devetzis et al. 2021). IHC sample preparation can be based on freshly tissue sections, frozen tissue  and formalin-fixed tissue sections. In order to find cancerous or abnormal cells biopsy slides are stained with IHC dyes. This allows the pathologist to detect abnormalities in the sample tissues (Jakobsen et al. 2016). However, there is still a need for further novel biomarkers for improving the detection of clinically significant cases and other clinical needs (e.g., malignancy). Furthermore, the phenomenon of therapy resistance e.g., castration resistant or hormone refractory prostate cancer needs to be addressed in future work.

Computational methods for the histologic assessment of prostate tissue samples

Different computational methods of microscopy image analysis for detection of prostate cancer have been discussed in literature. In general, computer aided methods for the histologic assessment of prostate tissue samples contribute to high accuracy, reproducibility and time-efficiency in the image analysis. We’ll discuss two techniques for the automated analysis of prostate tissue images in the following - semantic segmentation and image classification.

Semantic segmentation models for prostate cancer detection and grading 

The detection and grading (e.g., Gleason grading) of prostate cancer require speed and objectivity, which can be obtained by incorporating digital image analysis methods in prostate tissue analysis. The diagnosis is determined by pathologists manually, which is not only a complex but also a time-consuming task. Pathologists use a grading and/or classification system to qualitatively assess tumor histologies. To reduce the workload for pathologists, an automatic classification system would be of great use. In terms of prostate cancer, the severity is based on the Gleason score, which indicates different growth patterns of the tumor glands. The score gets calculated by adding the two most prominent Gleason grades (on a scale from 6-10 overall) gathered through prostate tissue analysis. Figure 1-4 show different Gleason grades. To date it is the best indicator for patients' outcomes. However, it is also necessary to advance technology in order to adapt to clinical needs and improve patient management. Deep learning techniques are able to recognize cell types and diagnose a disease based on the extraction of information on the parameters of cells and other morphological features. Consequently, the specimen can be either classified as cancerous or benign, depending on the present morphological characteristics. Further, cancerous and benign samples can be distinguished. In addition, deep learning methods allow for an accurate gland detection, where gland boundaries are preserved (Ing et al. 2018). With these techniques the following patterns can be typically recognized (depending on the respective Gleason score):

  • Histologic pattern
  • Glomeruloid pattern 
  • Stromal pattern 
  • Atypical gland pattern 
  • Cribriform pattern

In diagnostic pathology, the pathologists make a diagnosis based on viewing a set of biological samples (tissue stained with different markers) and evaluating many specific features of the objects (such as size, shape, color, texture etc.). This process is an important part in clinical medicine and can be enhanced by providing pathologists with automated quantitative data extracted from the images using deep learning algorithms/techniques (Vu et al. 2019). It captures entire slide images with a scanner and stores it as a digital image, with the aim of applying image analysis technology using machine learning to determine the presence and or absence of disease such as cancer (Daisuke & Shumpei 2017). The most common tasks in prostate tissue image analysis are the segmentation of microscopic structures, such as nuclei and cells, in cancer and non-cancerous regions and the classification of image regions and whole images. This process is vital to extracting and interpreting morphological information from digital slide images as e.g., cancer nuclei differ from normal nuclei in many ways. Thus correct quantitative characterization and extracting nuclear, cytoplasmic and intraluminal features, e.g., size and number of epithelial nuclei, size and number of lumina, shape of nuclei and lumina, roundness or circularity are key components of the analysis (Vu et al. 2019). 

One technique for the automated analysis of prostate tissue samples is image classification. Image classification can be done with or without segmentation. Image classification assigns a class label to an image or an image region. If a pathologist has to classify images into certain classes manually, it would definitely take up a large amount of time. Therefore an automated model is a key component in computing a categorization via imaging features of patients into classes. One way to do this is to teach the model using a set of sample images/data. The second method is to let the model learn by itself. The final output from the image classification process is a class which is given a label by the model. This can be done by using a convolutional neural network (CNN).  In terms of prostate tissue analysis such a trained deep learning network can classify the nuclei in prostate tissue images into classes. In a next step the algorithm can use this nuclear classification to classify regions, e.g., glandular regions according to the Gleason score and/or their malignancy grade (Gunashekar et al. 2022). 

Another technique for automated analysis of prostate tissue images is semantic segmentation. This algorithm is also based on deep learning. Semantic segmentation means labeling each pixel in the image and knowing to which class (determined through image classification) it belongs. Semantic segmentation performs pixel classification (local) using features of a broader area (Isaksson et al. 2017). 

Those deep learning algorithms allow pathologists to assess a potential invasion into the extracellular space, lumen appearance and spacing, appearance and arrangement of epithelial cells and nuclei. This is vital for determining the presence of prostate cancer and the prediction of the histological grades present in a prostate biopsy (Isaksson et al. 2017). Multiple patterns of images extracted from the digital slides of a prostate biopsy can be classified based on the Gleason grading system (Bhattacharjee et al. 2020). 

However, there are also some challenges in prostate tissue classification and segmentation. Despite a large body of research on image classification and segmentation, it remains a difficult task to extract and interpret information from digital slide images. Thus, there are a number of challenges and issues that need to be addressed by segmentation and classification algorithms. First, the development of accurate and efficient algorithms for these tasks is a challenging issue because tissue morphology is complex and tumors are heterogeneous, not just in prostate tissue but in other tissues as well. A single tissue specimen contains a variety of nuclei and other structures. Therefore, algorithms need to take this variety into consideration and need to dynamically adapt to such variations (Vu et al. 2019). As a result, an algorithm can do well for one image but may not be successful for another. Second, nuclei touch or overlap each other. Therefore, the segmentation process is difficult when nuclei are clumped. Moreover, digital slide tissue images have a high resolution and sometimes do not fit in main and GPU memory on most machines. Consequently, image classification might not be possible on the whole image at once. This brings up the need to design an algorithm which is able to work on multiple resolutions or image tiles (Vu et al. 2019).

The automated feature classification in prostate tissue images

Classification methods go even a step further by assigning morphological objects or areas on the histologic image to a specific class like cell type, lumen regions, cytoplasm regions etc. Classification results are based on the appearance, color and texture of objects and regions as well as on the grounds of morphological patterns visible in the images. Thus, feature classification comes into play, which is a pattern recognition technique primarily used to categorize data into different classes, e.g., cell types. For example, cell type classification is classification used to identify cells which share morphological features. The prostate gland consists of multiple cell types (e.g., epithelial, stromal, immune cells). In case of cancer in the prostate, complex interactions of tumor cells with the surrounding epithelial and stromal cells are involved, where also cellular state transitions are possible towards carcinogenesis. Further, also certain regions of interest can be identified and analyzed through regions of interest classification. Classification methods allow to differentiate between different elements of images, for example stroma regions, benign regions and tumor regions, resulting in improved diagnosis and accuracy in clinical pathology. Biopsy tissue images are already widely used for prostate cancer diagnosis and to determine the level of malignancy for cancerous tissue. To specifically assign the histological grade, biopsied tissue is stained with H&E and viewed under a microscope. Digital image processing allows pathologists to work more efficiently and faster. There are different techniques in use to extract different types of information present in histological biopsy images. The current method of grading prostate cancer in histology is the Gleason system, which describes different malignant stages of cancer according to a qualitative analysis of tissue architecture. However,  the Gleason score has been dependent on inter- and intra-observer-variability, which makes it difficult to determine an accurate diagnosis. New methods for automated prostate cancer grading based on image classification are on the rise, categorizing specimens into a Gleason grade using image classification and semantic segmentation. Modern research suggests that the current Gleason grading system and diagnosis of prostate cancer in general can be improved by developing quantitative image analysis algorithms to aid pathologists in determining an accurate and improved diagnosis as offered in  the IKOSA platform (Salman et al. 2022).

Test automated prostate tissue analysis methods directly in the IKOSA Platform

With the help of the IKOSA Platform you can perform complex segmentation analysis techniques on prostate tissue image data. On top of that, our advanced algorithm training tool IKOSA AI allows you to design your own image analysis applications tailored to your prostate cancer research project. If you have questions regarding the implementation of our tools in your research, feel free to contact us at


Abishek P, Singh SK, Khamparia A. Detection of Prostate Cancer Using Deep Learning Framework. IOP Conf. Ser.: Mater. Sci. Eng. 2021;1022. 

Bhattacharjee S, Prakash D, Kim CH, Choi HK. Multichannel Convolution Neural Network Classification for the Detection of Histological Pattern in Prostate Biopsy Images. Journal of Koreak Multimedia Society. 2020 Dec 31;23 (12). Pp.1486-95.

Bill-Axelson A, Holmberg L, Ruutu M, et al. Radical prostatectomy versus watchful waiting in early prostate cancer. N Engl J Med, 2005;352: 1977-1984.

Carneiro, A., Barbosa, Á., Takemura, L. S., Kayano, P. P., Moran, N., Chen, C. K., Wroclawski, M. L., Lemos, G. C., da Cunha, I. W., Obara, M. T., Tobias-Machado, M., Sowalsky, A. G., & Bianco, B. The Role of Immunohistochemical Analysis as a Tool for the Diagnosis, Prognostic Evaluation and Treatment of Prostate Cancer: A Systematic Review of the Literature. Frontiers in oncology, 2018;8(377).

Daisuke Komura and Shumpei Ishikawa. Machine learning methods for histopathological image analysis. 2017. arXiv preprint arXiv:1709.00786. 

Devetzis K, Kum F, Popert R. Recent Advances in Systematic and Targeted Prostate Biopsies. Res Rep Urol. 2021;13:799-809.

Gleason DF. Classification of prostatic carcinomas. Cancer Chemother Rep. 1966 Mar;50(3):125-8. PMID: 5948714.

Gunashekar, D.D., Bielak, L., Hägele, L. et al. Explainable AI for CNN-based prostate tumor segmentation in multi-parametric MRI correlated to whole mount histopathology. Radiat Oncol 17, 65 (2022).

Humphrey PA. Histopathology of Prostate Cancer. Cold Spring Harb Perspect Med. 2017 Oct 3;7(10):a030411. doi: 10.1101/cshperspect.a030411. PMID: 28389514; PMCID: PMC5629988.

Ing N, Ma Z, Li J, Salemi H, Arnold C, Knudsen BS, Gertych A. "Semantic segmentation for prostate cancer grading by convolutional neural networks", Proc. SPIE 10581, Medical Imaging 2018: Digital Pathology, 105811B (6 March 2018);

Iqbal S, Siddiqui GF, Rehman A, Hussain L, Saba T, Tariq U, Abbasi AA. "Prostate Cancer Detection Using Deep Learning and Traditional Techniques," in IEEE Access, vol. 9, pp. 27085-27100, 2021, doi: 10.1109/ACCESS.2021.3057654.

Isaksson J, Arvidsson I, Åaström K, Heyden A. Semantic segmentation of microscopic images of H&E stained prostatic tissue using CNN. International Joint Conference on Neural Networks (IJCNN), 2017, pp. 1252-1256, doi: 10.1109/IJCNN.2017.7965996.

Jakobsen NA, Hamdy FC, Bryant RJ. Novel biomarkers for the detection of prostate cancer. J Clin Urol. 2016 Dec;9(2 Suppl):3-10. doi: 10.1177/2051415816656121. Epub 2016 Dec 1. PMID: 28344810; PMCID: PMC5356177.

Martini A, Tewari AK. Anatomic robotic prostatectomy: current best practice. Ther Adv Urol. 2019 Jan 8;11:1756287218813789. doi: 10.1177/1756287218813789. PMID: 30671135; PMCID: PMC6329013.

Mikuz G. Histologic classification of prostate cancer. Anal Quant Cytopathol Histpathol. 2015 Feb;37(1):39-47. PMID: 26072633.

Rocco B, Sighinolfi MC, Sandri M, Spandri V, Cimadamore A, Volavsek M, Mazzucchelli R, Lopez-Beltran A, Eissa A, Bertoni L, Azzoni P, Reggiani Bonetti L, Maiorana A, Puliatti S, Micali S, Paterlini M, Iseppi A, Rocco F, Pellacani G, Chester J, Bianchi G, Montironi R. Digital Biopsy with Fluorescence Confocal Microscope for Effective Real-time Diagnosis of Prostate Cancer: A Prospective, Comparative Study. Eur Urol Oncol. 2021 Oct;4(5):784-791. doi: 10.1016/j.euo.2020.08.009. Epub 2020 Sep 18. PMID: 32952095.

Salman, M, Çakar GC, J, Kösem M, Cedimoğlu IH. Automated prostate cancer grading and diagnosis system using deep learning-based Yolo object detection algorithm. Expert Systems with Applications, 2022;201:117148,

Streicher J, Meyerson BL, Karivedu V, Sidana A. A review of optimal prostate biopsy: indications and techniques. Ther Adv Urol. 2019;11:1756287219870074. Published 2019 Aug 28. doi:10.1177/1756287219870074.

Tătaru OS, Vartolomei MD, Rassweiler JJ, Virgil O, Lucarelli G, Porpiglia F, Amparore D, Manfredi M, Carrieri G, Falagario U, Terracciano D, de Cobelli O, Busetto GM, Del Giudice F, Ferro M. Artificial Intelligence and Machine Learning in Prostate Cancer Patient Management-Current Trends and Future Perspectives. Diagnostics (Basel). 2021 Feb 20;11(2):354. doi: 10.3390/diagnostics11020354. PMID: 33672608; PMCID: PMC7924061.

Vu QD, Graham S, Kurc T, To MNN, Shaban M, Qaiser T, Koohbanani NA, Khurram SA, Kalpathy-Cramer J, Zhao T, Gupta R, Kwak JT, Rajpoot N, Saltz J and Farahani K. Methods for Segmentation and Classification of Digital Microscopy Tissue Images. 2019. Front. Bioeng. Biotechnol. 7:53. doi: 10.3389/fbioe.2019.00053. 

Wilt T, Brawer MK, Jones KM, et al. Radical prostatectomy versus observation for localised prostate cancer. N Eng J Med. 2012;367(3):202–13.