Texture Characterization of Histopathologic Images Using Ecological Diversity Measures and Discrete Wavelet Transform

Cancer is a disease caused by an uncontrolled division of cells that can happen anywhere in the body, susceptible to invading neighboring tissues and organs  [20]. Cell proliferation does not necessarily imply the presence of malignancy but can simply respond to the body’s specific needs. More than a hundred known types of tumors affect people, but not all tumors are cancerous  [46]. Benign tumors grow in an organized, generally slow, expansive manner and have apparent limits. Although they do not invade neighboring tissues, they can compress adjacent organs and tissues and are hardly life-threatening. In contrast, malignant tumors manifest a greater degree of autonomy. They are capable of invading neighboring tissues and causing metastases, which may be resistant to treatment and cause the patient’s death  [26].

Breast cancer is the most frequent cancer among women globally, accounting for nearly a quarter of new cases each year  [27], and is the most prevalent type of cancer in nearly 140 countries [64]. In addition to resulting in death, related treatment costs are high. It impacts both the public and the private healthcare system, which, in turn, penalizes both the government and the population.

Digital mammography analysis is among the most widely employed and effective methods for the early detection of breast cancer. Mammography is a radiological test that produces images in gray-scale, which are analyzed by radiologists, who need to give more attention and spend more time reliably detecting the cancer information from the images [23]. Likewise, imaging examinations like computerized tomography or ultrasound can diagnose whether there are masses growing in breast tissue, despite the verification of either type of tumor can only be accomplished employing biopsy. Biopsies, in turn, last to provide a result because of the acquisition procedure, which may imply open surgical biopsy or fine-needle aspiration, tissue processing (preparation of slides with a staining process) as well as pathologist visual examination. Innately, pathologist examination is a time-consuming and highly specialized task prone to intra and inter-observer disagreement [7].

Histopathology is the study of how a specific disease affects a set of cells (tissue). Usually, a biopsy study is done using a microscope and dyes. It can also be done during surgery or at an autopsy (death investigation). Histopathologic images (HIs) are a type of medical imaging obtained via microscopy of tissues from biopsies, which allow specialists to observe tissue characteristics on a cell basis. This process consists of tissue processing by chemical fixation or frozen section slides (creating the slide with the staining process). Next, the slides undergo staining process with one or several pigments to envision the tissue by means of a microscope, aiming to reveal cellular components; counterstains are used to provide contrast, and finally, the pathologist analysis. The stain by and large used in histopathology is a combination of hematoxylin and eosin (H&E). The former is employed to stain nuclei (blue), while the latter stains cytoplasm and the extracellular connective tissue matrix (pink). However, the staining process may give rise to a variance in the analysis process because H&E is prone to produce different color intensities being conditioned by the temperature, storage conditions, and brand. Nevertheless, as stated by de Matos et al. [41], HIs continue to be the gold standard for evaluating several types of tumors for cancer diagnosis.

Computer-aided detection (CAD) and computer-assisted diagnostic (CADx) systems are continuously being developed to assist medical image analysis. Clinicians are heavily reliant upon CAD for cancer detection and monitoring. However, given the reliance on CAD and CADx for cancer detection, there is always an extra focus and need for developing systems that improve pathologists’ productivity and ameliorate the reliance on outcomes by adding consistency to the diagnosis process and reducing observer subjectivity. Machine learning (ML) approaches are increasingly being used in CAD and HI analysis to diagnose cancer in various tissues or organs, such as the breast, prostate, skin, brain, bones, liver, and so on. Furthermore, when used in HI analysis, ML approaches reveal potential benefits. As a result, they have seen a lot of use in tasks like feature extraction, classification, and segmentation. The visual properties of macro vision images used in other machine learning applications, such as scene reconstruction, object and face recognition, differ from those of HIs, which contain complex textures and rich geometric structures.

This paper proposes a novel method for texture characterization of HIs with a considerable success rate. We state that it is possible to quantify the intrinsic properties of HIs to the maximum extent by combining biodiversity and taxonomic measures and discrete wavelet transform. Thus, the main contributions of our research are: (i) an information-theoretical measure of ecological diversity indices and measures of biodiversity for texture characterization; (ii) the exploitation of independent wavelet subband coefficients’ non-linear interactions across time; (iii) the mixture of wavelet features and statistical properties of taxonomic indexes representing an unexplored method based on a non-deterministic system analysis; (iv) such a mixture characterizes HIs so that intrinsic properties have provided promising performance for real-world HI datasets such as the CRC and the BreakHis datasets.

This paper is organized as follows: Section 2 presents works that put forth descriptors more popular in medical image analysis. Section 3 puts forth the proposed approach as well as the concepts from which it originated. Section 4 presents the datasets, experiments results, and discussion. Finally, the last section presents the conclusion and perspectives of future work.

Several feature descriptors have been used to extract a relevant and discriminant information from images. Some are based on shape, texture, fractal, or combination of those mentioned above. Besides natural images, texture descriptors are becoming increasingly popular in medical image analysis, particularly in HIs due to the variability of texture that such images exhibit. Therefore, researchers have been studying a broad range of textural descriptors for the classification of HIs, which are expected to be invariant to translation, scale, rotation, and intensity changes.

Characterizing morphological features from structures observed in HIs, as well as exploring higher-level representations capable of capturing relevant information for medical diagnosis purposes, is one of the most preeminent challenges in extracting features from such images. The above characteristics are related to recognizing tissue alterations (such as cell density or aberrant cell quantity) or cellular changes (e.g. malformed nuclei) caused by mitotic phases. Furthermore, morphological traits are related to how pathologists investigate HIs, looking for specific reasons to categorize them. High-level features, on the other hand, are generalizations of all structures in HIs, not solely cell structures. As a result, most researchers exploit representations or texture descriptors in the frequency domain [41]. This section presents some state-of-the-art works for feature extraction from HIs.

Several authors have utilized the descriptors based on the grey-level co-occurrence matrix (GLCM) to represent texture in HIs. [36] employed GLCM for feature extraction with a segmentation process through unsupervised mean-shift clustering. The latter minimizes color variety to facilitate the segmentation of the image using thresholds. Following that, nuclei are identified, and overlapping is reduced employing contour and area constraints. [10] combined seven feature extraction methods, including GLCM, and constructed a kernel-based representation of the data for each feature type. The kernels are then used within an SVM to detect similarities between data for the implementation of a content retrieval mechanism. [19] proposed a feature extraction approach that uses frequency and spatial textons, implying that images are represented by a limited vocabulary of textures. Texton histograms and GLCM extracted from texton maps are used as classification features. Similarly, the effect of various colormaps on these procedures was assessed. Despite the fact that GLCM requires a gray-level image, the transformation of the H&E color image to gray-level is influenced by the staining color variability, which affects GLCM accordingly. [37] utilized a random forest to determine whether or not GLCM features are susceptible to image variations. The work also emphasized the significance of color normalization.

[57] focused on stroma maturity to assess breast cancer. Image convolution with a bank of derivatives-of-Gaussian filters, basic image features, and LBP with multiple scales for the neighborhood produce the feature vector. [16] proposed geometric- and texture-aware features based on Hu moments and fractal dimensions, respectively. The latter was used to distinguish between mitotic and non-mitotic cells by detecting changes in geometrical and textural nuclei. [15] proposed a patching method on HI slides, intending to create small regions and extract scale-invariant feature transform (SIFT), luminance level, and discrete cosine transform features to generate a bag-of-words.

The local binary pattern (LBP) is one of the most commonly used texture descriptors. LBP was used in the context of multispectral HIs by Peyret et al. [52]. An SVM was used for the evaluation, which aligns all spectra and uses pixels from all other bands. Similarly, it employs a multi-scale kernel size. When compared to the standard LBP and the concatenated spectra LBP, this feature extractor performed better. To deal with multi-scale HIs, Bruno et al. [8] used a curvelet transform. The LBP algorithm was used to extract features from curvelets coefficients, which were then reduced using ANOVA. [54] proposed an algorithm that finds nuclei areas using adaptive and iterative thresholding and extracts texture information using histograms of oriented gradients and LBP.

[25] proposed a two-step feature extraction method comprised of a receptive field for detecting regions of interest and sparse coding in their study. Sparse coding assembles features by combining patches from the same region. Furthermore, the mean and covariance matrix of receptive fields, as well as sparse coding, are utilized as final filters. [47] proposed an approach for distinguishing basal cell carcinoma tumors from squamous cell carcinoma tumors in skin HIs by using Z-transform features derived from a mixture of Fourier transform features. Wan et al. [67] employed a dual-tree complex wavelet transform to represent the HIs in breast cancer detection for mitosis detection. The feature vector is made up of parameters from the symmetric alpha-stable distribution and the generalized Gaussian distribution. Chan and Tuszynski [12] used fractal dimension features to detect breast cancer. These features distinguish between malignant and benign tumors satisfactorily on HIs at 40$times$ magnification. Finally, Ataky and Lameiras Koerich [2] presented a bio-inspired texture (BiT) descriptor based on biodiversity and taxonomic indices. The authors mapped breast cancer HI images to an abstraction model of an ecosystem, from which measures of species diversity, richness, evenness, and taxonomic distinctiveness were extracted. The resulting texture descriptor has been shown to be rotation, translation, and scale invariant. Experiments on HI datasets revealed that the BiT achieved competitive results when compared to deep methods, and outperformed traditional texture descriptors.

In the last years, deep features have become very popular in several image classification tasks, including HIs. Khalid Khan Niazi et al. [33] put forward a CAD system for bladder cancer with a focus on extracting epithelium features with segmentation through an automatic color deconvolution matrix construction. Spanhol et al. [60] employed deep features extracted with a pre-trained AlexNet for the classification of breast tumors into benign and malignant. The method proposed by Vo et al. [66] put forth a feature extraction based on the combination of CNNs and boosting tree classifiers. The latter employs an ensemble of inception CNNs for visual features extraction from multi-scale images. Firstly, data augmentation methods were employed. Subsequently, ensembles of CNNs were trained to extract multi-context information from multi-scale images. The last-mentioned stage extracted both global and local features of breast cancer tumors. George et al. [22] presented an approach for breast cancer diagnosis, which extracts features from nuclei based on CNNs. The methodology comprises different approaches for extracting nucleus features from HIs and selecting the most discriminative spatially sparse nucleus patches. A pre-trained set of CNNs was used to extract features from such patches. Afterward, features belonging to individual images are fused using 3-norm pooling to obtain image-level features.

Various works employed and combined different categories of features for capturing information from geometrical structures and textures from HIs. Leo et al. [37] introduced a method aiming to quantify features’ instability across a few prostate cancer datasets with known variations caused by staining, preparation, and scanning platforms. The author evaluated five groups of features, such as graph-based features, gland shape features, co-occurring gland tensor features, subgraph features, and Haralick texture features. Yu et al. [69] investigated features that characterize lung cancer the best. They extracted the objective quantitative image features such as Haralick texture features of the nuclei, nuclei edge intensity, texture features of the cytoplasm, etc.

The work of Caicedo et al. [9] put forth a low-level to high-level mapping to facilitate image retrieval. Such mapping stage consists of color and gray-level histograms, Tamura texture histogram, LBP, Sobel histogram, and invariant feature histograms. Pang et al. [51] raised a CAD system for lung cancer detection. Such a CAD system utilizes textural features, such as GLCM, Tamura, and LBP, and shape features, namely global features, SIFT, and morphological features. The work of Kruk et al. [35] employed textural, morphometric, and statistical (histogram) features to describe nuclei for clear-cell renal carcinoma grading. To this end, the Genetic algorithm and the Fisher discriminant were utilized to select essential features. Basavanhally et al. [5] introduced a multi-field-of-view classification method to determine low against high-grade ductal carcinoma from breast HIs. This method utilizes a multiple patch size procedure for WSI to analyze which textural, morphological, and graph-based features are the most relevant to each patch size. Tashk et al. [63] put forth a comprehensive framework for breast HI classification that evaluates mitotic pixels in L*a*b color space. A mixture of LBP, statistical features, and morphometrics is extracted from mitotic candidates. The work of Cruz-Roa et al. [15] introduced a patching approach on HI slides to form small regions and extract SIFT, discrete cosine transform, and luminance level features to produce a bag-of-words. Besides, semantic features represent high-level information that can be associated with HIs to aid their classification.

Orlov et al. [50] compared four color spaces (RGB, L*a*b, gray-scale and RGB) with H&E representation and eleven features such as GLCM, Zernike, Tamura, Chebychev, color histograms, Gabor, edge statistics, Chebyshev-Fourier, and others to represent lymph node HIs. De et al. [17] propose a fusion of several feature types for uterine cervical cancer HI classification. The authors employed a feature vector based on Delaunay triangulation, GLCM, and weighted density distribution. The work of Vanderbeck et al. [65] utilized textural, pixel neighboring statistics, and morphological features to represent seven categories of white regions of liver HIs. Kandemir et al. [30] put forth a MIL method to detect Barrett’s cancer from HIs. They utilized cell-level morphometric features, to wit, radius, perimeter, central power sums, area, the roundness of segments, and so forth, within regions and patch-level features such as color histograms, SIFT, LBP, and from segmented images employing the watershed algorithm.

Coatelen et al. [14, 13] proposed a feature selection method of liver HI classification based on morphometric features and graph-based features. Pair of greedy algorithms such as fselector and in-house recursive was employed to select features in a collection of 200 features where an SVM classifier implemented the fitness function. The work of Michail et al. [43] highlighted nuclei using connected-component labeling to classify non-centroblast and centroblast cells. Das et al. [16] presented the so-called geometric- and texture-aware features based on Hu moments and fractal dimensional, respectively. The features set thereof was applied to detect geometrical and textural changes in nuclei to discriminate between non-mitotic and mitotic cells. The method introduced by Kong et al. [34] classifies neuroblastomas utilizing textural and morphological features. The authors consider that pathologists employ morphological features for their analysis, and textural features can be easily extracted. GLCM features and sequential floating forward selection were then utilized.

These are some examples of works developed to detect breast cancer through HIs. However, in spite of the substantial efforts and countless methods that have been proposed in recent years, accurate classification of HIs remains a challenge. Existing approaches developed to cope with such a purpose still arise performance issues in light of noise, different image resolutions, and a lack of a considerable amount of data. Furthermore, methods based on CNNs present a lack of explainability and interpretability, which are both necessary for understanding the behavior of such models, and trusting their decisions. Likewise, pre-trained CNN architectures designed for object classification still require fine-tuning some of their layers on a large amount of data to achieve good performance, including tiny architectures such as T-CNN Inception and T-CNN [42, 3].

This work proposes an efficient method for texture characterization of HIs combining ecology diversity measures and multi-resolution analysis. Combining biodiversity and taxonomic measures with the discrete wavelet transform (DWT) makes it possible to quantify the intrinsic properties of HIs to the maximum extent. Thus, the fundamental research highlights of the proposed method are: (i) an information-theoretical measure of ecological diversity indices and measures of biodiversity; (ii) the exploitation of non-linear interactions of single and independent wavelet subband coefficients over time; (iii) the mixture of wavelet features and statistical properties of taxonomic indexes representing an unexplored method for non-deterministic pattern’s systems analysis.

This section presents the concepts of information-theoretical measures of ecological diversity, multi-resolution analysis, and DTW, which are the base concepts of our method, before presenting our method for texture characterization of HIs.

This section presents the two datasets used to assess the performance of the BiTW descriptor and the experimental protocol to evaluate the properties of the BiTW descriptor and its performance on classification tasks considering different classification and ensemble methods. We compare the BiTW descriptor’s performance with shallow and deep state-of-the-art approaches.

The current research leveraged the information-theoretical measure of ecological diversity indices together with a discrete wavelet transform to characterize texture across HIs. We explored the interactions of individual wavelet subband coefficients over time and modeled each as an ecosystem from which measures of biodiversity and statistical properties of taxonomic indexes are extracted to represent HI texture effectively. We stated that by combining measurements of biodiversity from each HI channel and taxonomic indexes extracted from different wavelet subbands, it should be possible to quantify the intrinsic properties of such images to the maximum extent. The mixture of wavelet features and statistical properties of ecology diversity indexes represent a novel method and a promising tool for quantifying intrinsic properties of texture across HIs. Wherefore, the experimental results have shown an increase in terms of texture discrimination over both HI datasets. Moreover, the proposed method outperformed several shallow and deep state-of-the-art methods.

In future work, we intend to further improve the classification accuracy by exploring various color spaces and different families of wavelets to find an optimized wavelet that will bring on discriminative information for characterizing HI categories. Furthermore, because an increasing number of DWT-level decompositions can result in high-dimensional feature vectors, we want to look into appropriate feature selection approaches or dimensionality reduction techniques to find feature subspaces that not only improve texture features but also improve the proposed method’s class discrimination capability.

This work was funded by the Regroupement Strategique REPARTI - Fonds de Recherche du Québec - Nature et Technologie (FRQNT) and by the Natural Sciences and Engineering Research Council of Canada (NSERC) under Grant RGPIN 2016-04855.

Supplementary material that may be helpful in the review process should be prepared and provided as a separate electronic file. That file can then be transformed into PDF format and submitted along with the manuscript and graphic files to the appropriate editorial office.