Blood Vessels Segmentation in Retinal Images

Introduction

Retinal blood vessel analysis provides a noninvasive window into systemic health. The morphology of retinal vessels — their width, tortuosity, and branching patterns — is used for the diagnosis and monitoring of diabetic retinopathy, hypertension, arteriosclerosis, and age-related macular degeneration. Automated early detection programs for diabetic retinopathy, fovea localization, arterial narrowing assessment, and laser surgery guidance all depend on accurate vessel segmentation.

Manual segmentation of retinal blood vessels is time-consuming and requires specialized training. The challenge is inherent to the data: vessels appear as dark lines on a relatively uniform background, but their shape, size, and brightness vary considerably, and junctions and bifurcations further complicate detection.

Most prior approaches extract a feature vector per pixel (using the pixel's neighborhood) and apply a classifier. Since a 21×21-pixel window yields 441 raw features, dimensionality reduction is essential to avoid the curse of dimensionality. This work addresses that challenge through a Gabor filter bank whose parameters are tuned end-to-end by a genetic algorithm.

Dataset: DRIVE

All experiments use the publicly available DRIVE (Digital Retinal Images for Vessel Extraction) database, an internationally recognized benchmark that enables direct comparison with published results. It consists of 40 fundus photographs (768×584 pixels, 8 bits per color channel, ~540-pixel circular field of view), each with expert manual segmentations as ground truth.

• 20 images for training/validation
• 20 images for testing (the standard evaluation set)
• ~12.7% of pixels are vessels in the test set (set A)

A second independent expert segmentation is also available, achieving an accuracy of 0.9473 against the first expert — this serves as an upper bound for automated methods.

Methodology

The proposed pipeline consists of three main stages: preprocessing, feature extraction via a filter bank, and entropy-based segmentation.

Stage 1 — Preprocessing

Color Channel Selection

The green channel is the most informative for vessel detection, providing the highest contrast between vessel and background pixels. This was confirmed quantitatively using the Bhattacharyya distance — a divergence measure between two probability distributions — computed between the normalized histograms of vessel pixels and background pixels across all three RGB channels:

The green channel has ~5× better class separability than red. An ANOVA test confirmed the differences are statistically significant (p < 0.01). The channel is also inverted so vessels appear brighter than the background.

Adaptive Histogram Equalization (CLAHE)

Retinal fundus images suffer from spatially non-uniform illumination, which reduces class separability at the pixel level. Standard histogram equalization fails under non-uniform illumination; instead, CLAHE (Contrast-Limited Adaptive Histogram Equalization) applies local equalization on a grid of rectangular regions, with a clip limit to suppress noise amplification in uniform areas.

CLAHE has two parameters: grid size and clip limit. Rather than choosing them arbitrarily, they were optimized by maximizing the average Bhattacharyya distance between vessel and background histograms across the training set. The optimal parameters found were: grid = 25 divisions, clip limit = 0.04.

After CLAHE, the average Bhattacharyya distance increases from 0.1445 to 0.3261 — more than doubling class separability.

A custom border extension technique was also developed to avoid edge artifacts: since the region of interest is circular, conventional rectangular padding methods are unsuitable. The image is transformed to polar coordinates and the boundary pixels are extrapolated outward radially, creating a smooth, uniform transition that prevents spurious vessel detections at the retinal boundary.

Stage 2 — Feature Extraction: Gabor Filter Bank

Gabor filters are bandpass filters with optimal joint resolution in the spatial and frequency domains (achieving the lower bound of the uncertainty principle). Their 2D extension mimics receptive fields in the mammalian visual cortex (areas V1/V2), making them well-suited for oriented line detection.

A novel modification is proposed: instead of standard Gabor filters, each filter in the bank is the sum of three Gabor filters, providing more degrees of freedom to fit the specific geometry of retinal vessels:

g(x,y) = g₁(x,y) + g₂(x,y) + g₃(x,y)

Each component g_i has its orientation offset by θ₀ from the previous one, enabling combinations of filters at different rotations.

Rotation Invariance

To detect vessels at all orientations, the filter bank includes 10 equally-spaced rotations of each filter at angles θ ∈ {0°, 20°, 40°, 60°, 80°, 100°, 120°, 140°, 160°, 180°}. With 4 filters × 10 rotations = 40 filters in total, the feature for each pixel is simply the maximum response across all 40 filtered images:

G(x,y) = max_k { I(x,y) * g_k(x,y) }

This single feature captures the similarity between the local image patch and the "most matching" vessel-like filter, regardless of orientation. After applying the filter bank, the Bhattacharyya distance increases further to 0.5025.

Genetic Algorithm Optimization

The parameters of all 4 filters (each described by the parameters of 3 Gabor components) are optimized using a genetic algorithm, with the average segmentation accuracy on the training set as the objective function. Key aspects of the genetic algorithm:

• Parameters are binary-encoded as chromosomes
• Population evolved over ~300 generations
• Stochastic universal sampling for parent selection
• Single-point and two-point crossover operators
• ANOVA tests confirmed no statistically significant overfitting between training, validation, and test sets (p > 0.05)

The optimal configuration found was 4 filters, each composed of 3 summed Gabor filters. The resulting filters resemble oriented Gabor filters with an intensity peak at the central pixel.

Stage 3 — Segmentation: Entropy Maximization

The final segmentation thresholds the single feature map G(x,y) obtained from the filter bank. Rather than a fixed threshold, an optimal threshold is found per image by maximizing the entropy of the co-occurrence matrix — an unsupervised approach that requires no labeled data at inference time, which is what makes it compatible with efficient genetic algorithm training.

The co-occurrence matrix captures the joint distribution of pixel value transitions in the image. Maximizing its entropy finds the threshold that best separates the two modes (vessel vs. background) in terms of transition statistics. This is the entropy-based classifier referenced in the thesis title.

Results

Segmentation Accuracy

The proposed method achieves an accuracy of 0.9462 ± 0.0074 on the 20-image DRIVE test set, matching the performance of the best published methods and nearly reaching the second human expert (0.9473).

Processing Speed

The proposed method is approximately 24× faster than the best-performing prior method (Soares et al.) while achieving comparable accuracy, and also faster and more accurate than a reference CNN implementation. The speed advantage comes from the combination of a compact filter bank (only 4 filters), the single-feature maximum-response design, and the unsupervised entropy thresholding which requires no inference-time training.

Error Analysis

Segmentation errors are primarily associated with the thinnest vessels, which have insufficient contrast to be reliably detected. Images with low overall illumination or significant retinal structural deformation also show lower accuracy. When comparing with Soares et al. pixel-by-pixel on the test set:

• 93.24% of pixels are correctly classified by both methods
• 1.38% are errors unique to Soares et al.
• 1.31% are errors unique to the proposed method
• Shared errors (4.07%) are mostly thin vessels — a fundamentally hard case for any method

Visual Results

The following images show an original retinal fundus photograph and the corresponding automated segmentation result. White pixels correspond to detected blood vessels; black pixels to background.

Conclusions

This thesis developed and validated an automatic retinal blood vessel segmentation method centered on a genetically optimized Gabor filter bank with entropy-based thresholding. The key contributions are:

• Quantitative channel selection: Bhattacharyya distance confirmed the green channel as optimal (5× better separability than red).
• Optimized preprocessing: CLAHE parameters tuned by maximizing Bhattacharyya distance doubled class separability before filtering.
• Novel filter design: Filters defined as sums of 3 Gabor filters, providing richer shape expressivity than standard Gabor filters.
• End-to-end optimization: Genetic algorithms tune all filter parameters directly against segmentation accuracy.
• Efficient unsupervised thresholding: Entropy maximization of the co-occurrence matrix finds the optimal threshold per image without supervision, enabling the fast genetic optimization loop.
• Speed: 5 seconds per image — 24× faster than prior state-of-the-art at equivalent accuracy.

Future work could parallelize the filter bank convolutions to further reduce processing time, and explore whether the Gabor-sum filter design transfers to vessel segmentation in other imaging modalities (cardiac, cerebral, pulmonary).

References

Full thesis available at the Universidad de Chile Repository. Evaluated on the DRIVE database.

Key references include: Soares et al. (Gabor + Bayesian classifier, Accuracy 0.9466), Niemeijer et al. (k-NN, Accuracy 0.9416), Staal et al. (ridge-based features + k-NN, Accuracy 0.9442), and Villalobos-Castaldi et al. (matched filter + entropy thresholding, Accuracy 0.9759).