Active learning for semantic image segmentation through conformal risk control

Abstract

Deep neural networks have achieved remarkable performance in semantic image segmentation, enabling dense pixel-wise understanding for applications such as autonomous driving, medical imaging, and robotics. However, these gains rely on large collections of finely annotated images in which each pixel is manually labeled, making dataset creation extremely time-consuming and costly. This annotation bottleneck limits the scalability of supervised learning and motivates methods that achieve high segmentation performance with substantially fewer labeled images. Active learning addresses this challenge by iteratively selecting the most informative samples for annotation from a large pool of unlabeled data, but its effectiveness depends critically on how sample informativeness and uncertainty are quantified. Existing active learning methods for semantic segmentation predominantly rely on heuristic uncertainty measures derived from softmax scores or gradient magnitudes. These methods suffer from the well-known calibration problem of modern neural networks: models tend to produce overconfident probability estimates even when predictions are incorrect. In dense prediction settings, this miscalibration is amplified by thousands of per-pixel decisions and strong class imbalance between frequent background categories and rare but safety-critical classes. As a result, heuristic uncertainty scores often fail to identify truly informative samples, especially under realistic annotation budgets. This thesis proposes Conformal Risk Controlled Active Learning (CRC-AL), a new framework for semantic image segmentation that provides statistically grounded uncertainty quantification. CRC-AL relies on conformal risk control, an extension of conformal prediction that guarantees upper bounds on expected loss. Instead of outputting a single argmax label per pixel, the segmentation model produces class-wise softmax probabilities that are calibrated using a dedicated calibration set. For each semantic class, CRC-AL learns a threshold that transforms the softmax scores into binary prediction masks. Stacking these masks produces pixel-wise multi-class prediction sets that encode which classes remain plausible at each pixel under the chosen risk constraint. To transform these calibrated prediction sets into actionable uncertainty information for active learning, CRC-AL introduces a two-stage representation. First, it constructs co-occurrence matrices that record how often pairs of semantic classes are simultaneously included in the prediction sets across an image and normalizes them into co-occurrence density matrices that summarize the structure of class confusions. Second, the co-occurrence density matrices are flattened into high-dimensional embeddings that represent each image as a point in a semantic confusion space, capturing global patterns of uncertainty beyond pixel-wise scores. Based on these risk maps and embeddings, CRC-AL defines two selection strategies. A baseline Top-K rule ranks unlabeled images according to their image-level risk scores. However, this approach can lead to redundant sampling when many high-uncertainty images are similar. To overcome this limitation, the thesis introduces the Top-Diverse-K algorithm, a physics-inspired selection strategy that balances uncertainty and diversity in the embedding space. Each image embedding is treated as a particle whose mass is proportional to its uncertainty score, and a barycenter-based distance metric measures the distance between particles while weighting contributions by their uncertainty. Top-Diverse-K iteratively selects images that maximize a combination of average and minimum distance to the set of already labeled and selected samples. A trade-off parameter is used to control the balance between pure uncertainty sampling and diversity-driven selection. The proposed CRC-AL framework is evaluated on two complementary benchmark datasets for semantic image segmentation that differ in spatial structure, class imbalance, and appearance variability. Experiments are conducted using two representative segmentation architectures for architectural diversity. A pool-based active learning protocol is adopted in which all methods start from the same small labeled subset and query equal-sized batches of images at each iteration. Across all configurations, CRC-AL consistently outperforms various kinds of baselines under identical training schedules and annotation budgets. It achieves approximately 95\% of the performance of a fully supervised model while using only about 30\% of the labeled training data. Learning curves demonstrate that CRC-AL maintains a clear and persistent advantage across active learning iterations. Furthermore, statistical analysis at the 95\% confidence level confirms that no competing method significantly outperforms CRC-AL in the considered settings. The thesis further investigates the impact of key design choices through ablation studies. Varying the uncertainty-diversity trade-off parameter reveals that moderate values yield the best balance: very low values reduce Top-Diverse-K to pure uncertainty sampling, while very high values overemphasize diversity and select visually distinct but less informative samples. Analyses of different risk levels and calibration set choices show that using the current labeled set for calibration, although slightly biasing absolute risk values, still produces stable uncertainty rankings and strong active learning performance. These findings underscore that relative uncertainty ordering, rather than exact risk values, is the critical factor for effective sample selection. In conclusion, this thesis demonstrates that conformal risk control provides a powerful foundation for active learning in semantic image segmentation. By calibrating class-wise prediction thresholds with explicit risk guarantees and exploiting co-occurrence-based risk maps and embeddings, CRC-AL delivers reliable, interpretable uncertainty estimates and a principled diversity-aware selection mechanism. The framework substantially reduces annotation requirements while preserving segmentation accuracy on challenging benchmarks, offering practical benefits for domains where dense annotation is costly.