In this paper we present a novel image classification framework, which is able to automatically re-configure and adapt its feature-driven classifiers and improve its performance based on user interaction during on-line processing mode. Special emphasis is placed on the generic applicability of the framework to arbitrary surface inspection systems. The basic components of the framework include: recognition of regions of interest (objects), adaptive feature extraction, dealing with hierarchical information in classification, initial batch training with redundancy deletion and feature selection components, on-line adaptation and refinement of the classifiers based on operators' feedback, and resolving contradictory inputs from several operators by ensembling outputs from different individual classifiers. The paper presents an outline on each of these components and concludes with a thorough discussion of basic and improved off-line and on-line classification results for artificial data sets and real-world images recorded during a CD imprint production process.status: publishe