This study examined the feasibility of deep learning–based concrete damage classification under small-scale data conditions. Four representative CNN models—GoogLeNet, ResNet-50, EfficientNet-B0, and MobileNetV2—were employed using a dataset of 3,000 images representing three types of damage: crack, efflorescence, and rebar exposure. The number of training images was varied at 100, 250, 500, and 1,000, and performance was evaluated in terms of accuracy, F1-score, training time, and t-SNE visualization. The experimental results showed that all models exhibited a performance saturation point around 500 images. ResNet-50 and EfficientNet-B0 achieved high accuracy (around 92%) and distinct cluster separability, while MobileNetV2 demonstrated real-time applicability owing to its lightweight structure and fast computation. Among the damage types, rebar exposure achieved the highest classification accuracy, whereas efflorescence showed relatively lower accuracy with greater variability. Overall, this study confirms that reliable classification performance can be achieved with more than 500 images, and provides practical criteria for selecting CNN models for field applications under limited data conditions. Future research should focus on expanding data diversity, validating with real-world images, and applying advanced preprocessing and augmentation techniques.