Deep learning-based automated detection of oral squamous cell carcinoma in histopathological images: a comparative study of five CNN architectures

Abstract

Oral squamous cell carcinoma (OSCC) is the most common malignancy of the oral cavity, and early diagnosis plays a crucial role in improving patient prognosis and survival rates. Histopathological examination remains the gold standard for OSCC diagnosis; however, this process is time-consuming and highly dependent on expert interpretation. With the rapid development of digital pathology and artificial intelligence, deep learning-based approaches have emerged as promising tools to support automated diagnostic systems. In this study, five convolutional neural network (CNN) architectures-VGG16, ResNet50, InceptionV3, EfficientNetV2S, and ConvNeXt-Tiny-were comparatively evaluated for the automated classification of OSCC using histopathological images. An open-access OSCC dataset was utilized, and two experimental scenarios were created using the original dataset and an augmented dataset. The dataset was divided into training, validation, and test subsets using a stratified approach. All models were trained under identical experimental conditions using ImageNet-pretrained weights and a unified classifier head in order to ensure a fair comparison. Model performance was assessed using Accuracy, Precision, Recall, Specificity, F1-Score, and ROC-AUC metrics. Additionally, Grad-CAM was applied to visualize the image regions influencing model predictions and to enhance interpretability. The results demonstrated that data augmentation significantly improved the performance of all models. Among the evaluated architectures, ResNet50 achieved the highest diagnostic performance on the augmented dataset, reaching an accuracy of 0.91 and a ROC-AUC of 0.88, followed by EfficientNetV2S and ConvNeXt-Tiny. Visualization analyses indicated that the models focused on histopathologically relevant regions associated with tumoral structures. Overall, the findings suggest that deep learning-based approaches can effectively support patch-level automated OSCC classification from histopathological images and may contribute to future development of clinical decision support systems in digital pathology.