Colorectal cancer stands as the third most prevalent cancer diagnosis worldwide, and an accurate histopathological classification is essential for the appropriate planning of treatment. Conventional diagnostic modalities are associated with inter-observer variability and are time-consuming, limiting their clinical utility. In this study, we introduce TransCapsNet, a hybrid learning framework designed for colorectal histopathological and gastrointestinal endoscopic image classification across multiple disease classes. Multiple pre-processing methods, such as normalization, rescaling, and data augmentation via rotation, flipping, shear transformations, and brightness modification, are used to improve the model’s robustness and generalization. The architecture uses VGG-16 to extract features, followed by transformer blocks for contextual modelling and capsule networks to capture spatial relationships in tissue organization. The model was evaluated on two publicly available datasets comprising three-class histopathological images and four-class endoscopic images. The proposed model was extensively assessed and benchmarked against four baseline models: MobileNetV2, VGG19, InceptionV3, and DenseNet169. The results show excellent performance, with test accuracies of 99.64% for the three-class histopathological classification and 99.67% for the four-class endoscopic disease detection, along with sensitivities, precisions, and F1-Scores of 1.00 for both datasets. To provide qualitative interpretation of model predictions, SHAP, LIME, Grad-CAM, and Grad-CAM + + were applied to visualise the image regions that contributed to the predicted classes. The system has been implemented as a web-based interactive prototype tool to support AI-assisted classification of colorectal histopathological and gastrointestinal endoscopic images.