Cancer remains a leading cause of mortality worldwide, with early detection and accurate diagnosis critical to improving patient outcomes. While computer-aided diagnosis systems powered by deep learning have shown considerable promise, their widespread clinical adoption faces significant challenges in maintaining consistent performance across diverse imaging modalities and datasets. This research addresses the critical challenge of developing robust, generalizable deep learning models by proposing a comprehensive framework utilizing seven diverse medical imaging datasets encompassing fundus photography, histopathology, endoscopy, and MRI, covering diseases such as ocular toxoplasmosis, endometrial cancer, colorectal cancer, gastrointestinal disease, breast cancer, brain tumor, and tympanic membrane conditions. Our methodology combines customized data augmentation strategies (photometric, geometric, and elastic transformations) with an optimized vision transformer with external attention (MViTX) architecture. The MViTX model demonstrated exceptional performance with test accuracies ranging from 94.1 to 99.1% across all datasets, achieving superior metrics in accuracy, precision, recall, F1-score, and AUC compared to state-of-the-art CNNs. The model's effectiveness was further validated through ablation studies and explainable AI techniques, while its practical utility was demonstrated through deployment as a user-friendly web application. Our research establishes the effectiveness of combining tailored data augmentation with attention-based transformer architectures for medical image analysis, representing a significant step toward enhancing healthcare professionals' diagnostic capabilities and ultimately improving patient care outcomes.