Tomatoes are globally valued for their nutritional benefits and unique taste, playing a crucial role in agricultural productivity. Accurate diagnosis of tomato leaf diseases is vital to avoid ineffective treatments that can harm plants and ecosystems. While deep learning models excel in classifying these diseases, distinguishing subtle variations remains challenging. This study introduces XSE-TomatoNet, an enhanced version of EfficientNetB0, incorporating Squeeze-and-Excitation (SE) blocks and multi-scale feature fusion to boost classification performance. XSE-TomatoNet extracts multi-scale features, refines them with SE blocks, and merges them through Global Average Pooling, providing detailed and broad insights for precise disease classification. Our approach achieves an impressive accuracy of 99.11%, with 99% precision and recall, outperforming models like MobileNet and VGG19, especially when combined with data augmentation and ablation studies. The model achieved an average training accuracy of 99.41% and a validation accuracy of 98.88% in 10-fold cross-validation, showing strong generalization to unseen data. We also used LIME and SHAP for model interpretability, offering insights into the decision-making process, and employed Grad-CAM and Grad-CAM++ to visually highlight key areas in leaf images. Finally, the best model was integrated into a web-based system for practical use by tomato cultivators.