The need for efficient disease management in crop production is highlighted by the economic relevance of agriculture. Various leaf diseases cause major losses for tea, a staple product in many places. In this paper, a deep learning-based method is provided for the automated identification of illnesses affecting tea leaves. Using the unique architectural benefits of four models: VGG16, ViT (Vision Transformer), EfficientNetB3, and Xception, their performance is examined. Each model’s accuracy and effectiveness in classifying diseases are evaluated by conducting a thorough series of experiments on a dataset of photos of tea leaves. The results show that ViT and VGG16 perform better than expected, with accuracies of 96.09% and 98.29%, respectively. These results highlight the effectiveness of transformer-based designs and conventional CNN in the identification of tea leaf disease. Our research also demonstrates how deep learning approaches can improve agricultural practices by enabling early and accurate disease identification, which would facilitate proactive disease control tactics and increase crop output.