Northern Bangladesh is home to the most lychee growing, which has a major economic impact. Lychee output and quality are reduced by various leaf and fruit diseases. Deep learning is used to construct a disease detection system for lychee diseases to detect them early and accurately. We correctly identified 6,000 photos of healthy and unhealthy lychee foliage and fruits. Six deep learning algorithms - VGG16, CustomCNN, MobileNet, InceptionV3, ResNet50, and Vision Transformer - classified the photos. Normalization and augmentation improved model resilience during data preparation. F1-score, recall, accuracy, and precision were utilized to train and evaluate the models. Vision Transformer (ViT) scores 99.91% in accuracy, recall, and F1-score, outperforming the other models. This achievement shows that ViT can better recognize complex lychee disease traits. The ViT algorithm for disease detection can help Bangladeshi farmers identify illnesses quickly and accurately. Reduced lychee loss and improved fruit quality and attractiveness should boost lychee sales in domestic and international markets.