Tree disease classification is needed to determine the affected leaves as it controls the economic importance of the trees and their products and decreases their eco-friendly eminence. The lychee tree is affected by some of the diseases named Leaf Necrosis, Stem Canker and leaf spots. Therefore, classifying the Lychee tree is essential to find the good and affected leaves. Our economic growth will be very high if we can adequately do the Lychee tree classification. In this paper, we tried to do a Lychee tree disease classification to make things easier for the farmers as they cannot correctly distinguish the good and bad leaves in an earlier stage. We have created a new data set for training the architectures. We have collected about 1400 images with three categories of pre-harvest diseases “Leaf Necrosis”, “Leaf Spots”, and “Stem Canker”. There are 1400 images in total, and out of those, 80% of the data is for training and 20% is for testing, this dataset has fresh and affected leaves and stems. For Lychee tree disease classification, we have chosen pre-trained CNN and Transfer Learning based approach to classify the layer of the 2D image by layer. This method can classify images efficiently from the images of disease leaves and stems. It will address disease from the images of the leaves and trees and determine specific preharvest diseases.