Leaf rot and foot rot are the common diseases of betel leaf, cause grievous economic losses to farmers. These diseases should be recognized accurately in the early stage to prevent the expansion of diseases to ensure the effective improvement of betel leaf production. This study presents an approach of early disease recognition for betel leaf to attain a satisfactory balance between accuracy and real-time recognition. First, a dataset of 10662 images of betel leaf has been established. Afterward, an improved convolutional neural network (CNN) based recognition model which contains three depth-wise separable convolutions and two fully connected layers, namely, betel leaf CNN (BLCNN), has been built from scratch that realizes 96.02% accuracy under the test set of 1031 images with the Swish activation function. Another CNN architecture built with normal convolution layer has achieved 89.53% test accuracy under the same training strategy but consumed more training time compared to BLCNN.