Potato is one of the most significant crops over the world. But production of potato is hampered due to some diseases which cause an increase of the cost as well as affect the life of the farmers. An automatic and early detection of these diseases will increase the production and help to digitize the system. Our main objective is to detect the potato diseases with a few leaf image data using advanced machine learning techniques. In this paper, we demonstrate that transfer learning technique could be used for early detection of potato diseases when it is difficult to collect thousands of new leaf images. Transfer learning uses already trained deep learning model's weight to solve new problem. The experiments included images of 152 healthy leaves, 1000 Late blight leaves, and 1000 early blight leaves. The program predicts with an accuracy of 99.43% in testing with 20% test data and 80% train data. We also compared sequential deep learning model with several pre-trained model applying transfer learning and found that transfer learning provided best result till date. Our output showed that transfer learning outperform all existing works on potato disease detection.