Magnifera Indica, traditionally known as mango, is a drupe found around the world in over 500 species. India has produced 19.5 million metric tons of mango in 2017. In Bangladesh, mango has been referred as the national tree and government has included endemic species of mango as geographical index (GI) of Bangladesh. Recognizing specific breeds has become a significant computer vision task. In this paper, we have proposed the convolutional neural network (CNN) based approach for detecting five mango species namely, Chosha, Fazli, Harivanga, Lengra and Rupali from 15000 different images. For better experimentation, we have applied three different models of CNN and analyzed the recognition rates with various criteria. For performance evaluation, we have utilized the classic metrics such as precision, recall, F1-score, ROC and accuracy. Among the experimented three models, the third model, outperformed in terms of accuracy with 92.80%.