The recognition of citrus fruit is one of the most challenging and crucial measures in citrus yield mapping. Several artificial vision systems have been proposed to solve the issue of fruits recognition problem with sundry effects. In this study, we developed an automated system to categorize citrus fruit images using Convolutional Neural Network. We categorized two different citrus fruits, Orange (Citrus Sinensis) and Kinnow (Citrus Reticulate). Firstly the images of Orange and Kinnow were collected and preprocessed. Secondly, the fruit images and their background were segmented by image segmentation and edge detection. Four main features of Orange and Kinnow fruit were extracted based on image segmentations such as fruit size, surface color fruit shape and fruit surface defects. These features were examined through Convolutional Neural Network. We implemented three separate Convolutional Neural Network models to further experiment and tested recognition rates for different parameters. We have used the classical measurements including precision, recall, F1 score, ROC and accuracy for performance evaluation. Among the three experimented models, the third model was outperformed by 92.25% percent accuracy.