Mostly in the agriculture sector, identifying rotten fruits has been critical. The classification of fresh and rotting fruits is typically carried out by humans, which is ineffective for fruit growers. Humans wear out by doing the same role many days, but robots do not. As a result, the study proposed a method for reducing human effort, lowering production costs, and shortening production time by detecting defects in agricultural fruits. If the defects are not detected, the contaminated fruits can contaminate the good fruits. As a result, we proposed a model to prevent the propagation of rottenness. From the input fruit images, the proposed model classifies the fresh and rotting fruits. We utilized three different varieties of fruits in this project: apple, banana, and oranges. The features from input fruit images are collected using a Convolutional Neural Network, and the images are categorized using Max pooling, Average pooling, and MobileNetV2 architecture. The proposed model's performance is tested on a Kaggle dataset, and it achieves the highest accuracy in training data is 99.46% and in the validation set is 99.61% by applying MobileNetV2. The Max pooling achieved 94.49% training accuracy and validation accuracy is 94.97%. Besides, the Average pooling achieved 93.06% training accuracy and validation accuracy is 93.72%. The findings revealed that the proposed CNN model is capable of distinguishing between fresh and rotting fruits.