Fruit classification is crucial in various industries. This hierarchy helps vendors in different supermarkets to identify the species of fruit and also influences the pricing accordingly. The classification approach makes it much easier to determine if the fruit is good or poor. It is important to be able to quickly detect fruits when exporting fruits. Our economic situation will deteriorate if we are unable to export fresh fruits. So, in this case, a fruit classification system can be useful in a variety of areas, including autonomous agricultural robots and the production of smartphone apps for identifying unique fruit species on the market. In this study, we have taken a total of 5658 fruits images which are based on 10 classes. In the agriculture sector, detecting rotten fruits has been crucial. Humans are typically used to classify fresh and rotting fruits, which is ineffective for fruit growers. Humans get exhausted after doing the same role many times, but Machines however do not. As a result, the initiative recommends a strategy for reducing human effort and lowering costs. By finding defects in agricultural fruits, the expense and time for processing may be minimized. In our proposed classification system, we have worked on five CNN models. The InceptionV3 model has the highest accuracy, at 97.34 %.