Accurate multi-class classification of fruits is vital for smart agriculture and also within intelligent food processing systems, where accurate classification is always required. This investigation presented a novel deep learning-based framework to classify eight classes of fruits, which are Bilimbi, Burmese Grape, Carambola, Elephant Apple, Emblica, Sapodilla, Tamarind, and Wood Apple, using different deep learning models, such as CNN and Vision Transformer. This investigation used a robust labeled image set, where accurate preprocessing was obtained, and different deep learning models were trained under exactly the same conditions. For accurate classification, different CNN models were used in this investigation. The best CNN model, which obtained higher accuracy than other CNN models, obtained CNN accuracy of 0.99, which signifies better performance in terms of local feature extraction capability. In addition, another model, such as Vision Transformer, was also used with better accuracy, which reached 0.99, signifying accurate performance globally using self-attention mechanisms. The training, validation, loss, and accuracy curves obtained using this model never crossed each other, signifying better convergence towards accurate generalization without any overfitting. For higher transparency and trustworthiness of deep learning models, the performance of this best-trained model was appropriately analyzed using LIME and Saliency Map techniques. In general, it can be concluded that the comparative analysis of CNN models and ViT models, with the integration of AI, offers a reliable and accurate solution, which can be applied in real-world intelligent agricultural applications.