The use of artificial intelligence (AI) in smart agriculture has received a lot of interest in enhancing food safety, quality and postharvest monitoring, and supply chain decision making. Proper fruit quality evaluation, and effective determination of the chemical adulterant like formalin by visualizing the fruit is still difficult because of the slight visual difference, a small set of labeled data, and model transparency is necessary. To cope with these issues, this paper will present FruitNet-V4X, an efficient but minimalistic hybrid deep learning system to implement automated fruit quality classification and chemical adulteration detection. Large-scale experiments were performed on our collected multi-fruit dataset (15 quality classes) using a five-fold cross-validation strategy which included fresh, rotten and formalin-treated fruits. The proposed FruitNet-V4X was overall better in comparison to the baseline models like MobileNetV2, VGG-19, and Inception-V3, with an accuracy of 95.51 % classification accuracy, balanced precision, recall, and F1-score. To make predictions more interpretable and credible, explainable AI (XAI) methods, such as Grad-CAM and Grad-CAM++, were used to visualize class-discriminative regions affecting predictions by the model. The findings indicate that FruitNet-V4X is an appropriate trade-off solution to the requirements of accuracy, efficiency, and interpretability, which is appropriate to be used in smart-agriculture to monitor fruit quality and detect chemical adulteration.