The precise forecasting of agricultural productivity under climate change is essential for food security where many areas are sensitive. In this work, a multiclass classification problem is shown for categorizing crop yield as Low, Medium and High using climate and environment variables of Bangladesh. We apply the proposed method to a dataset with features such as rainfall, average temperature, drought severity, flood impact, cyclone frequency and air quality comparing the performance of SAINT (self-attention for tabular data), NODE (Neural Oblivious Decision Ensembles) and Support Vector Machine (SVM). Furthermore, we also consider a stacked ensemble. Results indicate that 99% accuracy is achieved by the ensemble model, which is superior to three single models: SAINT (95.33%), NODE (96.83%) and SVM (95.50%). The feature importance analysis indicates that rainfall and average temperature are the most influential features for predicting agricultural yield. This study indicates that hybrid models are effective in monitoring the climate impacts on agriculture, which can provide decision-making supports for stakeholders, policy makers, and agronomists.