The diagnosis of glioma, a complex and often deadly brain tumor, involves extensive medical examinations. Still, accurately grading and classifying gliomas is difficult, as different areas within the same tumor can exhibit varying characteristics. The integration of radiomics, a clinically relevant feature extraction method, with machine learning (ML) is becoming increasingly popular in addressing this issue, but several research gaps persist. To this end, this study proposes a novel deep neural network, RGNN3D, that combines Graph Neural Networks with LSTM layers to precisely grade gliomas in 3D magnetic resonance imaging (MRI) data. To train our proposed model, we meticulously extracted 112 radiomic biomarkers. Utilizing the biomarkers, RGNN3D constructs a graph, channels essential information through its layers, and preserves only pertinent information via its integrated memory cells. The proposed framework attained an accuracy of 98.58%, aligning with the performance of previous state-of-the-art architectures and surpassing prior radiomic-based ML models. We further employed an explainable AI approach (LIME) to highlight the most significant features, assisting radiologists in making more informed decisions. In short, RGNN3D offers a reliable and robust computer-aided solution for potential clinical application in the automated identification of gliomas.