In natural language processing (NLP), text categorization is a significant field of study that has drawn a lot of interest from scholars lately. Transliteration, expressing one language using an alphabet of different languages while preserving its phonetic pronunciation, presents unique challenges in language identification due to the difference in representations of languages across different writing systems. In this study, we addressed this issue by leveraging the power of deep learning on four languages which are Bangla, Hindi, Tamil, and English. We proposed a novel method transliteration language dataset. Here, at first we created a transliteration language dataset of 4 different languages by extracting languages from 12 distinct datasets, after that we utilized a hybrid architecture combining BERT (Bidirectional Encoder Representations from Transformers) with a Bidirectional Gated Recurrent Unit (BiGRU). The BERT-BiGRU model achieves exceptional performance metrics, including 98.77% accuracy, 97.57% F1 score, 99.15% specificity, and 96.74% Matthews correlation coefficient (MCC). The findings show benchmark performance. This hybrid model performs better than baseline models in terms of performance. We employed Layer Integrated Gradients to analyze the model’s decision-making process and provide insight into the linguistic elements that support language identification in order to improve the explainability of our model. This confirms the effectiveness of our model as well. With the rapid advancement of text classification in low-resource languages, this work provides a useful framework for improved accuracy in transliteration tasks.