There are a lot of works on English handwritten materials but not seen so much work on Bangla handwritten recognition. Bangla handwritten digits have more complex and diverse shapes and also have high inter-class similarities. So it is more challenging to recognize Bangla handwritten digits. The paper focuses on improving the lack of Bangla handwritten digit recognition approach using deep learning models. In the research, we have developed three deep learning-based models, CNN, DenseNet121, and ResNet50 to recognize Bangla digits. To make our work efficient we have used two existing BanglaLekha-Isolated-Numerics and Ekush datasets along with a new dataset that is collected by our team. We tested the three models on each dataset and achieved remarkable performance in all the datasets. Among the models, DenseNet121 outperformed on every dataset. DenseNet121 achieved 99.09% test accuracy on the Ekush dataset and 97.65% test accuracy on our developed dataset.