Handwritten digit recognition has always a big challenge due to its variation of shape, size, and writing style. Accurate handwritten recognition is becoming more thoughtful to the researchers for its educational and economic values. There had several works been already done on the Bangla Handwritten Recognition, but still there is no robust model developed yet. Therefore, this paper states development and implementation of a lightweight CNN model for classifying Bangla Handwriting Digits. The proposed model outperforms any previous implemented method with fewer epochs and faster execution time. This Model was trained and tested with ISI handwritten character database Bhattacharya and Chaudhuri (IEEE Trans Pattern Anal Mach Intell 31:444–457, 2009, [1], BanglaLekha Isolated Biswas et al. (Data Brief 12, 103–107, 2017, [2]) and CAMTERDB 3.1.1 Sarkar et al. (Int J Doc Anal Recogn (IJDAR) 15(1):71–83, 2012, [3]). As a result, it was successfully achieved validation accuracy of 99.74% on ISI handwritten character database, 98.93% on BanglaLekha Isolated, 99.42% on CAMTERDB 3.1.1 dataset and lastly 99.43% on a mixed (combination of BanglaLekha Isolated, CAMTERDB 3.1.1 and ISI handwritten character dataset) dataset. This model achieved the best performance on different datasets and found very lightweight, it can be used on a low processing device like-mobile phone.The pre-train model and code for all these datasets can be found on this link https://github.com/shahariarrabby/Bangla_Digit_Recognition_CNN.