Bangla handwriting recognition is becoming an important issue in several years but it becomes a challenge to get good performance due to the alignment and many of them are similar. A simple, lightweight CNN model has been proposed in this paper for classifying Bangla Handwriting Character, which contains 50 basic Bangla characters (11 vowels and 39 consonants). Experiments have been made on three datasets along with the BanglaLekha-Isolated [1] CMATERdb [2] and the ISI [3] dataset. For character recognition, the proposed BornoNet model gets 98%, 96.81%, 95.71%, and 96.40% validation accuracy respectively for CMATERdb, ISI, BanglaLekha-Isolated dataset and mixed dataset. Also proposed model was trained with one dataset and cross-validated with other two datasets. Proposed model achieved the best accuracy rate so far for BanglaLekha-Isolated, CMATERdb and ISI datasets.