Handwritten letter classification of any given language has the potential to be used in various fields such as literature, educational institutions, digitization of government records etc. Bengali language with its complex sets of mixed characters, poses significant complexities in terms of automatic recognition of characters. In the Bengali character set, there are over 360 distinct characters among which a lot of similarities are present between different characters. Thus, the classification of these characters gets harder as the recognition system incorporates all these distinct characters. In recent years, a lot of research has been done to solve this problem on isolated datasets with significant results. Continuing the advancement in image processing, In this paper, we have proposed a custom CNN model which has been trained on Bangla Lekha Isolated dataset containing 1,66,106 images belong to 84 distinct classes with the capability to detect individual handwritten Bengali letters including digits, vowels, consonants and compound characters. Our proposed model was able to achieve 94.73% accuracy while using less number of parameters compared to existing popular models