In this paper, we have demonstrated how to apply CNN (Convolutional Neural Network) structured model and transfer learning to identify the ethnicity of Bengali people and it's a systematic process of gender classification too. We also applied several models of transfer learning like VGG16, Mobilenet, Resnet50, etc. to find out which model is more convenient to get our desired accuracy. But problems arise because there are many Indian people who look like and get dressed up like Bengali since in India many Bengali dwell in when many of them speak Bangla as well! (people of Kolkata along with some other provinces). So, the Bengali people are not only found in Bangladesh but also elsewhere in the world. That's why our model is based on facial images along with the tradition of their costumes. We tried to build a sophisticated model using CNN and transfer learning for this purpose and we got some tremendous performances applying transfer learning.