Sign language is a medium of communication for a person with an auditory and verbal disability or deficiency. Therefore, it is essential to understand their hand gestures without difficulty in order to have effortless and improved communication. Hand gesture detection is a challenging task. In this paper, we proposed an efficient method to recognize and classify images that contains hand gesture, using image Segmentation and the Bottleneck feature from a pre-trained model of Deep Neural Network. Our model achieved a descent accuracy over 96% therefore can be used to build an efficient system which can work as an interpreter between the disabled person and the other party. A comparison between conventional CNN (Convolutional Neural Network) model and our model is also shown to measure the effectiveness of our proposed method.