Pepper is cultivated all over the world and many farmers' subsistence depends on these crops. But unfortunately, farmers who are involved in the cultivation of pepper, have to fall on a huge loss because of the low production of pepper caused by several diseases of pepper. If the diseases can be detected accurately and in a short time, then the losses can be prevented. The incorrect identification and time needed process can't release from the diseases and also can't help reduce the losses. For acquiring great accuracy within a short time to recognize the pepper diseases, multirecognition methods can give promising results to the users. In this study, several pre-trained deep learning models such as VGG-19, Xception, NasNet Mobile, MobileNet-V2, ResNet-152-V2, and Inception-ResNet-V2 have been used to extract the deep characteristic from the images and these models provide great accuracy. Most of the diseases of pepper are caused by a fungal and bacterial attack. In this study, 386 images are used for training, 63 images are used for validation and 107 images are used for testing 4 classes of pepper diseases and one healthy image of pepper for identifying the diseases types of pepper. The customized CNN models have achieved the highest accuracy and fulfilled the target of this study. The picking accuracy has been achieved from the VGG-19 and ResNet-152-V2 is 96.26%. Also, Xception has provided better accuracy than Inception-ResNet-V2, MobileNet-V2, and NasNet-Mobile and that is 93.46%.