The purpose of our model is to classify five types of flowers from input images. The flowers are Sunflower, Rose, Tulip, Daisy, and Lavender. We have also built our own CNN model for the task and compared it with the modified VGG16 network. Our modified VGG16 model gives better accuracy than the existing works. We have achieved a test accuracy of 96.64% by using the proposed model. As the accuracy is quite good, we were able to recognize the flowers accurately. Agriculture institutes and flower nurseries can be benefitted by using this model.