Sponge gourd, called as “ Open image in new window ” in Bangladesh, is scientifically known as Luffa cylindrical that belongs to Cucurbitaceae family. Sponge gourd or luffa gourd is one of the most easily found vegetable in Bangladesh. It is an edible wild vegetable that the plant can be seen anywhere around us during late summer till late autumn in Bangladesh. Cooked sponge gourd or the curry is a bit sweetish in taste. Even though sponge gourd is kind of a wild vegetable plant, in recent time a lot of people are cultivating it in the countryside thinking of profit and commercial production since there is a market demand for it. Despite of having every opportunity to commercial benefit most of the farmers neglect the issue of insects and diseases attack on the plant resulting on huge loss in the business. Also lack of proper knowledge of related diseases, advance technology and trustable source the farmers lag behind the diseases detection process to use pesticides or different methods of reducing diseases attack. If necessary steps can be taken to prevent the insects and diseases attack at the very beginning of cultivation, then the profits will increase as the crop yields increase. This research paper attempts to detect the leaf and flower diseases of sponge gourd using Convolutional Neural Network (CNN) and image processing techniques. CNN and image processing are one of the most recently introduced technologies using in the agriculture sector in Bangladesh ensuring highest accuracy. This system will take leaf images as input and after examining them healthy or detected diseases will be shown as output which has diffrent true and false values for different diseases and reached to the average accuracy of 81.52%.