Cauliflower, a winter seasoned vegetable that originated in the Mediterranean region and arrived in Europe at the end of the 15th century, takes the lead in production among all vegetables. It's high in fiber and can keep us hydrated, and have medicinal properties like the chemical glucosinolates, which may help prevent cancer. If proper care is not given to the plants, several significant diseases can affect the plants, reducing production, quantity, and quality. Plant disease monitoring by hand is extremely tough because it demands a great deal of effort and time. Early detection of the diseases allows the agriculture sector to grow cauliflower more efficiently. In this scenario, an insightful and scientific dataset can be a lifesaver for researchers looking to analyze and observe different diseases in cauliflower development patterns. So, in this work, we present a well-organized and technically valuable dataset “VegNet’ to effectively recognize conditions in cauliflower plants and fruits. Healthy and disease-affected cauliflower head and leaves by black rot,downy mildew, and bacterial spot rot are included in our suggested dataset. The images were taken manually from December 20th to January 15th, when the flowers were fully blown, and most of the diseases were observed clearly. It is a well-organized dataset to develop and validate machine learning-based automated cauliflower disease detection algorithms. The dataset is hosted by the Institute – National Institute of Textile Engineering and Research (NITER),the Department of Computer Science and Engineering and is available at the link following: https://data.mendeley.com/datasets/t5sssfgn2v/3.