Widely known as the "Golden Fiber", jute is regarded as one of the most important and profitable crops in many countries including Bangladesh. Jute and jute-based commodities can bring a lot of foreign income and eventually boosts the overall economy of the country. However, many a time, jute production gets hindered due to many harmful pests and insects. Even though farmers identify and take actions against these pests following a manual procedure, it is often tedious and time-consuming. That is why it may be very beneficial to have a machine learning-based approach towards pest detection. This paper proposes a deep CNN model named "PestDetector" that can correctly identify 4 major types of jute pests (Field Cricket, Jute Stem Weevil, Spilosoma Obliqua, and Yellow Mite) with substantial accuracy. The work is done on a total of 2200 images separated into 3 categories: Training, Validation, and Testing. The model ultimately demonstrates 99.18% training accuracy and 99.00% validation accuracy. Additionally, the model's overall performance has been assessed using precision, recall, F1-score, and confusion matrix.