Diseases and pests of jute hinder the quality production of fiber which is a malignant threat to the jute industry, causing severe financial losses to cultivators. Early recognition of diseases and pests of jute plant is highly vital for preventing the spread of diseases and pests which will ensure the quality improvement of the jute industry. This paper addresses a robust hybrid model, namely JuteNet, is a multi-scale feature fusion approach for early recognition of jute diseases and pests. First, a dataset of 56,108 images of jute leaves and stems is generated. Afterward, the fusion of extracted features from images by deep neural networks such as Xception, InceptionResNetV2, and InceptionV3 was used to develop JuteNet that obtained 99.47% accuracy in recognizing 2803 images of six classes of the testing set. Moreover, Xception, InceptionResNetV2, and InceptionV3 separately acquired 91.83, 96.11, and 98.86% of test accuracy, which validates the recognition efficiency of JuteNet.