Cardiac disorders can be fatal if not identified and treated in the primary stage. Cardiovascular disorders are categorized using electrocardiograms (ECGs), and doctors and clinicians frequently use paper-based ECG images to identify the patient’s condition. This research intends to achieve the best precision and least time complexity when classifying heart disorders into five classifications utilizing paper-based ECG images and a deep learning method. This study takes a two-pronged strategy. Five deep learning models are used in the first method: InceptionV3, ResNet50, MobileNetV2, VGG19, and DenseNet201. The proposed model is undergoing an ablation study, which modifies some of its elements and hyperparameters, further improving performance. Multiple image pre-processing methods are used to get rid of artifacts and improve the image quality before the model is trained. With a testing accuracy of 98.34%, our anticipated hybrid InRes-106 model outperformed the competition. The InceptionV3 model acquired a testing accuracy of 90.56%, the ResNet50 89.63%, the DenseNet201 88.94%, the VGG19 87.87%, and the MobileNetV2 achieved 80.56% testing accuracy. Our model is trained using a k-fold cross-validation procedure using various k values to assess the resilience further. Our suggested method, which is based on a variety of picture pre-processing techniques, model fine-tuning, and ablation studies, may accurately identify cardiac illnesses, even though the ECG dataset only contains a small number of complex ECG images.