The Internet of Medical Things (IoMT) has become an attractive playground to cybercriminals because of its market worth and rapid growth. These devices have limited computational capabilities, which ensure minimum power absorption. Moreover, the manufacturers use simplified architecture to offer a competitive price in the market. As a result, IoMTs cannot employ advanced security algorithms to defend against cyber-attacks. IoMT has become easy prey for cybercriminals due to its access to valuable data and the rapidly expanding market, as well as being comparatively easier to exploit.As a result, the intrusion rate in IoMT is experiencing a surge. This paper proposes a novel Intrusion Detection System (IDS), namely SafetyMed, combining Convolutional Neural Networks (CNN) and Long Short-Term Memory (LSTM) networks to defend against intrusion from sequential and grid data. SafetyMed is the first IDS that protects IoMT devices from malicious image data and sequential network traffic. This innovative IDS ensures an optimized detection rate by trade-off between False Positive Rate (FPR) and Detection Rate (DR). It detects intrusions with an average accuracy of 97.63% with average precision and recall, and has an F1-score of 98.47%, 97%, and 97.73%, respectively. In summary, SafetyMed has the potential to revolutionize many vulnerable sectors (e.g., medical) by ensuring maximum protection against IoMT intrusion.