Scopus Indexed Publications

Paper Details


Title
TransCall: A Transformer-Driven Framework for Zero-Day Malware Detection Using System Call Sequences

Author
Abdullah Al Siam, Nuruzzaman Faruqui,

Email

Abstract

The growing complexity of contemporary malware, along with the widespread use of polymorphism, obfuscation, and environment-aware evasion, has significantly diminished the effectiveness of conventional signature-based and static detection methods. Behavioral analysis, especially via system call sequences, provides a robust depiction of program intent; however, current machine learning and recurrent neural network (RNN) methods struggle to capture long-range dependencies and often do not generalize well to new, unseen threats. This paper presents TransCall, a streamlined Transformer-based framework for detecting zero-day malware by analyzing system call sequences. It utilizes multi-head self-attention to capture global contextual relationships within syscall streams, facilitating the practical identification of malicious behavioral patterns, even in the presence of unknown malware families. The suggested framework integrates a highly effective embedding module, a streamlined Transformer encoder, and a refined classification head designed for real-time inference in endpoint security systems. Experimental evaluations on UNM-style syscall datasets show that TransCall outperforms traditional baselines, including Random Forest, LSTM, and GRU models. In a standard train-test division, It reaches an F1-score of 0.8571 and an AUC of 0.8743. In a challenging zero-day environment where entire malware families are omitted from the training process, TransCall attains an impressive F1-score of 0.9286, highlighting its strong generalization capabilities. Moreover, visualizing attention weights improves comprehension by emphasizing crucial syscall interactions linked to potentially harmful activities. In therefore, TransCall delivers accurate, clear, and efficient detection of zero-day malware, presenting a practical and scalable solution for modern cybersecurity environments.


Keywords

Journal or Conference Name
2026 IEEE 5th International Conference on AI in Cybersecurity, ICAIC 2026

Publication Year
2026

Indexing
scopus