Explainable Vision Transformer Malware Classification

Malware continues to pose a significant threat to individuals, organizations, and critical infrastructure. As the number and complexity of malware samples continue to grow, automated malware family classification has become an important tool for supporting malware analysis, reverse engineering, and incident response. Visualization-based malware classification has become a widely studied approach by transforming executable files into images that can be processed using computer vision models. While Convolutional Neural Networks (CNNs) have traditionally dominated this research area, Vision Transformers (ViTs) have recently emerged as a promising alternative due to their ability to capture long-range relationships through self-attention. Despite their strong classification performance, Vision Transformers remain largely black-box models, making it difficult to understand which characteristics of an executable influence the model’s decision. Existing explainability methods typically produce visual heatmaps that highlight important image regions, yet these explanations remain disconnected from the underlying executable code. Bridging this gap is essential if explainable machine learning is to support practical malware analysis and reverse engineering.

Malware-to-Image Encodings

Visualization-based malware classification transforms executable files into image representations that can be processed using computer vision models. Malware binaries are encoded as grayscale or color images, allowing deep learning models to automatically learn discriminative patterns directly from the executable [1]. Different encoding strategies preserve different characteristics of the underlying executable. While the simplest approaches directly map raw byte values to grayscale or RGB images, more advanced representations capture statistical or semantic information, such as byte-transition probabilities, entropy patterns, or opcode relationships [1–6]. Consequently, the generated images capture different structural characteristics of the executable, which may influence both classification performance and model interpretability. To investigate this effect, we systematically compare six representative malware-to-image encodings: Gray, RGB, Markov Bytes, Entropy MALFCS, MCTVD, and SimHash Opcode under identical experimental conditions using the same Vision Transformer architecture on the BIG2015 and BODMAS-11 datasets.

Vision Transformer Explainability

For malware classification, achieving high prediction accuracy alone is not sufficient. Malware analysts need to understand why a model assigns a sample to a particular malware family in order to validate predictions and support subsequent reverse engineering. Explainability methods address this challenge by highlighting the image regions that contribute most strongly to a model’s prediction. However, existing Vision Transformer-based malware classification studies primarily rely on qualitative visual inspection, such as attention or activation map visualizations, while systematic quantitative comparisons of explainability methods remain limited.

To address this gap, we compare three explainability methods: Grad-CAM, Attention Rollout, and Grad-SAM. Their explanations are evaluated using complementary quantitative metrics for faithfulness, sparsity, consistency, stability, and computational efficiency, together with qualitative visual analysis. This allows a systematic comparison of explainability methods for Vision Transformer-based malware classification.

Reverse Traceability

Explainability methods provide visual heatmaps that identify influential image regions. However, malware analysts typically rely on executable artifacts to understand malware behavior and identify malicious functionality rather than on image pixels. As a result, a representation gap exists between image-based explanations and the executable artifacts that analysts use to understand malware behavior.

To bridge this gap, we propose a reverse-traceability framework that automatically maps influential image regions back to their corresponding byte offsets in the original executable. These byte ranges are subsequently linked to Portable Executable (PE) sections and, where possible, to functions recovered through static reverse engineering, enabling image-based explanations to be interpreted directly in terms of executable artifacts.

Experimental Setup

To evaluate the influence of malware-to-image encodings and explainability methods, experiments were conducted on two publicly available malware datasets: BIG2015 and BODMAS-11. BIG2015 is a widely used benchmark containing 10,801 labeled malware samples from 9 malware families, enabling direct comparison with prior work. BODMAS-11 is a more recent dataset containing 22,616 malware samples from 11 malware families and provides access to the original Portable Executable (PE) files, which is essential for the proposed reverse-traceability framework.

All six encodings were evaluated under identical experimental conditions using the same pretrained Vision Transformer (ViT-B/16) architecture. Classification performance was assessed using accuracy, macro F1-score, false positive rate, and preprocessing time. Explainability methods were evaluated using faithfulness, sparsity, consistency, stability, and computational efficiency, followed by reverse-traceability and reverse-engineering case studies.

Results

Classification Performance

Under identical experimental conditions, the investigated malware-to-image encodings achieved remarkably similar classification performance, contrasting with the substantially larger performance differences reported by many individual encoding studies. Although no single encoding consistently outperformed all others across both datasets, Markov Bytes achieved the highest classification accuracy, while Gray, RGB, and Entropy MALFCS remained highly competitive. On BIG2015, five of the six encodings achieved competitive results, although Entropy MALFCS exhibited a noticeably lower macro F1-score despite maintaining high overall accuracy. In contrast, MCTVD showed a more pronounced performance decrease on BODMAS-11, suggesting that opcode-based representations are more sensitive to dataset characteristics. More pronounced differences emerged in preprocessing efficiency than in classification effectiveness, highlighting that selecting an encoding strategy primarily involves balancing computational cost and predictive performance.

Table 1. Classification performance on the BIG2015 dataset

Table 2. Classification performance on the BODMAS-11 dataset

Explainability Evaluation

Existing Vision Transformer-based malware classification studies primarily rely on qualitative visual inspection of explanation maps. The visualization below compares Grad-CAM, Attention Rollout, and Grad-SAM for the same correctly classified malware sample across the investigated malware-to-image encodings. Because each encoding represents different characteristics of the executable—from raw bytes to byte transitions, entropy patterns, and opcode relationships—the resulting explanations highlight different structural aspects of the malware. While these heatmaps already reveal noticeable differences between the explainability methods, qualitative analysis alone makes it difficult to objectively compare explanation quality across different encodings.

To move beyond subjective visual inspection, we systematically evaluate the explainability methods using complementary quantitative metrics measuring faithfulness, sparsity, consistency, stability, and computational efficiency. Overall, transformer-specific explainability methods consistently outperform Grad-CAM across all investigated malware-to-image encodings. Attention Rollout and Grad-SAM produce substantially more faithful, sparse, consistent, and stable explanations than Grad-CAM. Among the evaluated methods, Attention Rollout provides the best overall trade-off by combining competitive explanation quality with the highest stability and significantly lower computational cost.

Table 3. Quantitative comparison of Grad-CAM, Grad-SAM, and Attention Rollout across the investigated malware-to-image encodings.

Reverse Traceability

While quantitative explainability metrics identify the most suitable explanation method, they do not reveal what the highlighted image regions represent inside the executable. To bridge this gap, we apply our reverse-traceability framework.

The proposed reverse-traceability framework successfully mapped highly influential image regions back to executable artifacts. Although only a limited proportion of image patches were directly mappable for many malware samples, the highest-ranked explanation regions could be linked to PE sections in the majority of cases. Attention Rollout achieved a substantially higher Top-1 mapping success than Grad-SAM (86.9% versus 77.1%), demonstrating that explainability methods with similar quantitative explainability performance can differ considerably in their suitability for reverse traceability. The overall reverse-traceability workflow is illustrated below.

Applying the framework revealed recurring attribution patterns across malware families. While many families primarily emphasized executable code contained in the .text section, others focused on resource, packed, or family-specific sections such as .rsrc, UPX1, and CODE. These recurring attribution patterns indicate that the recovered explanations correspond to meaningful executable structures rather than random image regions.

Most frequently recovered PE sections among the highest-ranked explanation regions for each malware family.

Reverse Engineering Case Study

To demonstrate the practical applicability of the proposed framework, we analyzed representative malware samples by tracing highly attributed image regions back to recovered executable functions. While function-level traceability remained challenging, only 14.1% of traced regions could be resolved to individual functions. Nevertheless, these recovered functions provided a focused starting point for subsequent malware analysis.

We next assessed whether the recovered functions were genuinely relevant to the model’s prediction by progressively removing (ablating) the highest-ranked functions. As shown below, removing the most highly ranked functions caused the model confidence to decrease substantially faster than removing lower-ranked or randomly selected functions, confirming that the recovered artifacts capture information that is important for the classifier.

Perturbation experiment using instruction-confirmed functions. Removing the highest-ranked recovered functions causes a substantially larger decrease in model confidence than lower-ranked or randomly selected functions.

Finally, we manually reverse engineered the highest-ranked recovered functions. The recovered artifacts were linked to recognizable malware behavior, including propagation, unpacking, API resolution, filesystem traversal, and network communication. Together, these findings demonstrate that the proposed reverse-traceability framework translates Vision Transformer explanations into executable artifacts that malware analysts can directly interpret.

References

[1] Lu, Q., Zhang, H., Kinawi, H., & Niu, D. Self-Attentive Models for Real-Time Malware Classification. IEEE Access, 10, 95970–95985, 2022.

[2] Fu, J., Xue, J., Wang, Y., Liu, Z., & Shan, C. Malware Visualization for Fine-Grained Classification. IEEE Access, 6, 14510–14523, 2018.

[3] Yuan, B., Wang, J., Liu, D., Guo, W., Wu, P., & Bao, X. Byte-Level Malware Classification Based on Markov Images and Deep Learning. Computers & Security, 92, 101740, 2020.

[4] Xiao, G., Li, J., Chen, Y., & Li, K. MalFCS: An Effective Malware Classification Framework with Automated Feature Extraction Based on Deep Convolutional Neural Networks. Journal of Parallel and Distributed Computing, 141, 49–58, 2020.

[5] Deng, H., Guo, C., Shen, G., Cui, Y., & Ping, Y. MCTVD: A Malware Classification Method Based on Three-Channel Visualization and Deep Learning. Computers & Security, 126, 103084, 2023.

[6] Ni, S., Qian, Q., & Zhang, R. Malware Identification Using Visualization Images and Deep Learning. Computers & Security, 77, 871–885, 2018.