Current plant disease detection systems rely on generic visual learning without incorporating botanical expertise, failing to capture anatomical relationships critical for accurate diagnosis. This limitation stems from the disconnect between machine learning architectures and the structured knowledge that plant pathologists use to identify diseases. We introduce SwiftM, a framework that incorporates botanical knowledge distillation, the novel integration of plant anatomical structure into deep learning for agricultural diagnostics. SwiftM introduces three key components: ‘botanical knowledge distillation’, incorporating plant-specific anatomical priors through multi-scale feature synthesis, ‘structure-aware graph representation’, modelling spatial relationships between biologically meaningful leaf regions (vein, margin, lamina) as graph nodes and ‘sparse swin attention over botanical graphs’, applying attention to anatomical structures rather than arbitrary patches. Unlike conventional approaches treating plant disease detection as generic image classification, SwiftM presents a biology-informed approach that combines agricultural expertise with machine learning. The framework demonstrates significant cross-domain generalization: 99.19% accuracy on sweet orange diseases, 99.81% on mango diseases and 100% on rice diseases while maintaining efficiency with only 2.69M parameters. This work explores how domain-specific biological knowledge can be integrated into neural architectures for agricultural AI. SwiftM contributes toward more interpretable, domain-aligned AI systems by moving from purely data-driven detection toward knowledge-guided diagnosis.