Breast cancer is a complex and often fatal malignancy in women worldwide, requiring thorough medical examinations. Accurately detecting breast cancer is challenging due to its diverse forms, stages, symptoms, and diagnostic techniques. With advancements in artificial intelligence, an automated computerized method can potentially aid radiologists in the early detection of breast cancer. This study presents a novel and robust deep neural network, EAH-Net, for breast cancer diagnosis using ultrasound images. The EAH-Net architecture comprises an ensemble attention module, a modified UNet model that performs segmentation by isolating regions of interest, and a hybrid approach to classify breast cancers accurately. Besides, we employed explainable AI techniques to highlight the most significant regions, assisting radiologists in making more informed decisions. The proposed segmentation framework yields promising outcomes across Jaccard, Precision, Recall, Specificity, and Dice metrics, averaging 89.26 ± 0.36, 91.79 ± 1.13, 92.98 ± 1.08, 99.38 ± 0.35, and 95.26 ± 0.45 percents, respectively. The hybrid classification framework demonstrates outstanding performance with an accuracy of 98.48 ± 0.18%. Overall, EAH-Net offers a reliable and robust computer-aided solution for automated breast cancer diagnosis.