This research performed a comparative study of five CNN architectures in the classification of American Sign Language (ASL) alphabets from a dataset that contained 87,000 images belonging to 29 classes. The models were compared and evaluated using accuracy, precision, recall, F1-score, and training time. The optimal performance was achieved by achieving a 99% with all the evaluation measures. VGG16 and MobileNet followed closely with precision, recall, and F1-score accuracies of 99%, thus being more appropriate for real-time use where computation is fast. LeNet-5 also performed reasonably well with accuracy of 98% and precision of 99%, indicating the possibility of utilizing simpler architectures. ResNet50 performed poorly with accuracy of 73% and F1 of 74%, possibly because of overfitting or poor hyperparameter tuning. Thus, the outcome suggests that more profound models such as AlexNet can work but light models such as MobileNet provide the best optimal gains in the trade-off of working capacity and resource thriftiness in real-time or edge-based ASL recognition systems. In spite of the good performance here, it is only restricted to in-distribution testing on a controlled dataset. Further research will also bring this analysis with the out-of-distribution testing and comparison to newer lightweight architecture to assure high levels of robustness in their real-world application.