Low contrasts and visual similarity between different skin conditions make skin disease recognition a challenging task. Current techniques to detect and diagnose skin disease accurately require high-level professional expertise. Artificial intelligence paves the way for developing computer vision-based applications in medical imaging, like recognizing dermatological conditions. This research proposed an efficient solution for skin disease recognition by implementing Convolutional Neural Network (CNN) architectures. Computer vision-based applications using CNN architectures, MobileNet and Xception, are used to construct an expert system that can accurately and efficiently recognize different classes of skin diseases accurately and efficiently. The proposed CNN architectures used a transfer learning method in which models are pre-trained on the Imagenet dataset to discover more features. We also evaluated the performance of our proposed approach with some of the most popular CNN architectures: ResNet50, InceptionV3, Inception-ResNet, and DenseNet, thus establishing a comparison to set up a benchmark that will ratify the essence of transfer learning and augmentation. This study uses data from two separate data sources to collect five different types of skin disorders. Different performance evaluation indicators, including accuracy, precision, recall, and F1-score, are calculated to verify the success of our technique. The experimental results revealed the effectiveness of our proposed approach, where MobileNet achieved a classification accuracy of 96.00%, and the Xception model reached 97.00% classification accuracy with transfer learning and augmentation. Moreover, we proposed and implemented a web-based architecture for the real-time recognition of diseases.