Automatic detection of road surface condition and classified data storage is proposed using a customized VGG16 model. One of the primary concerns affecting safety in transportation is road surface distress. The first sign of an asphalt pavement’s catastrophic collapse is a surface crack, which can later progress to become a pothole and incur expensive repairing costs. Conventional detection methods of road surface cracks or degradation that included manual checking by humans adds to additional time and resource cost which can be eradicated by replacing the monitoring system with an automated computer program that we are proposing in this study. A deep neural network that is trained on custom dataset collected manually by taking photos of roads to successfully detect smooth and cracked or damaged road surfaces is proposed on this domain. VGG16 with a custom input layer has been proven to achieve 97.52% accuracy in detection of both smooth and damaged surfaces which, compared to others, is much better than the other models. The classified images from the model can later be used by specific authorities trying to maintain the road infrastructures.