The contagious Corona Virus (COVID-19) transmission can be reduced by following and maintaining physical distancing (also known as COVID-19 social distance). The World Health Organisation (WHO) recommends it to prevent COVID-19 from spreading in public areas. On the other hand, people may not be maintaining the required 2-m physical distance as a mandated safety precaution in shopping malls and public places. The spread of the fatal disease may be slowed by an active monitoring system suitable for identifying distances between people and alerting them. This paper introduced a deep learning-based system for automatically detecting physical distance using video from security cameras. The proposed system employed the fine-tuning YOLO v4 for object detection and classification and Deepsort for tracking the detected people using bounding boxes from the video. Pairwise L2 vectorized normalization was utilized to generate a three-dimensional feature space for tracking physical distances and the violation index, determining the number of individuals who follow the distance rules. For training and testing, we use the MS COCO and Oxford Town Centre (OTC) datasets. We compared the proposed system to two well-known object detection models, YOLO v3 and Faster RCNN. Our method obtained a weighted mAP score of 87.8% and an FPS score of 28; both are computationally comparable.