Increasing demand for faster transportation has resulted in a rapid growth of rail track networks countrywide. The irregularities of properly monitoring rail tracks have been often reasons behind collisions of trains and obstacles, such as humans, animals or even other vehicles during crossing the railway track. Proper monitoring can cause a huge extra manpower which also refers to a magnificent increase in maintenance cost for the authorities. For reducing human interaction, multiple studies of applying deep learning approach to monitor rail tracks have been performed. Differences in surrounding environment and arrangement of rail crossings in different locations may have different impacts over the accuracy of the models. In our study, a convolutional neural network-based transfer learning model has been implemented on a custom dataset. Our proposed model “MobileNetV2” has proven its efficiency of using resources and able to detect obstacle 97.00% accurately. Though we have also applied YOLOv5, Resnet50, VGG19, VGG16 on our collected dataset but MobileNetV2 performed better than other selected models. The potential of being able to be used in low configured devices ensures that the model can maintain a balance among detecting speed and processing.