The Autonomous machineries are changing our thinking level day by day. Autonomous driving system is a field of likelihood of computer vision research. The flourished countries has already come to the fore and using semi autonomous driving vehicle. They constructed the driving model as their eco-friendly and with their road traffic data. But in Indian sub-continent countries like Bangladesh, there has a little research work and implementation on it. In our research we have made a full size vehicle where we can set up our sensors, cameras and other hardwares to collect the visual training data to train any kind of vision based autonomous vehicle algorithms. We also proposed a CNN model to classify the real-time objects with our dataset. Finally, tested the trained model using the vehicle in real environment and got expected output. Here we used a rickshaw (Bangladeshi traditional three wheeler) as our base vehicle and use some custom made hardware to full-fill our purpose.