A lot of people in Bangladesh suffer from mobility impairments and being neglected in many steps of their life. Despite having the ability to do something outstanding, it is not possible to express that for this huge portion of people due to the physical challenges. Nowadays, technology is considered a blessing not only for normal peoples but also for physically challenged people. The rapid advancement of technology has opened new opportunities for people with physical challenges. Here, we have sketched a state-of-the-art solution to help physically challenged people. To do this, we have proposed an algorithm namely the Gesture-based Smart Wheelchair (GBSWC) control algorithm. This research-based project focuses on building a gesture-based smart wheelchair that is controllable by the hand movement of the user using IoT technology. The customized smart wheelchair has the ability to move forward, right, left, and stop of the wheelchair based on the command given to the system by the hand motion. As well as if at any time the user faces any kind of problem, difficulty, critical situation, or needs someone emergency then a simple single tap on the sensor will help to send an emergency text to given emergency contacts. On the other hand, a simple double tap on the sensor will stop the movement of the smart wheelchair. The prototype device is thoroughly tested here and the obtained result is significant enough which is presented in the result analysis section.