Lung cancer is increasing day by day. According to the report published by WHO in 2017, number of death people due to lung cancer in Bangladesh reached 12,075 or 1.53% of total deaths. It is a matter of great sorrow that this lung cancer detection does not occur at an early stage for many people. It is very important to recognize lung cancer and detect the location as well as an accurate prediction. Many researchers applied techniques like Fast Fourier Transform (FFT) for image enhancement, thresholding for the purpose of segmentation and binarization system for extraction, etc. Basically recognition process consists of three stages like enhancement, segmentation and feature Extraction, lung cancer images can be used as inputs and maintaining these stages give more quality and accuracy in the detection of lung cancer. Approaches developed by the earlier researchers fail to produce accuracy in real-time applications. Hence, in order to mitigate the drawbacks of these approaches a hybrid method to detect lung cancer. Gabor filter has been used for image enhancement of input image. Marker-Controlled Watershed algorithm has been applied for segmentation purpose which will help to provide the exact location of the infected region in the input pictures as well as Out-of-bag (OOB) error rate for different iteration (trees) has been observed graphically according to Random Forest Ensemble and OOB rate for the different class also been determined. When the number of tree is increasing, OOB rate in percentage is decreasing. The highest OOB value is 23.40% which provides an initial tree. Distribution curve of Mean decrease in accuracy, Mean decrease in Gini index and standard error of importance measure according to Random Forest Ensemble has also been observed graphically. RUSBoost algorithm has been introduced finally in order to evaluate accuracy. Performance of RUSBoost algorithm has been visualized in this paper for the purpose of Subtlety, Spiculation, Sphericity, Texture, Margin, Malignancy, Lobulation, where Texture provides the highest accuracy and Spiculation provides second highest accuracy. © 2019 - IOS Press and the authors.