The mortality prediction model in the Intensive Care Unit (ICU) can be a great tool for assisting physicians in decision-making for the optimal allocation of ICU according to the patient's health conditions. Traditional scoring-based systems for mortality prediction don't provide good predictive performance in the case of a large dataset. Moreover, machine learning models can also provide poor performance for the lack of proper feature selection. A comparison of the performance of machine learning models with and without feature selection was explored in this study. Principal Component Analysis (PCA) was used to choose features for this investigation. For the classification job, the most widely used and diversified classifiers from the literature were used, including Logistic Regression(LR), Decision Tree (DT), K Nearest Neighbours (KNN), and Support Vector Machine (SVM). The Medical Information Mart for Intensive Care III (MIMIC-III) dataset was used to collect data on heart failure patients. On the MIMIC-III dataset, it was discovered that feature selection significantly improved the performance of the described machine learning models. Without feature selection, the accuracy of LR, DT, KNN, and SVM models was 86.66%, 80.12%, 85.13%, and 86.49%, respectively, however with PCA, the accuracy was improved to 88.0%, 80.46%, 86.83%, and 87.34%, respectively with only 5 principal components. Finally, the model's decision-making process was analyzed with explainable artificial intelligence using Local Interpretable Model-agnostic Explanations (LIME). This analysis can help to understand the feature's contribution to the model's prediction process. It was also observed that the features involved in the prediction process were mostly common with the first 15 features found in feature importance hierarchy