Millions of people are infected by the coronavirus disease 2019 (COVID-19) around the world. Within three months of its first report, it rapidly spread worldwide with thousands of deaths. Since that time, not only underdeveloped and developing countries, but also developed countries have suffered from insufficient medical resources and diagnoses. In this circumstance, researchers from medical and engineering fields have tried to develop automatic COVID-19 detection toolkits using machine learning (ML) techniques. The dataset is the fundamental element of any detection tool; therefore, most of the ML-based COVID-19 detection research was conducted used chest x-ray and computed tomography (CT) image datasets. In our study, we collected a series of publicly available unique COVID-19 x-ray and CT image datasets, then assessed and compared their performances using our proposed 22 layer convolutional neural network model along with ResNet-18 and VGG16. We investigated eight individual datasets known as Twitter, SIRM x-ray, COVID-19 Image Repository, EURORAD, BMICV, SIRM CT, COVID-CT, and SARS-CoV-2 CT. Our model obtained classification accuracy of 91%, 81%, 59%, 98%, 58%, 79%, and 97%, respectively. Our proposed model obtained the highest classification accuracy using four datasets (Twitter, COVID-19 Image Repository, COVID-CT, and SARS-CoV-2 CT). Similarly, ResNet-18 only utilized three (EURORAD, BMICV, and SIRM CT), whereas VGG16 only utilized the SIRM x-ray dataset. Results of this investigation indicate a significant comparison chart among the performance of the datasets. Indeed, our study is a large-scale assessment of existing COVID-19 x-ray and CT image datasets. And to the best of our knowledge, this is the first performance comparison study that includes all publicly available COVID-19 datasets.