The onset of the coronavirus disease (COVID-19) pandemic and its overwhelming physical and mental health burden can result in stigmatization toward the disease and those affected. This study aimed to measure the prevalence of COVID-19-related stigma and its associated factors among older people in Bangladesh. This cross-sectional study was conducted among 1,045 Bangladeshi older adults aged 60 years and above through telephone interviews in September 2021. The outcome was measured using an eight-point Stigma Scale, adapted to the Bengali language. Level of stigma was indicated by the cumulative score of the eight-items, ranging from 0 to 8, with a higher score indicating a higher level of stigma. On average, participants had stigmas on three of the eight items, and 62.6% had a high stigma score. The most prevalent stigmas were as follows: COVID-19 is a punishment from God (79.3%), patients with previous COVID-19 must be isolated (67.3%), and people infected with COVID-19 did not meet hygiene standards (63.9%). Participants who lived in rural areas (β: 0.67, 95% CI: 0.39 to 0.95) and who perceived needing additional care during the pandemic (β: 0.35, 95% CI: 0.09 to 0.60) had a higher average stigma score, whereas stigma scores were lower among unemployed/retired participants (β: −0.22, 95% CI: −0.45 to 0.00). The study findings suggest implementing interventions to raise awareness through appropriate health literacy interventions and mass media campaigns.