Indian subcontinent is a birthplace of multilingual people, where documents such as job application form, passport, number plate identification, and so forth are composed of text contents written in different languages or scripts. These scripts consist of different Indic numerals in a single document page. Recently, deep convolutional neural networks (CNN) have achieved favorable result in computer vision problems, especially in recognizing handwritten digits but most of the works focuses on only one language, i.e., English or Hindi or Bangla, etc. However, developing a language-invariant method is very important as we live in a global village now. In this work, we have examined the performance of the ten state-of-the-art deep CNN methods for the recognition of handwritten digits using four most common languages in the Indian sub-continent that creates the foundation of a script invariant handwritten digit recognition system. Among the deep CNNs, Inception-v4 performs the best based on accuracy and computation time. Besides, it discusses the limitations of existing techniques and shows future research directions.