Handwritten Bangla digit recognition is one of the most challenging computer vision problems due to its diverse shapes and writing style. Recently deep learning based convolutional neural network known as deep CNN finds wide-spread applications in recognizing different objects due to its high accuracy. This paper investigates the performance of some state-of-the-art deep CNN techniques for the recognition of handwritten digits. It considers four deep CNN architectures, such as AlexNet, MobileNet, GoogLeNet (Inception V3), and CapsuleNet models. These four deep CNNs have been experimented on a large, unbiased and highly augmented standard dataset, NumtaDB and confirmed that the AlexNet showed the best performance on the basis of accuracy and computation time.