This paper presents a thermal instability analysis for the natural convective heat transport within a horizontal layer filled with nanoparticles using non-homogeneous dynamic model, which is performed analytically and numerically. For the linear stability analysis, both the oscillatory and non-oscillatory convection for free-free boundaries is investigated. The normal mode procedure is used for the linear stability analysis, whereas the curtailed Fourier series, comprising only two terms, is applied for the nonlinear stability analysis. The neutral and oscillatory stability boundaries for the heat transfer and onset of convection throughout the nanofluid layer over periods of time are obtained numerically. It is found that the critical conditions of stability depend on the influence of the thermophoresis and Brownian motion on the thermal energy and concentration equations, as well as the nanofluids thermophysical properties. It is also found that the system is eventually stable for increasing values of the thermal Rayleigh number and the solutal Rayleigh number. The higher wave number helps the system to remain stable. Adding nanoparticles to the base fluid enlarges the stable region of the system. The heat transfer of the system is initially unstable for a small period of time, and then it becomes stable for the rest of the time.