Doped lanthanum niobate thin films were deposited using magnetron sputtering technique. W and Mg cathodes were used for doping LaNbO4 thin films. Thin films were deposited on two types of substrates: amorphous optical quartz (SiO2) and polycrystalline Alloy 600 (Ni-Cr-Fe). The structural and morphological analysis was performed using X-ray diffraction (XRD), scanning electron microscopy (SEM), energy dispersive spectrometry (EDS), atomic force microscope (AFM). The nonequilibrum growth of the thin films and the nonlinear dynamics of the sputtered atoms and their oxides is observed. XRD analysis revealed that Mg promotes crystallite formation in La1-xMgxNbO4 thin films and W inhibits crystallite formation in LaNb1-xWxO4 thin films. EDS analysis showed that La1-xMgxNbO4 and LaNb1-xWxO4 thin films are nonstoichiometric. They have deficit or excess oxygen and Nb (compared to lanthanum). It was noticed that the nonstoichiometry has influence on the morphology of thin films. The cross section of formed thin films analysis showed that grains or fibers are not visible in the thin films with excess oxygen concentration and thin films with deficit of oxygen has fiber type structure. Oxygen concentration also influences the surface morphology. The roughness of the surface increases (0.2 nm ÷ 0.3 nm for La1-xMgxNbO4 and 0.2nm ÷ 0.8 nm for LaNb1-xWxO4) with decreasing oxygen concentration