This thesis presents a new experimental facility that has been developed in the course of the PhD of the author together with the experimental measurements of the dipole moments for transitions between 5P3/2 and nD5/2 states in Rb atoms, where 20 ≤ n ≤ 48. A laser cooling setup for rubidium atoms was designed and built that allows to produce 3 x 10 atoms at temperature of few microkelvins. A laser system for coherent excitation to Rydberg states was realised that exploits a two-photon ladder excitation scheme. Such setup allowed to directly measure transition dipole moments from the first excited state to high n Rydberg states. The electromagnetically induced transparency and Autler-Townes splitting phenomena were exploited in order to obtain the values of the dipole moments, which were found to be in an excellent agreement with the theoretical values calculated using the quasiclassical approach by our collaborators in Novosibirsk. To the author's knowledge, these are the first direct measurements of such dipole moments. The data shown in the thesis proves the reliability of the measurement method. The obtained results are extremely useful to validate theoretical models and gain insight in the physics of the interactions that will be an object of subsequent studies. The methods presented in the dissertation will be used in the future to detect in a non-destructive way Rydberg-Rydberg interactions within mesoscopic samples
