To be sensitive to an effective Majorana neutrino mass, 〈m_ν〉, on the order of 0.1 eV, the NEMO 3 double beta (ββ(0ν)) decay experiment requires precise knowledge and control of the backgrounds. The effect of neutrons and γ-rays from the Fréjus underground laboratory (LSM, Laboratoire Souterrain de Modane) has been studied during 10 700 h of data collection. The data were taken with a NEMO 2 prototype detector using different shield configurations. Monte Carlo calculations with GEANT/MICAP code and a new library of γ-rays from neutron captures are presented. The implied consequences for the NEMO 3 detector which is under construction are discussed. The neutrinoless double beta decay background induced by neutrons and γ-rays within the LSM will be suppressed to the very suitable level of 0.1 event in a life time of five years given the appropriate shield and magnetic field.