Because long gamma-ray bursts (GRBs) are tightly linked to core-collapse supernovae, GRBs, their afterglows and host galaxies offer a unique opportunity to probe the interstellar medium and the environments where stars form throughout the observable Universe. The luminous afterglow in particular is a powerful messenger that pin-points and lights-up distant galaxies and star-forming regions that can not be studied by other means. GRB afterglows and hosts are thus powerful laboratories with which to study GRB progenitor models, the star-formation history of the Universe, its enrichment with metals and the evolution of galaxies. I will review recent progress and highlights from GRB host studies and briefly discuss from an observational point of view.