The nature of dark matter is one of the most thrilling riddles for both cosmology and particle physics nowadays. While in the typical models the dark sector is composed only by weakly interacting massive particles, an arguably more natural scenario would include a whole set of gauge interactions which are invisible for the standard model but that are in contact with the dark matter. We present a method to constrain the number of massless gauge bosons and other relativistic particles that might be present in the dark sector using current and future cosmic microwave background data, and provide upper bounds on the size of the dark sector. We use the fact that the dark matter abundance depends on the strength of the interactions with both sectors, which allows one to relate the freeze-out temperature of the dark matter with the temperature of this cosmic background of dark gauge bosons. This relation can then be used to calculate how sizable is the impact of the relativistic dark sector in the number of degrees of freedom of the early Universe, providing an interesting and testable connection between cosmological data and direct/indirect detection experiments. The recent Planck data, in combination with other cosmic microwave background experiments and baryonic acoustic oscillations data, constrains the number of relativistic dark gauge bosons, when the freeze-out temperature of the dark matter is larger than the top mass, to be N<14 for the simplest scenarios, while those limits are slightly relaxed for the combination with the Hubble constant measurements to N<20. Future releases of Planck data are expected to reduce the uncertainty by approximately a factor of 3, which will reduce significantly the parameter space of allowed models.