The motion of atoms in a solid always responds to cooling or heating in a way that is consistent with the symmetry of the given space group of the solid to which they belong1, 2. When the atoms move, the electronic structure of the solid changes, leading to different physical properties. Therefore, the determination of where atoms are and what atoms do is a cornerstone of modern solid-state physics. However, experimental observations of atomic displacements measured as a function of temperature are very rare, because those displacements are, in almost all cases, exceedingly small3, 4, 5. Here we show, using a combination of diffraction techniques, that the hexagonal manganites RMnO3 (where R is a rare-earth element) undergo an isostructural transition with exceptionally large atomic displacements: two orders of magnitude larger than those seen in any other magnetic material, resulting in an unusually strong magneto-elastic coupling. We follow the exact atomic displacements of all the atoms in the unit cell as a function of temperature and find consistency with theoretical predictions based on group theories. We argue that this gigantic magneto-elastic coupling in RMnO3 holds the key to the recently observed magneto-electric phenomenon in this intriguing class of materials6.