Planar crack propagation under pure tension loading (mode I) is generally stable. However, it becomes universally unstable with the superposition of a shear stress parallel to the crack front (mode III). Under this mixed-mode (I+III) loading configuration, an initially flat parent crack segments into an array of daughter cracks that rotate towards a direction of maximum tensile stress1. This segmentation produces stepped fracture surfaces with characteristic ‘lance-shaped’ markings observed in a wide range of engineering2, 3, 4, 5, 6, 7 and geological materials1, 8. The origin of this instability remains poorly understood and a theory with which to predict the surface roughness scale is lacking. Here we perform large-scale simulations of mixed-mode I+III brittle fracture using a continuum phase-field method9, 10, 11 that describes the complete three-dimensional crack-front evolution. The simulations reveal that planar crack propagation is linearly unstable against helical deformations of the crack front, which evolve nonlinearly into a segmented array of finger-shaped daughter cracks. Furthermore, during their evolution, facets gradually coarsen owing to the growth competition of daughter cracks in striking analogy with the coarsening of finger patterns observed in nonequilibrium growth phenomena12, 13, 14. We show that the dynamically preferred unstable wavelength is governed by the balance of the destabilizing effect of far-field stresses and the stabilizing effect of cohesive forces on the process zone scale, and we derive a theoretical estimate for this scale using a new propagation law for curved cracks in three dimensions. The rotation angles of coarsened facets are also compared to theoretical predictions and available experimental data.