The spin of an electron is a natural two-level system for realizing a quantum bit in the solid state1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16. For an electron trapped in a semiconductor quantum dot, strong quantum confinement highly suppresses the detrimental effect of phonon-related spin relaxation1, 2, 3, 4, 5, 6, 7. However, this advantage is offset by the hyperfine interaction between the electron spin and the 104 to 106 spins of the host nuclei in the quantum dot. Random fluctuations in the nuclear spin ensemble lead to fast spin decoherence in about ten nanoseconds8, 9, 10, 11, 12, 13, 14. Spin-echo techniques have been used to mitigate the hyperfine interaction14, 15, but completely cancelling the effect is more attractive. In principle, polarizing all the nuclear spins can achieve this16, 17 but is very difficult to realize in practice12, 18, 19. Exploring materials with zero-spin nuclei is another option, and carbon nanotubes20, graphene quantum dots21 and silicon have been proposed. An alternative is to use a semiconductor hole. Unlike an electron, a valence hole in a quantum dot has an atomic p orbital which conveniently goes to zero at the location of all the nuclei, massively suppressing the interaction with the nuclear spins. Furthermore, in a quantum dot with strong strain and strong quantization, the heavy hole with spin-3/2 behaves as a spin-1/2 system and spin decoherence mechanisms are weak22, 23. We demonstrate here high fidelity (about 99 per cent) initialization of a single hole spin confined to a self-assembled quantum dot by optical pumping. Our scheme works even at zero magnetic field, demonstrating a negligible hole spin hyperfine interaction. We determine a hole spin relaxation time at low field of about one millisecond. These results suggest a route to the realization of solid-state quantum networks24 that can intra-convert the spin state with the polarization of a photon.