Spontaneous symmetry breaking occurs in a system when its Hamiltonian possesses a certain symmetry, whereas the ground-state wave functions do not preserve it. This provides such a scenario that a bifurcation, which breaks the symmetry, occurs when some control parameter crosses its critical value. It is unveiled that the ground-state fidelity per lattice site exhibits such a bifurcation for quantum lattice systems undergoing quantum phase transitions. The significance of this result lies in the fact that the ground-state fidelity per lattice site is universal, in the sense that it is model independent, in contrast to (model-dependent) order parameters. This fundamental quantity may be computed by exploiting the developed tensor network algorithms on infinite-size lattices. We illustrate the scheme in terms of the quantum Ising model in a transverse magnetic field and the spin-1/2 XYX model in an external magnetic field on an infinite-size lattice in one spatial dimension.