The recently discovered three-dimensional or bulk topological insulators are expected to exhibit exotic quantum phenomena. It is believed that a trivial insulator can be twisted into a topological state by modulating the spin-orbit interaction or the crystal lattice, driving the system through a topological quantum phase transition. By directly measuring the topological quantum numbers and invariants, we report the observation of a phase transition in a tunable spin-orbit system, BiTl(S(1-δ)Se(δ))(2), in which the topological state formation is visualized. In the topological state, vortex-like polarization states are observed to exhibit three-dimensional vectorial textures, which collectively feature a chirality transition as the spin momentum-locked electrons on the surface go through the zero carrier density point. Such phase transition and texture inversion can be the physical basis for observing fractional charge (±e/2) and other fractional topological phenomena.