X-ray spectral reconstruction from transmission data was investigated by direct resolution of the matrix system A*F = T using spectral algebra. Theoretical, numerical, and physical conditions were studied to obtain the physical solution of the problem. It is shown why the least-square-fitting methods may not give good results for high energy photon spectra above several MeV. The proposed method does not need a preshaped starting spectrum. Simulation studies have been made with arbitrary but realistic spectra of different shapes and maximum energies simulating the reality and using different materials as attenuator like carbon, aluminum, and even water. Conditions on the attenuator choice to obtain the physical solution of the system are given. The reconstruction method provides good results in the radiotherapy energy range despite measurement errors. The computation is quick and implementable on a personal computer with a mathematical coprocessor.