I was fed the time-differential argument when I was younger: your ears tell your brain the difference in the time it takes for a sound to arrive at your left and right ear, and this determines the position.  But this system is way, way underdetermined.  I know it has to do with the pinna, but there's still a lot to explain.

 It seems that Planck's E=hv, combined with the knowledge that light carries momentum, necessitates this.