I have the same problem. I have TONS of Mathematics (mainly) and Physics books, but I've not really worked through most of them. I've spent a fortune on them so far. I always keep saying that I'll take a couple of years off to concentrate on nothing but hard science, but I know that will never happen.

I actually have a theory that good programmers are frustrated Mathematicians and Physicists who couldn't stomach the real thing and chose the easier way out.