Programming is easy. But most people are not interested in learning it because they have other things they would rather do. But of those interested in learning it, they come forward quite quickly. As opposed to those trying to learn physics or math, both of which are hard subjects.
Programming is neither harder nor easier than mathematics: like mathematics, the difficulty is in the problems attacked. In both, the big advances come from asking new questions, and finding the right abstraction to deal with the new question. (Many people would say that programming <em>is</em> mathematics, albeit a new and in many ways distinct branch of mathematics).
Well, that is true, but you are forgetting that the "other things" are what the program is for. Physicists may write dodgy FORTRAN but that's fine, because the programs are only a side-effect of doing their real work. I would go so far as to say most of the people writing code (if not most of the code that exists) was written by people who aren't programmers.
