Worth noting that complex numbers are not avoided entirely in Smith's DSP book, they are introduced in the last few chapters. FWIW Smith's book was the one that helped me (who didn't take any math classes after high school) get a basic grasp on audio DSP principles.