Without a separate CLA or copyright assignment, the contributor retains the copyright to their code, and licenses it to the project under the terms of the GPL (or whatever license).

The license is basically a legally-binding promise not to sue, as long as you follow the rules in the license. So the general consensus is that they can't terminate the license unless you violate the license. (Though there are dissenting opinions on that.)