This is the correct answer. Labor and materials costs, nationwide, exceed $300 per square foot (and often much higher) for a high-rise project [1]. Once you factor in efficiency, land, insurance, financing, sales and marketing, and all of the other costs of a project, it's difficult to make a profit until costs start pushing the $1000/sq.ft. range. That's luxury housing.
However broken SF's development process may be, it isn't the limiting factor to new development. SPUR estimates the regulatory costs for a San Francisco project at about 20% of the total costs [2], and most of that is the low-income housing set-aside (which is ~10-15%, varying by project). The majority of the costs of any new project are land, labor and materials. And San Francisco truly is an exception, in that it's one of the few cities in the world where there's no more undeveloped land.
People love to make generalizations based on community meetings (which are always a circus, in any city) and the cost to do a bathroom renovation or alter a garage door, but to a large construction project, these kinds of bureaucratic costs are a fixed expense. Land, labor and materials drive everything.
Edit: I'm stating facts here, and citing sources for those facts, directly from developers. I have no idea how you can downvote factual information. If you want to have a reasonable conversation about housing in SF, you have to take these basic economic questions into account. They matter.
"The majority of the costs of any new project are land, labor and materials."
That's highly misleading, because if development gets easier then rents go down, and if rents go down then land prices go down. The paper at http://faculty.washington.edu/te/papers/Housing051608.pdf estimates that excessive regulations are responsible for half of San Francisco's housing costs. That was in 2006, so it's almost certainly more now.
"The data shows, clear as daylight, that prices are only ever this high in cities which refuse to build any new housing"
That graph shows only that housing is more expensive in cities that haven't built a lot (it's actually not clear, since there's no R^2, so you can't interpret it). It doesn't tell you anything about why those cities haven't built.
"In Houston, which famously has no zoning, you can buy a condo in a downtown luxury high-rise for $200 per square foot"
Houston is over 600 square miles, with few geographic impediments to construction. San Francisco is 50 square miles, surrounded on three sides by water. There's a latent variable you're ignoring.
"That's highly misleading, because if development gets easier then rents go down, and if rents go down then land prices go down."
Development doesn't "get easier" as rents go down. It gets harder, because it's harder to justify the costs of a project to investors.
The $300/sq.ft. estimate is a nationwide baseline estimates for high-rise construction. They're labor and materials costs, and are unrelated to land, rents, local policy, etc. High-rise construction becomes especially difficult as rents drop, which is why you don't see it outside of the most expensive markets in the world.
"Downtown San Francisco is full of undeveloped surface parking lots. Here are some maps I made..."
Highlighting every parking lot you see in SF (which is what you've done here) isn't a counter-argument. They're land that is being used for a current economic purpose (unlike, say, most of the land surrounding Houston, which is being used primarily for jackrabbits and snakes).
In fact, you've highlighted a number of lots in my own neighborhood that are currently in the process of being developed. Why weren't they being developed before? It wasn't worth it until residential demand in the neighborhood went up. You're assuming that these lots weren't developed "because regulation", when the answer is "because nobody wanted to build there".
"The paper at...estimates that excessive regulations are responsible for half of San Francisco's housing costs."
As far as I can tell, it's not a reliable source. It's a paper about Seattle, which runs a regression based on a small sample of land-use restrictions from a Wharton database, and claims (without proof) that regulations added $409,000 to housing costs in SF from 1989-2006. Aside from not passing the data analysis smell test, that number includes all regulations -- including state and zoning restrictions, many of which are perfectly reasonable.
The numbers I'm citing are directly from SF developers, via SPUR. It's possible that "unreasonable restrictions" drive up land prices more generally in SF, but you'd need a better analysis than this to prove it.
However broken SF's development process may be, it isn't the limiting factor to new development. SPUR estimates the regulatory costs for a San Francisco project at about 20% of the total costs [2], and most of that is the low-income housing set-aside (which is ~10-15%, varying by project). The majority of the costs of any new project are land, labor and materials. And San Francisco truly is an exception, in that it's one of the few cities in the world where there's no more undeveloped land.
People love to make generalizations based on community meetings (which are always a circus, in any city) and the cost to do a bathroom renovation or alter a garage door, but to a large construction project, these kinds of bureaucratic costs are a fixed expense. Land, labor and materials drive everything.
Edit: I'm stating facts here, and citing sources for those facts, directly from developers. I have no idea how you can downvote factual information. If you want to have a reasonable conversation about housing in SF, you have to take these basic economic questions into account. They matter.
[1] https://westnorth.com/2014/03/18/between-rocks-and-a-tall-pl... [2] http://www.spur.org/publications/urbanist-article/2014-02-11...