> Of course, all the property, tax incentives, and power you ever want won't do cloud providers much good if it's in the middle of nowhere. Latency is still a factor and cloud providers like to spread their datacenters out to serve broad geographic regions. In the case of the Phoenix, that's the American Southwest, which is home to just shy of a fifth of the US population.
I'm kind of put off by the title - which sounds as if they should not build in hot areas. It should be "Here's why cloud providers keep building datacenters in the America's hottest city"
Exactly. The article is just saying random things. The title and body have no logical connection to each other, and it reads like it is trying to second guess itself on every point it tries to make.
American South West is Vast expanse. Depends on how you count it too. If DFW, Austin, San Antonio count as American South West for that 20% number, which is arguable, the Phoenix location does not make sense.
Phoenix datacenters are largely a SoCal play, the cost of investing in SoCal versus on the other side of the border with tax incentives a bit more business friendly environment and near zero land costs (compared to Coastal California metro).
When it comes to water management, the desert towns like Las Vegas, Phoenix do a far better job than coastal cities.
note: 80% of American population lives East of 98th parallel, that is roughly East of I-35 and majority of the remaining 20% is along pacific coast.
Do not put Phoenix and Vegas in the same category for water management. Vegas has done a decent job, Phoenix, and the sprawling suburbs around it, have done a terrible job and are in for a very rude awakening.
Arizona is actually working on a plan to bring in seawater from the Sea of Cortez, people forget that the ocean is only ~62 miles from Arizona’s southern border. It’s actually one of the more realistic and economically viable desalination proposals out there.
The main obstacle are environmentalists. Unfortunately desalination (like Nuclear Energy) is to environmentalists like garlic is to a vampire.
Right, because desal is totally environmentally sound. And sourcing your water through a country that kinda hates you (for flooding their country with guns, refugees, high fructose corn syrup, etc) is also a great idea.
> If DFW, Austin, San Antonio count as American South West for that 20% number, which is arguable, the Phoenix location does not make sense.
Right. As measured from the center of the Texas triangle, Phoenix is 915 miles away. Chicago is 910, and Jacksonville, FL on the Atlantic ocean is 900.
Sufficiently big AC systems often use water cooling because water is more effective at transferring heat than air, is cheaper than other liquids, and unlike other liquids is not a massive safety hazard if it leaks.
It is why if you want to defrost something quickly, leave it in water rather than in the open air.
Basically, yes. It's about cooling efficiency, cost-effectiveness, practicality and scalability. Water has excellent heat transfer properties, which allows for efficient cooling. As I understand it, solar will have limited capacity and depend on sunlight availability..which obviously fluctuates based on weather conditions, whereas data centers need continuous power supply. Also, cost of installing a large-scale solar power system and maintaining battery storage can be high.
Using solar without batteries to supplement baseload requirements is a great cost-cutting exercise, because it reduces grid draw when data center power requirements are highest (business hours) and electricity is most expensive. You can also add batteries later if they start to make more business sense.
Ah interesting, make sense. You're saying use solar most abundant and cost-effective, effectively optimizing energy consumption and saving money by leveraging solar during high-demand periods? Just spitting it back to confirm I got what you're saying.
Most homes use air conditioning. Often times installed by a crane on SFH roofs. I don't think efficiency is a word most people bother with, unfortunately.
That area of the country is also pretty seismically stable and generally doesn't have other sorts of natural disasters. (Some wildfires but not everywhere.)
The Register misquotes itself by writing that “Google's datacenters in Dallas, Texas consumed more than a quarter of the city's water supply”.
Clicking through to the quoted article [1], the figure is actually for The Dalles, Oregon, a city with a population about 80 times smaller than Dallas’.
Journalists love writing articles people want to read. People like crapping on tech companies because they look at them everyday in their phones so it rings a bell. The average person rarely thinks of smelters or is confronted with one.
And what does “too high” mean, anyway? Using water in The Dallas is a totally different story than using it in Death Valley. Part of the reason the data center is there is the abundance of water.
We do it in New Zealand. The power source is renewable and dirt cheap thanks to the games Rio Tinto play, which have resulted in the population subsidising them.
Rio Tinto have a poor record and leave toxic waste in various places. It seems possible that the taxpayer will be tidying up their mess.
Quebec too. And Quebec is far from landlocked, so an abundance of hydroelectricity has alternative markets.
The subsidies are staggering:
> The total cost of $2.7 billion comes to $274,338 per job per year during 35 years for the 740 jobs in the new plant. If we use the figure of 10.0 cents/kWh, which is the expected cost of new projects under study, the cost per job per year rises to $370,864
And these are 2007 number!!! And no, the smelters aren't paying their employees FAANG wages.
> It is far more profitable to export electricity directly
through interconnections than indirectly through aluminum ingots.
Alcoa had a pretty large aluminum smelter on the west coast in Ferndale WA, but it shut down near the beginning of covid. The bonneville power contracts with Alcoa were pretty generous, somewhere around $0.035/kWh for ~300MW, and even then, they couldn't make it work.
A PE firm is trying to buy it up, but the bonneville administration isn't playing ball and giving them the same rates, so it'll likely sit empty.
Cheap power requires good infrastructure which countries with cheap labor generally dont have. There are locales in US with pretty good electricity rates
This makes the datacenter out to be a massive water user. Which it is, in some sense. It's also worth considering that this volume of water usage -- 274.5 million gallons per annum, or 842 acre-feet -- is about 1/3 that of an average-size (445 acre) farm.
From personal experience, most auto-fill fields will correctly find Dallas for "dal", but will quickly update to Dalles for "dall". Which is weird since "dalla" comes before "dalle" alphabetically which makes me think their "learned" use implies more people looking for Dalles type "dall".
I've been caught out by this on multiple occasions.
Every time I want to launch Photos.app or Photo Booth.app through Spotlight, it keeps flicking between one and the other almost with every key press... then when I think it's done, and am about to press enter, it updates the top result a millisecond before I hit enter.
Last week, I helped a friend update their new to them older mac book air. I used my machine to download the most recent OS compatible with their model. Now that's completed, I tried removing the installer from my machine, only part of it will not delete because it has the restricted flag set. According to the internet, I can only delete this file by rebooting into special mode disabling SIP, delete the file, then reboot into normal mode. WTF is that bullshit? This isn't a critical installed file that's a crucial part of the OS. It's just an external file used to install an OS. Totally baffling
Sure, it’s hilarious as long as it’s not you. Otherwise, it’s beyond frustrating.
Not sure about the hard to track down issue. If you have a Mac and the App Store, they’re there for the downloading. Trying to download to run in a VM on windows or Linux violating license agreement then it is not mine nor Apple’s issue
For VMs, usually on Macs but also on licence violating machines.
Whatever the need, it’s of no consequence to Apple that it was a struggle. Having it download as an app is irritating too as it adds steps with the conversion.
There are people who will run their graphical session as root, and throw in a "chmod -R 777 /" for good measure; and there are people who will go through convoluted steps to disable runtime kernel module loading, mount / read-only, and run their web browser in a container. Now I'm definitely not on the latter extreme, but if I can have a decently-hardened system out of the box, why would I throw that away just to remove a builtin app?
Are we still talking about macOS? If so, sudo rm -f does not work on a file with the restricted flag set. Based on that, I don’t see how modifying it with a child would work either. SIP is powerful
If it offers up "dallas" for "dal", but the user keeps typing, then it's reasonable to assume "dallas" wasn't what they were looking for, or they'd have picked it after "dal".
Exactly. I'm not typing D...pause...A...pause...L wait for suggested value. I'm typing dall by the time the first suggestion has had a chance to make its appearance. Then again, I'm on a real computer with a keyboard, and not some mobile device where it's impossible to type properly and forces those pauses between letters where the autocomplete might be noticeable.
And The Dalles, unlike Dallas, has a huge source of water available: the mighty Columbia River, currently flowing 140,000 to 240,000 cubic feet per second (cfs). (Snowmelt causes daily surges.)
The Trinity River at Dallas, TX is currently running at 400 cfs, LOL.
You mean The Dalles that sits on the banks of the humongous Columbia River? That’s the one they are worried about using too much water? The Columbia River pours more water into the Pacific Ocean than any other river in the Western Hemisphere. Are they worried about the Pacific Ocean going dry?
I get that it's common usage (and that's what language basically is), but when I see "80 times smaller" my brain screams "80 times what? Wouldn't `1 times smaller` be zero?" Yes, brain, yes it would.
> And while at $0.078 a kilowatt, Phoenix's power isn't as cheap as Washington State, which is closer to $0.04 a kilowatt, but it's still competitive, Howard said.
JFC. Meanwhile, Hetzner charges 26 ct/kWh in Finland and 53 ct/kWh in Germany [1]. At these prices (and yes, I'm aware these also include the cooling costs) it's completely inexplicable to me how Germany can ever get actually competitive with US services.
Oh, wait … you’re telling me they just phased that out? :(
But yeah, as it turns out energy prices matter; it’s not just corporate overlords of the fuel companies spreading astroturf when people point out the still-high costs of using renewables. It matters to homes and cars and industry too, and when energy costs more, the nation will have less.
The myth that nuclear is cheap needs to die[1]. Nuclear is only "cheap" because of government subsidies that mandated its low price per kWh. The actual cost is far above most other options. So if we subsidize anyways, why not subsidize cheaper energy production methods?
On the other hand, nuclear is only “expensive” because we make power plants pay for ~all externalities and put aside huge amounts of money in advance to cover the (projected, though sometimes not comprehensive) cost of clean up and decommissioning.
Yes; no nuclear proponent asks for that process to change for nuclear but only for the same to be applied to other sources. I just listened to an NPR program about methane leaks in Louisiana from fly-by-night operators that drill, make a profit while prices are high, then abandon shop leaving the drill sites spewing methane without capping a thing off when they shut their business down because the market price makes it unprofitable to continue drilling and selling compared to other energy sources. Then rinse and repeat. Apparently hundreds of thousands of these (growing by thousands more a year) and Louisiana hasn’t wizened up to asking for a full deposit of the costs to cap off a site before granting a drilling license; the cost is punted to the taxpayers and even then the state only gets around to capping off a tiny percentage of the abandoned sites.
If you know of any defunct methane sites i'd like to photograph them. I'm in central Louisiana, only the top and bottom eastern side are kinda "too far" for a trip to photograph in a day.
Here’s a map of abandoned oil wells; they’re referred to as “orphaned wells” in the official nomenclature and there’s a state-maintained registry so they should be easy to find.
They’re all over the state. Probably can’t walk far without tripping over one!
I am writing this from a naïve perspective, but isn’t nuclear more competitive when you factor the negative externalities that come with fossil-fuel generation sources? Wind/PV etc. are far cheaper, yes, but are also subject to more fluctuation in availability. Though I suppose PV/wind plus battery or geothermal take care of that concern.
Who do you think cover any and all negative externalities when wind and PV combined "fluctuate" to pretty much zero for a solid 96 hours, as happened in the UK in winter 2021? Batteries for that scale are a simple fucking lie.
The non-intermittents get to name their price. Except for nuclear, of course. EDF were forced to sell at 45 Euros / MWh when the market rate was over 600 last summer, because the private sector is more efficient, so can't be allowed to compete with reactors whose capital cost was paid 40 years ago by a government.
The whole European energy system is a fucking mess, trying to ignore this obvious fact that grids cannot be allowed to fail. Now, go to an manufacturing conference in Europe and the atmosphere is a fucking funeral. Try selling to customers when your plant produces half of what a US or China plant does, at irregular intervals, because it's at the mercy of metereologists. Go on. Try it.
See that would be reasonable, but Germany decided to go one step further, to throw away even all the CAPEX they had already spent in the past.
Also, all of this due to pressure from the Greens party, which must be respected for the ruling coalition to operate. I will never understand why Greens think that nuclear reactor problems are just as bad as global warming. Tried talking to some voters, got nothing back except "humans are bad". I guess we're far, far away from a solution if the political parties built around global warming do not understand the problem.
The actual cost of renewable energy is barely relevant. What is relevant is that German energy politics has been dominated by Russian oligarchs who want nothing more than to sell Germany as much natural gas as possible. This high cost is not the cost of renewables, because they don't have any - it's the cost of non-renewables meant to be sourced from Russia, and the cost of replacing those with non-Russian sources or Russian laundered sources.
They now buy from the Nuclear Power plants a few km nearby the German Borders. The Prices are that high because of Taxes.
They also probably get cheaper Industry Prices for electricity, but aren't allowed to resell that cheap energy to their customers.
And they sell to the country with the nuclear power when it gets too hot in the summer and the country with the nuclear power can't safely run it without killing their wildlife.
Hetzner competes mainly on price. And completely clears the market on that metric.
My guess is it's because the US competitors are run by incompetent people that see no need to improve anything because they have non-competitive lock-in.
Or because major American competitors do not compete on price, but rather on providing different features? Competing for low cost is not always a good idea, especially if you want to provide more services than bare bone hosting. The proof is on the scale, American corporation dominate market share internationally, and dwarf the likes of Hetzner. (not a dig against hetzner, I for sure would use them more than I'd use AWS if I was looking at building a budget conscious infra)
If anything, European operators have been utterly incompetent in that regard. And not for lack of trying. The only European players left are scaleway and ovh.
On the other hand, you can find tons of cheap operators in the US, they just don't get a similar rep to hetzner here on HN. Hetzner also found a very good niche with their second hand server that you can bid on, for sure. And to be clear, it provides a great service for the price.
I was planning on renting a 1/3 rack at Hetzner Finland, but power is cheaper in Bulgaria now (I'm in Bulgaria). Currently 19c/kwh for businesses and looking like it's coming down still (used to be closer to 10c I believe). It's also 80% nuclear/renewables, better than Finland, seems.
I have three Asus ESC8000 G3 with 8*3080 GPUs each. I have been hosting in the basement but looking at colocation or renting a space. Colocation pricing doesn't seem to favor power-hungry GPU servers though.
I have a source for cheap Asus ESC8000 G3 and cheap 3080 turbo cards, in case anyone needs to set up machines for ML inference, mail me.
In all the datacenters i've used in the US they don't charge per kwh, they charge per max amperage, so you can get dual 15A 240VAC or dual 45A or whatever you need, and that's baked into the cost. Each rack has a power budget due to cooling concerns, etc.
On the one hand, the bill is always the same amount, regardless of power consumption. On the other hand, you gotta make sure you use your power budget or you're wasting money.
You're comparing apples and oranges. The cost you're paying Hetzner is not just for the power they are buying from the electric company, but the cooling of that same number of watts as well as other building maintenance, networking, etc.
Don't forget we have a huge energy crisis right now due to the cutting of Russian gas. It's not directly affecting electricity production but overall energy pricing is severely affected due to people switching away from gas.
What was more shocking to me in that article was that google used 25% of Dallas's water supply.
Between power use and water use, while I appreciate teaching people to be mindful and responsible, I feel like caring slightly less about where I set my thermometer or if I have a long shower when our country's resources are being depleted by big tech and alfalfa shipments to China and almond harvests in California.
Perhaps you haven't lived the same life they have, where receiving condescension for using your normal weights and measures is par for the course. Not saying it's justified, but when it's something you expect to receive it's not unheard of to head it off ahead of time with the same tone that you tend to receive.
I do agree about the date formats but Fahrenheit makes a lot of sense for day to day casual use. Below 0 is pretty fricking cold and above 100 is pretty fricking hot and the degree increments are about twice as granular. For a lot of scientific computations an absolute scale like Kelvin would make more sense but that wold obviously be ludicrous for day to day use. Celsius as an absolute scale actually seems inferior to me and, as another commenter noted, people in the US doing engineering etc. are perfectly comfortable with using SI units as appropriate.
Really? Almost nobody uses reasonable date formats. YYYY-MM-DD is the only reasonable format, but afaik, US and EU both tend to put month and day first, day/month/year is consistent, but still unreasonable.
Caught up? In most metrics the US leads, so I'm not seeing a great argument for forcing people to switch to Celsius and metric. And you're assuming we don't already use metric extensively.
I think the only people who -really- care about it are non-US people. Here in the US we know how to do conversions as necessary, and we don't bitch about it.
Increased regulatory burden for US companies, increased labour enforcement targeting foreign companies in general, and preferential treatment for DE/EU companies codified in law.
EU is actually doing the opposite because they don't want to be cut off from the US internet companies. For instance, now allowing EU citizens data to be transferred to and stored in the US.
I guess the answer is regions. If they built all their DCs in Washington there wouldn't be any DCs in south middle area of US. And since temperatures are largely based on proximity to the equator, it would mean zero DCs near the equator.
But I really don't know what the article is implying. We should built zero DCs anywhere hot? Or DCs use lots of resources, like water and power which are in short supply in deserts?
If you go near the equator but go up in altitude then it won't be hot. There are places in Costa Rica and Columbia where it is basically 72-degrees every day all year.
I wonder what impact cosmic radiation has at those altitudes. I seem to recall experiments somewhere in Colorado that demonstrated a need for ECC, etc.
True, but wouldn't water be harder to get? From my understanding tops of mountains would have significantly less water reserves than valleys. Although they may have more then deserts...
Well, Costa Rica has a long rainy season with torrential downpours. In fact, they have so much hydropower that they are a net energy exporter. Here is a paragraph I found at https://www.hydropower.org/country-profiles/costa-rica:
> Costa Rica has a notable geographic advantage in that its concentration per capita of rivers, dams and volcanoes allow for high levels of renewable energy production. This abundance of natural resources and favourable rainfall levels are beneficial for great amounts of hydropower generation.
Long story short: lot's of rainfall and lots of mountains means lots of water flowing downhill.
This article made me dizzy. It keeps going back and forth and destroying every point it tries to make.
>A state bill signed into law in early 2021, extended tax breaks on the use, installation, assembly, repair or maintenance of datacenter equipment, for most Arizona datacenter
>However, cheap land and favorable tax incentives aren't the only thing cloud providers are looking for.
>Access to power is another important factor.
>Phoenix's power isn't as cheap as Washington State
>Phoenix, that's the American Southwest, which is home to just shy of a fifth of the US population.
>However, proximity won't do much good if you don't have adequate connectivity to support them.
This points to the overall conversation about how fresh water is valued.
If the sliding scale for fresh water prices was more aggressive, this would change the design of cooling systems for industry and large buildings.
It is straight forward to design air-cooled refrigeration systems... even for operation in very hot/desert environments. With this in mind, I think pricing water to make it more cost effective to use water cooling towers, rather than entirely air cooled systems, is very poor policy. The amount of water used to provide cooling for Las Vegas, for instance, is absolutely ridiculous.
Every time these kind of stories come up, I just think of that guy in Chandler, AZ that just hears wump wump wump wump of the air conditioner all year round. I feel for him. Hopefully he moved.
Well, I tried to find the specific story but I can't. However since that article first came out, apparently it highlighted the issue and now articles like this seem to be coming up:
Data centers live off connectivity. You can't just build it anywhere. It's not just about the cloud provider.
What about other cloud providers, ISPs, exchanges, peering endpoints, etc? There needs to be good fiber routes to other cities or data centers in the region.
You can't just pick a random city and hope AT&T, Verizon, NTT or whoever would turn up. You can surely pay for it but then that might cost more than the power and other concerns. If there's not enough bandwidth you wouldn't even survive a DDoS attack.
But I often think surely there must be some viable economic model in which a data center is built in some remote area with plentiful hydroelectric or geothermal power that would largely go unharnessed, and that cheap source of power makes building specific connectivity infrastructure to that location feasible, no? Someone correct me...
Thermally there are spots in western Colorado or eastern California that would be better. The problem is that if you look at maps of major internet thoroughfares in the US, there aren’t many going through Colorado. Which is probably more Kansas’ fault than Colorado.
They're talking about some special evaporative cooling system. Thousands of datacenters around the country run on garden-variety HVAC systems that don't require any water, just regular refrigerant.
It is typically much more expensive to dig down than build on the surface, and in the Phoenix area ground temps are still very warm. A well insulated building on the surface will be easier to build, maintain, and won’t require significantly more energy to keep cool.
(And with a data center, the majority of the heat is coming from inside the facility, not the environment outside.)
> And while at $0.078 a kilowatt, Phoenix's power isn't as cheap as Washington State, which is closer to $0.04 a kilowatt, but it's still competitive, Howard said.
what kind of writing is that? It’s unreadable, in addition to mixing kW and kWh
You missed the paragraphs leading up to this, which mentioned tax incentives. What they were probably trying to say was that the tax incentives were good, and the even though the electricity prices weren't the lowest in the country, the two combined makes it an unbeatable combo.
It's that the structure of the sentences is garbage. I don't think the one quoted here is even the worst offender. Every comma in this article is an adventure. Some of them are okay, some should be deleted, and some should be periods.
Its writer actually used so many chained clauses that they ended up with a syntactical error... in English you can't say, "Although it isn't low enough by itself, but it is such-and-such."
It's tricky in the sense that anyone could get away with saying, "Although it isn't x, still, nonetheless, it is y," which means the same thing as, "Although it isn't x, it is yet y;" and, in some contexts, 'yet' and 'but' are allowed to be treated as interchangeable synonyms.
They probably just didn't have enough time or energy to finish the thought with enough attention, and punctuation has such usage conventions in order to let it aid rather than to confuse comprehension and attention. Commas are deceptively easy to get into without overhead, but not to get out of without it. We could simply abuse the commas and still choose either one of the following thoughts, but the Register ended up splitting the difference:
At $0.078 a kilowatt, Phoenix's power isn't as cheap as Washington State['s], which is closer to $0.04 a kilowatt, but it's still competitive, Howard said.
And, Howard said, while Phoenix's power, at $0.078 a kilowatt, isn't as cheap as Washington State['s], which is closer to $0.04 a kilowatt, it's still competitive.
Which is fantastic because Arizona will then make up that deficit by pulling in federal funds, part of which will disproportionately be paid by residents of Washington.
Damn I’m paying 0.104/kWh in Jacksonville, FL. We have a public utility company (JEA) that NextEra Energy (FPL’s parent) has been trying to get its hands on. They promise lower rates, yet in parts of Florida where FPL is the utility company, people often complain of the high rates. I believe FPL’s rate is technically lower than 0.104 but is higher in practice when you factor in all their bullshit fees.
That's getting to the point that it could literally be economic to run a small generator on natural gas, or that plus solar plus batteries. It would take too many $500-$1000 electric bills to really pay down some personal infrastructure.
Yeah I mean they're not great, but for that much money one could buy a pair of the Honda 2000w inverter generators and run them on natural gas and cycle between them quite handily. Have the "on" generator and the "just off of the latest rebuild cycle" generator.
I've also thought that if one could figure out some of the power electronics a wrecked Prius engine just ticking over at idle would probably provide most of your off-peak power and charge batteries and then during peak times it could just ramp up a bit. I've never heard of a natural gas conversion though so it would be a long road I suspect.
> Of course, all the property, tax incentives, and power you ever want won't do cloud providers much good if it's in the middle of nowhere. Latency is still a factor and cloud providers like to spread their datacenters out to serve broad geographic regions. In the case of the Phoenix, that's the American Southwest, which is home to just shy of a fifth of the US population.
I'm kind of put off by the title - which sounds as if they should not build in hot areas. It should be "Here's why cloud providers keep building datacenters in the America's hottest city"