> (We also instituted pairing on production proxy maintenance. I'm not a fan of pair programming but pair maintenance is great.)
It's a great opportunity to share knowledge and techniques and I very much recommend doing so. It's an important way to get people familiar and comfortable with what the documentation says. Or, it's less scary to failover a database or an archiving clutser while the DBA or an archive admin is in a call with you.
Also reminds me of an entirely funny culture shock for a new team member, who was on a team with a much worse work culture and mutual respect beforehand. Just 2-3 months after she joined, we had a major outage and various components and clusters needed to be checked and put back on track. For these things, we do exactly this pilot/copilot structure if changes to the system must go right.
Except, during this huge outage, two people were sick, two guys had a sick kid, one guy was on a boat on the northern sea, one guy was in Finland and it was down to 3 of the regulars and the junior. Wonderful. So we shoved her the documentation for one of the procedures and made her the copilot of her mentor and then we got to work, just calmly talking through the situation.
Until she said "Wait". And some combined 40 - 50 years of experience stopped on a dime. There was a bit of confusion of how much that word weighed in the team, but she did correctly flag an inaccuracy in procedure we had to adress, which saved a few minutes of rework.
It depends on the complexity and if your LLM-driven changes fight the architecture of the project.
Some changes are in the area of "Well no one did that yet because no one needed it or had time for it", or "Well shit, no one thought of that". If Claude Code did these changes with good documentation and good intent and guidance behind it, why not? It is an honest and valid improvement to the library.
Some other changes rip core assumptions of the library apart. They were easy, because Claude Code did the ripping and tearing. In such a case, is it really a contribution to improve the library? If we end up with a wasteland of code torn apart by AI just because?
That ship has sailed, but I consider Unicode a good thing, yet I consider it problematic to support Unicode in every domain.
I should be able to use Ü as a cursed smiley in text, and many more writing systems supported by Unicode support even more funny things. That's a good thing.
On the other hand, if technical and display file names (to GUI users) were separate, my need for crazy characters in file names, code bases and such are very limited. Lower ASCII for actual file names consumed by technical people is sufficient to me.
At $work we're hosting business knowledge databases. Interestingly enough, if you need to revert a day or two of edits, you're better off to do it asap, over postponing and mulling over it. Especially if you can keep a dump or an export around.
People usually remember what they changed yesterday and have uploaded files and such still around. It's not great, but quite possible. Maybe you need to pull a few content articles out from the broken state if they ask. No huge deal.
If you decide to roll back after a week or so, editors get really annoyed, because now they are usually forced to backtrack and reconcile the state of the knowledge base, maybe you need a current and a rolled-back system, it may have regulatory implications and it's a huge pain in the neck.
I preach to everyone to fail as loudly as possible and as fast as possible. Don't try to "fix" unknown errors in code. It often catches fresh graduates off guard. If you fail very loud and fast most issues will be found asap and fixed.
I had to help out a team in the cleanup of a bug that corrupted some data silently for a while before being found. It was too long out to roll back and they needed all help to identify what was real or wrong data.
I agree. I generally don't ask "Do we need to future proof this?", I rather ask: "Do we paint ourself into a corner?"
For example, if you implement a job framework of things calling a REST API to do stuff async, a sufficient first implementation is a simple monolith doing the stuff and some retries or periodic calls. Because, if it does not scale, we can start thinking about replacing that with something to put things into queues and scaling stuff-doers horizontally and such. But often enough, these simple things scale quite the distance.
On the other hand, if you start introducing in-memory caches into a singular instance to taper around database performance... that's a huge issue. That always leads to pain.
Additionally though, I"ve started to keep notes about people doing simple, efficient things without fanfare. This way, if my boss asks me what Bob did over the year, I can tell them that Bob made these problems disappear and how Bob is starting to handle this area of topics more and more. Suddenly Bob is growing responsible for this area, and if my boss asks Bob about these topics they did well in an annual review, Bob can shine. Hope they learn though.
> The higher paid engineers i've worked with are always worth their salary/hourly rate because of the way they approach problems and the solutions they come up with.
I'm honestly just happy at the moment, because our two junior admins/platform engineers have made some really good points to me in preparation for their annual reviews.
One now completed his own bigger terraform project, with the great praise of "That looks super easy to maintain and use" from the other more experienced engineers. He figured: "It's weird, you actually end up thinking and poking at a problem for a week or two, and then it actually folds into a very small amount of code. And sure, Copilot helped a bit with some boilerplate, but that was only after figuring out how to structure and hold it".
The other is working on getting a grip on running the big temperamental beast called PostgreSQL. She was recently a bit frustrated. "How can it be so hard to configure a simple number! It's so easy to set it in ansible and roll it out, but to find the right value, you gotta search the entire universe from top to bottom and then the answer is <maybe>. AAaah I gotta yell at a team". She's on a good way to become a great DBA.
> Agents are great at building out features, i'm not so sure about complex software that grows over time. Unless you know the right questions to ask, the agent misses alot. 80/20 doesn't work for systems that need 100% reliability.
Or if it's very structured and testable. For example, we're seeing great value in rebuilding a Grafana instance from manually managed to scripted dashboards. After a bit of scaffolding, some style instructions and a few example systems, you can just chuck it a description and a few queries, it just goes to successful work and just needs a little tweaking afterwards.
Similar, we're now converting a few remnants of our old config management to the new one using AI agents. Setup a good test suite first, then throw old code and examples of how the new config management does it into the context and modern models do that well. At that point, just rebuilding the system once is better than year-long deprecation plans with undecided stakeholders as mobile as a pet ferret that doesn't want to.
It's really not the code holding the platform together, it's the team and the experiences and behaviors of people.
> He figured: "It's weird, you actually end up thinking and poking at a problem for a week or two, and then it actually folds into a very small amount of code. And sure, Copilot helped a bit with some boilerplate, but that was only after figuring out how to structure and hold it".
Let me just get you that Fred Brooks quote, now where was it...? Ah, yes, here's one:
It makes sense for junior admins and junior platform engineers to leverage LLM's but I'd be highly skeptical for the future skillset of any junior software engineer who leverages LLM's right off the bat, unless we have already moved that goalpost.
Depends how they use them as arguably was the case with stack overflow or other resources in the past. E.g. an LLM can be a valid and useful way to start discovering or understanding code and architecture. You can ask for a summary , distill important concepts and then read in more detail about them.
You’re not wrong. You’re not the only one saying this either. Though, I’m currently of the mind that the concern is overblown. I’m finding Opus 4.6 is only really capable of solving a problem when the prompt explains the fix in such concrete detail that coding is incredibly straightforward. For example, if the prompt has enough detail that any decent human programmer would read it and end up writing basically the same code then Claude can probably manage it too.
While I haven’t used other models like Codex and Gemini all that much recently, Anthropic’s is one of the top-tier models, and so I believe the others are probably the same in this way.
A junior’s mind will not rot because the prompt basically has to contain detailed pseudocode in order to get anywhere.
Also, I have been called a bit of a hard-ass for this, but if the junior author of some piece of code is not able to explain to me why it is written that way or how they would extend it in a few reasonable cases, I consider that a problem.
This is orthogonal to both if it is well thought-out/naive/really strange code, or LLM generated/LLM assisted/hand written code. If there is a good understanding of the task and the goals behind it, the tools become secondary. If skills are lacking, it will end up a mess no matter the tools and it needs teaching.
Most of us could run stable servers with just ssh and vi. Would suck a lot though.
> The primary motivating factor by far for these CLI agents always seems to be expedience in completing the task (to a plausible definition of "completed" that justifies ending the turn and returning to the user ASAP).
Curiously enough, step one of becoming a good system operator is to learn how to do things. Step two is learning when not to do things and how to deal with a user trying to force you to do things. And step three is learning how to do things you should not do, just very carefully. It can be a confusing job.
But that's why any kind of AI agent stays very far away from any important production access. People banging configs in uncontrolled ways until something beneficial happens is enough of a problem already.
One uncomfortable, but wise truth is: Actual security is bound to the number of minutes until people with big guns arrive. A lot of other measures just exist to bridge time and limit damages until that happens.
We learned this during a funny situation when a customer sent us the wrong question set for vendors. We were asked to clarify our plans for example for an armed intrusion by an armed, hostile force to seize protected assets from us. After some discussion, we answered the equivalent of "Uh Sir. This is a software company. We would surrender and try to call the cops".
During some laughter from the customer they told us, the only part missing from that answer was the durability rating of our safes and secure storages for assets, of which we had none, because they just had to last until cops or reinforcements arrived. That was a silly day.
It's a great opportunity to share knowledge and techniques and I very much recommend doing so. It's an important way to get people familiar and comfortable with what the documentation says. Or, it's less scary to failover a database or an archiving clutser while the DBA or an archive admin is in a call with you.
Also reminds me of an entirely funny culture shock for a new team member, who was on a team with a much worse work culture and mutual respect beforehand. Just 2-3 months after she joined, we had a major outage and various components and clusters needed to be checked and put back on track. For these things, we do exactly this pilot/copilot structure if changes to the system must go right.
Except, during this huge outage, two people were sick, two guys had a sick kid, one guy was on a boat on the northern sea, one guy was in Finland and it was down to 3 of the regulars and the junior. Wonderful. So we shoved her the documentation for one of the procedures and made her the copilot of her mentor and then we got to work, just calmly talking through the situation.
Until she said "Wait". And some combined 40 - 50 years of experience stopped on a dime. There was a bit of confusion of how much that word weighed in the team, but she did correctly flag an inaccuracy in procedure we had to adress, which saved a few minutes of rework.
reply