I supervise the contributions that juniors make to our codebase. To do this effectively, I use three tools:
1. Scoping. I give juniors projects that are big enough to be valuable, but not so big that they get lost.
2. Instruction. I give juniors frequent, bite-sized lessons related to their tasks. In this, I invest in their skills.
3. Linting. Anything that could be considered a “nit” in a code review is relegated to machine lints. For example, I will never talk about formatting in a code review. That is solved by tooling.
This article makes me wonder if there’s some sort of machine-readable lint for costly abstraction? In a TS project, is there some way to say, “the shadow DOM under this component is too deep.” Or “this object’s inheritance tree is too long.”
Or for zero/cost abstraction languages, is it feasible/productive to lint “this packages dependency tree is too deep”?
How do you identify abstractions that are too leaky to be helpful?
In all seriousness, I’d take a “s3 is all you need for durable workflows” and use it in data processing applications that move data from s3 -> s3 with no other dependencies.
Yep. But we all know that one machine can and will fail (or be patched and restarted), so the log needs to be distributed.
Different workflows should probably go in different buckets or "topics" for clarity. Since it's distributed, the system must guarantee that the log items are stored in the same ordering ("offsets") among the nodes.
Are logs all you need for durable workflows? I'm confused here. How'd persist and query nested or related data over logs? By logs I assume you mean something like elasticsearch or meilisearch?
Pretty much every durable system has an intent log of some sort. The log provides durability, the database system just integrates that log into a more queryable format.
I assume they meant a log like a WAL. A WAL should be (quite literally?) all you need for durable workflows.
A distributed WAL (to survive a machine death) would also probably be something I'd want, and … something I'm not sure you're getting directly from SQLite.
Folks this is meant to be an honest question, not a snarky comment. I'm not a DBA, I'm DevOps/SRE and logs for me always meant execution logs. I'm just curious if between those involved in database domain logs is used to refer to WAL.
I think the original poster in this thread was joking. A fair number of databases use "logs" as a core mechanism for storing and sorting data. "Logs" in this context is not to be confused with stdout/stderr output that you may collect from a running program and forward somewhere like Cloudwatch/Elasticsearch etc. "Logs" in the context of databases here refers to the data structure; which can generally be defined as simply an append-only "file" (I put file in quotes because just because something is a log does not mean it is necessarily written to disk yet - that's why write-ahead logs exist). It's not just write-ahead logs.
Google "Log-structured merge trees" if you want an interesting read.
Pardon my ignorance trying to follow up on what is most likely sarcasm but is this not Kafka's claim to fame?
I am joining a new project and need to know to what extent Kafka is still a part of the future for new big data projects. It doesn't seem like there are alternatives at the high end but instead the question is when other technologies (that are easier to manage, require less compute, etc.) max out.
> Pardon my ignorance trying to follow up on what is most likely sarcasm but is this not Kafka's claim to fame?
Yes
> I am joining a new project and need to know to what extent Kafka is still a part of the future for new big data projects.
It's not gonna win or die on merit.
If I were to sit here and propose that you mutate bank accounts in-place (which I'm more or less doing by analogy right now over on https://qht.co/item?id=48339103), and you just need safe enough locking technology to do so, I'd be immediately and rightfully shouted down that no one does that and you write down the money transfers and derive the final balances from that.
Manage any other kind of state, however, by appending state-changes and rolling them into a derived state, and then you're chasing fads and doing resume-driven development. So I'm skeptical about the future of this way of doing things.
The exploit they chose assumes ASLR is disabled for simplicity's sake, but if you read the full writeup they say they could've used the vulnerability to map memory layout. It's nice to have ASLR but some types of vulnerabilities can be used to bypass it.
This is exactly what stood out to me, too. Before this Tweet, my feelings towards Coinbase were completely neutral. After this Tweet, I want nothing to do with it.
> Over the past year, l've watched engineers use Al to ship in days what used to take a team weeks. Nontechnical teams are now shipping production code and many of our workflows are being automated.
> The database can be dumb and fast because the application layer is smart and careful.
I’ve always baked important invariants directly into the database with constraints and triggers. Maybe this is because I work on internal apps, where the data is more important than the presentation. Maybe it’s from my functional programming experience and some need to make invalid states unrepresentable.
Regardless, I believe that the data layer should be the most carefully designed part of an app.
This creator has fascinating insights into design and UX. This team’s synthesis is more insightful and professional than anything else I have been able to casually consume.
This particular video argues against the notion of a “happy path.” In reality, “edge cases” are just as valid _states_ as the “happy path.”
At work, I am rewriting a Salesforce Implementation that failed largely due to poor UX. The first solution was built by contractors who later ditched. They built the “happy path.” After they left, I had to support the software. When we launched, I had conversations with users EVERY DAY where I had to say, “sorry, we didn’t plan for that edge case.” It turns out that edge cases aren’t really “edge” at all. It was just a poorly designed system.
Watching this video made me redouble my efforts to plan for every state. I want to be able to respond to support questions with, “we designed for that.”
What do you think? How can we balance good design with time and money requirements? Any thoughts about the video’s suggestions at the end?
1. Scoping. I give juniors projects that are big enough to be valuable, but not so big that they get lost.
2. Instruction. I give juniors frequent, bite-sized lessons related to their tasks. In this, I invest in their skills.
3. Linting. Anything that could be considered a “nit” in a code review is relegated to machine lints. For example, I will never talk about formatting in a code review. That is solved by tooling.
This article makes me wonder if there’s some sort of machine-readable lint for costly abstraction? In a TS project, is there some way to say, “the shadow DOM under this component is too deep.” Or “this object’s inheritance tree is too long.”
Or for zero/cost abstraction languages, is it feasible/productive to lint “this packages dependency tree is too deep”?
How do you identify abstractions that are too leaky to be helpful?
reply