> I think this is a matter of perspective about what counts as "cutting corners".
The nature of probabilistic sampling practically guarantees that corner cutting is always just a few samples away. Certain sampling strategies can mitigate this, but there's no way to fully eliminate it, without fully eliminating it from the training data and guarding against it during training. Model reasoning can help by giving the model space to draft and review its approach before it executes, but models still aren't guaranteed to follow their own thinking. A mistake or shortcut can always simply slip in during generation and it won't always be caught and corrected.
- Good logits and sampling strategy can make cases like those exceptionally unlikely -- sufficiently so for one to assume it won't realistically happen.
- Once a bad path is sufficiently taken, it tends to be a lot more likely to continue.
This leads to real-world advice:
- If a model refuses your request, do not argue with the refusal; edit your original message or otherwise regenerate -- the presence of refusal tells the model it should continue refusing.
- More generally, don't allow the model context to get contaminated with behavior or commands you don't like -- describing what to do is more effective than describing what NOT to do.
- It's rude to push unreviewed model outputs onto others.
I'm not saying corner-cutting is a thing that is necessarily all over the place (even though there are countless examples in the wild). I'm also not saying it always results in random stops, or that doing one thing bad makes everything else bad. What I'm saying is that bad decisions could be hidden anywhere, at any time, even if everything else looks fine. Such is the nature of current LLMs.
> - Once a bad path is sufficiently taken, it tends to be a lot more likely to continue.
no they demonstrably self correct with injected bad tokens
> - If a model refuses your request, do not argue with the refusal; edit your original message or otherwise regenerate -- the presence of refusal tells the model it should continue refusing.
wish i could do that with humans :P
> - It's rude to push unreviewed model outputs onto others.
Yes but that's not why.
Same real-world advice applies to stuff random fresh graduates make. I remember being one of those.
indeed. That being said, there is a psychological effect of reviews that doesn't exists with LLM. One of the thing that make reviews effective is that when we code, we know someone will look at our code and judge it. It might be subconscious, but it's there.
How on earth are you going to move the needle on "honesty or effort put forth by an LLM" at the logit-choosing stage, though?
To me that's like inferring that you can keep a GPS system from choosing suboptimal cross-country routes by affixing a camera to the grill of the car with sufficient visibility to examine the next fifty feet of pavement.
The nature of probabilistic sampling practically guarantees that corner cutting is always just a few samples away. Certain sampling strategies can mitigate this, but there's no way to fully eliminate it, without fully eliminating it from the training data and guarding against it during training. Model reasoning can help by giving the model space to draft and review its approach before it executes, but models still aren't guaranteed to follow their own thinking. A mistake or shortcut can always simply slip in during generation and it won't always be caught and corrected.