Being humble and being observant are both important but in most interesting design challenges the data is sparse relative to the space of possible design solutions. Heuristics and opinions are a necessary part of the process.
Good designers listen and inject well formed options into the product.
While 3 is a bit on the low end, user studies are not required to have large sample sizes. You're not trying to estimate what percentage of users are running into a particular issue, but only which issues many users run into. Whether it affects 10% or 30% of users does not really influence whether it should be fixed.
And this is why the field of design gets shit on so much. Jakob Nielsen might have a PhD, but he’s much better at PR than he is at conducting actual research. It’s mind boggling to me that people will still quote his opinion pieces 20 years later as if there was any actual science behind them.
There’s a few other fields that happen to conduct a lot of studies around human behavior. It turns out, there’s actually an established process for calculating the confidence level of your findings:
Is 3 or 5 better than nothing? Sure! But don’t trick yourself into thinking your findings actually mean anything because a Danish dude drew a graph 20 years ago.
So again, a large margin of error and a low confidence level are fine, because it doesn't matter if in reality, 80% of your users run into something rather than the 60% that came out of user testing. In fact, we don't care about the specific numbers at all. All we want to know is what things are unclear to many users. Often, you test a design on five users, and all five of them miss the same thing - that gives you plenty of confidence that you'll want to try something else there.
User testing is not a statistical test, and it's not a way to learn about human behaviour (so indeed, in that sense it doesn't "mean" anything); it's to give insight into your designs. I'd highly recommend everyone to take part in conducting user research sometime, as it will quickly become obvious just how easy it is to miss "obvious" things in something you're closely involved with, and how you need to have just few people interact with it to find out which.
Anecdotal, but I've observed quite a few in-person usability studies. After the the third or fourth person you've usually discovered all your're going to with your current test script and stimuli.
You probably could design a study that would continue to uncover new issues after many tens of people, but the test would probably be two hours long and would be unreliable based on that fact alone.
The NN Group article linked by the sibling comment holds true in my experience.
if, as a UX designer, you don't have opinions, then all you're doing is blindly optimising for metrics that may have nothing to do with actually good experiences. This is why platforms like twitter and facebook are such cacophonous trashfires: UX "Designers" chasing "engagement", and holding no opinions. you might as well just hand your job over to an AI, you're contributing nothing to the process.
