> So, about four years ago, I went on one of my week-long retreats, where I just take a computer and a stack of reference materials and I spend a week kind of reimplementing the fundamentals of the industry. And getting to the point where it’s like, ‘All right, I understand this well enough to have a serious conversation with a researcher about it.’ And I was pretty excited about getting to that level of understanding.
As much as I respect Carmack as a computer graphics expert, I really doubt his competence in machine learning. He doesn't have a single notable paper published. If he really thought that implementing gradient descent and basic stuff in a week long retreat gave him the chops to have serious conversations with AI researchers, he is really deluded.
Unless he can produce something that outdoes stable diffusion, chatgpt, alphago etc he should just hand over technical leadership of his start up to a leading AI researcher. Even Yann Le Cun at Meta is struggling to make any progress and is keeping himself busy by calling every other research labs output pedestrian. We cannot take any of Carmacks AGI predictions seriously, he simply lacks any expertise in the field.
> As much as I respect Carmack as a computer graphics expert, I really doubt his competence in machine learning. He doesn't have a single notable paper published.
Publishing papers is the way the academic/scientific world measures notability and/or competence. It's not the way the engineering world that Carmack comes from measures it. They measure it by building. But you're right, we kind of have to just trust that he has the expertise he says he does by his statements since he has not built any modern AI programs (that I know of at least).
> If he really thought that implementing gradient descent and basic stuff in a week long retreat gave him the chops to have serious conversations with AI researchers, he is really deluded.
This is not an accurate account of how he said he developed his knowledge base. Just how he got started so he could have conversations. He said that he spent a retreat learning the basics and then later in the interview he said he took the time to understand the 40 most essential papers in the field as related to him by a well known researcher. He has since largely put the last 4 years of his professional life into this. While we have no proof of his knowledge, given his intelligence and high competence in computer programming and math, I have no doubt that if he did put in the work he could achieve an understanding equivalent to that of your average AI researcher.
That said, of course it makes sense to be skeptical.
> This is not an accurate account of how he said he developed his knowledge base.
I quoted him directly, because I was expecting this kind of response. He took a week off and implemented some stuff from the ground up and was ready to have serious conversations with AI researchers. The 40 papers by Ilya came later. I have read a 100 ML papers and reviewed preprints. That's quite a low bar, especially if you are prone to skip the math and simply read the abstract and conclusions.
His whole approach gives me a ML for hackers vibe and his thoughts on AGI, if it had come from anyone else, would have been described as word salad.
> The 40 papers by Ilya came later. I have read a 100 ML papers and reviewed preprints.
I would say it’s more likely John Carmack is capable of learning the state-of-the-art of AI from 40 papers than a random (pun intended based on username) from 100.
Sure, he must be faster than Geoff Hinton too and it took Hinton a life time.
Funnily enough, I am able to publish ML papers - but John Carmack isn't. I wonder why. I would also like to learn more about all the computer graphics algorithms Carmack has invented before I trust him to invent AGI.
Here is one example of a person I am familiar with - Math Olympiad bronze medalist. Princeton PhD in ML Theory. AI researcher in Google.
Your OP and replies kind of just come across as jealous that John Carmack’s opinion on this stuff is taken seriously whereas relatively unknown folks’ (yourself, the researcher you mentioned) opinions, are not despite the fact that he’s not traditionally credentialed. Like I said in my original post, we should be skeptical of him and his claims. But the way you are dragging this thread out feels like sour grapes. Of course one of the most famous programmers in the world is at least heard out when he dedicates years of his life to a programming adjacent topic. That’s just how fame works.
And calling out Princeton, Google, etc. further exemplifies an academic bubble kind of credentialism. I suspect you don’t realize how it sounds because you’re so in that credential filled world.
I don't have any credentials really. If I had to be jealous I would be jealous of Carmacks work in id and oculus.
I am definitely irritated by the fact that he is able to pull 20M in funding. He has been giving interviews left right and center. We are upvoting Altman, sam Harris,Carmack but nobody cares about the actual AI researchers, all academics, who have brought about the Deep learning revolution.
Your comments about academic bubbles and researchers gives you away. All of the revolutions in AI have been brought about by the academics inside bubbles you are jeering at. The biggest example of academic inside a bubble is Geoff Hinton.
We have to sit here and listen to word salads from
Carmack, Altman (Ilya - who doesn't get interviewed is the actual researcher behind chatgpt), Sam Harris etc, who have very little insights making bold AGI predictions.
I asked to learn about all the algorithms Carmack invented in computer graphics, but haven't heard back. But looks like he has a good grasp on solving AGI by 2030 according to himself. After all, he is a really great programmer.
> I don't have any credentials really. If I had to be jealous I would be jealous of Carmacks work in id and oculus.
Fair, you come across as jealous of him for something... could be that.
> I am definitely irritated by the fact that he is able to pull 20M in funding. He has been giving interviews left right and center. We are upvoting Altman, sam Harris,Carmack but nobody cares about the actual AI researchers, all academics, who have brought about the Deep learning revolution.
There it goes again. Fame is fame is fame is fame. Like I said earlier. That's just how it works. No reason to be against someone for being famous for their non-academic accomplishments if they have something valuable to contribute.
> Your comments about academic bubbles and researchers gives you away. All of the revolutions in AI have been brought about by the academics inside bubbles you are jeering at. The biggest example of academic inside a bubble is Geoff Hinton.
I didn't jeer at anybody I just tried to give you some perspective about how your comments came across. I appreciate all of the researchers but I also live in the real world and understand that people flock to personalities and front-people. That's not necessarily good, but that's the way it is. Just like nobody gives credit to under-secretary of state for coming up with a great new foreign policy.
Waving around great university A or great company B does not make someone any more right, just like being John Carmack doesn't make him any more right.
> We have to sit here and listen to word salads from Carmack, Altman (Ilya - who doesn't get interviewed is the actual researcher behind chatgpt), Sam Harris etc, who have very little insights making bold AGI predictions.
I agree with you. I never said they were right about AGI. Maybe you should be more generous with my replies and actually think I was trying to give you another perspective about how your comments came across.
> I asked to learn about all the algorithms Carmack invented in computer graphics, but haven't heard back. But looks like he has a good grasp on solving AGI by 2030 according to himself. After all, he is a really great programmer.
Again, you're confusing who he is (go back to my original comment in the thread where you jeered at him because he didn't publish any notable papers). He is not a researcher and he is not really a computer scientist. Engineers don't come up with novel algorithms for the most part. He builds things. And being a really good builder is a different but still valuable perspective. But that doesn't mean he's right about AGI.
AI researcher perhaps, but almost none of them understand cognition. They're focussed on getting something that vaguely resembles a part of the brain to predict the next most likely token. Their idea of cognition apparently stops at Skinner.
Any AI researcher worth their salt knows that those models aren't representative of how the human brain works... but they're just the kind of models that work best out of what we know how to implement right now. There are models with stronger cognitive inspiration, but their performance is worse.
Anything you study for a few months you can become the world's leading expert on. It's a lesson I learned while doing my Ph. D. That's all it takes. After a few weeks, you get to the point where there are only a few others in the world that have read and are able to understand what you've read. A few months on, you are generating new ideas and insights. They might be wrong. But they won't be uninformed.
John Carmack did not start from zero. He already has a firm grasp on algorithms related to linear algebra. Basically machine learning is a whole bunch of matrix manipulation. He's been doing that for 3 decades. The rest is just absorbing concepts about how to apply linear algebra to ML. I'd say he's probably uniquely qualified to really absorb a lot of knowledge quickly on this. It's not about publishing papers, it's about reading and understanding the right papers. I have no doubt he can chew his way through lots of research material in a week or so.
If it is simply about linear algebra, can you please read this ML Paper[1], go through all the proofs and lemmas over a week? You already have a PhD in ML, should be easy. Every kid graduating in STEM understands/should understand linear algebra. Knowing linear algebra is such a low bar.
Frankly, one does not need this paper to get towards the AI. Adam the optimization algo you might need (and even there I am not sure). And it is very readable. The fact that this particular proof of Adam's convergence is complicated is largely irrelevant.
That's not really an argument for why understanding this is needed to move the field forward.
Even your point rests on an assumption, that there's no proof for Adam convergence, that a high school student could understand, which is just a guess at best.
He’s been working on machine learning long enough now to have some chance of success. It may go the way of his rocket ambitions (nothing comes out of it) but let the man try
Yep - he also failed with his lean, simple, first-principles approach with Armadillo Aerospace. The guy is proudly uncreative, and so he could have never come up with Scaled's design of a variable geometry rocket ship launched from a jet mothership.
As much as I respect Carmack as a computer graphics expert, I really doubt his competence in machine learning. He doesn't have a single notable paper published. If he really thought that implementing gradient descent and basic stuff in a week long retreat gave him the chops to have serious conversations with AI researchers, he is really deluded.
Unless he can produce something that outdoes stable diffusion, chatgpt, alphago etc he should just hand over technical leadership of his start up to a leading AI researcher. Even Yann Le Cun at Meta is struggling to make any progress and is keeping himself busy by calling every other research labs output pedestrian. We cannot take any of Carmacks AGI predictions seriously, he simply lacks any expertise in the field.