‘Keep AI on a Leash’: OpenAI Co-Founder Warns Against Blind Trust in Agents
Andrej Karpathy, co-founder of OpenAI and a well-known figure in the computer science community, has recently delivered a stern message about not overestimating the current capabilities of artificial intelligence, particularly large language models (LLMs). During a keynote session facilitated by Y Combinator, Karpathy highlighted the need for developers to remain cautious and not let the fast-paced advancements in AI cloud their critical judgment.
“We need to keep AI on the leash,” Karpathy asserted during his presentation, which was later shared on Y Combinator’s official YouTube channel.
Karpathy acknowledged the impressive nature of current AI models, but he pointed out their susceptibility to elementary mistakes that no human would typically make.
Furthermore, he characterized large language models as “people spirits”—simulations of human intelligence that demonstrate uncanny behavior yet lack key human abilities like memory retention, self-awareness, and fact-checking capabilities. He provided examples, noting, “They will insist that 9.11 is greater than 9.9 or that there are two R’s in ‘strawberry’. “
Despite the models’ proficiency at generating thousands of lines of code within seconds, Karpathy emphasized the importance of continuous human oversight. “I’m still the bottleneck,” he admitted. “I have to make sure this thing isn’t introducing bugs. It gets way too overreactive.”
Karpathy advocated for a more systematic approach to working with AI, urging developers to adopt meticulous, iterative prompting rather than vague or excessively broad instructions. “I always go in small incremental chunks. I want to ensure that everything is sound,” he stated. “It makes a lot more sense to invest a bit more time in being more specific with your prompts, which boosts the probability of successful verification, allowing you to progress.”
Earlier this year, Karpathy coined the term “vibe coding” to describe the practice of prompting AI to generate code. In a February post, he referred to it as a process where developers could “fully give in to the vibes” and “forget the code even exists.” Nonetheless, his recent comments highlighted the need for staying grounded and responsible when integrating AI into development workflows.
Other experts supported his call for caution. Bob McGrew, the former head of research at OpenAI, stated on Sequoia Capital’s Training Data podcast that human engineers remain crucial—not only to guide AI but also to intervene when tasks become too complex for AI systems to manage independently. “When something goes wrong or if a project becomes too complicated for AI to understand, a human engineer can help break the problem down into parts for an AI to solve,” McGrew explained.
Kent Beck, a pioneering software engineer and original signatory of the Agile Manifesto, likened AI agents to mythical genies. Speaking on “The Pragmatic Engineer” podcast, Beck cautioned, “They will not do what you mean. They have their own agenda. And the best analogy I could find is a genie. It grants you wishes, and then you wish for something, and then you get it, but it’s not what you actually wanted.”
Beck further noted that working with AI might sometimes resemble gambling rather than engineering due to unpredictable outputs.
Despite these warnings, major tech companies continue to forge ahead with AI advancements. Alphabet CEO Sundar Pichai revealed during the company’s latest earnings call that over 30% of its new code was authored by AI systems—an increase from 25% in the previous year.
