The Trick That Agentic Frameworks Pulled On Us

The greatest trick the Devil ever pulled was convincing the world he didn't exist…

You've heard the famous quote, "The greatest trick the Devil ever pulled was convincing the world he didn't exist."

Theological references are relevant here because more than any technological advancement in recent times, the mania surrounding Generative AI is more religion than science or engineering.

The purpose of this blog is not to fight the hype train (I suggest Gary Marcus) for a counterweight. This is a critical analysis of the snake oil being sold to the unsuspecting consumer. I have a simple thesis. Agentic AI frameworks are unreliable and fundamentally unrealistic abstractions. To build complex, chained, and repeatable flows using LLMs requires knowledge, skill, and hard work.

It's tempting to look for a shortcut, a silver bullet; but you can either ride the hype train until the tracks fall off or build the foundations for the long haul.

What is Agentic AI?

Differentiating between wheat and chaff

“Agentic AI is a groundbreaking advancement in artificial intelligence. A combination of different AI techniques, models, and approaches, it empowers a new breed of autonomous agents that can analyze data, set goals, and take action to achieve them—all with minimal human supervision. Agentic AI allows these autonomous agents to achieve near-human cognition in many areas, turning them into problem-solving machines that thrive in dynamic environments and constantly learn and improve with every interaction.

Wow. Amazing. Mindblowing.

Absolute nonsense as well... Let me try,

Agentic AI is the use of Generative AI in complex workflows. Outputs that are produced by LLMs may or may not be relevant to the task assigned. Agentic AI inherits the sophistication, intelligence, and reliability of its output from system instructions and programming, most of which is highly abstracted, written by novices, and will result in a barely functioning system.

Agentic AI is great for making demos for unsophisticated consumers, yet generally bad for implementing in any use case where the outcome is actually important. It won't work at all unless you are an expert, but you almost certainly aren't.

Let me be clear, I'm not skeptical about Gen AI in the slightest, I think used expertly, and with tooling that doesn't yet exist, we can implement workflows that will change the way we perform tasks. Perhaps not to the extent the AI doomsayers talk about, but definitely to the extent of a paradigm shift for many ways of working.

It is no coincidence that the first killer use case for Gen AI is programming because programming will always be the first domain to get proper tooling and infrastructure. The reason for this is simple. We build the tooling and infrastructure for programming tools with, errrm, programming. Programming is a low-level concern, a perfect first use case. It is also a skill which will by the way, still be a (if not 'the') relevant and most coveted ability for days, weeks, months, years, and probably generations to come.

Coding tools have proven the potential of Gen AI. I've got two separate LLM tools that I (pay for and) use daily for coding because they often get blocked up by overwhelming demand and green infrastructure. I use them concurrently in different IDE windows. This is because they are new, thus slow and sometimes flaky.

Even like this, they have revolutionized my workflow. I'm having a bad day when I boot up my computer and Claude is busy so I'm talking to o1-mini instead. And this is the beginning of the shifting ways Gen AI will change the way you work…

These tools are being built from the ground up; they are brand new and not reliable yet. This is exciting because it is the beginning. We are in the middle of a new wave. Which brings me to the subject of this blog. The lie and the overreach that is Agentic AI.

The Lie of Agentic AI

A simple abstraction of a complex unreliable API.

Many years ago, I was running a development agency, and we had a freshly graduated developer join the team. I set them a reasonably simple task (to build a web form in PHP). I'd check in regularly.

“How is it going?

“Really well!

Days and weeks passed, and tap tap tap went the keyboard. It was taking too long, so I went over to the programmer and asked them to show me the code they'd been working on. It was a nested layer of forEach loops, reaching into thousands of lines of code.

My stomach sank (the client was waiting for this…), and I was lost for words. It was a physical blow, everything clicked. Was it my fault for not providing guidance? Was the programmer a lost cause? What the blazing hell am I going to tell the client who is supposed to pay for this? (We billed time, not results…)

The reason I digress is because this is exactly the same feeling I get when I read the docs for any Agentic Framework I have come across. I'm not alone.

I don't want to pick on any specific framework (I'm yet to find any Python Agentic Framework I can't apply this to), but when your docs start talking about roles and workers and bosses and backstories, the whole concept begins to read like a fantasy novel.

The reason is, the whole thing IS a fantasy novel. It's a revolutionary and sophisticated tool built on pioneering technology, that works out of the box, for beginners. And every bit as convoluted and contradicted as that sounds.

I'm not saying that complex LLM workflows cannot work; I'm saying that the expertise required and level of control of nuance are extremely high, and the creation of 'simple and accessible' frameworks is simply impossible until tooling is more mature and foundations are more solid.

There are very few people in the world right now (although they do walk among us!) with the expertise to create and maintain reliable, complex iterative workflows (based on Gen AI or not!) that work and are true to the definition of Agentic workflows that are being sold the majority.

But here's the thing that bugs me; these few pioneers are working on cutting-edge use cases that we probably aren't aware of. They aren't building in public on Twitter, nor writing blogs and how-to guides. They are not focused on creating a framework so you can easily automate your LinkedIn posts. They are getting on with things quietly and behind closed doors.

I get it, I truly do.

People want an accessible, easy-to-use framework that will allow them to build reliable and complex workflows with real business results. I want many things. Elixir of life. Grow a few centimeters taller. Become the tennis champion in my sports club. It is more likely that I will become the tall local tennis champion with eternal youth than for any of these agentic frameworks to work for the average user and believe me and anyone who has seen me play tennis when I say that this is extremely unlikely.

Agentic frameworks are the attempt to build a WYSIWYG website builder before we've even agreed on a single unified standard for HTML. They are premature.

This isn't a case of the cart before the horse; this is trying to build a rocketship before we've discovered electricity. An opportunistic land grab that has already succeeded.

Layers of Abstractions

A broken tool.

The problem with providing a tool or framework that allows you to abstract functionality is that it comes with a set of assumptions. When I buy a hammer, I assume it will work. When I buy a pressure cleaner, I assume it will work.

The problem is that when I use a framework, I assume it will work. But this is quite literally impossible given the maturity of the underlying technology. Far from increasing adoption, Agentic Frameworks are selling an illusion on top of highly controlled demos and finite use cases that will never actually work in the hands of the typical user (and there are millions…).

The damage they are causing is significant. Much as developers these days learn React before they learn javascript, or adopt frameworks before learning about programming principles. A generation of enthusiastic technologists is jumping on the hype train to learn the 'x' framework. Here's the thing; it won't work (to any reasonable standard), in fact, building a demo or prototype is quite the achievement in itself.

But that doesn't mean that Gen AI or LLMs "don't" work, even though that will become the assumption in the short term. It simply means that we haven't reached a maturity level of the underlying technology to be able to safely build solid abstractions on top of it. There are too many problems to solve first. A few fundamental issues that need to be solved at a deep level by LLM providers include but are not limited to:

Latency

Availability

Safety

Reliable output

Scale

So yes, what I'm saying is that the underlying APIs are often overwhelmed by usage; they are slow, vulnerable to hacking (especially prompt hacking), and introduce breaking changes on a weekly basis. The true innovators are working hard behind closed doors on these problems. But until we have more progress, good luck building a reliable production system, much less a framework.

If you don't have an understanding of the underlying APIs (which by the way, is the actual valuable breakthrough we are all working with), how can you possibly build a system on top of it? Some APIs don't require us to understand their limitations mainly because they are a black box that just works. It will take years if not a generation for LLMs to reach this level of maturity, until then it should come with a warning sign "Production use for Experts only."

It's impossible to build a reliable abstraction on top of shifting sands. In fact, this breaks the basic tenet of computer programming. Abstraction in computer science is the process of removing elements that distract from more important elements. We can't focus on high-level details when the low-level system is not reliable; we need to understand nuance and use within a sophisticated paradigm. Not a low/no-code magic framework (with fantastical abstractions).

This is driving us into the trough of disillusionment, which while not in itself a problem as it provides space to build things of true value, is still disappointing to observe.

The Greatest Trick

Inventing the wheel in the Jurassic era.

The greatest trick the Agentic AI frameworks have ever pulled is to convince you that they are necessary. They are the devil of the LLM landscape.

You don't need crews, agents, backstories, and pretty graphical interfaces showing you things that will never work.

You need to understand that LLMs are powerful tools that you should understand without stories from a fantasy novel. Your Hello World and demo apps should be built by hand, in code. When you understand the how, the why, and the what and can control the inputs and outputs of Gen AI models, then you will have a foundation to go out into the world and build something useful. And it will probably take a while.

Not with wizards and managers and agents and dragons, but with the precise deterministic uses of a breakthrough in AI.

Originally published: https://tyingshoelaces.com/blog/agentic-framework-lie