The language used to articulate AI is increasingly selling mathematics as magic and agency as intelligence; the words we use have power and meaning

The emerging technologies of AI, especially agentic AI, have many of us reaching for metaphors to make sense of them. I have seen many compare the technology to autopilot in planes. While this might seem to be a helpful parallel for understanding “how” someone might use agentive solutions in their work, it is flawed and misleading. There are very few metaphors that really adequately explain how AI works, let alone agentic AI, and most fail to capture what is truly happening under the hood.

In the future, we will use AI the same way that pilots use autopilot systems.

There are a few reasons why this is a misguided comparison. First, colloquialisms and complexity render certain technologies as, essentially, magic. Second, most of us don’t actually understand how autopilot really works. Finally, hardly anyone has an appreciation for the difference between simple and complex systems.

Before I continue, I feel the need to briefly caveat what I am not saying. I use AI tools in my work today. I use them to augment and accelerate certain elements of my workflow in the same way I might use Figma, Excel formulas, spell check, or StackOverflow, but I am careful with that use. I am firmly in the camp of “AI can improve the lives of humans, but only if it is democratized and the benefits are socialized for all”. The ethics of how AI scrapes and plagiarizes the work of others or the energy demands of such tech are not on trial here. What I am mostly concerned about in this piece is how the technology is being sold to the public and how it possibly serves to further accelerate technofudalism, to borrow from Yanis Varoufakis. I use Google products, TikTok, and Meta despite my deep distrust of their data privacy because I am personally comfortable(ish) with the social contract at play, while advocating for a change to that power dynamic and process. These are revolutionary technologies; our business practices and social structures need to be updated to protect the public.

It is also worth noting that these are my own opinions and don’t necessarily reflect the positions of my employer. With that out of the way, let’s continue!

Linguistic magic

I am always skeptical of anyone who starts a statement with “In the future, we will…” in reference to the emergence of new technologies. Very rarely are these assessments correct. Despite my best efforts, I’m sure my hubris has still led me to make such assertions, but believing that you alone have divine premonitions into the future of technological evolution is vanity and arrogance. However, I appreciate the desire to reach for a metaphor to help others conceptualize a complex topic, but that first requires an understanding of both the technology at hand and the origin of the metaphor. Have you ever paused to consider just how many sailing colloquialisms we use in corporate speech?

“You just need to learn the ropes.”

“She runs a tight ship.”

“That team is even keeled.”

“Let’s jury rig a quick fix.”

“Be sure to keep everything above board.”

You may not need to understand sailing to make sense of these terms, but that is mostly because the terms are no longer contextualized to their original meanings. I’ve never managed a 16th-century sailing vessel, and that is not a prerequisite to making sense of these common terms. The term “autopilot” has entered a similar linguistic zeitgeist; most everyone is familiar with the concept (even children), despite very few of us having a pilot’s license.

At face value, this parallel might seem like an endorsement for “autopilot” as an acceptable metaphor, but there is an important distinction here. When one “learns the ropes,” the ramifications of managing the sails of a vessel aren’t at play; it is simply a colloquialism. If you “run a tight ship” at home, the term doesn’t directly imply a degree of seafaring in your home. These terms are safely used without any inherent power dynamic at play. One might also say that these phrases have become so completely detached from their original meaning that they no longer pertain to their original context. It might seem like I am splitting hairs over semantics, but in the case of AI and how we describe it or define it, the language used by leaders and those in decision-making positions represents a power imbalance. One can look to the work of Noam Chomsky to gain a better understanding of this; having the power to name something means one controls the narrative.

One only has to consider a few other colloquial terms to see the difference and spectrum. Euphemisms have the power to cloak their subject in ambiguity. Famously, the term “Carbon Footprint” was created by the gas and oil industry. Nazi Germany deployed the euphemism of the “Final Solution” to commit their atrocities and convince their citizens to acquiesce. Many authoritarian governments have used “re-education camps” to jail and torture dissidents. “Layoffs” or “right-sizing” dehumanize how some corporations chew through staff and normalize it as part of the business cycle. A century ago, “entitlements” meant that a citizen was entitled to certain rights and benefits; now the term has come to be conflated with “being entitled,” which is a negative connotation and weaponized against those who depend on the welfare state. “The Cold War” was a useful term to justify the proliferation of nuclear weapons, U.S. hegemony, and the military-industrial complex. There are countless other examples.

Another pressing example is the rise of “gamble on anything” markets. It’s no secret that the legalization of sports gambling led to an explosion of gambling addictions, especially in young men. The rise of Kalshi and other applications allows users to bet on nearly anything now, such as what song Bab Bunny would play first at the Super Bowl or if the United States will invade Iran or Venezuela. Kalshi and other such apps get around regulations by calling their services “prediction markets” rather than “gambling”. Casinos and even sports betting are subject to much more scrutiny as a result of this semantic difference in naming. This has resulted in rampant insider scams on Kalshi. If you want to learn more about these, check out the work of Alex Falcone on sports gambling and prediction markets. This simple language distinction has resulted in Kalshi ballooning into a multi-billion-dollar company almost overnight.

https://medium.com/media/72377c176d9193da2c7f3789ad80d589/href

Artificial Intelligence is benefiting from a few different etymological tricks as well. The term “Artificial Intelligence” anthropomorphizes the technology. Humans anthropomorphize anything that can replicate speech, purely because we understand that speech makes our species unique. I have written about this before, but tools like ChatGPT are essentially very sophisticated parrots. Furthermore, the name suggests a deep, underlying “intelligence” that is rooted in science fiction. Similarly, as the TikToker Adam Aleksic, otherwise known as Etemology Nerd, points out, AI tools are deploying “sparkles” in iconographic communication. This, along with the use of the color “purple,” has successfully framed AI as “magic” to the general public and not just extrapolated statistics and mathematics. Arthur C. Clark noticed this in the 1960s when he said: “Any sufficiently advanced technology is indistinguishable from magic.” While AI is not a scam like Kalshi or sports betting, its market perception is altered by this language, which shapes and defines how we make sense of this technology.

In short, language is a vessel for communicating, and the semantic meaning, etymology, and epistemology associated with these words and phrases can provide cover-fire for ideas and concepts that might otherwise be dangerous or nefarious.

Autopilot is not magic

It’s safe to say most people do not really understand autopilot technology. As such, many of us embrace Clark’s third law, and essentially believe that the plane flies itself and that pilots are just there to monitor stuff and turn on and off the seatbelt sign. In actuality, autopilot is a complex decision-making machine that provides pilots with feedback and information for joint process management with ground control and other flights. As Tech Insider notes, the autopilot system is a negative feedback loop, which seeks equilibrium. Sensors across the plane are taking continuous readings and feeding them back to a central computer. That data is processed and returned to the system to seek stability across multiple axes (vertical, horizontal, etc).

https://medium.com/media/3026652d63f6530756b90f1a5b157cd3/href

Earl Wiener, a former pilot and aviation scholar, coined the “Two D’s of Automation”. Autopilot is “Dumb and Dutiful,” meaning it will accept any sort of acceptable input (even if it is illogical), and always follow its core objective (flying, landing, etc). In fact, this has become known as Wiener’s Law. This means that a pilot must understand all of the available inputs and readings and constantly check for irregular outcomes or behaviors. Understanding acceptable altitude levels, pitches and angles, speeds, wind directions, and much more allows the pilot to sort of act as the “boss” of the autopilot. As Trevor Aldridge, a professional pilot, notes, autopilot frees a pilot up to conduct higher importance activities, keep situational awareness, and anticipate problems.

https://medium.com/media/61551f749c5eb4ae17b0f246a9628618/href

Routes and flight plans are required to deploy the system effectively, and the pilot has to understand these. Once these are input, the plane can largely fly itself; however, there is constant human surveillance through that process. If the plan is not executing correctly, pilots will turn it off and re-input the plan, and start it up again. Pilots can override the autopilot when needed as well. Aldridge again notes that the pilot often is needed to fly the plane in suboptimal conditions when things aren’t working as expected. In this sense, one of the core skills of a pilot is dealing with uncertainty and complexity while mitigating risk.

The technology behind autopilot can also be conceptualized like the Advanced Driver Assistance System, or ADAS, in most modern cars. ADAS can drive your car on the freeway through the use of sensors and onboard distance calculations of your surroundings. Nothing about that system is “intelligent,” however. It simply reads the distance between you and other objects and takes into account your velocity and the calculated velocity of other objects. The self-driving functions of Teslas or Waymos behave similarly to ADAS, with a layer of complex rules that are layered on top of machine learning algorithms. In fact, we see Weiner’s “Dumb and Dutiful” laws play out in the ways that self-driving cars have still caused accidents and fatalities and become the subject of protests. These systems thrive on structure, rules, and predictability. Even in the cases where they are adapting to irregularities, they handle them through cascading logical processes and failsafes. Said, in a different way, these sophisticated machines aren’t trained to walk your dog or mow your lawn; they were never created to handle commands outside of their domain of skills. Throw anything utterly random at them, and the system will fail to respond adequately.

Agents and AI in complex systems

Agentic AIs, LLMs, small language models (SLM), Gen AI, machine learning; if you’re having trouble keeping track of the different flavors, you’re not alone. The latest fad is agentic AI, which really gained speed over the summer of 2025. Predictably, this is when I started to see the “autopilot” analogy applied to agents. Agentic AI allows an autonomous agent to behave on behalf of a human to conduct various tasks. A crude and oversimplified example is an email bot. This “goals-oriented” reward system means that agents can be trained to carry out a wide range of tasks, or so we are told. Most of us seem to conceptualize an agentic AI as a chatbot (thinking of it as an “assistant” or “agent”) when the actual meaning of the term is that it has “agency” to decide without input. Furthermore, you can chain together multiple agents to create a sort of “mesh” that contains both agency and the ability to maintain internal equilibrium. Think of this like baking in a “devil’s advocate” (another colloquialism with meaning lost to time) that ensures any assumptions are challenged and checked.

“Explainability” is the core issue with AI, and agentic solutions aren’t immune to this. Explainability, said another way, is the paper trail of clear reasoning used to arrive at a solution or outcome. Think of it like “showing your work” on your math homework in high school (sorry if that was triggering for anyone). For the most part, AI cannot actually explain the reasoning behind what it generates. The “boundaries” are defined by how well the prompt is crafted, and this also applies to agentic solutions. Telling an agent to “reply to my emails” isn’t a great prompt for, I hope, obvious reasons. However, prompting an agent to “surface my most important emails every morning when I log into my computer” still introduces a challenge: who is defining “important”? Sure, you could define that in your prompt, but even still, how is the agent really interpreting the meaning of your ask? Even if you provided training data for “importance” based on data from your inbox, how can it possibly handle the complexity of managing human interactions via email? What if “important” changes in meaning? If you have a new boss, can the agent adjust to this newly introduced variable? This is where training, recalibration, tokenization, and the appreciation of reinforcement learning are really key. If you simply believe it is all “magic,” then you cannot troubleshoot or adjust the agent.

This is why autopilot and most AI applications are not even remotely similar. All of the inputs, variables, and data points in an autopilot are discrete in nature and very explainable. At any given time, a pilot can see exactly what that autopilot system is doing and how it is reacting, along with live feedback. As Don Norman might say, we have clear affordances and causality in all of the instruments and screens providing feedback to the pilot. The parameters of flight are wholly and completely defined for the autopilot and guarded by mathematics and the laws of physics. In this case, “dumb and dutiful” is a beautiful thing; it means it is fully comprehensible to a user.

Contradicting myself, maybe AI can be autopilot

In September, Nvidia dropped a bombshell paper about the effectiveness of SLMs to perform tasks at a microscopic fraction of the energy demands of Gen AI. It has since surfaced that most AI pilots are failing at large companies or at least not generating the returns expected. These two items aren’t surprising to me. Agents and SLMs are best suited for hyper-focused tasks. The limited scope of what these tools are allowed to engage with prevents them from collapsing under the weight of complexity and possibility. What is a better solution for turning on and off a light: a Boolean on/off line of code or an autonomous AI? This is over-simplified, but there is safety baked into the limited scope of the Boolean. In fact, it might be easier to stitch together a series of micro tasks with limited autonomy to get better results. Call that AI prompt engineering, product design, or good old-fashioned coding, but it properly places governance and structure around the behavior of these tools.

In my own case, I use FigmaMake, a UI development tool that can convert prompts into basic interfaces. The power in this tool is in the limited library of variables it has access to and in the strict output method of the tool; it only spits out screens and code. The tool does best at building basic screens for testing and early prototyping, but really struggles with specifics, tiny details, or integrating existing products. These elements require extensive prompting, lots of tokens, and usually still encounter glitches. This tool is extremely powerful in the hands of a competent UX designer who appreciates accessibility, design systems, interaction design, and so forth. In other words, it works best in the hands of an experienced pilot.

I’ve seen examples from various developer teams where agentic “mesh” architectures can accomplish moderately complex autonomous builds with a little guidance and governance. Truthfully, I believe this is AI done right; focused, moderated, explainable, and limited in scope. It probably goes without saying, but this is anything but “intelligent” in the human sense. It’s a collection of predictive models that can work in concert to pinpoint ideal outcomes within a given set of constraints.

At this point, I fear I have come full circle. A collection of agents with limited scope and autonomy, stitched together and feeding data back to a central user, sure sounds a lot like, well, autopilot. But this is exactly why I am concerned about the semantic meaning of “autopilot” for the general public and the actual behavior of autopilot as a system and tool. Yes, there are AI tools that can and do behave like autopilot; they are best leveraged in the hands of a skilled and highly trained professional with an appreciation for what is happening under the hood, and is capable of responding and adjusting to feedback and irregularity.

The general public is being sold AI as if it is some sort of magical being that will do everything for them under the same guise that planes fly themselves. In fact, I’ll take this further: most of us are utterly unaware of the degree of complexity, thought, logistics, and labor that is involved in every single product, service, and consumable we interact with daily. AI is just the latest convenience that is being sold to us as if it were magic rather than the product of human labor. Perhaps it is the job of designers such as myself to call attention to the complexity of the world that is hidden in plain sight. We should highlight how much of our daily lives is dependent on a concert of decisions, technology, and people, if only because it allows users and consumers to feel more empowered with their choices and actions.

The cloud isn’t magic; it is literal server farms that exist somewhere and consume electricity and require rare minerals to manufacture. Plastic bottles are crude oil that has been dug out of the ground, shipped around the world, refined into petroleum, and integrated into other chemicals to produce the end product. The lettuce in your fridge was sown and picked by hand somewhere, likely by an underpaid immigrant worker, shipped across the country, shelved at your grocery store, and sold to you at a price that is calculated based on complex markets. In broader terms, Kohei Santo articulates it more clearly in his book Slow Down: The Degrowth Manifesto: the “core nations” and people of modern society are unaware of the true cost of our lifestyles because the impacts largely fall on the “periphery nations” and the Global South; out of sight and out of mind. This is hardly new either. I’ve referenced “The Story of Stuff” in some of my other writing; this piece of media is now over 16 years old!

https://medium.com/media/60cedc99731e8f4b3cb985078af63001/href

All of these exceedingly complex markets, systems, and products dazzle like a magician performing a sleight-of-hand magic trick. Demystifying this, otherwise fantastical consumerist market, might just give all of us a healthy dose of skepticism and shrewdness needed to break the spell.

Autopilot, agentic AI, and the dangers of imperfect metaphors was originally published in UX Collective on Medium, where people are continuing the conversation by highlighting and responding to this story.