The age of software decision making

Talking with my friend Sebastian we hit the topic of things that AI was helping us to achieve and also things that we dodged the bullet by steering the AI agent in the right direction, avoiding a bad “long-term” choice. These were situations where the AI agent went in one direction that was not the right one, or let’s say, that we know would have bad consequences…

Those situations where you ask yourself “But does it work?” and the answer is yeah, for sure… but is not the right choice because it’s either a problem in the long term, or a dead end, or maybe it’s way too… “myopic”?

In some few cases could just be that the model might not be as strong as we think it is, but in most cases is definitely lack of context. Is the AIs fault for not knowing what is our final vision of the solution? well… that’s pretty unfair. We are the only ones knowing the end goal of what we want to achieve, the AI only executes whatever we ask to solve “right now”. Same thing happens in all our (human) interactions, and has been happening forever.

Let’s use this example: asking AI to generate a dashboard or a website. Everyone is doing this, and the non‑technical conclusion is always the same: “So fast! We don’t need developers anymore!” but how many users will consume that dashboard? One? Ten? Ten thousand? A million? The use case changes the implementation drastically. And what happens when we have 300 metrics? Where does metric 301 go? Does it belong on the first page? Is it redundant? Does it break the layout? The bigger the solution gets, the more complex developing and maintaining it becomes.

These are not new problems…

Written in 1986 “No Silver Bullet—Essence and Accident in Software Engineering” is a paper on software engineering written by Fred Brooks that addresses the general problem. In that paper he brings two key concepts: “accidental complexity” and “real complexity”. ¹

Accidental complexity is basically problems that engineers create and can fix. The technical details, the glue code, the integrations, the mechanics of walking the path toward the solution.

Essential complexity is caused by the problem to be solved and nothing can remove it. If users want a program to do 30 different things, then those 30 things are essential and the program must do those 30 different things.

Clearly Mr Brooks had no idea what AI would look like, but given how general the problem is, it’s very relevant. AI addresses the accidental complexity and, given the right instructions and context, nails it but we’re still left with a big chunk of the problem to solve.

The gap between knowing and understanding

In Star Trek: The Next Generation there’s an episode where Data (Lieutenant Commander Data which, if you never watch it, number one shame on you, number two he is an android) builds another android that he aids with cognitive and standard behavioral algorithms. During an interaction Data explains something that fits a problem perfectly but Picard (Commander Jean-Luc Picard) challenges him about whether the new android was learning, or whether she was simply executing algorithms to which Data responds:

“It is the difference between knowledge and experience.” ²

And I feel that’s exactly what happens with AI today.

The AI agent produces analysis, design, code, tests, even full PRs. But it never experienced the problem, or any similar problem. It doesn’t understand the long‑term vision, the constraints, the tradeoffs which are basically the intuition behind the design. And that means the quality of the solution is directly proportional to the quality of the instruction.

Conclusion

I feel it’s all about the final vision and the design to arrive to the solution because of factors like technology limitations and, more important, the essential complexity of each problem. The narrative that anybody can build software solutions is just very naive because it assumes the AI agent will magically have enough context to make decisions and fill the gaps.

Can this ever happen? hard to predict… it will definitely be alleviated by having much more processing power and not having to worry about who is the target of the solution, but the essential complexity will always be there.

It’s as unavoidable as the need to solve new problems.