Just in Time Software

What happens when Software no longer needs code

First, a big thank you and shout out to Aditya Agarwal, who originally encouraged me to write this as well as helped refine and edit it down to something more readable.

Since the watershed moment of ChatGPT—which sparked a minor existential crisis for me and many technologists—questions have loomed: Will we become irrelevant? Will anyone need to code anymore? If AI does everything, what will we do?

I've been obsessed with computers and technology since I could form thoughts. What truly drew me to tech was the ability to model and simulate—to create worlds, test ideas, and build products that solve real problems. In what now feels like the true dawn of AI, these possibilities seem more tangible than ever. And I believe we have been entering a new paradigm of software development that I’ve been calling "Just in Time Code," or more generally "Just in Time Software" for short.

"Just in Time Software" is both my prediction for how technology will transform in this AI-powered age, and also serves as a possible answer to the existential question. Just in time Software shows how AI can transform society for the better by achieving goals and delivering utility much more directly.

Not to be confused by AI code-generation, Just in Time Software differs from simply using tools like Cursor or Copilot to generate code. JITS avoids the need for a persistent codebase; In fact, JITS doesn’t have a persistent codebase - or code base at all - instead generating the code it needs to run on the fly to then be executed and discarded.

Software Without Code?

Often Code is seen as a kind of up-front investment followed on by on-going maintenance costs. Fundamental components and modules are strung together with functional logic to create software flexible to the evolving needs of it’s users. Codebases are developed over long periods by oft-changing teams, which institute both superficial and programmatic conventions such as choice of programming language or frameworks. These choices can subsequently have significant downstream effects on everything from the success of the project to hiring and execution speed.

As such, software that can be in theory be decoupled from a codebase would no longer have to consider or be held back by such things. After all, software fundamentally distills down to which instructions are presented to processor units and when. Software without an "underlying stack," could utilize computational resources more directly, freeing us from the limitations and costs of the underlying stack.

Moreover, a piece of software that provides utility without a codebase would entirely bypass said upfront investment and on-going maintenance costs of a codebase. This might sound too good to be true, but consider young children making up games on the fly or complex business processes where workers learn tasks quickly, obviously without them having to be explicitly programmed.

Everything we now use software for used to be done manually by humans, and now with the natural interfaces to computation made possible with AI, we can start treating our software in the same way.

So How Is This Not Agentic AI?

If you've followed recent AI discourse, you've encountered the concept of agentic AI—AI agents that can perform tasks autonomously. While agentic systems are ideal to tackle complex workflows, their usefulness is limited by the interface provided to interact and manage them. Spawning an army of AI assistants to call every restaurant to find you a reservation is an oft cited use case, but lest the restaurants be overrun by nonstop calls they would also need to institute agents to answer and field these calls and both sides of the equation need the involvement of busy and distracted humans.

As a result - applications like OpenTable will remain relevant because whether or not they use agents behind the scenes, they provide this utility directly. The difference between agentic AI and Just in Time Software is that in JIT Software, code is generated on the fly to provide the interface to the underlying workflow, whether agentic or not. For instance, a Just in Time OpenTable would generate an interface of various restaurants for you to choose from, then engage underlying workflows to book your reservation. While such a system would still need constituent parts like a database, bank accounts etc., the actual utility would be executed by code generated on the fly, some of which may employ AI agents or even human counterparts.

How Would This Even Work?

Currently, JIT Software isn't fully feasible because it requires generating and executing code as fast humans can interact with the system. Initially, JIT Software will likely start piecemeal, such as generating code to execute API calls “just in time” and transforming the resulting structured data within the context of an existing framework or codebase.

The most relevant aspect of large scale connectivity models such as LLMs to consider in the context of the prospect of just-in-time software is that they are in effect extremely capable at generating code. Modern LLMs¹ are stellar at approximating the most likely next “token” which we’ll simply for the purpose of this discussion as a word or symbol in the context of a programming language.

So lets take a simple idea, say a shopping app —as a user, I’m searching for a pair of comfortable tennis shoes. An initial interface may be nothing more than a prompt or chat interface. Apart from that, there is no code base. While an existing AI / agentic system would take such a prompt and run it against various predefined workflows in the context of available capabilities, instead a JIT Software system would accomplish the task as follows:

• Determine the top online shoe retailers

• Generate the API request code for each of these retailers’ data interfaces (or scrape the data from their website)

• Execute the API request code / scraper code to collect structured data across the disparate data sources

• Generate the code to display the results to the user (for example static html driven by the results of the API requests)

• Execute the display code and present to the user

While simplistic, this example illustrates the pattern of Just in Time Software: determining utility, generating code to address it, and executing the code.

This will take forever! What about bugs or unexpected issues? This is where I’ll take the leap of faith and say that those are simply issues to be solved. If time is not a concern, the above is possible today — either using feedback loops, Chain of Thought, and/or RAG - it could be possible to build this as a server side rendered application today, although unlikely to be competitive with current ecommerce offerings.

But it will get faster, and faster — capabilities will get distilled into smaller networks, and computers will continue to get faster, have denser compute, become more power efficient, and inexpensive — storage continuously getting cheaper and cheaper. This is the inevitable next step in the evolution of software and how people leverage machines to do their bidding.

The Inevitability of Just in Time Software and What's After

Programming has always been about raising the level of abstraction to increase the velocity of development. Punch cards that input machine code directly led to languages like C, quickly followed by object-oriented programming with languages like Java and C++. Scripting languages like Python and JavaScript introduced higher-level abstractions and reflection capabilities profoundly increasing development speed and productivity, and optimizations such as Just-in-Time compilation made the modern web possible.

Regardless of what they’re made of and how they are programmed, the role of computers has always been to convert human “want” or “utility” into results. Even before electronics, humans would fill warehouse sized rooms working on solutions to complex mathematical equations and instructions. These people were in fact called computers, and their instructions looked much more like the prompts we associate with AI today.

Now, with expressive compositional capacity of LLMs, we can envision converting natural language directly into code. Recent research even demonstrates the possibility of transforming English directly into machine code, skipping the intermediate of code to execute altogether.

If natural language can be converted into computer instructions directly, there is no actual need for a codebase. The programming of computers will simply transform into instructing them directly with English, and the notion of Just in Time Software becomes inevitable.

The Magic Notebook

So, what does this mean for software developers and the noble software development profession? As a kid, I dreamed of a "magic notebook" where anything I thought up could become reality. I would spend countless hours coming up with ideas in paper notebooks, pretending that one day simply drawing a helicopter would make it real. I always think about this when building software. Enabling human creativity has always been the point, not the code. Over the last few decades folks began to accept that such projects required large teams, budgets and timeframes.

The goal of software development has always been solving real-world problems. The process of software development has inverted this as of late, forcing the process to be the goal - so not only is the magic notebook of Just in Time Software inevitable, it is sorely needed to keep going.

With Just in Time Software, developers can engage directly with the problem space, instead of spending months building something to only then be able to test its utility. This future doesn't render developers obsolete; rather, it amplifies their ability to tackle challenges previously out of reach to such a degree that spending time writing code will be reserved for low-level or fundamental components, or as a study of computer science and engineering, which will only increase in its value and utility.

I, for one, welcome this new future and am extremely excited about it. Also, it will significantly amplify the value of study and learning, and encourage folks to get out of echo chambers and diversify their exposure to the various aspects of human knowledge, which will also be assisted by this technology to further expand and broaden their horizons.

Also, thank you to Stacey Beck, Margaux Pelen, Yi Ding, Abie Katz, Keith Teare, Danny Hamady and Ben Altieri for reading drafts of this and providing feedback.

1. LLMs, however, are fundamentally limited by the probability distribution of said next token as conditioned by said context. Natural language, as found on the open internet, appears to have a limit of how closely we can “solve” this problem, in other words we hit a limit in the form of the natural underlying entropy of language. At the moment, it’s unclear how models will overcome this barrier; one approach is to use a constrained / more structured language, which we have in the form of code. Code is inherently lower entropy, more predictable, and it’s been shown that LLMs are not just good at generating code they’re also great at in turn utilizing the code that they’ve generated.