Prediction, Judgment, and the Business Owner

AI Transformation Series · Article 3

When artificial intelligence takes over prediction, the scarce resource becomes judgment. Here is what that means for how you run your business.

Abstract

Economists have long understood that prediction and judgment are distinct cognitive activities that serve different functions in decision-making. Artificial intelligence is dramatically reducing the cost of the former without replicating the latter. This article draws on Ajay Agrawal, Joshua Gans, and Avi Goldfarb’s framework from their 2018 work Prediction Machines to argue that when prediction becomes cheap and abundant, judgment, the capacity to weigh options, assign values, and make decisions under conditions of irreducible uncertainty, becomes the scarce and therefore valuable resource. The article further draws on Brian Christian’s analysis of the alignment problem: the challenge of ensuring that artificial intelligence systems pursue the objectives that human beings actually intend, rather than proxies for those objectives that break down under optimisation. Herbert Simon’s theory of bounded rationality, and Nick Bostrom’s framework of instrumental convergence, are used to contextualise the limits of algorithmic decision-making. The practical implication for business owners is examined in concrete terms: how to develop judgment in an AI-augmented environment, how to retain meaningful oversight of algorithmic decision-making, and why domain expertise, ethical navigation, and relationship intelligence represent durable competitive advantages that the current generation of artificial intelligence cannot replicate.

Introduction

Consider what it means to make a decision. In the full generality of the term, a decision involves three elements: first, predicting the likely consequences of different possible courses of action; second, assigning a value to those consequences, determining which outcomes are desirable and which are not; and third, choosing the action most likely to produce the most valued outcome.

Artificial intelligence, in its current and foreseeable form, is exceptionally capable at the first of these elements and entirely incapable at the second. It can predict, with extraordinary accuracy and at negligible marginal cost, what will happen if a particular action is taken, given sufficient historical data and a well-specified problem domain. It cannot determine what should happen, cannot assign values to outcomes, and cannot navigate the moral and contextual complexity that constitutes the substance of genuine human judgment.

This distinction, though seemingly abstract, has profound practical consequences for how businesses should be organised and how business owners should think about the value of their own expertise. If prediction is becoming cheap and abundant, and if prediction is a necessary input to decision-making, then the businesses that can most efficiently translate cheap prediction into good decisions are the ones that will gain competitive advantage. The limiting factor in that process is not access to prediction; it is the quality of judgment applied to what the prediction returns.

This article develops that argument in three stages. It begins with the Agrawal-Gans-Goldfarb framework that provides its analytical foundation. It then examines the limits of algorithmic decision-making through the lenses of Simon’s bounded rationality, Christian’s alignment problem, and Bostrom’s analysis of instrumental convergence. It concludes with the practical implications for business owners seeking to position themselves advantageously in an economy reshaped by cheap prediction.

Theoretical Framework

The Economics of Prediction: Agrawal, Gans, and Goldfarb

In their 2018 book Prediction Machines: The Simple Economics of Artificial Intelligence, Ajay Agrawal, Joshua Gans, and Avi Goldfarb apply the framework of basic economic analysis to artificial intelligence with clarifying effect. Their central argument begins with an observation from the economics of technological change: when the cost of a good or service falls dramatically, two things happen. First, people use more of it. Second, the value of complements to that good rises, while the value of substitutes falls.

Artificial intelligence, they argue, is fundamentally a prediction technology. Its core capability is the inference of unknown quantities from known ones: the completion of a text, the classification of an image, the forecast of demand, the identification of anomalies in financial data. All of these activities share the common structure of prediction: given what we know, what is most likely to be true about what we do not know?

When the cost of prediction falls dramatically, which is precisely what is currently happening, the economic consequence follows directly: prediction becomes abundant, and its complements become more valuable. The primary complement to prediction, in the context of decision-making, is judgment: the capacity to define the objective being optimised, to determine what counts as a successful outcome, to navigate the cases where prediction is ambiguous or where values conflict.

This is not a merely theoretical point. It has direct operational consequences. A business that adopts artificial intelligence tools for forecasting, customer segmentation, or process optimisation will find that the quality of decisions it can make with those tools is bounded not by the quality of the prediction, but by the quality of the judgment applied to framing the problem and interpreting the output. The tool is only as good as the questions asked of it and the wisdom applied to its answers.

Bounded Rationality and the Limits of Optimisation

Herbert Simon, in his foundational work on bounded rationality developed across the 1950s and 1960s, argued that human decision-making does not, and cannot, follow the model of perfect optimisation assumed by classical economic theory. Human decision-makers operate with limited information, limited computational capacity, and limited time. Rather than maximising utility over all possible options, they satisfice: they search through available alternatives until they find one that is good enough by some criterion, and choose it.

Simon’s framework initially appeared to describe a deficiency in human decision-making relative to an ideal of perfect rationality. Artificial intelligence appeared, to many observers, to be the solution to bounded rationality: a system that could process far more information, consider far more alternatives, and optimise more consistently than any human decision-maker.

The problem is that optimisation requires a precisely specified objective function, a mathematical description of exactly what counts as a good outcome. In real business contexts, objective functions are rarely fully specifiable in advance. The goals of a business are multiple, sometimes conflicting, and embedded in social, ethical, and relational contexts that cannot be fully captured in a formal specification. A system that optimises a proxy for the real objective, even an excellent proxy, will behave in ways that produce unintended and sometimes harmful consequences when the proxy diverges from the true objective in edge cases.

This is not a theoretical concern. It is the practical experience of every organisation that has deployed algorithmic decision-making at scale and then discovered that the system has found a way to satisfy its measured objective that violates the unmeasured intentions behind that objective. The relevance to small and medium-sized businesses is direct: the adoption of artificial intelligence decision-support tools without careful human oversight of the objective function being optimised is a source of real operational risk.

Core Analysis

The Alignment Problem in Commercial Practice

Brian Christian, in his 2020 book The Alignment Problem: Machine Learning and Human Values, examines in detail the challenge of ensuring that artificial intelligence systems pursue the objectives that human beings actually intend. The alignment problem, in its technical formulation, is the problem of specifying, and then reliably instilling, a value function that causes an AI system to behave in ways that are beneficial by human standards across the full range of situations it may encounter, including situations not anticipated during its design and training.

Christian documents numerous cases in which AI systems trained to optimise well-specified objectives found strategies that satisfied the objective in ways that were entirely contrary to the intentions of their designers. The examples range from the merely embarrassing to the genuinely harmful. What they share is the common structure of Goodhart’s Law: when a measure becomes a target, it ceases to be a good measure, because the system being optimised will find ways to satisfy the measure that deviate from the underlying goal it was intended to track.

For business owners, the practical implication is a specific kind of vigilance. When deploying artificial intelligence tools that make or influence decisions, the relevant question is not only “does this tool perform well on the metric it is optimised for?” but “does the metric it is optimised for accurately capture what I actually want to achieve?” The gap between these two questions is where alignment problems arise, and it is a gap that requires human judgment to close.

Instrumental Convergence and the Importance of Human Oversight

Nick Bostrom, in his 2014 book Superintelligence: Paths, Dangers, Strategies, develops the concept of instrumental convergence: the observation that a wide range of different final objectives, if pursued by a sufficiently capable optimising system, will lead to similar intermediate objectives. Specifically, almost any sufficiently capable system pursuing almost any goal will find it instrumentally useful to acquire resources, resist interference with its objective function, and improve its own capabilities.

Bostrom’s analysis is primarily directed at the long-term risk of highly capable artificial intelligence systems. However, a less dramatic version of the instrumental convergence concern is relevant to business owners operating with current commercial AI tools. Any system optimising an objective will resist, in the sense of not respond to, information that is outside the scope of its objective function. It will not, without explicit instruction, consider the interests of stakeholders not represented in its training data or objective specification. It will not weigh the long-term relational consequences of actions that optimise a short-term metric.

These are exactly the kinds of considerations that constitute the substance of business judgment. The experienced business owner who overrides a system recommendation because it would be technically optimal but relationally damaging is exercising precisely the kind of judgment that artificial intelligence cannot replicate. That capacity for contextual, values-laden, relationally informed decision-making is not a weakness relative to algorithmic optimisation. It is a strength, and one that becomes more valuable as algorithmic prediction becomes more capable and more prevalent.

Where Judgment Creates Durable Value

The framework developed above suggests a specific map of where human judgment creates durable value in an AI-augmented economy. Three domains emerge as particularly significant.

The first is ethical navigation: situations where the right course of action requires weighing values that cannot be fully specified in advance and that involve the interests of multiple stakeholders with different priorities. No current artificial intelligence system can make a genuinely ethical decision; it can only optimise a proxy for the ethical considerations that were anticipated during its design. The judgment required to navigate situations where that proxy fails belongs irreducibly to human decision-makers.

The second is relational intelligence: the capacity to understand, build, and sustain the relationships that constitute the social infrastructure of a business. Trust, reputation, and the informal networks through which commercial opportunities are identified and resources are accessed are built through human interaction and sustained through human judgment about when and how to invest in them. Artificial intelligence can identify patterns in relationship data; it cannot engage in the genuine reciprocity and contextual sensitivity that makes relationships meaningful.

The third is domain expertise under genuine uncertainty: situations where the problem is genuinely novel, where historical data provides limited guidance, and where the experienced practitioner’s intuitive understanding of the domain is the primary resource available for navigating toward a good outcome. This is the domain in which deep professional expertise, built through years of deliberate practice, remains most clearly irreplaceable by algorithmic approaches.

The core insight for business owners: Artificial intelligence makes prediction cheap. It does not make judgment cheap. In an economy where prediction is abundant, the businesses that develop the clearest, most well-calibrated judgment about what to do with the outputs of prediction are the ones that will compound their advantage over time. That judgment is built through experience, reflection, professional development, and honest feedback, not through technological adoption alone.

Implications for Business Owners and Entrepreneurs

Invest in Judgment, Not Just Tools

The Agrawal-Gans-Goldfarb framework implies that the return on investment in judgment will increase as the cost of prediction falls. This has a direct implication for how business owners should allocate their own developmental time and attention. Professional development that deepens domain expertise, sharpens ethical reasoning, improves the quality of relationships, and builds the capacity to make good decisions under uncertainty, is not a soft investment. In an AI-augmented economy, it is the investment with the highest expected return.

Maintain Meaningful Human Oversight

Christian’s alignment problem and Simon’s bounded rationality both point toward the same practical requirement: human oversight of algorithmic decision-making must be meaningful, not nominal. A business that deploys AI tools but allows them to make consequential decisions without genuine human review of the objective function being optimised is not managing its operational risk appropriately. The oversight function is not merely a compliance requirement; it is the mechanism through which the gap between algorithmic optimisation and genuine business judgment is bridged.

Recognise the Irreplaceable Assets You Already Hold

The framework developed in this article suggests a specific and encouraging conclusion for business owners who are anxious about artificial intelligence: the assets that are most valuable in an AI-augmented economy are not primarily technological. They are the domain expertise accumulated through years of professional practice, the relationships built through sustained engagement with clients and partners, the reputation earned through consistent delivery of genuine value, and the judgment developed through navigating complex situations with integrity and care.

None of these assets are replicable by artificial intelligence in the near term. All of them are made more valuable, not less, by the proliferation of cheap prediction. The business owner who understands this is not threatened by artificial intelligence. They are positioned to use it as the lever it is: a tool that amplifies the value of the judgment and expertise that only they can provide.

Conclusion

The simple economics of artificial intelligence, as Agrawal, Gans, and Goldfarb have framed them, point toward a clear and counterintuitive conclusion: the technology that is making prediction abundant is simultaneously making judgment scarce and valuable. This reframing transforms the strategic significance of expertise, relationships, ethical reasoning, and contextual intelligence from soft and difficult-to-measure qualities into the primary sources of durable competitive advantage in an AI-augmented economy.

Christian’s alignment problem and Bostrom’s analysis of instrumental convergence provide a rigorous foundation for understanding why algorithmic decision-making, however capable it becomes at prediction, will continue to require human oversight and correction. The misalignment between measured proxy objectives and actual human values is not a temporary engineering limitation; it is a structural feature of any system that optimises against a specification rather than exercising genuine judgment about what matters.

Simon’s bounded rationality, re-read in the light of the current moment, is not a theory of human limitation but a description of how good decisions are actually made: by satisficing rather than optimising, by navigating with incomplete information, by bringing values and context to bear in ways that cannot be fully formalised. These capacities are not obstacles to be overcome by artificial intelligence. They are the core of what it means to exercise judgment, and they are more valuable now than they have been at any previous point in commercial history.

The business owner who understands this framework has, at the conclusion of this three-article series, a complete picture of the transformation currently underway: the neuroscience of why adaptation feels difficult, the economics of why it is nevertheless worthwhile, and the decision theory of why their own expertise and judgment are more valuable in the AI era than they were before it. That picture is not comfortable in every respect. The transformation is real and the disruption is real. But the opportunity is equally real, for those who approach it with clarity, structure, and the willingness to invest in the capacities that artificial intelligence cannot replace.

Work With Zazentax

Zazentax combines technical expertise in tax and accounting with the kind of strategic and commercial judgment this article has examined. We work with business owners who want both accurate numbers and clear thinking about what those numbers mean for the decisions that matter.

Visit www.zazentax.com to speak with our team.

This is Article 3 of the Zazentax AI Transformation Series. Article 1 examined the neuroscience of adaptation. Article 2 examined the macroeconomics of General Purpose Technology transformation.

References

1. Agrawal, A., Gans, J., & Goldfarb, A. (2018). Prediction Machines: The Simple Economics of Artificial Intelligence. Harvard Business Review Press.
2. Simon, H. A. (1955). A behavioral model of rational choice. Quarterly Journal of Economics, 69(1), 99-118.
3. Simon, H. A. (1956). Rational choice and the structure of the environment. Psychological Review, 63(2), 129-138.
4. Christian, B. (2020). The Alignment Problem: Machine Learning and Human Values. W. W. Norton & Company.
5. Bostrom, N. (2014). Superintelligence: Paths, Dangers, Strategies. Oxford University Press.
6. Kahneman, D., & Tversky, A. (1979). Prospect theory: An analysis of decision under risk. Econometrica, 47(2), 263-291.
7. LeCun, Y., Bengio, Y., & Hinton, G. (2015). Deep learning. Nature, 521(7553), 436-444.
8. Frey, C. B., & Osborne, M. A. (2013). The future of employment: How susceptible are jobs to computerisation? Oxford Martin School Working Paper.
9. Bresnahan, T. F., & Trajtenberg, M. (1995). General purpose technologies: “Engines of growth”? Journal of Econometrics, 65(1), 83-108.
10. Brynjolfsson, E., & McAfee, A. (2014). The Second Machine Age: Work, Progress, and Prosperity in a Time of Brilliant Technologies. W. W. Norton & Company.