A recent experiment with a digital store run by an AI agent is both an exciting and cautionary tale.
The agent ran the store on its own. It priced products, issued discounts, found suppliers and adjusted to users. But, ultimately, the AI agent ran the store into the red because it gave away too much, priced some products too low, and invented payment methods.
The fact that we’re already at the stage where an AI agent can begin to run a small digital store underscores how fast AI technology is progressing. Already, AI agents are being used and tested for everything from booking travel to checking for fraud to optimizing fleets, and other supply chain issues. Gartner predicts that, by 2028, AI agents will make 15% of day-to-day work decisions autonomously.
Automation without safeguards is not efficiency—it’s liability.
AI with guardrails, especially for CFOs
Procurement lives at the intersection of cost, control and compliance. It impacts a company's profitability, efficiency, and ability to adapt to market changes. Procurement also connects companies with their suppliers and is far more than just buying the right item. Purchases need to align with budgets. Enterprises need to pay contracted rates–not full prices. Vendors need to be approved for all kinds of reasons, including quality controls. Transactions need to meet gross margin thresholds, and in highly regulated industries, audit trails need to be clear.
As such, CFOs must ensure that any autonomous systems used in procurement are not only engineered for automation, but also have guardrails to ensure good business judgement.
In short, for AI agents to succeed at procurement and enrich the enterprise, they need to be able to do much the same things as humans do. This means prioritizing financial and operational integrity over guesswork and making well informed decisions. Like humans, AI agents should rarely act alone. Each action needs to be logged, traceable, and, if thresholds or rules are triggered, be governed by human review.
As such, design principles of finance-ready autonomy must include:
1. Persistent memory: AI agents need to recall decisions, know pricing benchmarks, preferred vendors and other contract terms. When an AI agent re-issues discounts, for example, it loses money. In the real world, mistakes such as double-billing or missed rebates will significantly harm business relationships and reputations.
2. Guardrails to prevent bad spend: Before spending, procurement agents must validate pricing and ensure the economic viability of what they’re doing. This means agents need to be able to reason amid live pricing changes, cost validations, and budget context. If a transaction breaks a budget rule, for instance, the agent should halt, flag, or escalate. Agents without embedded financial judgment will continue transacting, even when doing so generates losses.
3. Human confirmation: Autonomous agents need brakes. Transactions above certain spend limits, or outside data and policy norms, should trigger a human review.
4. A single source of truth: This is critical whether AI agents are involved in the finance function or not. No one can make a good decision if data in a spend management system differs from data kept elsewhere. Errors are more likely when people–or AI agents–rely on differing data sources.
AI finance agents evolve
This year is ushering in early adoption of AI agents as they pilot low-risk categories like travel. Next year, expect integrated agent ecosystems managing cross-functional workflows, impacting procurement, treasury and so on.
The experiment for instance, showed that profit-and-loss literacy needs to be coded, not assumed. This means finance bots need explicit margin checks and real-time profit-and-loss monitoring. Thresholds that shouldn’t be crossed need to live as machine-ready policy objects and be auditable like any other code. Also, real-time data feeds are great, but decisions made only off of them may miss important nuances. As a result, AI agents need to wrap queries in provenance checks. Despite constant feedback, the AI agent in the experiment kept repeating mistakes. This means even AI agents need scheduled retraining, clear rollback playbooks, and audits that check their behavior.
Finally, assess risk vs. reward. If an agent messes up, you want it to be on a low-risk task, not on a bet-the-company trade. By starting with decision-support roles and achieving success, you’ll be well positioned to graduate to autonomous actions after guardrails, audit trails, and profit tests are proven.
Design for trustworthy outcomes, not just scale
Given the speed of AI advancement, finance leaders need to get agent strategies in place now, and ensure that they include economic viability checks, dynamic exception handling, and audit-grade traceability. AI agents will drive real savings in procurement, but only if they’re deployed strategically and are governed by financial logic.
