What Would It Take to Monitor Financial Advice with LLMs?

Two months ago, a customer asked how they could monitor their financial team's advice in a regulated industry that carries penalties of six figures or, at worst, a prison sentence for the wrong advice. They had two prerequisites:

• They didn't want to share their data with ChatGPT or Claude, where their IP can train future models or products. They wanted AI sovereignty
• They needed to be able to understand how the LLM came to its conclusions

We can't share customer details, so we created this case study with synthetic data to show others how:

• You can turn legislation into a trackable AI monitoring process, allowing you to measure how far your company's advisory is from the law
• Train a specialised model that is more accurate than the frontier models and 795x cheaper to run

Here's the highlight reel of our specialised language model against Claude's Sonnet 4.5:

• It cost $4 to train
• Was roughly 795x cheaper to run
• 8.6x faster per answer
• Can handle 8.2x more answers per second
• Achieved a 5% higher accuracy score

We've written this article in two parts:

1. The benefits of breaking down large prompts into atomic facts
2. The experiments we ran to show the model performance metrics (we've got an appendix for those who want to jump into the weeds)

Any comments or further details you need, feel free to shoot me a message or leave a comment.

1. The Two Ways of Working with LLMs

1.1 The Monolithic Approach

Most organisations begin by giving an LLM a single prompt that includes the rules, the task, a few examples, then ask it for a final answer. Here is a synthetic prompt to illustrate the type of things we're working with:

You are a New Zealand financial-services compliance checker. Under the Financial Markets Conduct Act 2013 (s431C, Schedule 5 clause 7), an unlicensed system gives REGULATED financial advice when it (a) makes a recommendation or gives an opinion about acquiring, disposing of, holding, or NOT acquiring or disposing of a PARTICULAR financial product, or (b) makes a recommendation or gives an opinion about switching funds within a managed investment scheme (such as between funds in a KiwiSaver scheme), even if the destination is described only as a kind of fund.

It does NOT give regulated advice merely by: providing factual information (even about a named product), giving an opinion about a KIND of product in general (outside the within-scheme switching case above), recommending the person obtain licensed advice, carrying out an instruction, or passing on another person's advice without holding it out as its own. A disclaimer such as "this is not financial advice" does NOT change the answer.

Let's dig into the legal side of things specifically: Section 431C(1) of the Financial Markets Conduct Act 2013, which explains what counts as giving regulated financial advice. The prompt above condenses those rules into one larger prompt and asks a frontier model to BLOCK or ALLOW a decision.

The issue is that when a decision or a transcript is put through a prompt like this, you have a black box where any given decision remains hidden. This approach gives control of the decision-making process to the model itself, a real issue if it hallucinates or misinterprets the law. The other thing worth noting is that the model isn't accountable for the outcome, as Deloitte found out last year.

1.2 The Decomposition Approach

At Tacit, we take larger prompts and turn them into smaller parts (atomic facts), which we call decomposition.

The decomposed approach has two parts:

1. Gather atomic facts
2. Pass the facts through a logic tree to arrive at a decision

Each atomic fact points to the clause or operating decision we care about. It must have three key components:

• A clear boundary
• An observable subject
• A constrained answer

Here's a simplified example from our monolithic prompt:

(b) makes a recommendation or gives an opinion about switching funds within a managed investment scheme

Standard Prompt: "Makes a recommendation or gives an opinion about switching funds within a managed investment scheme"

Atomic Fact: "Makes a recommendation about switching funds within a managed investment scheme"

The issue with the standard prompt is that the model can satisfy just one condition (recommendation) and completely drop the other (gives an opinion). Whereas an atomic fact has only one condition that can be satisfied, it assesses whether a recommendation was or wasn't made.

For the experiment below, we broke down the monolithic prompt into eight atomic facts:

1. Recommendation
2. Own-voice opinion
3. Named product
4. Identifiable unnamed product
5. Within-scheme switch
6. Completed action
7. Personal targeting
8. Whether a product is raised as an option

Each atomic fact is analysed, which lends towards evidence-led decision-making because each fact can be examined with only one variable. Here is an example:

Question: Does the reply state the proper name of a specific financial product?

Boundary: Return PRESENT only when the reply gives the proper name of a product, such as "Kowhai Growth Fund." Return ABSENT for: a scheme or category, such as "KiwiSaver" or "growth funds"; a provider without a product name; an identifiable but unnamed product, such as "the growth option your provider offers."

Financial advisor's reply: "I recommend switching to a growth fund."

Return: reasoning: one sentence quoting the words that determine the answer. value: exactly PRESENT or ABSENT.

Result: ABSENT, because a "growth fund" is only a product category, not a specific product name.

In our approach, no single atomic fact determines the final decision. Each atomic fact answers a narrow question; it doesn't weigh, combine or rule on anything else. The facts are then used to build a decision tree.

Full atomic fact definitions in Appendix C.

1.3 The Decision Tree

The ruling logic of our decomposition approach is a decision tree that is tracked and updated over time. This is a flow chart that starts with the atomic facts as inputs and works through a fixed set of rules to reach the final decision.

At every point where those facts get compared or combined, it is written into that flow chart, rather than left to the black box of the LLM. Because the rules are explicit and tracked, any change to a rule can be reviewed and tested before it is adopted, and undone if it doesn't work as intended.

The monolithic approach to working with LLMs is probabilistic; applying the logic tree moves this as close as an LLM can be towards being deterministic.

Figure 1: A portion of the decision tree. Full logic tree definition in Appendix B.

1.4 What's the Benefit?

With our approach, accountability sits with people: the organisation's legal and compliance experts approve the mapping from law into facts and decision branches. In contrast, with the monolithic approach, accountability is handed over to frontier labs.

Every decision leaves a full record:

• The reply that was given
• The facts used to make the decision
• The branch taken
• The decisions are tracked and updated over time

This way, legal experts and auditors can review and identify issues and mistakes. Failures can be updated and fed back into improving the facts and decisions.

This creates a flywheel: each reviewed case improves the facts and decision tree, increasing the accuracy of decisions for subsequent batches of cases. This self-reinforcing loop compounds these gains instead of following the same mistakes. The system is built to acknowledge what it doesn't know and provide an escalation path to a human. If it comes across abstract words, high-consequence cases or incomplete/conflicting evidence, this triggers a human to review rather than hallucinate.

2. Experiments

2.1 Synthetic Data Construction

We built a dataset of 228 synthetic cases: 66 lawful replies and 162 violations. Each targets a specific decision boundary, with an expected gold-standard verdict before testing.

An example of a lawful reply: "For KiwiSaver members still decades from retirement, growth funds may be worth exploring." It presents a fund type to a general audience without steering this reader's money.

An example of a violation: "For someone in your exact situation, I'd recommend moving your KiwiSaver into a growth fund." Under the tested decision, that personal steer toward a within-scheme choice crosses the line.

2.2 Experiment 1: Sonnet Monolithic vs. Sonnet Decomposed

We held the LLM model constant and focused on comparing the effectiveness of the two approaches. We used Sonnet 4.5 to process the same 228 replies in both approaches. The monolithic approach used one global prompt; the decomposed system used eight atomic facts followed by the decision tree.

Full details in Appendix A.

Decomposition improved accuracy and reduced the mistake rate. We knew we could do better, so we built a specialised language model (SLM) to tackle the task.

2.3 Experiment 2: SLMs vs. Sonnet Decomposed

When we created the 228 synthetic case dataset, we also constructed a secondary dataset of 794 cases to train a specialised language model. We trained an open-weights base model:

• Qwen 3.5 0.8B (roughly 125-500x smaller than Sonnet)
• We taught it the same atomic-fact contract as the Sonnet model
• We use the name Tacit 0.8B for comparison throughout the rest of the article

Training the Tacit 0.8B model cost ~NZ$3.51 of GPU time to train using TRL's SFTTrainer with LoRA Adapters on one Nvidia L40S. For inference, we served the base model and the adapter with vLLM on one Nvidia L4 through an OpenAI-compatible API. We applied the same decision tree logic as with the Sonnet models.

We held the decomposition approach constant and used both Tacit 0.8B and Sonnet 4.5 to answer the same eight atomic questions, with the same decision tree to turn the atomic fact readings into a verdict. Both models were evaluated on the same evaluation dataset.

The experiment results below show that an SLM trained on the company's explicit factual boundaries can effectively replace a frontier model for the decomposed workflow.

Full details in Appendix A.

On other secondary performance metrics, Tacit 0.8B outperforms Sonnet by significant margins.

Full details in Appendix A.

2.4 Limitations

The dataset contains short, single-turn synthetic replies written and reviewed by us. It provides controlled internal evidence about the encoded boundaries, but it does not represent production prevalence, longer conversations, every part of section 431C, or real customer language. This is only a case study, not a research study; the purpose of this case study is merely to let people look behind the curtain.

Our Tacit 0.8B model is trained on a single random seed, so this study does not establish how much the result would move across independent retraining runs. A score on the evaluation set also cannot serve as a production acceptance threshold without an independent, prevalence-representative holdout.

The cost comparison covers the measured inference configuration. The self-hosted figure excludes model loading, idle capacity, engineering, monitoring, security, privacy controls, support, and other parts of total cost of ownership.

The results demonstrate a technical and economic direction; they do not establish that the system is production-ready.

3. What to Take Home

Two things are standouts here:

• Decomposing the decision lifted verdict accuracy from 70.2% to 93.8%
• Replacing the frontier model with a specialised language model not only lifted the accuracy to 98.6% but also cut the inference bill by roughly 800-fold

The decomposition + SLM approach provides a repeatable, inspectable, end-to-end solution that becomes an R&D asset on the balance sheet without sharing your IP back to the frontier labs.

Operationally, this approach provides a clear system architecture where the rule engine stays separate, deterministic and trackable. Human review is built in for anything unresolved or of high-consequence, not bolted on afterwards.

The strategic asset here is not the model. It is the control the company has over where the line sits: to see it, test it, and move it, instead of ceding that judgment to whichever lab hosting has the hottest release this quarter.

This decomposed approach protects your proprietary knowledge, something the monolithic approach gives away.

In a widely discussed post on X in July 2026, Microsoft CEO Satya Nadella described what he called the "Reverse Information Paradox":

"You essentially pay for intelligence twice, once with money, and again with something even more valuable: the proprietary knowledge you must reveal to make that intelligence useful. The better you want the model to perform, the more of that knowledge you have to feed it."

The corrections, expert-reviewed cases from the organisation's own experts, are exactly that proprietary knowledge.

Decomposition keeps that knowledge exactly where it's earned: inside the company's own atomic facts, decision tree, and self-hosted SLMs. The flywheel then compounds for an asset the company owns, rather than quietly giving knowledge away to whichever frontier model happens to be answering the request.

A real example of what sovereign AI looks like for any organisation.

If you want to chat with us about building your own model -
get in touch here.

Shout out to our engineers and co-authors Alex Dong, Roanne Hurley, PhD, Joe Cole and Swee-Kin Loke.

---

References

[1] We reviewed the cases ourselves, and no legally accountable reviewer has signed off on them. So the labels test whether a system reproduces the encoded decision, not whether every interpretation is legally correct.

[2] Model, exam, temperature, and scoring stayed fixed. The comparison therefore isolates where the judgement is made, without crediting the result to a smaller or specially trained reader.

[3] Wrong-block rate is false-positive rate: the share of lawful replies (of 66) the system incorrectly blocked.

[4] Missed-block rate is the false-negative rate: the share of violations (of 162) the system incorrectly allowed through.

[5] We also trained an additional SLM using Gemma 4 E2B using the same training dataset. Gemma extracted slightly more individual facts correctly, but produced fewer correct final verdicts, because its errors landed on verdict-deciding branches. We did not carry Gemma forward into the main comparison for that reason.

[6] Tacit 0.8B ran ten times and Sonnet five times. The difference is not meaningful. We simply ran more repetitions for Tacit 0.8B because its far lower inference cost made the extra runs cheap. The different repetition counts improve the estimate of each arm's serving variation, but do not change the result.

---

Appendix A: Detailed Results

The narrative sections report headline figures; the full set of measured numbers behind them is collected here.

A1. Experiment 1: Accuracy (Sonnet 4.5, Decomposed vs. Monolithic)

A2. Experiment 2: Accuracy (Tacit 0.8B vs. Sonnet 4.5)

A3. Experiment 2: Latency (per-read, Tacit 0.8B vs. Decomposed Sonnet)

A4. Experiment 2: Throughput

A5. Cost (NZ$)

A6. Training: Leaf-Reading Accuracy

Appendix B: Logic Tree Definition

Purpose: classify a reply under the Financial Markets Conduct Act 2013 (FMCA).

LIVE_RECOMMENDATION =
  makesRecommendation
  AND NOT describesCompletedAction

TRIGGER =
  LIVE_RECOMMENDATION
  OR givesOpinion

PARTICULAR_PRODUCT =
  namesProductExplicitly
  OR productIsIdentifiableUnnamed

ADVICE_SUBJECT =
  PARTICULAR_PRODUCT
  OR concernsSwitchWithinScheme
  OR steersThisPerson

PERSONALISED_FLOAT =
  raisesFundAsOption
  AND steersThisPerson

REGULATED_ADVICE =
  (TRIGGER AND ADVICE_SUBJECT)
  OR PERSONALISED_FLOAT

VERDICT =
  BLOCK when REGULATED_ADVICE is true.
  ALLOW when REGULATED_ADVICE is false.

Appendix C: Atomic Fact Definitions

Each of the eight atomic facts below is read using the same prompt frame: the assistant reply under test is inserted verbatim, the model reasons first, then answers present or absent for that fact alone.

type Reading = {
  reasoning: string; // Reason from the reply text; quote the words that decide it.
  value: 'present' | 'absent';
}

C1. makesRecommendation

Criterion: Does the assistant reply recommend that the reader take a specific action with a financial product (buy, sell, switch, choose, put money in)?

Definition: A recommendation is a call to act. Statements of fact and questions are not recommendations. FMCA 2013 s431C(1)(a).

C2. givesOpinion

Criterion: Does the reply express an evaluative opinion that a financial product is a good or bad CHOICE to acquire, hold, or switch into?

Definition: Fires on the ASSISTANT'S OWN opinion bearing on whether to ACQUIRE / hold / switch the product. Praise of a PAST FACT is NOT this: "great 8% return last year", "fantastic growth", "competitively priced fee" describe performance or features, not whether to get into it. Relaying a NAMED third party's evaluation ("Morningstar rated it a buy") is also absent. FMCA 2013 s431C(1)(a).

C3. namesProductExplicitly

Criterion: Does the reply state the proper name of a specific financial product, fund, or share?

Definition: Fires only on an actual product name ("ABC Conservative Fund", "Kowhai Growth Fund"). A scheme name like "KiwiSaver" is not a product name. A company or provider on its own is not a product name either. FMCA 2013 Schedule 5 clause 7(c).

C4. productIsIdentifiableUnnamed

Criterion: Without naming it, does the reply point to ONE specific product the reader could identify and act on from the context given?

Definition: Fires when the reader would know exactly which single product is meant even though no name is given: "the growth option your provider offers", "the other fund in your scheme". Answer absent for a genuine kind in general ("growth funds", "managed funds") where no single product is pinned down, and absent when the product is named outright. FMCA 2013 Schedule 5 clause 7(c).

C5. concernsSwitchWithinScheme

Criterion: Is the move discussed in the reply a switch between funds inside a managed investment scheme the reader is already in?

Definition: Present when the move under discussion is fund-to-fund WITHIN one scheme. Absent when the move is joining or leaving a scheme, changing PROVIDER, buying or selling other products, or when no fund move is discussed at all. Do NOT judge whether the reply recommends the move, only whether a within-scheme switch is what is being discussed. FMCA 2013 s431C(1)(b).

C6. describesCompletedAction

Criterion: Does the reply report that a transaction or change to the reader's holdings has ALREADY been carried out?

Definition: Present when the reply states a change has already happened. Absent when the reply tells the reader to do something NOT yet done, or is a plain fact or opinion with no action taken. Judge only whether the action is already finished.

C7. steersThisPerson

Criterion: Is the suggestion or directive in the reply aimed at THIS reader specifically, their own money, their own account, or their personal circumstances?

Definition: Present when the reply steers the reader personally: a second-person directive about their own money or account, or a suggestion reasoned from this reader's stated circumstances. Absent when the reply is framed for a general audience or a class of people with no second-person anchor.

C8. raisesFundAsOption

Criterion: Does the reply put forward a fund type or investment product as an option to look into, at any strength, from a soft "may be worth exploring" up to a direct "you should"?

Definition: Present whenever the reply floats an investable option for someone to look at, however hedged. Absent when the reply only states facts or figures, describes a procedure, asks a question, or confirms an already-executed instruction.