How to Reduce AI Costs with Jev: From a Simple Decision to a Smarter AI Workflow

Start with one question: how much intelligence does this task actually need? Then build a workflow that spends accordingly.

Imagine a support assistant receiving three requests:

All three concern an order. They require very different work.

The first may need a database lookup. The second may need a policy passage and a short explanation. The third may require several facts, overlapping policies, and a careful decision about what the application is allowed to do.

Yet many AI applications send all three to the same powerful model, with the same long prompt and workflow. It is convenient to build. It can also be expensive to run.

The opportunity is to match the work to the request: use code for a known lookup, an economical model for a routine explanation, and a stronger model when the problem needs deeper reasoning.

Jev can help make the decisions between those paths. To see how that affects the bill, we first need to understand what it does.

What is Jev, in plain language?

Jev is TypeSafe’s model for making structured decisions. You supply the information needed for a judgment and a clearly defined question. It returns a result that application code can use directly. TypeSafe introduction

For our support assistant, the question might be:

Which handler should receive this request: order tracking, policy explanation, or complex support?

Jev selects a path. The application sends the request to the handler that does the work.

TypeSafe calls the information you supply the state. Here, the state could contain the customer’s message and relevant order facts. The question describes the decision to make.

There are three question types:

Question type	What it does	Example in our assistant
Choice	Selects from defined options	Choose the support handler
Score	Evaluates information against a rubric	Assess a policy passage’s relevance
Noul	Returns a probability for a yes/no proposition	Assess whether a passage addresses a stated issue

These are building blocks for larger workflows. Choice can support routing; Score can help rank evidence; Noul can support a narrow check. A result from any of them still needs to be interpreted in the context of the task.

When we refer to a Jev framework in this article, we mean the application design built around these decisions. Jev is the model; your code supplies the handlers, checks, budgets, and fallback rules.

Now we can see where the savings come from.

Why a small decision can reduce a large bill

An AI application spends money sending input to models, generating output, retrieving information, and invoking tools. A single request may trigger several of these steps, especially when an agent can decide what to do next.

A routing decision adds one step. It creates savings when it prevents more expensive work later.

For “Where is my order?”, our assistant could retrieve the authenticated customer’s tracking record and display a fixed response. There may be no reason to generate a new answer. If a trusted task identifier already identifies the request, ordinary code can select this path without calling Jev.

For “Explain your return policy,” a small model may be sufficient when given the correct passage. The damaged-order question can go directly to the stronger path.

This follows the central principle of the original cost-optimization approach:

Use the least expensive execution path that reliably meets the task’s requirements.

Choosing a cheaper model is only part of that decision. A weak answer that needs two retries and a human correction may cost more than a good first answer.

Measure the cost of getting the job done:

Cost per successful task
  = total spending on all attempts / number of successful outcomes

Define success explicitly. For tracking, it means returning the correct authorized record. For a policy explanation, it means an accurate answer supported by the current policy. Merely producing text is not enough.

Give the assistant a sensible first route

Before building a router, inspect real requests. Identify recurring tasks, tasks that need reasoning, and tasks that code can already handle.

Our first workload map might look like this:

Request	Starting approach	What must be checked
Order tracking	Authenticated lookup and fixed response	Correct customer and order
Routine policy explanation	Retrieval and a small model	Current policy and supported claims
Conflicting policies or unusual exceptions	Stronger model with relevant evidence	Correct interpretation and missing facts
Refund execution	Authorized business workflow	Eligibility, permissions, and transaction rules
Unsupported or unclear request	Clarification or review	What information is missing

This map is a hypothesis. Test it against examples before deciding which requests a small model can handle reliably.

Add Jev where an unstructured message requires interpretation to choose a supported handler. Keep the question specific. “Which handler matches this request?” gives the application a concrete next step and is easier to evaluate than “How difficult is this problem?”

Include an uncertain or unsupported path. Otherwise, the router must select something even when none of its options fits.

Apply authorization in application code before any model-directed action. A routing decision cannot grant permission to view an order or issue a refund. Understanding that a customer wants a refund is different from establishing that the refund is permitted.

The first implementation can remain small: a few known paths, one decision, and clear rules for what happens next.

Let the small model try—and give it an exit

A route can be reasonable while the resulting answer is wrong. Our routine policy question might reach the small model, which then overlooks an exception or invents a deadline.

The workflow needs an output check and a fallback.

Figure 1 — Start with an eligible economical path, check its output, and escalate once when necessary.

This pattern is called a cascade: one model attempts the task, and a more capable model handles selected failures. Research such as FrugalGPT explores cascades for cost-efficient execution, although its results do not predict savings for Jev or for our assistant. FrugalGPT paper

The check should match the task. Structured extraction can use required fields, data types, and arithmetic checks. A policy answer needs checks for factual support and whether it addresses the whole question. Valid JSON proves that an answer has the right shape; it does not establish that its contents are correct.

Our assistant can escalate a failed small-model answer once. The stronger answer still needs appropriate checks. If it also fails, ask for missing information or send the task to review.

That exit matters. Repeatedly asking models to try again can turn one difficult request into an uncontrolled sequence of paid calls.

We now have a workable design. Is it economically worthwhile?

Follow the money through 100 requests

Instead of assuming a universal savings percentage, let’s price the paths we have built.

The numbers below are hypothetical assumptions, not Jev price quotes or measured results.

Step	Assumed cost
Routing decision	$0.0002
Small-model attempt	$0.0020
Check after the small attempt	$0.0003
Strong-model path, including its normal checks	$0.0200

Suppose all 100 requests need model-based handling. Routing sends 70 to the small model and 30 directly to the stronger one. Seven of the 70 small attempts fail validation and escalate.

We pay for:

The total is $0.921 for 100 requests, compared with $2.00 if every request took the strong path. That is a modeled reduction of 53.95%, assuming comparable accepted quality and no additional retries or review costs.

At 500,000 monthly requests, the same assumptions produce a bill of $4,605 instead of $10,000. New engineering, monitoring, and operating expenses would reduce those gross savings.

Now change one assumption: suppose 63 of the 70 small attempts escalate. The total becomes $2.041 per 100 requests, slightly more than the original design. Most small-path requests pay for both attempts.

Savings depend on how many requests finish on the economical path, not merely how many start there.

For engineers who want to generalize the calculation:

Average routed cost
  = routing cost
  + small-path fraction × (small attempt + its check)
  + strong-path fraction × strong-path cost

Strong-path fraction
  = direct strong fraction
  + small-path fraction × escalation rate

Use measured costs for each path. Escalated requests may have longer prompts and higher costs than requests sent directly to the strong model. A single average can hide that difference.

How does the router know when to trust its decision?

The damaged-order question may clearly need complex support. “Can I return this?” is less clear: the purchase date, item category, and applicable policy may be missing.

Uncertainty is useful here. Jev’s Choice answers include probabilities and a confidence value that the application can use as a routing signal.

But confidence needs careful interpretation. TypeSafe documents Choice confidence as a statistic derived from the option probabilities. With three options and probabilities 0.8, 0.1, 0.1, it is 0.7, rather than 0.8. Noul returns a probability without a separate confidence field. TypeSafe confidence documentation

A confidence of 0.95 does not establish that only 5% of your assistant’s answers will be wrong. It concerns the routing question, and its usefulness depends on performance on your data.

Collect labeled support requests and test several thresholds. At each setting, measure:

• - Coverage: how many requests qualify for the economical path?
• - Error: how many accepted answers fail the quality standard?
• - Escalation: how often does the small attempt need the stronger model?
• - Cost and latency: what does the complete workflow consume?

A stricter threshold may reduce mistakes but send more traffic to the expensive path. A looser one may increase rework. Choose a setting that meets the quality and response-time requirements of the application.

As the system matures, inspect results separately for languages, request types, and customer groups. A strong overall average can conceal a weak segment. Tune on one set of examples and confirm on a separate test set, so the threshold is not simply fitted to familiar cases.

This is how calibration becomes operational: you establish how the decision signal behaves on the workload where you intend to use it.

Better routing still needs better context

Return to the policy question. Even if it reaches the right model, the assistant might retrieve twenty passages and send all of them into the prompt.

Some repeat the same information. Some concern another country. One contains the exception that matters.

Retrieval-augmented generation, or RAG, means finding source material and providing it to the model before it answers. That material affects both accuracy and cost. A small model with precise evidence may perform better than the same model buried in irrelevant text.

Figure 2 — Keep the evidence needed to answer the question, including exceptions. Remove repetition and irrelevant material.

First, use trusted metadata to retrieve the appropriate policies: country, product category, effective date, and access scope. Remove duplicates. Then compare conventional ranking methods with Jev for a narrow relevance decision or score.

For the promotional-credit refund question, the assistant may need both the damage policy and the credit policy. Removing one because it seems less relevant can make the answer cheaper and wrong.

Preserve source identifiers and versions so the answer can cite its evidence. If the selected passages do not answer the whole question, retrieve more information or ask for clarification.

The filtering step also needs a financial test. If it costs $0.002 and avoids $0.018 in generator input, it has room to create savings. If the removed input was already billed at a discounted cached rate and would cost only $0.0018, the same filter loses money before considering errors.

Measure dollars avoided, not just tokens removed.

Reuse work, then limit what remains

Our assistant now selects a path and supplies focused evidence. Two further changes can help: reuse valid results and bound repeated execution.

Caching means saving something that can safely be reused. Repeated questions might reuse an embedding, a retrieval result, or a decision. Final answers need stricter freshness and personalization checks.

A cached policy answer may become invalid when the policy changes. A cached order lookup must not cross customer boundaries. Include relevant access scope, source version, question definition, and model or policy version in the cache identity.

Provider prompt caching is different: it can reuse computation for repeated input rather than reuse the final answer. Its eligibility and billing rules vary, so verify actual cache hits before estimating savings.

Now consider an assistant that acts as an agent: it calls tools, inspects results, and decides what to do next. Each additional step can spend money. Give the entire request a shared budget:

• - A maximum number of model and tool calls.
• - A token or monetary allowance.
• - A deadline and limited retry count.
• - A stopping rule when the task succeeds or cannot proceed.

In advanced systems, several branches may run at once. Reserve budget before launching each branch, then reconcile with actual usage. Otherwise, each branch may believe it can spend the same remaining allowance.

Count retries inside SDKs as well as your own code. For actions such as refunds, use transaction and idempotency controls to prevent a timeout from triggering a duplicate action.

The customer’s wait matters too. A request that tries the small model and then escalates experiences both stages. Track end-to-end p95 latency—the time within which 95% of requests finish—rather than only average call speed.

A small Python example: turn a decision into a path

We can now express the basic idea in code. This example asks Jev to select a support handler, then applies a simple confidence rule.

It follows the documented typesafe-sdk pattern, which reads TYPESAFE_API_KEY from the environment. Running it requires the package, credentials, and network access. TypeSafe Python quickstart

from math import isfinite
from typesafe_sdk import Choice, TypeSafeClient

message = "Can you explain the return policy for an unopened item?"

with TypeSafeClient(model="jev-latest") as client:
    response = client.system_one(
        state={"customer_message": message},
        questions={
            "handler": Choice(
                instructions="Choose the support handler for this request.",
                criteria={
                    "tracking": "Find the delivery status of an order",
                    "policy": "Explain a routine published policy",
                    "complex": "Resolve exceptions or overlapping issues",
                    "review": "Unclear request or no supported handler",
                },
            )
        },
    )

    decision = response.choices["handler"]
    handler = decision.choice
    confidence = decision.confidence

# Illustrative threshold: select it using your evaluation data.
valid_confidence = isfinite(confidence) and 0 <= confidence <= 1
if handler == "review" or not valid_confidence or confidence < 0.90:
    path = "clarification_or_review"
elif handler == "tracking":
    path = "authenticated_order_lookup"
elif handler == "policy":
    path = "retrieve_policy_then_small_model"
elif handler == "complex":
    path = "strong_model_with_relevant_evidence"
else:
    path = "clarification_or_review"

print(path)

The script chooses a path; it does not perform the lookup or generate an answer. Those handlers need the authorization, evidence, output checks, and fallback behavior developed above.

Handle API failures within a fixed deadline and use a review or permitted fallback path. Record the model and question version used. Pin an available model version when you need reproducible comparisons.

The design choice is visible: the model supplies a bounded judgment, while code controls what that judgment can cause.

Prove the improvement before expanding it

Our assistant now has several opportunities to save money. Deploying them all at once would make it difficult to discover which change helped or introduced a problem.

Start with one expensive, repeatable workload. Compare the original and proposed workflows on the same representative requests.

Measure	What you want to learn
Cost per successful task	Did the complete outcome become cheaper?
Answer quality	Are responses correct and supported?
Completion and review rates	Are savings coming from efficiency or unfinished work?
Escalation rate	Can enough requests finish on the small path?
End-to-end latency	Do fallback calls make customers wait too long?
Human effort and operating cost	Did the expense move elsewhere?

Audit accepted answers as well as failures. A validator can miss an error, so reviewing only escalations gives an incomplete picture.

A useful first production step is shadow routing: calculate the new decision while the existing workflow still serves the customer. Test alternate paths on a sample to observe what they would cost and whether they would succeed. A routing label alone cannot reveal the quality of an unexecuted path.

Then release to a small fraction of traffic with clear rollback criteria. Reevaluate when policies, model versions, or customer behavior change.

When Jev is worth adding

This design is promising when requests vary in complexity, decisions repeat, and expensive model calls make up a meaningful part of the bill. Support, document processing, enterprise search, and tool-using agents often have those characteristics.

The savings may be limited elsewhere. If almost every request needs deep reasoning, a small attempt may add cost and delay before the inevitable escalation. If traffic is low, the savings may not cover maintenance. If the main expense is idle GPU capacity, storage, or a search cluster, routing may barely move the total bill.

When a reliable rule already selects the handler, use it. Jev is most useful where a narrow semantic judgment is needed and its measured benefit exceeds its cost.

This returns us to the starting point: establish where the money goes before choosing how to optimize it.

Turn the idea into a 30-day pilot

You do not need to redesign the entire AI platform to begin.

• - Week 1: Measure. Find one expensive workflow, define success, and collect representative requests with complete cost and quality data.
• - Week 2: Compare. Test simpler rules, alternative models, and a Jev decision. Identify which requests can safely take a cheaper path.
• - Week 3: Build. Add routing, task-specific checks, one fallback, and workflow limits. Run the new policy alongside the baseline.
• - Week 4: Validate. Compare cost per successful task, quality, latency, and human effort. Expand when the measured improvement holds.

Return to our three customers. The first gets a tracking answer through a lookup. The second gets a clear explanation grounded in the right policy. The third receives the additional reasoning and checks that the situation requires.

They do not need to know which model handled the request. They need the assistant to resolve it accurately and promptly.

That is the outcome a Jev-based workflow should work toward: spend more intelligence where it improves the result, remove unnecessary work elsewhere, and measure the cost of a successful resolution. Start with one decision that changes what the application does next. If that decision reliably avoids expensive work, you have a foundation worth building on.

Sources and further reading

Reduce AI Costs Without Compromising Quality

CloudVectra helps engineering and FinOps teams optimize AI spending through cost-aware model selection, intelligent routing, token optimization, and efficient RAG and multi-agent architectures.

Contact Us