Skip to content
All articles
AI Engineering

AI Automation vs Traditional Automation: When to Use Which

A framework for mixing deterministic automation with model calls: the four-stage spectrum, how to tier a pipeline so the model runs last and least, and a production dispute-detection system built exactly that way.

Rushit Jivani6 min read

The short version

  • AI automation and traditional automation are not rivals — they are stages on one spectrum: plain code, rules, small model calls, open-ended generation. Cost and unpredictability rise left to right.
  • Tier your pipelines: deterministic checks first, patterns second, a constrained model call only on the remainder, and a fail-closed default when the model errors.
  • The case study: a dispute-risk detector where record checks and regex handle the certain cases for free, a JSON classifier with a 0.75 confidence floor handles only the misses, and alerts ride the notification system the product already had.

"We should use AI to automate this" is the most common sentence in product meetings this decade — and the most common mistake inside it is treating AI automation as a replacement for the automation you already know. Cron jobs, queues, validations, pattern matching: that machinery is not legacy. It is the cheap, deterministic 80% of every automation problem, and the model is the expensive, probabilistic layer you add for the remainder that genuinely needs judgment.

This guide draws the line properly: what each kind of automation is actually good at, how to combine them into tiered pipelines, and a production walk-through — a dispute-detection system where the model is deliberately the last and least-used stage.

The spectrum, not the versus

Put the options on one axis and the "versus" dissolves:

The automation spectrum

1

Plain code

record checks, date math, dedupe ledgers

cost · free at runtime

predictable · total

2

Rules & patterns

regex on known phrases, thresholds, schedules

cost · free at runtime

predictable · total — within the rules you wrote

3

Small model calls

JSON classifiers, temperature 0, confidence floors

cost · cents, adds latency

predictable · high with a schema and a floor

4

Open-ended generation

chat answers, drafts, summaries

cost · most expensive, slowest

predictable · lowest — needs grounding and fallbacks

Work left to right. Every step you solve on the left is a step the model never gets a chance to get wrong — or to bill you for.

Plain code answers questions with exact answers: does this record exist, has the deadline passed, did we already process this. Rules and patterns encode human knowledge about inputs — regex on known phrases, thresholds, schedules. Still free, still predictable, now covering fuzzy-looking cases whose vocabulary is actually stable. Small model calls buy judgment on genuinely unbounded language: classification, extraction, scoring — constrained with schemas, temperature 0, and confidence floors. Open-ended generation writes text for humans, the most capable and least predictable stage, needing everything in the hallucination guide wrapped around it.

The engineering rule that follows: work left to right, and stop as early as the question allows. Every case resolved on the left is a case the model can never get wrong, never bill you for, and never slow down.

How you should design a tiered pipeline

Step 1: Inventory the deterministic facts

Before any model enters the design, list what your system already knows in records: timestamps, ownership, state flags, history. Most "AI automation" proposals contain a large deterministic core someone is about to re-ask a model. If the event already ended, if the sender is the account owner, if this exact action already ran today — those are database reads, and they should gate the pipeline: any miss ends processing before a single token is spent.

Step 2: Write down the patterns you are sure about

Every domain has vocabulary that means one thing: refund words, cancellation words, your product's own feature names. Encode those as patterns and treat a hit as high confidence — no model confirmation needed. Two properties make this tier precious: it costs nothing at runtime and it fails readably — when a pattern misfires you can point at the characters that did it, which is a luxury no model output offers.

The common objection — "regex can't understand language" — is true and misses the point. The regex is not there to understand language; it is there to remove the understood cases from the model's workload, so the model's error budget is spent only on genuine ambiguity.

Step 3: Constrain the model call that handles the remainder

For inputs the patterns miss, a small model call earns its cost — if you treat it like an unreliable microservice:

  • One narrow question, not an open conversation: "does this message express X?" — answered as JSON only, { match: boolean, confidence: 0-1 }.
  • Temperature 0 and a parse-or-reject contract — scan for the JSON object; anything unparseable counts as no.
  • A confidence floor — accept a match only above a threshold you chose (0.7–0.8 is a sane start).
  • A fallback provider if the first errors, and after both: fail closed. A model outage must degrade to "patterns only," never to "alert on everything" or "alert on nothing the patterns knew."

Step 4: Keep actions on the infrastructure you already trust

Whatever the pipeline decides, the resulting action — notify, enqueue, write — should ride your existing machinery: the task queue, the notification system with its per-user preferences, the audit logs. And deduplicate at the action with a transactional claim (one action per entity per window), so retries and parallel workers cannot double-fire. None of that is AI; all of it is what makes the AI usable.

How I implemented it on a real project

The events platform I work on alerts organizers when a ticket-holder's message suggests they want a refund or will not attend — early warning that prevents payment disputes. It is an "AI reads the inbox" feature on the surface; underneath it is four tiers in exactly the order above.

Cheap checks in front of every model call

  1. 1Record checks firstEvent exists, hasn’t started, sender isn’t the host, the guest holds an active unrefunded ticket. Any miss ends it — no model call, no alert.
  2. 2Tier 1: regexThe wording we’re sure about — refund, money back, can’t attend plus money words. A hit is treated as high confidence and skips the model.
  3. 3Tier 2: classifier, only on a missA small JSON-only call, temperature 0. One hosted model first, a second as backup. It counts only above a 0.75 confidence floor; any failure means not flagged.
  4. 4Claim the 24-hour ledgerA transaction claims one alert per guest per event per day. A parallel worker that loses the claim stops quietly.
  5. 5Alert the host on existing channelsWhatsApp, then SMS, then email — the notification system the product already had. Dispute detection didn’t grow its own sender.
The model is the last resort, not the front door. A provider outage changes tier 2 only — the regex tier and the record checks keep working.

Tier 0: record checks gate everything

The evaluator returns early, with a logged reason, when the input is incomplete, the event does not exist or has already started, the sender is the host, the guest holds no active unrefunded ticket, or an alert for this guest on this event already went out in the past 24 hours. Note what never happens in those cases: no model call, no alert, no cost. A classifier that says "refund!" about a message from the host, or about yesterday's event, would not be smarter automation — it would be an extra bill you then have to ignore in code anyway. Ignore it before paying.

Tier 1: the patterns we were sure about

Two regex families: refund-or-cancel wording (refund, money back, reimburse, cancel my ticket) and non-attendance wording (can't make it, flight cancelled, got sick). The intent rule is stricter than the chat router that shares these patterns: a refund phrase alone counts, but a non-attendance phrase counts only when the same message also mentions money, tickets, or paying — "I can't attend" alone is life; "I can't attend, can I get my money back" is a dispute signal. A hit is treated as high confidence and skips the model entirely.

Tier 2: a JSON classifier, only on misses

Messages that pass the record gates but miss the patterns get one narrow question: does this guest message express inability to attend or a refund/cancellation request? JSON only — a boolean and a confidence. Gemini first at temperature 0 with a short output cap; OpenAI as backup if the key is missing or the call throws. The reply is scanned for a JSON object; a match counts only when the boolean is true and confidence ≥ 0.75. Parse failure, both providers down, low confidence — all resolve to not flagged. The regex tier keeps working through any outage, so the phrases we were sure about still alert.

Tier 3: dedupe and the boring action layer

Before notifying, a transaction claims a ledger document for that guest-event pair; a parallel worker that loses the claim stops silently. The notification then rides the platform's existing registry — WhatsApp, then SMS, then email, per the host's existing preferences. Dispute detection did not grow its own sender, and the chat reply never waits on any of this: the support bot enqueues the evaluation in the background after it has already answered (that agent's own design is here).

Worth naming the contrast: the platform also runs ordinary scheduled marketing automation — workflow tools and email sequences. Nobody calls it AI, it never touches a model, and it is not worse for that. The dispute detector is what AI automation should look like: traditional automation with one carefully fenced model call where language genuinely is unbounded.

Pitfalls to avoid

  • Model-first design. If the model is your first filter, you pay latency and tokens to rediscover facts your database knew.
  • Fail-open classifiers. An outage that pages every customer — or suppresses the alerts your patterns already understood — is a design choice. Choose closed.
  • Unversioned prompts and thresholds. The classifier prompt and the 0.75 floor are configuration; review changes like code.
  • Building a parallel action system. New AI feature, new notification path = two places preferences and audit live. Reuse the registry you have.
  • No early-exit logging. Record why each input stopped (which gate, which tier). It is your only debugging tool when someone asks "why didn't this alert?"

FAQ

Cost, latency, and failure coupling. The patterns handle the common certain cases for free and keep working during outages. The model earns its keep only on inputs the patterns cannot resolve.

  • #Automation
  • #AIWorkflows
  • #Classifiers
  • #Guide
  • #CaseStudy
Rushit Jivani headshot

Written by

Rushit Jivani

Senior Software Architect & AI Application Engineer. Architecture that shows up in crash rate, performance and how fast the team ships.