Skip to content

BA Playbook

The Batch Ran Successfully. So Why Is the Data Missing?

A green status proves the job finished. It doesn't prove the business outcome was correct.

Surya · August 13, 2026 · 5 min read · 9 practices

Data

For Business Analysts and Ops teams investigating a batch that reports SUCCESS but is missing data, BAs supporting reconciliation, data quality or control functions, QAs and developers trying to isolate which pipeline stage actually dropped records and Anyone who's had to explain "the job is green" to someone holding the wrong number.

It's 8:30 AM. Operations opens the daily report. Yesterday: 48,216 records. Today: 46,193 records. More than 2,000 records are missing. Someone checks the overnight batch. Status: SUCCESS. The developer says the job completed successfully. Operations asks where the data is. Both can be right — a successful job proves the process finished. It does not prove the business outcome was correct. Picture a bank processing yesterday's trades overnight: Trade Source → Extract → Transform → Validate → Load → Reconcile → Report. The batch normally handles about 50,000 trades. Today the scheduler is green — no obvious technical failure — but the report is missing 2,000 trades. Don't start by staring at the final report. Follow the data. The fix isn't arguing about whether the batch "worked." It's tracing the record count through every stage until you find the exact checkpoint where expected stops matching actual. That's what turns "some data is missing" into "2,000 expected records disappeared between Transform and Validate" — a problem someone can actually go fix.

Infographic summarizing the missing-data batch investigation framework: the golden rule to trust the data, follow the flow and prove the outcome; five common causes (bad or missing source data, filter or criteria issues, transformation logic issues, load or partial-load issues, and validation gaps); the seven-stage data flow from Source through Extract, Transform, Validate, Load, Reconcile to Consume; six common root-cause categories (data, filters, logic, load, validation, environment); an eight-point checklist to run before closing the issue; and the takeaway that a good BA validates the business outcome, not just the job status.
The whole framework on one page — from a green scheduler to a proven business outcome.
01

Checkpoint 01

SOURCE — Did the data exist?

Before blaming the batch, check whether the source ever had the missing records to begin with.

If the source contained only 48,000 records, the problem may be upstream, before your pipeline ever touched the data. If the source had 50,000 but only 48,000 were extracted, you've already narrowed the problem to something inside your own process.

At a glance

  • How many records were expected?
  • How many were actually available?
  • Did every source system deliver?
  • Did anything arrive late?
  • Were records incomplete, duplicated or malformed?
  • Was the extraction window correct?

Why it helps

Counts turn "some data is missing" into "2,000 expected records disappeared between Source and Extract" — a much easier thing to investigate.

02

Checkpoint 02

EXTRACT — Did we select everything we should?

A batch can run successfully while selecting the wrong population entirely.

The key question is: what selection criteria were actually used? Then compare Expected → Selected → Rejected.

At a glance

  • The query says trade_date = yesterday, but some trades arrived after midnight.
  • A filter excludes a newly introduced status.
  • Extraction uses created_date while the business expects trade_date.

Why it helps

A green extract can still be incomplete — the job succeeded at running the wrong query.

03

Checkpoint 03

TRANSFORM — Did business logic remove records?

This is where data gets mapped, enriched, joined, aggregated, converted, deduplicated and classified — and where records quietly leave the population.

The 1,800-record gap

50,000 entered transformation.
48,200 left transformation.

At a glance

  • A join requires reference data that doesn't exist for a new product.
  • An unmapped currency drops records.
  • Deduplication removes valid trades, not just duplicate ones.

Why it helps

Don't just ask whether transformation succeeded. Ask what rule caused these specific records to leave the expected population.

04

Checkpoint 04

VALIDATE — Did something fail quietly?

Most pipelines validate before loading — but what happens when a record fails validation matters as much as the validation itself.

Does the entire batch stop on a failure, or does it reject the record and continue? A job can show SUCCESS while thousands of records sit in a rejection table, an error queue, an exception file or a dead-letter queue.

At a glance

  • Account exists?
  • Mandatory fields populated?
  • Currency valid?
  • Status allowed?
  • Reference data available?

Why it helps

Don't ask only whether validation ran. Ask how many records passed, failed and were skipped.

05

Checkpoint 05

LOAD — Did everything actually land?

Records passing validation is not the same as records reaching the target.

The flow

InputInsertedUpdatedRejectedCommitted

At a glance

  • Constraint failures
  • Duplicate-key errors
  • Permission issues
  • Storage problems
  • Transaction rollbacks
  • Partial commits

Why it helps

"Load completed" is not enough on its own — you need the numbers at every one of these five states.

06

Checkpoint 06

RECONCILE — Find the first gap

This is usually the fastest way to locate the issue, and it's often skipped in favour of guessing.

Take the bank's overnight batch: Source 50,000, Extracted 50,000, Transformed 48,200, Validated 48,200, Loaded 48,200, Reported 48,200. The first gap appears during transformation — a new instrument type was introduced yesterday, the transformation joins every trade to a reference-data table, and that new instrument type has no reference mapping. The join drops those records. The batch still completes. Technically SUCCESS. Business outcome: 2,000 trades missing.

Why it helps

Without reconciliation, a team can spend hours checking the scheduler, the database and the report. With it, the rule is simple: find the first checkpoint where expected ≠ actual. That's usually where the investigation should begin.

Pro tip — You can now explain the incident precisely: new instrument → missing reference mapping → join drops records → downstream population incomplete. That's far more useful than "batch issue."

07

Checkpoint 07

CONSUME — Is the data missing, or just invisible?

Sometimes the data reached the target just fine. The user simply can't see it.

At a glance

  • Report filters are wrong
  • The dashboard cache is stale
  • Reporting dates differ from processing dates
  • Permissions hide records
  • The downstream extract hasn't refreshed
  • The report reads a different table or view

Why it helps

"Data missing from the pipeline" and "data missing from what the user sees" are different problems that need different fixes.

08

Checkpoint 08

What does SUCCESS actually mean?

This is the question that often changes the entire investigation.

Compare

Technical SUCCESS

Process started → no fatal exception → process ended.

Business SUCCESS

All expected records processed → exceptions identified → totals reconciled → downstream data available.

Why it helps

The developer and Operations aren't disagreeing about facts — they're using different definitions of the same word. One of the BA's jobs is turning technical success into measurable business success. If SUCCESS can mean the first definition while an important outcome is wrong, the success criteria itself is incomplete.

09

Checkpoint 09

Turn the incident into a better requirement

Fixing the issue is only half the job. Ask how this gets detected automatically tomorrow.

For example: "Alert Operations if loaded record count differs from extracted count by more than 0.5%."

At a glance

  • Source-vs-target reconciliation
  • Rejected-record counts
  • Control totals
  • Tolerance thresholds
  • Missing-data alerts
  • Exception monitoring

Why it helps

You didn't just help fix yesterday's problem. You reduced the chance of tomorrow's problem going unnoticed.

Use the closing checklist before you close the issue

A green status proves the process finished.

It doesn't prove the business outcome was correct.

Next time a batch reports SUCCESS but the numbers don't add up, don't stop at the scheduler. Trace the count through Source, Extract, Transform, Validate, Load, Reconcile and Consume, and find the first checkpoint where expected stops matching actual. That's where the investigation actually starts — and reconciling counts at every checkpoint is how you make sure it doesn't happen quietly again.

Take this with you

Missing Data / Batch Investigation Checklist

MISSING DATA / BATCH INVESTIGATION CHECKLIST

Batch / job name:
Expected record count:
Actual record count:
Gap:

CHECKPOINT COUNTS
Source:
Extract:
Transform:
Validate:
Load:
Report:

First checkpoint where expected ≠ actual:
Root cause type (source / selection / transformation / validation / load / consumption):

[ ] Root cause identified and explained
[ ] Counts reconcile at every checkpoint
[ ] Rejected or skipped records are understood
[ ] Missing records are restored or accounted for
[ ] Downstream output is verified
[ ] Monitoring or reconciliation controls added where needed
[ ] Stakeholders understand what happened
[ ] The learning is documented

Get new playbooks first.