SOLUTIONS

WHO WE SERVE

RESOURCES

ABOUT US

GET STARTED

SOLUTIONS

WHO WE SERVE

RESOURCES

ABOUT US

GET STARTED

MENU

On this page

At a galnce

Three routes

The benchmark

Three examples

Accurracy

Repeatablility & speed

Cost

Completeness

Which should you use?

The questions

Part 2

RESEARCH

Prepared data vs. raw filings:
an ABS benchmark for AI agents
(Part 1)

Prepared data vs. raw filings: an ABS benchmark for AI agents
(Part 1)

Prepared data vs. raw filings:
an ABS benchmark for AI agents
(Part 1)

Research team, Elimentary

•

Oct 19, 2026

•

10 min read

There are three common ways to point an AI agent at credit data. Let it search the web. Let it read the source filings when you ask. Or prepare the data first and let it query that. All three work some of the time. They fail in very different ways, and that is what matters when the number goes into a model or a report.

We tested all three on the same 142 questions about the 36 SEC-registered ABS deals priced in Q3 2026. Same model, same instructions. Only the route to the data changed. This is Part 1. Part 2 will test harder, multi-step work (see the end of this article).

There are three common ways to point an AI agent at credit data. Let it search the web. Let it read the source filings when you ask. Or prepare the data first and let it query that. All three work some of the time. They fail in very different ways, and that is what matters when the number goes into a model or a report.

We tested all three on the same 142 questions about the 36 SEC-registered ABS deals priced in Q3 2026. Same model, same instructions. Only the route to the data changed. This is Part 1. Part 2 will test harder, multi-step work (see the end of this article).

What we measured, at a glance

We scored each route on six things. The rest of the article shows how each number was worked out.

We scored each route on six things. The rest of the article shows how each number was worked out.

Dimension

What it tells you

Data layer

Raw filings

Web only

Accuracy

Share of checked answers that were right (251 per route)

100.0%

90.4%

82.5%

Completeness

Share of required facts stated in open-ended analyst answers

92.0%

87.0%

74.5%

Confident errors

Answers that gave a wrong figure as if it were right

0

15

24

Repeatability

Questions answered right on every try (of 106)

106

96

82

Speed

Median time per answer

8.5 s

16.9 s

45.3 s

Cost

Model spend per right answer

$0.029

$0.113

$0.182

Same model for all three routes, Claude Sonnet 5.5. 95% range on accuracy: 98.5 to 100, 86.2 to 93.5 and 77.3 to 86.7.

  • Raw-document failures look like answers. Most misses were a confident wrong figure, not "not found".

  • Structured data allows the use of a smaller model. A smaller, cheaper model on the data layer scored about the same as a larger model on raw filings, at about a third of the cost per right answer.

  • Our recommendation: prepared data for any number that goes into a report, model or covenant test. Raw filings are fine for one-off lookups of printed facts.

  • Raw-document failures look like answers. Most misses were a confident wrong figure, not "not found".

  • Structured data allows the use of a smaller model. A smaller, cheaper model on the data layer scored about the same as a larger model on raw filings, at about a third of the cost per right answer.

  • Our recommendation: prepared data for any number that goes into a report, model or covenant test. Raw filings are fine for one-off lookups of printed facts.

How the three routes work

The difference is when the reading happens.

The difference is when the reading happens.

Web search

Reads at question time

Raw filings

Reads at question time, every time

Prepared data layer

Reads once, before any question

  1. Search the open web

  2. Open what ranks

  3. Build deal lists by hand

  1. Search EDGAR

  2. Open filings 8,000 characters at a time

  3. Pick the figure from the text

  1. Query loan files stored as tables

  2. Pull deal terms stored as values

  3. Show the filing sentence behind each value

With raw filings, the agent pays to read again on every question. A 352 MB loan file can only be read a slice at a time, so its tens of thousands of loans can't be added up, and a figure 200 pages in is easy to miss. With a data layer, that work is done once. A question about 74,000 loans becomes one query. The full text of every filing stays in the layer too, for answers that only live in long paragraphs.

With raw filings, the agent pays to read again on every question. A 352 MB loan file can only be read a slice at a time, so its tens of thousands of loans can't be added up, and a figure 200 pages in is easy to miss. With a data layer, that work is done once. A question about 74,000 loans becomes one query. The full text of every filing stays in the layer too, for answers that only live in long paragraphs.

The benchmark

Item

Detail

Deals

36 SEC-registered ABS deals with final prospectus filed July to September 2026: auto loans and leases, CMBS, credit cards, equipment, utility recovery bonds

Checked questions

106, each with one right answer checked against the filings. Marked by script, no model grader.

Analyst questions

36 open-ended, with 384 required facts, each backed by a filing quote. 20 written by a credit analyst we work with, 16 by us.

Grading of analyst answers

Claude Opus 5.5 marks each fact, without knowing which route answered

Tries

2 to 3 per checked question (251 answers per route), 2 to 4 per analyst question (112)

Model

Claude Sonnet 5.5 for all three routes, same instructions

One caveat: the raw-filings agent worked the way a person on EDGAR would.

  • What it could do: run EDGAR's full-text search, list a company's filings, and read documents a slice at a time.

  • What it could not do: run code.

  • Why that matters: loan-level questions need thousands of loan records added up, which reading alone can't do.

  • What's next: a raw setup that can run code on the loan files would likely do better there. We plan to test that in Part 2.

One caveat: the raw-filings agent worked the way a person on EDGAR would.

  • What it could do: run EDGAR's full-text search, list a company's filings, and read documents a slice at a time.

  • What it could not do: run code.

  • Why that matters: loan-level questions need thousands of loan records added up, which reading alone can't do.

  • What's next: a raw setup that can run code on the loan files would likely do better there. We plan to test that in Part 2.

Three examples

What the difference looks like on single questions.

What the difference looks like on single questions.

Three questions, side by side.

The same question, put to each route.

Three questions, side by side.
The same question, put to each route.

What it tells you

What it tells you

What it tells you

The question

Balance-weighted average PTI ratio of Capital One Prime 2026-1, worked out from its loan file

Weighted average FICO in the Exeter 2026-4 prospectus. The summary says 585; the pool table says 566.

Which investors bought the Hyundai 2026-C Class A-3 notes? No filing says.

Data layer

CORRECT

The loan file is stored as a table. One query, under 10 seconds.

Correct: 566

FICO is stored as its own value, linked to the pool table.

Correctly declined

Stated the filings do not contain this, on all 14 such questions.

Raw filings

No result

Files up to 352 MB, read a slice at a time. Returned no figure after about a minute.

Incorrect: 585

Used the summary figure and did not reach the pool table.

Correctly declined

On every try, at a higher cost per answer.

Web only

No result

Cannot open loan files from search. 34.8% accuracy on loan-level questions overall.

Incorrect: 585

Used the same summary figure.

Declined on 93.3% of tries

On other tries it filled the gap with web content.

With raw filings, the agent pays to read again on every question. A 352 MB loan file can only be read a slice at a time, so its tens of thousands of loans can't be added up, and a figure 200 pages in is easy to miss. With a data layer, that work is done once. A question about 74,000 loans becomes one query. The full text of every filing stays in the layer too, for answers that only live in long paragraphs.

With raw filings, the agent pays to read again on every question. A 352 MB loan file can only be read a slice at a time, so its tens of thousands of loans can't be added up, and a figure 200 pages in is easy to miss. With a data layer, that work is done once. A question about 74,000 loans becomes one query. The full text of every filing stays in the layer too, for answers that only live in long paragraphs.

Accuracy: when an agent got it wrong, what went wrong exactly?

We opened the log behind every one of the 68 wrong answers and read what the agent did.

We opened the log behind every one of the 68 wrong answers and read what the agent did.

This is the distinction that matters in production. "Not found" costs an analyst a few minutes. A wrong figure stated with confidence can end up in a model, a memo or an investor report with no one noticing.

The misses come down to two things.

The agent never reached the data (11 raw-filings misses, 28 web). Loan files were too big to open. Searches missed the filing. Lists of "every deal this quarter" left some out, which happened nine times on the web route alone.

The agent reached the data but took the wrong number (13 raw-filings misses, 16 web). It used a similar measure, such as loans 31 or more days late when the question asked about 30. It took a figure from a different report. A few times it named the underwriters as the buyers, missed a value that was in the filing, or used a non-SEC site.

The data layer avoids both by design. Loan files are tables it can query, every deal is a row, and each stored value points to the sentence it came from. One reader assigned these causes, so treat the counts as a guide.

This is the distinction that matters in production. "Not found" costs an analyst a few minutes. A wrong figure stated with confidence can end up in a model, a memo or an investor report with no one noticing.

The misses come down to two things.

The agent never reached the data (11 raw-filings misses, 28 web). Loan files were too big to open. Searches missed the filing. Lists of "every deal this quarter" left some out, which happened nine times on the web route alone.

The agent reached the data but took the wrong number (13 raw-filings misses, 16 web). It used a similar measure, such as loans 31 or more days late when the question asked about 30. It took a figure from a different report. A few times it named the underwriters as the buyers, missed a value that was in the filing, or used a non-SEC site.

The data layer avoids both by design. Loan files are tables it can query, every deal is a row, and each stored value points to the sentence it came from. One reader assigned these causes, so treat the counts as a guide.

Repeatability and speed: same answer every time, and how fast?

The web agent flipped between right and wrong on 13 questions, so a single web answer is hard to rely on. The data layer was right on every try for all 106.

The web agent flipped between right and wrong on 13 questions, so a single web answer is hard to rely on. The data layer was right on every try for all 106.

Cost: does the model matter? (Less than you'd think)

With a data layer, the agent no longer has to find a filing and pick the right number. So we tried a much cheaper model, Claude Haiku 4.5, on 75 of the checked questions.

With a data layer, the agent no longer has to find a filing and pick the right number. So we tried a much cheaper model, Claude Haiku 4.5, on 75 of the checked questions.

Route

Sonnet 5.5

Haiku 4.5

Haiku 95% range

Haiku cost per
right answer

Prepared data layer

100.0%

92.0%

83.6 to 96.3

$0.042

Raw SEC filings

87.8%

82.7%

72.6 to 89.6

$0.138

Web search only

79.0%

72.0%

61.0 to 80.9

$0.262

Same 75 questions. Sonnet asked 2 to 3 times each, Haiku once. Raw filings with Sonnet cost $0.128 per right answer on this subset.

  • Every route got worse with the smaller model, but the order held.

  • Haiku on the data layer (92.0%) scored above Sonnet on raw filings (87.8%) at a third of the cost. The ranges overlap, so the fair reading is "about the same accuracy, much cheaper".

  • Every route got worse with the smaller model, but the order held.

  • Haiku on the data layer (92.0%) scored above Sonnet on raw filings (87.8%) at a third of the cost. The ranges overlap, so the fair reading is "about the same accuracy, much cheaper".

Completeness: what about open-ended analyst questions?

These are the questions an analyst actually gets asked. Each one needs several documents, sometimes several deals, and a lot of reading. There is no single right number, so for each question we listed the facts a complete answer must include, each backed by a quote from a filing. The grader checks which of those facts the answer states. Three examples:

These are the questions an analyst actually gets asked. Each one needs several documents, sometimes several deals, and a lot of reading. There is no single right number, so for each question we listed the facts a complete answer must include, each backed by a quote from a filing. The grader checks which of those facts the answer states. Three examples:

Question

What a complete answer must cover

Why it's hard

Across Carvana 2026-P3, CarMax 2026-3 and Exeter 2026-4, how do the payment waterfalls differ, especially in allocating losses and redirecting cash to senior noteholders?

The payment order for each deal, how losses hit each class, and what changes after a trigger

Three prospectuses, explained in long paragraphs, not tables

For Carvana's latest deal as of September 30, 2026, which performance triggers or reserve requirements are breached or closest to breach, and what happens if they are?

Each trigger level from the prospectus, the latest reported value from the monthly reports, and the consequence

Two kinds of documents joined together, across six months of reports

Compare BMO 2026-5C16 and BMO 2026-C15: loan count, pool balance, weighted DSCR and LTV, and largest property-type concentration.

Five figures for each of two CMBS deals

Figures spread across two long prospectuses

Results on all 36:

Route

Key facts stated

Statements a filing contradicts

Time for all 36

Cost for all 36

Prepared data layer

92.0%

36

17.5 min

$5.03

Raw SEC filings

87.0%

44

36.4 min

$10.69

Web search only

74.5%

61

72.6 min

$9.63

The data layer covered more of the required facts, made fewer statements a filing contradicts, and took half the time and half the cost of raw filings. That matters because this is hours of analyst reading per question. The accuracy lead is smaller here, five points, and with 112 answers per route it is less certain than the checked results. The data layer helped most on triggers, waterfalls and credit enhancement, which prospectuses explain in long paragraphs. Raw filings did better on a few, such as Carvana's note terms. Even the data layer made 36 statements a filing contradicts, so analyst answers still need review.

The data layer covered more of the required facts, made fewer statements a filing contradicts, and took half the time and half the cost of raw filings. That matters because this is hours of analyst reading per question. The accuracy lead is smaller here, five points, and with 112 answers per route it is less certain than the checked results. The data layer helped most on triggers, waterfalls and credit enhancement, which prospectuses explain in long paragraphs. Raw filings did better on a few, such as Carvana's note terms. Even the data layer made 36 statements a filing contradicts, so analyst answers still need review.

Which should you use?

It isn't one or the other. It depends on what the number is for.

It isn't one or the other. It depends on what the number is for.

Going into a report, model, covenant test or LP letter?

Prepared data layer, with a source link for every number

Needs loan-level math or every deal in a set?

Prepared data layer. This is where raw reading missed most.

A one-off lookup of a printed fact?

Raw filings work, at higher cost and time.

Open-ended read of how a deal works?

Data layer plus full text, then analyst review

Web search alone came last on every measure. We would not use it for deal data.

Web search alone came last on every measure. We would not use it for deal data.

The questions

The 142 questions cover these themes:

The 142 questions cover these themes:

Theme

What it asks for

Questions

Deal and tranche facts

Coupons, ratings, CUSIPs, pool size, parties, from one deal's prospectus

23

Monthly performance

Pool balance, losses, delinquencies, reserve balances from monthly reports

15

Contract terms

Trustees, clean-up calls, risk retention, events of default

12

Loan-level analysis

Figures worked out from loan files, such as weighted FICO or state mix

9

Across deals

Comparisons and totals over several or all Q3 deals

13

Two documents together

Prospectus terms checked against a monthly report

4

Robustness

Earlier questions re-worded, and questions no filing can answer

30

Analyst questions

Waterfalls, triggers, credit enhancement, servicing terms, deal-to-deal comparisons

36

For the exact list of questions, or any more detail on the method, reach out to contact@elimentary.com.

For the exact list of questions, or any more detail on the method, reach out to contact@elimentary.com.

What the next iteration will include (Part 2)

Part 1 shows the gap on questions with one right answer. The data layer scored 100% on them, so the next step is harder work. Part 2 is in progress and will be published here.

Part 1 shows the gap on questions with one right answer. The data layer scored 100% on them, so the next step is harder work. Part 2 is in progress and will be published here.

What Part 2 adds

Why it matters

Multi-step workflows

Real analyst tasks chain several steps, such as building a tape, testing triggers, then writing up the result

Answers that need complex derived logic

A harder question set, so a perfect score is no longer within easy reach

A semantic layer on top of the data

Tests whether defining business terms (delinquency buckets, pool factor) cuts errors further

Code execution for every route

A stronger raw-filings baseline that can parse loan files itself

Multiple models from different providers

Shows whether the results hold beyond one model family, for the agents and the grader

Questions written by outside analysts

Removes the home-field advantage of writing our own questions

Private credit documents

Public ABS loan files follow an SEC format. Credit agreements and fund documents don't.

Part 1 is also clear about one limit: on 251 answers, 100% still has a 95% range of 98.5% to 100%.

Part 1 is also clear about one limit: on 251 answers, 100% still has a 95% range of 98.5% to 100%.

Run it on your own data

Elimentary builds AI agents for the credit middle and back office on a prepared data layer like this one. Every number links to its source.

Elimentary builds AI agents for the credit middle and back office on a prepared data layer like this one. Every number links to its source.

BOOK A DEMO

LET'S TALK

Ready to see what Elimentary can do for your institution?

BOOK AN INTRO CALL

COMPANY

About Us

Case Studies

RESOURCES

Newsroom

Case Studies

Research

SOLUTIONS

Deal Diligence

Portfolio Monitoring

Regulatory Reporting

TEAMS

Private Credit Funds

Originators

Servicers

© 2026 Elimentary. All rights reserved.

Trust Centre

Privacy

Terms

LET'S TALK

Ready to see what Elimentary can do for your institution?

BOOK AN INTRO CALL

COMPANY

About Us

Case Studies

RESOURCES

Newsroom

Case Studies

Research

SOLUTIONS

Deal Diligence

Portfolio Monitoring

Regulatory Reporting

TEAMS

Private Credit Funds

Originators

Servicers

© 2026 Elimentary. All rights reserved.

Trust Centre

Privacy

Terms

LET'S TALK

Ready to see what Elimentary can do for your institution?

BOOK AN INTRO CALL

COMPANY

About Us

Case Studies

RESOURCES

Newsroom

Case Studies

Research

SOLUTIONS

Deal Diligence

Portfolio Monitoring

Regulatory Reporting

TEAMS

Private Credit Funds

Originators

Servicers

© 2026 Elimentary. All rights reserved.

Trust Centre

Privacy

Terms