Every Fix Leaves a Test Behind

On July 24th a bug report landed in the EdgarTools issue tracker about Coeur Mining's FY2025 10-K (accession 0000215466-26-000004). Asking for Item 7 — Management's Discussion and Analysis — returned 257,440 characters, roughly double the reporter's estimate of what the section should hold. The extraction had blown through Items 7A, 8, and 9A and kept going all the way into the director-signature block at the end of the document.

The reporter, Steven Fitzpatrick, hadn't stumbled on this by reading one filing. His team had pinned EdgarTools at an old version, drifted 295 commits behind, and built an audit that swept their stored corpus — 31,691 documents — for sections that looked structurally wrong. The sweep surfaced 277 candidates. Two days after that first report, his fix was merged and released to every user of the library.

A two-day turnaround on a parsing bug in a 10-K section extractor sounds like heroics. It isn't — it's infrastructure, and almost none of it is specific to SEC filings. If you maintain a library, a service, or a data pipeline, the machinery in this post is a pattern you can lift whole: a naming convention, one extra assertion per fix, and a recording of the network traffic.

Every fix leaves a test behind

Since September 2025, every bug fixed in EdgarTools has left behind a test named after its issue. They live in tests/issues/regression/, follow the pattern test_issue_NNN_description.py, and there are 206 of them as of this writing. The chart below shows the pace — a steady 11 to 18 per month, with a spike to 41 in June when I cleared a backlog across three subsystems.

What makes these tests useful is what they pin. A regression test here doesn't check that a value exists; it asserts the exact number from the real filing that triggered the report, verified by hand against SEC EDGAR. Here is the heart of the test for issue #452, where Krispy Kreme's fiscal-year-end change caused the wrong period's revenue to surface:

# Expected value from SEC filing: $1.686104B
expected = 1_686_104_000
assert abs(revenue_2023 - expected) < tolerance

# Also verify we're NOT getting the wrong value
wrong_value = 1_530_000_000  # The incorrect value from Jan 1, 2023 period
assert abs(revenue_2023 - wrong_value) > tolerance

That second assertion is the part worth stealing. Most test suites assert what should happen; almost none pin the specific wrong answer that already happened once. When a future refactor reintroduces the period-selection bug, the failure message names the issue, the company, and both values — an incident report instead of a mystery. The whole convention costs a directory and a file-naming rule, and it converts your issue tracker from a history of apologies into an archive you can execute.

The tests replay the SEC

Tests pinned to real filings have an obvious problem: they need the real filings. SEC EDGAR, all million-odd entities of it, is rate-limited to ten requests per second, occasionally slow, and — because filings get amended and reindexed — subtly alive. Running hundreds of tests against it on every push would be both rude and flaky.

The escape is a tool worth knowing if you haven't met it: vcrpy, used here through pytest-vcr. Mark a test with @pytest.mark.vcr and the first run records every HTTP interaction into a YAML file called a cassette; every later run intercepts the same requests and replays the recorded responses, so the test runs offline, deterministically, at disk speed. A cassette is not magic — it's a reviewable artifact that reads like this:

interactions:
- request:
    method: GET
    uri: https://www.sec.gov/Archives/edgar/data/215466/000021546626000004/0000215466-26-000004.txt
  response:
    status:
      code: 200
    body:
      string: "...the filing, byte for byte, as the SEC served it on recording day..."

The cassette recorded for the Coeur fix is 31 MB, because it contains the entire 10-K exactly as EDGAR returned it. There are 175 cassettes in the suite today, and the battery runs on a weekly schedule plus every push that touches the parsing subsystems.

If your tests depend on any rate-limited, changing third-party service — a payments sandbox, a geocoder, a government data feed — this is the same trade: record once against the live service, replay forever in CI. Your assertions stay pinned to reality without putting reality on your critical path. And for anyone downstream of you, it converts a familiar anxiety — we're 295 commits behind; is it safe to update? — into a command they can run. The battery is the changelog you can execute.

The bug report that does half the work

Steven's two issues (#904 for Coeur, #905 for a Freddie Mac 10-Q that scattered 165,000 characters of MD&A into a schema-impossible "Part I, Item 6") arrived in a shape I'd like to bottle. Each one had a specific accession number, the observed and expected values, a diagnosis pointing at the actual code paths involved, and a runnable repro script. When he asked whether that format was what I wanted, my answer was easy:

Both issues had what makes a filing bug actionable: a specific accession number, the observed and expected character counts, and enough of a mechanism to check. Keep writing them exactly like that.

The pull request that followed, #906, had the same discipline: one commit per mechanism, a before/after validation table in the description, and — because the contribution guidelines ask for it — its own regression tests and cassettes. Coeur's Item 7 went from 257,440 characters to a correctly bounded 91,682, and the tests pinning those numbers landed in the same PR that fixed them.

Notice that none of that shape is EdgarTools-specific. The accession number generalizes to the exact input; the observed and expected counts to both values; the repro script to itself. An issue that arrives in that shape is already halfway to being a regression test, because the test needs exactly the same three ingredients — which is a good argument for encoding the shape in your own issue template rather than hoping reporters discover it.

The battery cuts both ways

The part of this story I find most satisfying is that the battery caught a bug in the fix itself. During review, test_issue_891 — a regression test from an earlier, unrelated report involving Nathan's Famous — passed on main and failed on the branch. The looser header detection that repaired Coeur's 10-K had collapsed Nathan's correctly parsed section map, silently destroying its MD&A section. A bug fixed months ago, for a different company and a different reporter, stood guard against a new fix regressing it.

Steven shipped a targeted correction the same evening, and honesty compels me to complete the picture: during the second review round I discovered my own review table had asserted something false about an Exxon filing because I'd read across a row from the JPMorgan column. I shipped my own correcting commit twelve minutes after the merge. Reviews are part of the verification surface too, and they fail like everything else — the difference is whether the failure gets caught and recorded.

This is where the naming convention pays compound interest. Every test in the battery is a past incident standing guard over every future fix, and the more specific each one is, the more precisely it objects. A generic "sections parse without errors" test would have passed happily on both branches while Nathan's MD&A quietly vanished.

Why CI checks cassettes but never records them

Steven asked one genuinely hard process question: shouldn't CI generate the VCR cassettes, rather than trusting whatever a contributor recorded? My answer became the cassette policy in CONTRIBUTING:

Recording in CI would put live SEC on the critical path of every merge — flaky and rate-limited — and, worse, it would overwrite rather than check. I spot-checked Coeur Item 7 against the live filing and got 91,682, matching your recording exactly.

Contributors record cassettes from the live SEC against main, never edit them afterwards, and submit them as part of the PR, where they get reviewed and spot-checked like any other artifact. CI's job is to verify the recordings are safe and the tests still pass against them — a checker, never an author.

If you adopt cassettes, you will hit this exact question within your first month, and it deserves a deliberate answer rather than a default. The rule that generalizes: recordings are evidence, and evidence gets submitted and cross-examined — the moment your pipeline can regenerate its own fixtures, nobody is checking anything.

Agents on both sides

The LinkedIn-friendly framing of this story is "AI-assisted open source," and that's accurate as far as it goes. Steven's sweep pipeline and dense technical artifacts were drafted with an LLM; most of my fixes this year are pair-written with Claude; the June spike of 41 regression tests exists because agents make the fix-plus-test loop cheap enough to run dozens of times a month.

But the interesting part is where the humans stayed. Before automating anything, Steven asked whether I wanted more reports like these, and his sweep runs every candidate through a live-verification gate — re-extracting each suspect section against current code — before anything reaches me. That triage step, which filters out the already-fixed cases, is the most valuable part of the whole pipeline, and it's the piece to copy if you're pointing agents at a codebase — yours or anyone else's. Verifying one representative filing per ticker first is the difference between fixing 65 bugs and fixing 6 bugs with 65 confirmations, and the judgment about what deserves a human's attention is precisely the part that never got delegated to the machines.

The current state of that pipeline: of the original 277 candidates, everything the fixes could clear has cleared, leaving 99 findings across 91 filings, grouped into four root-cause buckets. Each one that proves real will become an issue, then a fix, then a permanent test.

Where the pattern goes next

Writing this post forced an audit of the process, and four improvements came out of it — each one the same discipline applied to its own next bottleneck. An issue template will encode the shape that made #904 and #905 actionable, so every reporter, human or agent, starts from it. A batch-intake convention will handle the 99-candidate backlog, because findings that arrive 99 at a time need different plumbing than findings that arrive one by one. A scaffolding script will stamp out the regression-test skeleton from an issue number, since at 15 to 40 tests a month the boilerplate cost is real. And cassettes will get an expiry report, because a recording of the SEC from last year is a fact about last year — recorded verification must expire, and it's time the tooling enforced it.

None of this needed to be invented for SEC data, and none of it depends on scale. A directory, a naming rule, one extra assertion per fix, a recording of the traffic: that's the entire battery, and it would bolt onto most existing test suites in an afternoon. The compound interest starts accruing with the first bug you fix afterwards.

The library treats filings as data objects rather than documents, if you have not met it. And if you use EdgarTools in production and find a filing that parses wrong, you now know exactly what to send: the accession number, what you observed, what you expected. It will get fixed, and the fix will still be guarded years from now — because around here, every fix leaves a test behind.