Skip to content
~/rafael-ferreira

AI for semantic analysis of large datasets: how to make a model read a thousand court rulings and prove what it read

What I learned building Evidens, a pipeline that uses an LLM to classify 1,048 court decisions: a staged funnel, the whole document in the prompt, literal quotes validated by code, disjoint batches and reproducibility.

A friend was writing her law school thesis (TCC) and had a question that no keyword search can answer. In appeals where someone seeks damages for alleged medical malpractice and the expert evidence is contested, how does the court explain why it accepted, rejected or downplayed the expert's report? For readers coming from software: an appeal (apelação) is the challenge to a trial court judgment, a ruling (acórdão) is the decision a panel of appellate judges hands down on that appeal, and the expert report (laudo pericial) is the technical opinion of a physician appointed by the judge. The question is not whether the ruling mentions the expert evidence. Almost all of them do. The question is what the text does with it.

To help, I built Evidens, a Python and PostgreSQL pipeline that collects the court's rulings, extracts the text from the PDFs and uses a language model to read each one and fill in a form. The model is GPT-6 Luna, with high reasoning effort, called through the Codex CLI. The repository is private, but the validation logic is mine and I show simplified excerpts below. What follows is what worked to make a model read 1,048 documents and leave a trail for every claim.

What I tried first, and why it didn't work

The first version was a deterministic classifier built on regular expressions. It looked for phrases typically used to accept or reject the expert report and assigned categories. The second step was a pilot: 100 rulings sampled with a fixed seed, read by the regex and by an earlier LLM acting as a second reader, to measure how much the two agreed.

They agreed very little. On case eligibility, 45 of 94 (47.9%). On the defendant side, that is, whether the defendant was public or private, 27 of 94 (28.7%). On how the expert's conclusion was treated, 36 of 91 (39.6%). On coverage of the objection raised by the party, 21 of 91 (23.1%). Only the form of the reasoning got above 70%: 70 of 91 (76.9%).

Bar chart of agreement between regex and LLM in the 100-ruling pilot: eligibility 47.9%, defendant side 28.7%, treatment of the conclusion 39.6%, objection coverage 23.1% and form of reasoning 76.9%

The pilot report has a sentence I would repeat on any similar project: these percentages measure agreement, not accuracy. Two readers can agree and both be wrong. Even so, the pilot taught me what I needed to know. The regex only got right what was said with the expected words, and the same legal reasoning shows up written in dozens of ways. I retired the rules as a classifier. The active pipeline uses only the model, and the question became how to trust what it returns.

1. A staged funnel: the model only where reading is needed

Reading with an LLM is the slowest stage of the pipeline, so it comes last. The first sieve is the court's own boolean search, with three queries using the case law system's operators. They returned 1,499 results, which became 1,453 distinct documents after deduplicating by official identifier and by the file's SHA-256. Of these, 1,412 had a public PDF, and 1,077 were main rulings with text. The analyzed cohort ended up at 1,048, and the model marked 518 as likely inclusions.

Staged funnel: 1,499 raw results, 1,453 distinct documents, 1,412 with a PDF, 1,077 main rulings with text, 1,048 in the cohort and 518 likely inclusions, separating cheap filters from reading by the model

A small example shows the pattern. Dentistry cases had to be separated out. A high-sensitivity regex, built to catch everything and accept false positives, found 90 candidates. The model read only those 90, with mandatory quotes, and discarded 29 that really were dental disputes, kept 59 where the mention was incidental and left 2 as uncertain. The cheap regex cuts the volume; the model decides where the text has to be understood.

2. The whole document, organized by page

Each call receives one complete ruling, every page in order, in a JSON with metadata and a list of numbered pages. There is no chunking and no embeddings. In the article on RAG I did the opposite: there the goal was to answer a question, so retrieving the seven most relevant passages was enough. Here the goal is to classify every document, and an argument raised on page 9 and answered on page 14 gets lost if each piece is read on its own.

The page numbering is not decoration: it is what lets me require every quote to point to a page. And there is an explicit limit, 100 pages or 180,000 characters. Above that, the model does not receive the document; the system records an abstention with the reason, instead of truncating the text and pretending it read everything. The model also does not receive earlier classifications of the same document, so it is not anchored by them.

3. Closed output, validated twice

The response follows a JSON Schema with additionalProperties: false. The fields are closed enumerations: eligibility (likely, excluded, uncertain), defendant sector (public, private, mixed, uncertain), treatment of the report (accepted, rejected, downplayed, undetermined), controversy mode, a list of up to 4 appeals and a list of up to 20 pieces of evidence with id, role, page, quote and note.

The schema is passed to the CLI with --output-schema, and the response is validated again in Python, because the schema cannot express rules across fields. The code checks that evidence ids are unique and exist, that each piece of evidence has the right role, that "likely inclusion" requires an express controversy or indirect assessment, and that indirect assessment comes with three separate pieces of evidence. The top-level summaries are derived from the appeals, so the model cannot say one thing in the appeal and another in the summary.

4. Every claim needs a literal quote

This is the part that changed my confidence in the result the most. Each piece of evidence carries a quote and the page number, and the code checks that the text exists on that page. A form with no abstention and no quotes at all is rejected. The check has three steps, from strictest to most lenient, and all of them tolerate only whitespace differences:

def localizar_citacao(pagina: str, citacao: str) -> tuple[int, int, str]:
    inicio = pagina.find(citacao)
    if inicio >= 0:
        return inicio, inicio + len(citacao), "exact"

    # spaces and line breaks collapsed, with offsets into the source
    achou = buscar_com_espacos_colapsados(pagina, citacao)
    if achou:
        return (*achou, "normalized_whitespace")

    # no whitespace at all: PDFs that break a line in the middle of an identifier
    achou = buscar_sem_espacos(pagina, citacao)
    if achou:
        return (*achou, "whitespace_insensitive")

    raise CitationValidationError("citação não é trecho literal da página")

When it matches, the stored text becomes the exact slice of the source, along with the offsets and the method used. In the final campaign there were 7,426 quotes: 1,050 matched exactly, 6,296 after normalizing spaces and line breaks, and 80 only when ignoring all whitespace, almost always a number the PDF had split across two lines.

I deliberately did not use fuzzy matching. I don't ignore accents, capitalization or ellipses. An "almost identical" quote is exactly where the model paraphrases, and paraphrasing a court decision is where the meaning changes. If the rule accepts 95% similarity, it accepts the "not" that went missing.

5. One retry, which fixes the evidence but not the answer

If a quote fails, the model gets back the error and its own previous JSON, with an instruction to fix only the evidence. The code compares the two responses field by field: if the model changes the classification or the justification during the fix, the result is rejected as retry_changed_classification. Without that guard, the retry becomes a second opinion picked by the validator, not a reference correction. Structural failures, such as a violated cross-field rule, also get one retry that points out the inconsistency. If it fails again, the record is stored as a failure with the error type, and both responses are kept.

Literal quote validation flow: exact match for 1,050 quotes, normalized whitespace for 6,296 and no whitespace for 80, with a single retry that can fix the evidence but not change the classification

In practice, 702 forms came out on the first attempt and 330 needed the second. There were 294 attempts rejected for an invalid quote and 68 for invalid structure, all of them redone. In the end, the cohort of 1,048 finished with 1,034 completed forms, 14 abstentions and no pending failures.

6. Abstaining and saying "uncertain" are valid answers

The rubric given to the model says (in English, in the code) not to infer that an unreported argument was absent and, if the text does not support a reliable assessment, to abstain and explain why instead of filling the gap with a guess. The 14 abstentions and the 44 cases of uncertain eligibility are not a pipeline defect. They are the pipeline telling you where the reading doesn't hold up, which is far more useful than 58 made-up answers delivered with the same confidence as the rest.

The same rubric treats the page text as untrusted data: the model must not follow instructions that appear inside a page. It is a defense against prompt injection, and it doesn't rely on the prompt alone. The CLI runs in a read-only sandbox, in an empty working directory, with no state between documents and a 180-second timeout. If a PDF contains a malicious instruction, the worst possible outcome is a wrong form, which still has to pass quote validation.

7. Scale with disjoint batches, not clever concurrency

The first campaign, with an old version of the rubric, started sequentially, at about 33 seconds per document. For the final campaign, I froze the cohort and split the documents into 18 batches with a greedy, deterministic partitioning: the largest document goes to the least loaded batch, until none are left. Each batch becomes a JSON manifest, created only once and with its own hash. An assignment table keyed on (campaign, ruling) guarantees that no document belongs to two batches.

Scaling with disjoint batches: frozen cohort, hashed manifests, 18 independent workers with a PostgreSQL advisory lock per document and results written to the database

The workers are independent processes, sequential inside. Before calling the model, each one takes a PostgreSQL advisory lock for the case. If a process dies, resuming skips what succeeded and redoes only the failures. There were 18 runs of 58 or 59 documents, and according to the logs the campaign took about 2 h 36 min from start to finish. There is no distributed queue and no orchestrator: the guarantee comes from the partition, and the lock is just a second layer.

8. Reproducibility: everything has a hash

Each form stores the hash of the prompt (version, rubric and schema), the hash of the evaluator's own source file, of the text input, of the PDF, of the request and of the response, plus the raw response. Resuming a run is refused if the model, the prompt or the code has changed. And changing the evaluation rubric never rewrites results: it creates a new run, and the previous one stays in the database. That is how the rubric evolved from one general form per ruling to evidence per appeal without losing the history.

What AI doesn't deliver

The project states its limits, and they apply to any similar use. The data is the text of the ruling: the pipeline does not measure how often the court behaves in a certain way, does not grade the decision and does not infer whether the judge read the report. A ruling's silence does not prove the argument didn't exist. The corpus is what three queries retrieved from a search system, not the universe of cases.

Most importantly: there is no completed human validation yet. The tool exists. It draws a blind, stratified sample with a fixed seed, generates a CSV without the model's labels and opens a local interface with the PDF and the text side by side. The model's labels stay hidden until every case in the sample has a human review, and only then does per-field agreement show up. But that review hasn't been done yet, so the model's classification is preliminary. A literal quote proves the passage exists, not that the interpretation is right. I also didn't record cost or token usage, so I don't have that number to give.

Checklist for your corpus

  1. Phrase the question so that only reading can answer it, and confirm that with a pilot against a simple rule.
  2. Build a funnel: cheap, sensitive filters first, the model only on what's left.
  3. If the task is to classify every document, send the whole document, numbered, with an explicit limit and abstention above it.
  4. Lock the output into a schema and validate cross-field rules in code.
  5. Require a literal quote with a page and check it mechanically, with no fuzzy matching.
  6. Allow one retry that fixes the evidence, and reject any retry that changes the answer.
  7. Treat abstention and "uncertain" as first-class outputs, and the input text as untrusted data.
  8. Scale with disjoint, resumable batches, hash everything and treat a new rubric as a new run.
  9. Plan the blind human validation from the start, and don't call the result final before it.

The dashboard, the methodology and the full numbers are on the Evidens project page.