Evidens
Collection of TJRJ appellate rulings and LLM semantic analysis, with every claim backed by a verified verbatim quote
- Python
- Playwright
- BeautifulSoup
- PostgreSQL
- psycopg 3
- PyMuPDF
- Codex CLI
- Flask
- React
- TypeScript
- Vite
- Recharts
- Docker
- pytest
Evidens is a Python pipeline that collects appellate rulings (acórdãos) from the Rio de Janeiro State Court of Justice (TJRJ), extracts the text from the PDFs and has a language model read each full decision, backing every claim with a verbatim quote. I built it to help a friend with her law school thesis, which studies how the court justifies accepting, rejecting or downplaying expert evidence in appeals over alleged medical malpractice.
For those outside the field: an acórdão is the decision a panel of appellate judges issues when ruling on an appeal, and expert evidence (perícia) is the technical examination carried out by a specialist appointed by the judge, recorded in a report. The code is private and no case data appears here on purpose; what follows is how I solved each part.

Overview
Each stage is a command in a custom CLI: collect, download, extract, screen for scope, analyze, draw the blind audit sample and generate the report. PostgreSQL 16 stores everything in three schemas: raw with the evidence as collected, core with the unique documents and analysis with the model's records and provenance. It is plain SQL with psycopg 3, in 13 idempotent migrations.

Auditable collection from a site not built for it
The court's search is an ASP.NET form. Playwright checks "Acórdão", unchecks "Decisão Monocrática" (single-judge decision) and picks the year; since the site sometimes resets the year after a postback, the collector double-checks and retries up to three times. Each search is a record with state and can resume from the last saved page; if the total reported by the site changes midway, the search is aborted, because pagination can no longer be trusted.
The HTML of each page is saved immutably, addressed by its SHA-256, with atomic writes. The BeautifulSoup parser raises an explicit error when the layout changes instead of returning empty fields. PDFs are fetched over HTTP with a host allow-list, mandatory HTTPS, a 50 MB limit and a body that must start with %PDF-. Four threads download bytes; every write to disk and to the database happens on the main thread.

Never fetching the same thing twice
Deduplication uses two keys: the document's official identifier, with a UNIQUE constraint, and the file's SHA-256. The same ruling found by different queries becomes a single document, but every result from every search is preserved, and the upsert with COALESCE never overwrites metadata that is already known. Sealed cases and results without a PDF are recorded with the reason and never classified. The repair-pdfs command reapplies the current parser to the saved HTML, offline, and only goes back to the site for whatever is still missing a PDF.
Extracting text without making things up
PyMuPDF extracts the text page by page, and a state machine separates the headnote, the case report, the opinion and the operative part. If the PDF lacks the headnote, the one from the search result is used, flagged with its own method. A PDF with too little text becomes ocr_needed; I did not implement OCR, and the program does not fill in missing text.
Having a model read 1,048 rulings and prove what it read
The whole decision, with an output contract
The model is GPT-6 Luna with high reasoning effort, called through Codex CLI in a stateless subprocess per document, in a read-only sandbox. The input is the entire ruling, with no chunking or embeddings; above 100 pages or 180,000 characters, the system abstains and records the reason. The versioned instructions treat the page text as untrusted data, forbid concluding that an argument was never made just because the ruling does not mention it and require abstaining when the text does not support the assessment.
The output follows a closed JSON Schema, revalidated in Python: eligibility, defendant side (who the defendant is: a public entity, a private one or both), treatment of the expert report, up to 4 appeals and up to 20 pieces of evidence. Consistency rules in code prevent the summary from contradicting the analysis of each appeal.
A verbatim quote or nothing
Each quote must appear on the cited page. Verification has three steps: exact match, then with whitespace and line breaks collapsed, then ignoring whitespace entirely. There is no fuzzy matching and no ellipses, and the stored text becomes the exact excerpt from the source, with offsets. On failure, there is a single retry to fix only the evidence; if the model changes the classification in it, the result is rejected. A second failure is recorded, along with both responses.

Each record stores the hash of the prompt, the evaluator code, the input, the PDF and the raw response. Resuming a run is refused if the model, prompt or code has changed.

Scaling with disjoint workers
The cohort is frozen and split deterministically by text length, with the largest document going to the least loaded worker. A hashed manifest per worker, an assignment table and a PostgreSQL advisory lock per case guarantee that no document is read twice. The final campaign ran in 18 runs of 58 to 59 documents, in about 2 h 36 min. Before that, a cheap regex finds candidates and the model only reads where it is needed: when screening dentistry cases, it discarded 29 of the 90 candidates.
The numbers
- 1,499 raw results, 1,453 distinct documents and 1,412 extracted PDFs (15,635 pages).
- 1,077 main rulings with text and 1,048 analyzed: 1,034 records and 14 abstentions.
- 518 likely included, 486 excluded and 44 uncertain.
- 7,426 verified quotes; 330 records needed a second attempt.

The public dashboard, in React 19 with TypeScript and Recharts, shows the 518 cases. It is read-only, never exposes the model's raw response and does not host PDFs: the link points to the court. In the images on this page, case numbers, names and excerpts from the decisions have been masked.

What did not work
The first version was a regex classifier. In a pilot with 100 randomly drawn rulings, I compared the rules against an earlier model: agreement was 47.9% on eligibility and 28.7% on defendant side. That measures agreement, not accuracy, but it was enough to see that rules were not capturing the legal reasoning. I retired the rules as a classifier and the pipeline switched to using only the model, with validated quotes.
Limitations
The human audit has its infrastructure in place, with a stratified blind sample and a Flask interface that hides the model's labels until the review is done, but human validation has not been completed. The model's classification is preliminary.

The study is exploratory: it does not measure how often the court behaves a certain way, does not assess whether the decision was right and does not infer the judges' intent. A ruling's silence does not prove an argument was absent, and the corpus is whatever three queries retrieved.
Further reading
I wrote a post about the analysis method: AI for semantic analysis of large datasets: how to have a model read a thousand rulings and prove what it read.
// screenshots