Skip to content
~/rafael-ferreira

Evidens

Collection of TJRJ appellate rulings and LLM semantic analysis, with every claim backed by a verified verbatim quote

private repository
  • Python
  • Playwright
  • BeautifulSoup
  • PostgreSQL
  • psycopg 3
  • PyMuPDF
  • Codex CLI
  • Flask
  • React
  • TypeScript
  • Vite
  • Recharts
  • Docker
  • pytest
Cover of the Evidens project

Evidens is a Python pipeline that collects appellate rulings (acórdãos) from the Rio de Janeiro State Court of Justice (TJRJ), extracts the text from the PDFs and has a language model read each full decision, backing every claim with a verbatim quote. I built it to help a friend with her law school thesis, which studies how the court justifies accepting, rejecting or downplaying expert evidence in appeals over alleged medical malpractice.

For those outside the field: an acórdão is the decision a panel of appellate judges issues when ruling on an appeal, and expert evidence (perícia) is the technical examination carried out by a specialist appointed by the judge, recorded in a report. The code is private and no case data appears here on purpose; what follows is how I solved each part.

Home screen of the Evidens dashboard, the expert evidence observatory

Overview

Each stage is a command in a custom CLI: collect, download, extract, screen for scope, analyze, draw the blind audit sample and generate the report. PostgreSQL 16 stores everything in three schemas: raw with the evidence as collected, core with the unique documents and analysis with the model's records and provenance. It is plain SQL with psycopg 3, in 13 idempotent migrations.

Pipeline diagram: collection, download, extraction, scope screening, model analysis, blind audit and dashboard, on top of the raw, core and analysis schemas

Auditable collection from a site not built for it

The court's search is an ASP.NET form. Playwright checks "Acórdão", unchecks "Decisão Monocrática" (single-judge decision) and picks the year; since the site sometimes resets the year after a postback, the collector double-checks and retries up to three times. Each search is a record with state and can resume from the last saved page; if the total reported by the site changes midway, the search is aborted, because pagination can no longer be trusted.

The HTML of each page is saved immutably, addressed by its SHA-256, with atomic writes. The BeautifulSoup parser raises an explicit error when the layout changes instead of returning empty fields. PDFs are fetched over HTTP with a host allow-list, mandatory HTTPS, a 50 MB limit and a body that must start with %PDF-. Four threads download bytes; every write to disk and to the database happens on the main thread.

Auditable collection diagram: Playwright search, immutable hashed HTML, parser, results, PDF download with checks and content-addressed file, deduplicated by official identifier and SHA-256

Never fetching the same thing twice

Deduplication uses two keys: the document's official identifier, with a UNIQUE constraint, and the file's SHA-256. The same ruling found by different queries becomes a single document, but every result from every search is preserved, and the upsert with COALESCE never overwrites metadata that is already known. Sealed cases and results without a PDF are recorded with the reason and never classified. The repair-pdfs command reapplies the current parser to the saved HTML, offline, and only goes back to the site for whatever is still missing a PDF.

Extracting text without making things up

PyMuPDF extracts the text page by page, and a state machine separates the headnote, the case report, the opinion and the operative part. If the PDF lacks the headnote, the one from the search result is used, flagged with its own method. A PDF with too little text becomes ocr_needed; I did not implement OCR, and the program does not fill in missing text.

Having a model read 1,048 rulings and prove what it read

The whole decision, with an output contract

The model is GPT-6 Luna with high reasoning effort, called through Codex CLI in a stateless subprocess per document, in a read-only sandbox. The input is the entire ruling, with no chunking or embeddings; above 100 pages or 180,000 characters, the system abstains and records the reason. The versioned instructions treat the page text as untrusted data, forbid concluding that an argument was never made just because the ruling does not mention it and require abstaining when the text does not support the assessment.

The output follows a closed JSON Schema, revalidated in Python: eligibility, defendant side (who the defendant is: a public entity, a private one or both), treatment of the expert report, up to 4 appeals and up to 20 pieces of evidence. Consistency rules in code prevent the summary from contradicting the analysis of each appeal.

A verbatim quote or nothing

Each quote must appear on the cited page. Verification has three steps: exact match, then with whitespace and line breaks collapsed, then ignoring whitespace entirely. There is no fuzzy matching and no ellipses, and the stored text becomes the exact excerpt from the source, with offsets. On failure, there is a single retry to fix only the evidence; if the model changes the classification in it, the result is rejected. A second failure is recorded, along with both responses.

Verbatim quote validation diagram: three matching steps, one retry that cannot change the classification and the failure record

Each record stores the hash of the prompt, the evaluator code, the input, the PDF and the raw response. Resuming a run is refused if the model, prompt or code has changed.

Ruling reading page in the dashboard: header, type of dispute over the expert evidence and section-by-section reading, with case number, reporting judge and text masked

Scaling with disjoint workers

The cohort is frozen and split deterministically by text length, with the largest document going to the least loaded worker. A hashed manifest per worker, an assignment table and a PostgreSQL advisory lock per case guarantee that no document is read twice. The final campaign ran in 18 runs of 58 to 59 documents, in about 2 h 36 min. Before that, a cheap regex finds candidates and the model only reads where it is needed: when screening dentistry cases, it discarded 29 of the 90 candidates.

The numbers

  • 1,499 raw results, 1,453 distinct documents and 1,412 extracted PDFs (15,635 pages).
  • 1,077 main rulings with text and 1,048 analyzed: 1,034 records and 14 abstentions.
  • 518 likely included, 486 excluded and 44 uncertain.
  • 7,426 verified quotes; 330 records needed a second attempt.

Funnel: 1,499 raw results, 1,453 distinct, 1,412 PDFs, 1,077 main rulings with text, 1,048 analyzed and 518 likely included

The public dashboard, in React 19 with TypeScript and Recharts, shows the 518 cases. It is read-only, never exposes the model's raw response and does not host PDFs: the link points to the court. In the images on this page, case numbers, names and excerpts from the decisions have been masked.

Dashboard: filters, total cards, rulings by month of judgment and defendant profile

What did not work

The first version was a regex classifier. In a pilot with 100 randomly drawn rulings, I compared the rules against an earlier model: agreement was 47.9% on eligibility and 28.7% on defendant side. That measures agreement, not accuracy, but it was enough to see that rules were not capturing the legal reasoning. I retired the rules as a classifier and the pipeline switched to using only the model, with validated quotes.

Limitations

The human audit has its infrastructure in place, with a stratified blind sample and a Flask interface that hides the model's labels until the review is done, but human validation has not been completed. The model's classification is preliminary.

Local human review interface: blind sample queue, extracted ruling text (masked) and the analysis record with its five dimensions

The study is exploratory: it does not measure how often the court behaves a certain way, does not assess whether the decision was right and does not infer the judges' intent. A ruling's silence does not prove an argument was absent, and the corpus is whatever three queries retrieved.

Further reading

I wrote a post about the analysis method: AI for semantic analysis of large datasets: how to have a model read a thousand rulings and prove what it read.

Gallery

Other projects

Cover of the Markmedi project
markmedi.com · private ★ featured

Markmedi

My main project: scheduling, medical records, telemedicine, finance, CRM and marketing for clinics, in a modular Laravel and Vue monolith with queues, real-time updates and per-clinic encryption.

  • Laravel
  • Vue 3
  • Inertia.js
  • TypeScript
  • Tailwind CSS
Cover of the B2B platform for ticket and course sales project
b2b-ticketing-and-courses-platform · private ★ featured

B2B platform for ticket and course sales

A B2B platform that sells tickets and courses, validates event entry with signed QR codes and offline check-in, and lets companies buy licenses and track each employee's progress.

  • Laravel
  • Vue 3
  • Inertia.js
  • TypeScript
  • Tailwind CSS