Skip to content
~/rafael-ferreira

Queues in Markmedi: how Horizon keeps patient reminders apart from email campaigns

Fixed-capacity supervisors, tested timeout invariants, a global rate limit for SES and an idempotency ledger: how I organized Markmedi's queues with Laravel Horizon.

In a clinic system, the job queue carries things with very different levels of urgency. The reminder for tomorrow's appointment has to go out on time. Transcribing a consultation can take minutes of AI processing. An email campaign to four thousand contacts can tie up a worker for half an hour. And a patient's photo progression video uses ffmpeg and memory. If all of that shares the same queue, the marketing campaign delays the reminder, and the patient doesn't show up.

Markmedi runs on Redis with Laravel Horizon, and organizing the queues was one of the parts I struggled with most before arriving at the current design.

Fixed-capacity supervisors

Instead of letting Horizon balance processes automatically, each supervisor has a fixed number of processes (minimum equal to maximum) and consumes a small set of queues. This trades a bit of efficiency for predictability: I know exactly how many workers exist for each type of work and how much memory the container will use.

Supervisor

Queues (in priority order)

Processes

Purpose

clinic

critical, default

2

Reminder and confirmation triggers, provider webhooks, transactional notifications and the clinic's everyday work

ai

ai

2

Transcription, summaries, assisted patient intake, analyses

emails

emails

1

Campaign dispatch and email marketing delivery

media

media

1

Recordings, progression videos, medical record export

The important part is what sits inside the clinic supervisor. The critical queue receives the jobs that trigger reminders and confirmation requests, plus the processing of WhatsApp, SMS and email webhooks. Since the supervisor consumes critical before default, those jobs jump ahead. The transactional notifications themselves (the WhatsApp reminder, the confirmation email) go out on the same supervisor's default queue. There is no dedicated WhatsApp queue. What exists is isolation of whatever is heavy and not urgent: AI, media and marketing never consume the clinic's capacity.

Four fixed-capacity Horizon supervisors: clinic with critical and default, ai, emails and media

I also configured wait thresholds per queue, which raise an alert when a job sits idle for too long: 30 seconds for critical, 60 for default, 2 minutes for emails and 5 minutes for AI and media.

A timeout invariant that became a test

A classic Redis queue bug is a job being delivered twice. It happens when the job takes longer than the connection's retry_after: Redis assumes the worker died and hands the job to another worker while the first one is still working. So that this doesn't depend on someone remembering, there is a configuration test with a table of every relevant job. It checks each one's queue and the chain of timeouts:

job timeout < supervisor timeout < connection retry_after

The job timeout is lower than the supervisor's, which is lower than the connection's retry_after; if the job exceeds retry_after, Redis hands it to another worker

The same file checks the total number of workers and ensures media and campaigns don't consume the clinic's capacity. Changing the Horizon configuration without updating the test breaks CI, which is exactly what I want.

A global rate limit for Amazon SES

Each campaign recipient is a job, and those jobs go through a rate limiter called ses-send. It is global rather than per clinic, because the SES sending limit belongs to the whole account, not to each customer. A per-clinic limit would allow ten clinics sending at the same time to blow through the account quota and start getting throttling errors.

The send job has several safeguards around this: a capped number of exceptions, progressive backoff (30 s, 2 min, 5 min) and a maximum 12-hour window for retries. When a campaign is paused, the job doesn't spend a rate limiter slot just to find out it is paused: it checks the state, goes back to the queue for later and extends its own deadline. The full architecture of the email module is in BYOD email marketing with Amazon SES.

A job that knows it's stale

A reminder is scheduled for 24 hours before the appointment. If the front desk reschedules the appointment after that, the old job is still in the queue. The solution was to give the appointment a reminder_version that changes on every reschedule. The job carries the version it saw and, when it runs, discards itself if it has changed. It also gives up if the appointment has already been completed or cancelled, if the patient can't be contacted, or if the clinic's plan doesn't include that feature.

After a reschedule, reminder_version changes from 1 to 2; the old job notices the difference and discards itself, the new one sends the reminder

This pattern of "checking whether it still makes sense" at execution time is more robust than trying to cancel jobs in the queue, which is something Redis doesn't do well.

A ledger so nothing is sent twice

Even with all of that, a job can run twice: a worker dies after sending and before acknowledging, or a deploy restarts the process at the worst possible moment. For patient messages, sending twice is a real problem. Every outbound message goes through a ledger table with a unique idempotency key, derived from what the message represents, such as "prescription X is ready" or "step N of sequence Y".

try {
    $message = OutboundMessage::create(['idempotency_key' => $key, ...]);
} catch (UniqueConstraintViolationException) {
    return DispatchResult::alreadySent(); // lost the race: another worker already sent it
}

The channel first checks whether the key exists and, if it loses the race on the insert, treats the uniqueness violation as "already sent". The SMS fallback follows the same idea: after a WhatsApp send, an SMS job is scheduled a few minutes later and reuses the idempotency key from the original send, so it stays tied to the same logical message.

Nothing goes out before the commit

Another concern is the ordering between the database and the queue. Notifications and domain events are dispatched only after the transaction commits. Without that, a job could start before the transaction finishes, look for the appointment that was just created and not find it, or worse, send the confirmation for a booking that was rolled back.

Testing the real queue

In regular tests the queue runs in sync mode, which is great for testing business rules and useless for testing reliability. That's why CI has a separate job that spins up real Redis and Supervisor and runs queue lifecycle tests:

  • a job is only redelivered after the worker-crash window;
  • a SIGTERM during a deploy shuts the worker down without losing the backlog;
  • recovery after a crash doesn't duplicate the job's effect;
  • jobs for the same report run serialized.

This kind of test covers what no unit test can reach. One example is the Supervisor configuration: the shutdown wait time has to be longer than the longest job, otherwise a deploy kills the worker in the middle of its work.

Real-time status

Some long-running jobs report progress to the interface. The clinic positioning analysis, for example, emits progress events over WebSocket with Reverb, and the screen has a fallback polling mechanism in case the connection drops. The details are in real time with Laravel Reverb.

Summary

I split work by urgency and cost, not by technology. I fixed each supervisor's capacity so the server behaves predictably. The rules most likely to cause incidents (timeouts, isolation, idempotency) became tests. Horizon handles what it does well, visibility and supervision, and the correctness of each send lives in the application code.

This post is part of the series about Markmedi. To see what happens before a job enters the queue, read architecture, DDD and domain events.