@haatlabs-core/queue
BullMQ work queues on Redis: the publisher, the retrying consumer (with a priority lane) and
the outbox relay. A project supplies its topology, meaning its queue names, the routing-key
prefixes each takes, and its key prefix, through its own connectQueue:
const route = routeByPrefix<QueueName>({ 'shopify-sync': ['shopify.'], imports: ['import.'] });
export const connectQueue = (url: string, options: { prefix?: string } = {}) =>
connect(url, { prefix: options.prefix ?? 'shop:queue', route });
ioredis is a peer dependency.
@haatlabs-core/queue/contracts (0.12.1): QUEUE_ERRORS, QueueErrorCode,
createQueueError and the QueueError type, with nothing of Node: a project's contracts, which its pages load, import from here and never the root, which is for Node.
Install
pnpm add @haatlabs-core/queue
It leaves one package to your app, so the app holds one copy: ioredis.
Polling without holding a worker slot
A handler that waits on something external (a bulk operation, media processing) calls
pollAgain(message, { delayMs, deadlineMs, state, onDeadline }). The job is put back as a
delayed job, its worker slot is free meanwhile, it keeps its attempt count and its lane, and
message.poll (round, startedAt, deadlineAt, state) is there on the next round.
delayMsis 250 ms to 15 min a round. Every poll has a deadline, set on the first round and shared by all later rounds and phases; a round can shorten it, never extend it. Once it has passed,onDeadline()is thrown and the job fails.- Each round runs the handler again from the top. The work before
pollAgainmust be idempotent, or branch onmessage.poll.state. stateis JSON of at most 16 KiB; more is aRangeError.onDeadline()'s error decides what happens next: a transient one fails the attempt and the retry starts the flow afresh (the poll is cleared); a permanent one keeps the job as failed. APollAgainthrown directly past the deadline fails asPOLL_TIMED_OUT(QUEUE_ERRORS), which is transient.RestartPollfails the attempt as transient and clears the poll, so the retry starts afresh.- A poll round is logged at debug and its
queue.processspan ends with outcomepoll, not as an error.
Delayed and de-duplicated publishes
publish(key, payload, { delayMs, deduplication: { id } }) delays a message, or coalesces the
publishes that share an id. The job holds the id until it completes or fails for good:
- waiting for its first run (delayed or not), a later publish is dropped: the job has not started, so it will see the newer state;
- running, the newest later publish is kept and queued when it finishes or fails;
- delayed again, by a
PollAgainround or a retry's backoff, a later publish is dropped, though the job has already read the state.
So use deduplication only for handlers that never poll and recompute from scratch on every
attempt. The consumer warns, once a poll, when a de-duplicated job polls.
Such publishes are direct, never through the outbox: an outbox row carries no delayMs and no
deduplication. If the process dies between the commit and the publish, the message is lost.
A caller that needs it to survive publishes through the outbox, without them, and coalesces in
the handler instead.
Failed jobs are kept 1,000 per queue for 14 days.