cd..→blog

Architecting Resilient LLM Gateways with Bun and BullMQ

const published = "Aug 13, 2026, 05:18 PM";const readTime = 5 min;
BunBullMQLLM InfrastructureNode.jsDistributed Systems
Learn how to build a high-performance LLM gateway using Bun and BullMQ to handle provider rate limits, implement intelligent retries, and ensure reliable asynchronous inference.

Architecting Resilient LLM Gateways with Bun and BullMQ

As of August 2026, the bottleneck in AI-integrated applications has shifted from model capability to infrastructure reliability. Engineering teams are moving away from direct SDK calls to centralized LLM Gateways. This architectural shift addresses the inherent instability of upstream providers, strict rate limits, and the need for unified observability.

In this post, we will build a high-performance gateway using Bun, a fast JavaScript runtime with a built-in bundler and test runner, and BullMQ, a message queue for handling distributed jobs based on Redis. This combination provides the low-latency I/O required for proxying and the robust persistence needed for long-running inference tasks.

The Problem: The Fragility of Direct Inference

Directly calling an LLM provider (like OpenAI or Anthropic) within a standard request-response cycle introduces several failure modes:

  1. Head-of-Line Blocking: A slow 30-second generation blocks a worker thread or connection slot.
  2. Rate Limit Exhaustion: Without a centralized coordinator, multiple microservices can simultaneously hit 429 errors, triggering aggressive backoffs that degrade user experience.
  3. Orphaned Requests: If a client disconnects, the backend often continues to pay for the full token generation because there is no cancellation propagation.

Designing the Gateway Architecture

A resilient gateway acts as a buffer. It should provide a synchronous endpoint for low-latency tasks (like classification) and an asynchronous, queue-backed flow for heavy tasks (like RAG synthesis or agentic loops).

Why Bun?

Bun's native implementation of the Web Standards fetch API and its highly optimized HTTP server (Bun.serve) make it ideal for proxying. In our testing, Bun's overhead is significantly lower than Node.js when handling high-concurrency streaming responses, which is critical when your "database" is actually a third-party API with variable latency.

Why BullMQ?

BullMQ provides the state management that raw HTTP lacks. It handles parent-child job dependencies, delayed retries with exponential backoff, and rate-limiting at the queue level. This ensures that even if an upstream provider goes down for five minutes, your system doesn't lose the user's intent.

Implementation: The Async Inference Pattern

Instead of waiting for the LLM, the gateway accepts the request, validates the schema, and pushes a job to BullMQ. The client receives a 202 Accepted with a job ID.

import { Queue } from 'bullmq';
import { Redis } from 'ioredis';

const connection = new Redis(process.env.REDIS_URL!);
const inferenceQueue = new Queue('inference-tasks', { connection });

Bun.serve({
  port: 3000,
  async fetch(req) {
    const url = new URL(req.url);
    
    if (url.pathname === "/v1/chat/completions" && req.method === "POST") {
      const body = await req.json();
      
      // Enqueue the job for background processing
      const job = await inferenceQueue.add('generate-text', {
        payload: body,
        priority: body.is_premium ? 1 : 10
      }, {
        attempts: 5,
        backoff: {
          type: 'exponential',
          delay: 2000,
        },
      });

      return Response.json({ id: job.id }, { status: 202 });
    }
    
    return new Response("Not Found", { status: 404 });
  },
});

Handling Rate Limits Globally

One of the most powerful features of BullMQ is the ability to limit the rate at which jobs are processed across the entire cluster. If your Anthropic tier allows 1,000 Requests Per Minute (RPM), you can configure the worker to respect this strictly.

import { Worker } from 'bullmq';

const worker = new Worker('inference-tasks', async (job) => {
  const response = await fetch("https://api.anthropic.com/v1/messages", {
    method: "POST",
    headers: {
      "x-api-key": process.env.ANTHROPIC_API_KEY!,
      "Content-Type": "application/json",
    },
    body: JSON.stringify(job.data.payload),
  });

  if (response.status === 429) {
    // This triggers the BullMQ retry logic with backoff
    throw new Error("Rate limited by provider");
  }

  const result = await response.json();
  // Store result in Redis or push to a webhook
  await storeResult(job.id, result);
}, {
  connection,
  limiter: {
    max: 1000,
    duration: 60000, // 1 minute
  },
});

Streaming and the "Dangling Connection" Problem

For real-time UI, we need streaming. However, streaming through a queue is non-trivial. The pattern here is to use the Gateway as a Smart Proxy.

When a streaming request hits the Bun server, we use AbortController to detect client disconnects. If the client leaves, we immediately abort the upstream fetch to save costs. Simultaneously, we log the partial response to a tracing system like LangSmith or Arize Phoenix for debugging.

Implementation of the Smart Proxy

Bun.serve({
  async fetch(req) {
    const controller = new AbortController();
    req.signal.addEventListener("abort", () => {
      console.log("Client disconnected, aborting upstream...");
      controller.abort();
    });

    const upstreamResponse = await fetch("https://api.openai.com/v1/chat/completions", {
      method: "POST",
      headers: { "Authorization": `Bearer ${process.env.OPENAI_API_KEY}` },
      body: req.body,
      signal: controller.signal,
    });

    // Bun's native handling of ReadableStream makes this efficient
    return new Response(upstreamResponse.body, {
      headers: upstreamResponse.headers,
    });
  }
});

Observability and Cost Tracking

A production gateway must inject metadata. By intercepting the request/response in Bun, we can calculate token usage locally using libraries like tiktoken or by parsing the usage field in the LLM response. This data should be pushed asynchronously to a time-series database (like ClickHouse) to avoid adding latency to the critical path.

Tradeoffs and Considerations

  1. Complexity: Introducing BullMQ and Redis adds moving parts. For small apps, this is overkill. For systems spending >$5k/month on API costs, the reliability gains outweigh the maintenance.
  2. Latency: Every hop adds milliseconds. Bun's native code execution helps, but the network overhead between the Gateway and Redis is real. Keep your Gateway and Redis instance in the same VPC/region.
  3. State Management: If using the async pattern, you need a way to notify the client when the job is done. WebSockets or Server-Sent Events (SSE) are the standard choices here, which Bun supports natively via Bun.serve({ websocket: { ... } }).

Conclusion

Building a resilient LLM gateway is no longer just about proxying strings; it's about managing distributed state and respecting upstream constraints. By leveraging Bun's high-performance I/O and BullMQ's robust task scheduling, you can build an infrastructure layer that turns brittle AI features into reliable production services.

cd..→back to blog