The Common BFF Pattern and Its Pitfalls

Many applications employ a Backend for Frontend (BFF) architecture where a Next.js frontend communicates with a separate inference service, often built with FastAPI. This setup is standard for routing user requests to machine learning models. However, when these ML models, particularly those for tasks like audio generation with high sample rates (e.g., 44.1kHz), require extensive processing time on the CPU – potentially spanning minutes or even hours – a critical issue emerges. The frontend application eventually receives a 502 Bad Gateway error, even though the inference process on the backend has not yet completed.

This problem stems from the underlying HTTP client used by Node.js for making outgoing requests. In modern Node.js versions, this is typically undici, a high-performance HTTP/1.1 and HTTP/2 client. While efficient for standard API calls, undici has default timeouts that are too short for lengthy ML inference tasks. When the inference takes longer than this default timeout, undici terminates the connection prematurely, leading to the 502 error on the frontend, irrespective of the backend service's status.

The inference service logs might show that the process is still running, but the connection from the Next.js API route has already been severed. This creates a disconnect: the user sees an error, but the work might still be in progress on the server. This is not an issue with the ML model itself or the inference service's ability to process the request, but rather a limitation in the communication layer between the Next.js API route and the inference backend.

Diagram illustrating Next.js API route proxying to a separate FastAPI inference service

Understanding Undici and Default Timeouts

undici is designed for speed and efficiency, often outperforming Node.js's built-in http module. It handles connections, requests, and responses with a focus on performance. Part of its default configuration includes connection and request timeouts. These timeouts are sensible for typical web API interactions, which are expected to complete within seconds. For instance, a default request timeout might be set to 30 seconds, and a keep-alive timeout might be around 5 seconds.

When a Next.js API route acts as a proxy, it makes an outgoing HTTP request to the inference service. If this request takes longer than undici's configured timeout, undici will abort the request. The Next.js server, receiving this abortion from its own outgoing client, translates it into a 502 error for the original frontend client. The original request from the user's browser to the Next.js server might have a longer client-side timeout, but the server-side proxy request is what fails first.

The surprising detail here is not that timeouts exist, but how aggressively they can cut off processes that are legitimately long-running. For developers accustomed to quicker API responses, these extended inference times can be a blind spot, leading to a frustrating debugging experience where the error appears to originate from the ML service when the root cause is the proxy's communication timeout.

Configuring Undici for Long-Running Tasks

The solution involves configuring undici to accept longer timeouts. Since Next.js API routes are essentially Node.js serverless functions or server-side code, you can influence the underlying HTTP client's behavior. The challenge is that Next.js abstracts away much of the direct Node.js server configuration.

For Next.js projects, especially those deployed on platforms like Vercel, direct manipulation of global Node.js HTTP agent options can be tricky due to the serverless environment. However, when running Next.js in a custom Node.js server environment (e.g., using Express), or when targeting specific deployment configurations, you can often pass custom options to the HTTP client.

One approach is to leverage the http.request options when making the proxy call. If you are using a library like node-fetch (which under the hood can use undici in newer Node.js versions), you might be able to pass options related to timeouts. For direct undici usage, you can set the requestTimeout option. For example, to set a timeout of 1 hour (3,600,000 milliseconds):

import { request } from 'undici';

async function proxyRequest() {
  try {
    const response = await request('http://your-inference-service.com/generate',
      {
        method: 'POST',
        body: JSON.stringify({ data: 'your_input' }),
        headers: { 'content-type': 'application/json' },
        // Set a long timeout for the request
        requestTimeout: 3600000 // 1 hour in milliseconds
      }
    );
    // Process response
  } catch (error) {
    // Handle timeout or other errors
    console.error('Request failed:', error);
  }
}

It's crucial to understand the deployment environment. Serverless platforms often have their own execution time limits for functions. If your ML inference takes longer than the serverless function's maximum execution time (e.g., Vercel's 60-second limit for hobby plans, extendable on paid tiers), you will hit that limit first. The undici timeout needs to be configured to be less than or equal to the maximum execution time allowed by your hosting provider. If the inference truly takes hours, a serverless Next.js API route is likely not the correct architecture; a dedicated, long-running server or a background job queue would be more appropriate.

Alternative Strategies and Considerations

Beyond simply increasing the undici timeout, consider other architectural patterns for long-running tasks:

  • Asynchronous Processing with Webhooks/Polling: Instead of waiting for a synchronous response, the Next.js API route can initiate the ML inference and immediately return a 202 Accepted status with a job ID. The frontend can then poll a status endpoint or receive a webhook notification when the job is complete. This decouples the request initiation from the response delivery, preventing frontend timeouts.
  • Background Job Queues: Integrate with a dedicated job queue system (e.g., BullMQ, Celery with a broker like Redis or RabbitMQ). The Next.js API route enqueues the ML task, and a separate worker process picks it up. The frontend can then query for job status.
  • Dedicated Inference Service Scaling: Ensure the inference service itself is robust and can handle long-running requests. If the inference service is also running in a short-lived environment, it might time out before undici even gets a chance.

The 502 error is a symptom of a misaligned timeout configuration between the client proxy and the long-running server process. By understanding undici's role and its configurable timeouts, developers can adjust their Next.js API routes to accommodate tasks that require significant processing time, or adopt architectural patterns that better suit asynchronous, long-running operations.