NUS Chiplets Speed Up LLM Inference with Heterogeneous Memory

Introducing CHIPSMORE: A Novel Approach to LLM Inference Acceleration

Researchers at the National University of Singapore (NUS) have introduced CHIPSMORE, an innovative accelerator designed to significantly boost the efficiency of Large Language Model (LLM) inference. This new architecture tackles a critical bottleneck in AI: the speed and cost of processing complex, multi-request workloads. Traditional systems often struggle to balance throughput, latency, and energy consumption when handling diverse inference tasks simultaneously. CHIPSMORE addresses this by employing a heterogeneous chiplet-based design that integrates compute-in-interconnect (CII) and compute-in-memory (CIM) technologies. This allows the system to perform computations directly where data resides or during data transfer, drastically reducing the need to shuttle data between separate compute and memory units. The system is engineered to support both base-mode inference and low-rank adaptation (LoRA) techniques, making it versatile for a wide range of LLM applications.

The Challenge of Multi-Request LLM Inference

Large Language Models, while powerful, are notoriously resource-intensive, particularly during the inference phase – the process of using a trained model to generate outputs. A significant challenge arises when a system needs to handle multiple inference requests concurrently. This is common in real-world applications like chatbots, content generation services, and recommendation engines, where many users might be interacting with the LLM simultaneously. Each request can vary in complexity, and applying techniques like LoRA, which allows for efficient fine-tuning of LLMs without retraining the entire model, adds another layer of computational demand.

Existing hardware architectures often face trade-offs. Dedicated AI accelerators might offer high performance for specific tasks but can be inflexible or power-hungry. General-purpose CPUs and GPUs can handle diverse workloads but often incur substantial data movement overhead, leading to higher latency and energy consumption. The sheer volume of data required for LLM parameters and intermediate computations means that memory bandwidth and access times become major performance limiting factors. This is where CHIPSMORE's heterogeneous, chiplet-based approach offers a compelling solution. By distributing computation closer to or within the memory fabric, it aims to minimize data movement, which is analogous to a chef performing prep work directly at their cooking stations rather than constantly walking to a separate pantry for each ingredient.

CHIPSMORE Architecture: Compute-in-Interconnect and Memory

The core innovation of CHIPSMORE lies in its integration of two key architectural paradigms: Compute-in-Interconnect (CII) and Compute-in-Memory (CIM).

Compute-in-Interconnect (CII)

CII enables computation to occur within the data pathways themselves. Instead of data packets traveling from a processing unit to memory and back, some of the necessary calculations can be performed as the data traverses the interconnect fabric. This is particularly effective for operations that can be parallelized or are part of data aggregation and communication between different processing elements or chiplets. For LLM inference, this can speed up the aggregation of results from distributed computations or the processing of intermediate layer outputs. Think of it like having small processing hubs embedded along the highways of a city, allowing for quick sorting and routing of goods without every truck needing to go to the central distribution center.

Compute-in-Memory (CIM)

CIM takes this a step further by performing computations directly within or adjacent to the memory arrays. This dramatically reduces the energy and time spent on data retrieval. For LLMs, where model weights (parameters) are massive and frequently accessed, performing operations like multiply-accumulate (MAC) directly on the data stored in memory cells can yield substantial performance gains. CHIPSMORE leverages CIM to accelerate key LLM operations, such as matrix multiplications, which are fundamental to neural network computations. This is akin to having a calculator built directly into your spreadsheet software, so you don't need to copy numbers to a separate app to perform calculations.