突破性硬件加速器 FlightLLM：大型语言模型推理的未来

近年来，大型语言模型（LLMs）在自然语言处理领域取得了卓越的成就。然而，这些庞大的模型在推理阶段面临着巨大的计算和内存开销，传统的硬件加速方案逐渐显得力不从心。在这一背景下，一项名为 FlightLLM 的研究在硬件加速领域掀起了一场革命，通过充分利用现代 FPGA（可编程逻辑门阵列）架构，为 LLMs 的推理阶段提供了一种创新性的解决方案。

论文下载：2401.03868.pdf 工程：https://github.com/FlightLLM/flightllm_test_demo (备份压缩：flightllm_test_demo-main.zip) 但是，遗憾的是，FPGA的源代码并没有开源。

FlightLLM 概述

FlightLLM 不仅仅是一种硬件加速器，更是一种重新定义了大型语言模型推理方法的技术。该研究通过细致入微的硬件优化和创新性的设计，为 LLMs 在 FPGA 上的高效推理创造了条件。

架构设计

FlightLLM 的核心架构采用了灵活的稀疏计算方法和混合精度策略，旨在最大程度地发挥 FPGA 的自定义能力。其中的三个主要组成部分为：

1. 计算核心（Matrix Processing Engine，MPE）：MPE 是 FlightLLM 的心脏，负责处理矩阵计算中的各种操作。从 GEMM、SpMM 到 GEMV、SpMV 和 SDDMM，MPE 打破了传统硬件的限制，引入了可配置的稀疏 DSP 链，通过优化路径配置，提升了稀疏计算效率。

2. 特殊功能单元（Special Function Unit，SFU）：SFU 专注于处理除矩阵和向量计算外的其他运算，如 softmax 和层归一化。多个 SFU 协同工作，共享参数和计算结果，从而减少端到端延迟和 FPGA 上的线路开销。

3. 始终在芯片上解码（Always-On-Chip Decode）：FlightLLM 引入了始终在芯片上解码的策略，通过巧妙的操作融合和内存缓冲，减少对外部内存的频繁访问，提高内存带宽利用率。

计算流程

FlightLLM 的计算流程分为解码阶段和预填充阶段。在解码阶段，采用操作融合和始终在芯片上解码的策略，降低了对外部内存的访问频率。同时，MPE 和 SFU 的协同工作提高了计算效率，使得 FlightLLM 在推理阶段表现卓越。

实现方法

FlightLLM 的实现方法深度优化了硬件资源的利用。具体而言：

• 计算核心优化：通过在 FPGA 上实例化多个计算核心，并将它们放置在不同的片上，充分利用 FPGA 并行计算的优势，降低了关键路径的影响。

• 内存控制器位置优化：将内存控制器放置在离 HBM 最近的 SLR0 上，提高了内存访问的效率。

• 功耗测量：利用供应商提供的 Xilinx Board Utility 工具 xbutil 对 FlightLLM 的功耗进行测量，为后续能效评估提供数据支持。

性能评估

FlightLLM 在多个方面超越了传统的 GPU 加速方案。通过与 NVIDIA A100 和 V100S 等 GPU 进行比较，FlightLLM 在端到端延迟、吞吐量、能效和成本效益方面都取得了显著的优势。与领先的领域特定加速器（DFX、FACT、CTA 等）相比，FlightLLM 在性能和能效方面也表现出色。

性能对比与突破

GPU 加速对比

FlightLLM 在不同输入和输出令牌大小的模型上，与 GPU 相比在 VHK158 上取得了更低的端到端延迟。在 OPT-6.7B/LLaMA2-7B 模型上，FlightLLM 在 U280 上相对于 V100S-naive 和 V100S-opt 分别提高了 1.5/1.6× 和 1.3/1.2× 的端到端延迟。表明 FlightLLM 在 FPGA 上的性能优势明显。

SOTA 加速器对比

与 DFX、FACT 和 CTA 等领域特定加速器相比，FlightLLM 在 OPT-6.7B 模型上实现了更为显著的端到端延迟速度提升。在 U280 和 VHK158 上，FlightLLM 的几何平均延迟速度提升分别为 2.7× 和 4.6×。同时，在相同硬件参数下，FlightLLM 实现了更好的计算资源利用率以及更高的带宽利用率。

能效和成本效益

FlightLLM 在 U280 上相对于 GPU 实现了显著的能效优势。对比 V100S-naive、A100-naive、V100S-opt 和 A100-opt，FlightLLM 实现了 6.7×、4.6×、6.0× 和 4.2× 的能效提升。从成本效益的角度看，FlightLLM 在 U280 上相对于 V100S-opt 和 A100-opt 实现了 1.9× 和 1.5× 的高性价比。

性能拆解与未来展望

FlightLLM 的端到端延迟拆解显示，在 FPGA 上相较于 V100S GPU，性能提升主要源于柔性稀疏方法和可配置的稀疏 DSP 链的引入。始终在芯片上解码的采用进一步提高了 FlightLLM 的性能。

结论

FlightLLM 的出现为大型语言模型的推理阶段带来了巨大的性能提升。通过充分发挥 FPGA 的灵活性和硬件自定义能力，FlightLLM 在计算效率和能效方面展现了令人瞩目的成果。这一研究不仅推动了 LLMs 硬件加速领域的发展，也为未来更高效的自然语言处理提供了新的思路和解决方案。通过与 GPU 和领先的领域特定加速器的对比，FlightLLM 在端到端延迟、能效和成本效益等方面均取得了令人瞩目的优势。随着硬件技术的不断发展，FlightLLM 为大型语言模型的未来发展指明了一条创新的道路。

In recent years, large language models (LLMs) have achieved remarkable success in the field of natural language processing. However, these massive models face significant computational and memory overhead during the inference stage, rendering traditional hardware acceleration solutions increasingly inadequate. In this context, a study named FlightLLM has sparked a revolution in the hardware acceleration field, providing an innovative solution for the inference stage of LLMs through the extensive utilization of modern Field-Programmable Gate Array (FPGA) architecture.

Overview of FlightLLM

FlightLLM is not just a hardware accelerator but a technological paradigm shift redefining the approach to large language model inference. The research, through meticulous hardware optimization and innovative design, creates conditions for efficient inference of LLMs on FPGA.

Architectural Design

The core architecture of FlightLLM employs flexible sparse computation methods and mixed-precision strategies, aiming to maximize the customization capabilities of FPGAs. The three main components include:

1. Matrix Processing Engine (MPE): MPE is the heart of FlightLLM, responsible for handling various operations in matrix computation. From GEMM, SpMM, to GEMV, SpMV, and SDDMM, MPE breaks traditional hardware limitations, introducing configurable sparse DSP chains to enhance sparse computation efficiency.

2. Special Function Unit (SFU): SFU focuses on handling operations other than matrix and vector computations, such as softmax and layer normalization. Multiple SFUs collaborate, sharing parameters and computation results, reducing end-to-end latency and on-chip overhead on the FPGA.

3. Always-On-Chip Decode: FlightLLM introduces the strategy of always-on-chip decoding, reducing the frequency of external memory access through clever fusion operations and memory buffering, thereby improving memory bandwidth utilization.

Computational Workflow

FlightLLM's computational workflow is divided into the decoding stage and the prefilling stage. In the decoding stage, the strategy of operation fusion and always-on-chip decoding reduces the frequency of external memory access. The collaborative work of MPE and SFU enhances computational efficiency, demonstrating outstanding performance in the inference stage.

Implementation Strategies

The implementation strategies of FlightLLM deeply optimize the utilization of hardware resources. Specifically:

• Optimization of Computing Cores: By instantiating multiple computing cores on the FPGA and placing them on different slices, FlightLLM maximizes the advantages of parallel computing, reducing the impact of critical paths.

• Optimization of Memory Controller Placement: Placing the memory controller on SLR0 closest to the High Bandwidth Memory (HBM) improves memory access efficiency.

• Power Measurement: FlightLLM's power is measured using the vendor-provided Xilinx Board Utility tool xbutil, providing data support for subsequent energy efficiency evaluations.

Performance Evaluation

FlightLLM surpasses traditional GPU acceleration solutions in various aspects. Through comparisons with GPUs like NVIDIA A100 and V100S, FlightLLM achieves significant advantages in end-to-end latency, throughput, energy efficiency, and cost-effectiveness. Compared with leading domain-specific accelerators (DFX, FACT, CTA, etc.), FlightLLM demonstrates outstanding performance and energy efficiency.

Performance Comparison and Breakthroughs

GPU Acceleration Comparison

FlightLLM achieves lower end-to-end latency on VHK158 compared to GPUs for models with different input and output token sizes. For OPT-6.7B/LLaMA2-7B models, FlightLLM on U280 improves end-to-end latency by 1.5/1.6× and 1.3/1.2× compared to V100S-naive and V100S-opt, respectively. This indicates a significant performance advantage of FlightLLM on FPGA.

Comparison with State-of-the-Art Accelerators

Compared with DFX, FACT, and CTA domain-specific accelerators, FlightLLM achieves a general speed-up in end-to-end latency for OPT-6.7B and LLaMA2-7B models. The geometric mean latency speed-ups of FlightLLM on U280 and VHK158 are 2.7× and 4.6× compared to DFX for OPT-6.7B, respectively. FlightLLM also achieves better utilization of computing resources and bandwidth under the same hardware parameters.

Energy Efficiency and Cost Effectiveness

FlightLLM achieves significant energy efficiency advantages over GPUs on U280. Compared with V100S-naive, A100-naive, V100S-opt, and A100-opt, FlightLLM achieves energy efficiency improvements of 6.7×, 4.6×, 6.0×, and 4.2× for OPT-6.7B. For LLaMA2-7B, FlightLLM on U280 achieves energy efficiency improvements of 6.0×, 4.4×, 5.5×, and 3.8×. In terms of cost efficiency (Token/s/dollar), FlightLLM on U280 generally outperforms GPUs.

Performance Breakdown and Future Outlook

FlightLLM's end-to-end latency breakdown reveals that the performance improvement on FPGA is mainly due to the introduction of flexible sparse methods and configurable sparse DSP chains. The adoption of always-on-chip decoding further enhances FlightLLM's performance by effectively reducing the overhead of off-chip memory access.

Conclusion

The emergence of FlightLLM brings a significant performance boost to the inference stage of large language models. By fully leveraging the flexibility and hardware customization capabilities of FPGAs, FlightLLM demonstrates remarkable achievements in computational efficiency and energy efficiency. This research not only propels the development of hardware acceleration for LLMs but also provides new insights and solutions for more efficient natural language processing in the future. Through comparisons with GPUs and leading domain-specific accelerators, FlightLLM achieves remarkable advantages in end-to-end latency, energy efficiency, and cost-effectiveness. As hardware technology continues to advance, FlightLLM points the way to an innovative future for large language models.