RaiderChip brings Meta Llama 3.2 LLM HW acceleration to low cost FPGAs

The company incorporates the latest model, presented by Meta less than a week ago, into the catalog of LLMs already accelerated on a wide range of FPGAs

Spain -- October 1st, 2024 -- Just 6 days after its launch by Meta, RaiderChip has added support for the new Llama 3.2 model to the list of LLMs hardware accelerated by its GenAI v1 IP core for FPGAs. The supported models, which already included previous models from the same company (such as Llama 2, Llama 3, and Llama 3.1), as well as models from other providers (Microsoft’s Phi-2 and Phi-3), have been expanded again, following RaiderChip’s strategy of adding the most relevant models as they hit the market.

RaiderChip’s Generative AI hardware acceleration IP core is designed to accelerate any model built on Transformers technology, which underpins the vast majority of LLMs. The selection of additional supported models on FPGA is determined by customer choices of specific target devices and foundational LLMs. Supporting a foundational model enables any customer-specific fine-tuned derivative to be accelerated seamlessly, without the need to share its weights.

It is important to note that the various FPGA options available on the market have different sizes and variable logic and memory capacities. These technical factors, combined with commercial decisions such as unit cost, power consumption, or final functionality, mean that the ideal FPGA and LLM vary for each potential final product. For instance, using “smaller” models (such as Meta’s Llama 3.2 1B and Microsoft’s Phi-2 2.7B) or 4-bits Quantization are ideal for products based on simpler, more economical FPGAs; while larger LLMs in their original format, preserving the model’s full floating point precision, require larger and more expensive FPGAs,” explains Victor Lopez, the company’s CTO.

Click to enlarge

GenAI v1-Q running the Llama 3.2 1B LLM model with 4 bits Quantization on a low-cost Versal FPGA with LPDDR4 memory

Like previous models, the new Llama 3.2 can be tested through an interactive demo based on a Versal FPGA. “The demonstrator exposes not only a simple local and remote API access to the accelerated LLM, but may also be run directly through a chat terminal application that allows users to interact with the different models, while they are HW accelerated by RaiderChip’s IP core on the FPGA. At RaiderChip, we invite our customers to personally experience the real performance of our product, assessing key aspects like intelligence, latency, or tokens per second in a live demonstrator, beyond the usual performance tables based on theoretical or simulation data, which are often confusing,” the team comments.

Companies interested in trying the GenAI v1-Q IP core may reach out to RaiderChip for access to our FPGA demo or a consultation on how our IP cores can accelerate their AI workloads.

More information at https://raiderchip.ai/technology/hardware-ai-accelerators