Large Model Memory Anxiety Solved on a USB Drive?
To address the memory and bandwidth challenges of large language model inference, SanDisk and SK Hynix are developing a new technology called High Bandwidth Flash (HBF). This approach adapts the advanced 3D packaging and stacking techniques used for High Bandwidth Memory (HBM) to NAND Flash, which is traditionally seen as slow for compute tasks.
The key insight is that while NAND Flash is slow for writes, its read speeds can be significantly enhanced through parallelization. By stacking up to 16 NAND Flash chips, the first-generation HBF aims for capacities up to 512GB and read bandwidths up to 1.6 TB/s, with roadmaps targeting 3.2 TB/s. This positions HBF not as a replacement for HBM, but as a complementary, cost-effective tier in a hierarchical memory system for AI inference.
During inference, model weights are largely static and read-intensive, making them suitable for HBF storage. This can alleviate the severe capacity bottleneck of expensive HBM, potentially reducing the number of accelerators needed per server, lowering overall system cost and power consumption. The technology, currently being standardized within the Open Compute Project, signals a shift towards more specialized memory architectures tailored for the specific demands of large-scale AI deployment.
marsbit07/20 00:18
