Efficient Modeling and Usage of Scratchpad Memory for Artificial Intelligence Accelerators

Abstract

Deep learning accelerators play a crucial role in enhancing computation-intensive AI applications. Optimizing system resources—such as shared caches, on-chip SRAM, and data movement mechanisms—is essential for achieving peak performance and energy efficiency. This paper explores the trade-off between last-level cache (LLC) and scratchpad memory (SPM) usage in accelerator-based SoCs. To evaluate this trade-off, we introduce a high-speed simulator for estimating the timing performance of complex SoCs and demonstrate the benefits of SPM utilization. Our work shows that dynamic reconfiguration of the LLC into an SPM with prefetching capabilities reduces cache misses while improving resource utilization, performance, and energy efficiency. With SPM usage, we achieve up to 13× speedup and a 10% reduction in energy consumption for CNN backbones. Additionally, our simulator significantly outperforms state-of-the-art alternatives, running 3000× faster than gem5-SALAM for fixed-weight convolution computations and up to 64,000× faster as weight size increases. These results validate the effectiveness of both the proposed architecture and simulator in optimizing deep learning workloads.