Efficient and Effective Sparse LSTM on FPGA with Bank-Balanced Sparsity

Efficient and Effective Sparse LSTM on FPGA with Bank-Balanced Sparsity
Slide Note
Embed
Share

Utilizing Bank-Balanced Sparsity, this work presents an efficient Sparse LSTM implementation on FPGA for real-time inference in various applications like machine translation, speech recognition, and synthesis. Through innovative design and evaluation, the model achieves high accuracy while maintaining hardware efficiency and low latency. The research addresses the challenge of growing model sizes for improved accuracy by focusing on low-latency inference of large LSTM models without batching. Additionally, weight pruning techniques are explored for learning both weights and connections in neural networks for enhanced efficiency.

  • Sparse LSTM
  • FPGA acceleration
  • Bank-Balanced Sparsity
  • Real-time inference
  • Machine translation
zayn_754
zayn_754 Follow

Uploaded on Feb 28, 2025 | 0 Views

Download Presentation

Please find below an Image/Link to download the presentation.

The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.

You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.

Download Presentation

The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.

E N D

Presentation Transcript

  1. Efficient and Effective Sparse LSTM on FPGA with Bank-Balanced Sparsity Shijie Cao1, Chen Zhang2, Zhuliang Yao3, Wencong Xiao4, Lanshun Nie1, Dechen Zhan1, Yunxin Liu2, Ming Wu2, Lintao Zhang2 1Harbin Institute of Technology, 2Microsoft Research Asia, 3Tsinghua University, 4Beihang University

  2. Outline Motivation Design Bank-Balanced Sparsity Pattern (BBS) Sparse Matrix Computation and Format for BBS BBS FPGA Accelerator Evaluation Model Accuracy Hardware Efficiency Conclusion

  3. Outline Motivation Design Bank-Balanced Sparsity Pattern (BBS) Sparse Matrix Computation and Format for BBS BBS FPGA Accelerator Evaluation Model Accuracy Hardware Efficiency Conclusion

  4. Real-time Inference of LSTM Machine Translation Speech Recognition Speech Synthesis

  5. Real-time Inference of LSTM Machine Translation Speech Recognition Speech Synthesis User-interactive and latency-sensitive applications

  6. Real-time Inference of LSTM Machine Translation Speech Recognition Speech Synthesis User-interactive and latency-sensitive applications Model size continues to grow to achieve higher accuracy

  7. Real-time Inference of LSTM Machine Translation Speech Recognition Speech Synthesis User-interactive and latency-sensitive applications Model size continues to grow to achieve higher model accuracy Low latency inference of large LSTM model with no batching

  8. Quick Intro to LSTM A popular type of RNN ?? ct 1: long-term information ct 1 ?? ?t ?? 1 The most computation-heavy part: Matrix-Vector Multiplication (MxV)

  9. Weight Pruning Han, Song, et al. Learning both Weights and Connections for Efficient Neural Networks, NIPS 15

  10. Weight Pruning Difficult to accelerate Prune away small weights Unstructured sparse matrices MxV SpMxV Han, Song, et al. Learning both Weights and Connections for Efficient Neural Networks, NIPS 15

  11. Accuracy and Speedup Tradeoff Fine-grainedCoarse-grained IrregularRegular Pros: High model accuracy High compression ratio Cons: Irregular pattern Difficult to accelerate Cons: Low model accuracy Low compression ratio Pros: Regular pattern Easy to accelerate

  12. How to Achieve Both? Model accuracy Add few constraints on the sparsity pattern Speedup Matrix partitioning for parallel computing Eliminating irregular computation and memory access

  13. Outline Motivation Design Bank-Balanced Sparsity Pattern (BBS) Sparse Matrix Computation and Format for BBS BBS FPGA Accelerator Evaluation Model Accuracy Hardware Efficiency Conclusion

  14. Outline Motivation Design Bank-Balanced Sparsity Pattern (BBS) Sparse Matrix Computation and Format for BBS BBS FPGA Accelerator Evaluation Model Accuracy Hardware Efficiency Conclusion

  15. Bank-Balanced Pruning Bank Split Dense Matrix

  16. Bank-Balanced Pruning Bank Split Dense Matrix Traverse all rows 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 Dense Matrix Row 0.8 -0.1 0.2 1.5 1.0 0.3 -0.4 -1.4 0.7 2.0 0.9 -0.5 1.2 -1.3 2.1 0.2 Fine-grained pruning inside each bank BBS Matrix Row 0.8 1.5 1.0 -1.4 2.0 0.9 -1.3 2.1 Threshold percentage to obtain identical sparsity ratio among banks

  17. Bank-Balanced Sparsity (BBS) Bank partitioning for parallel computing Fine-grained pruning inside each bank for maintaining accuracy

  18. Weight map visualization Visual comparison

  19. Weight map visualization Visual comparison Bank 0 Bank 1

  20. Weight map visualization Visual comparison Bank 0 Bank 1 Effect on model accuracy in evaluation results

  21. Outline Motivation Design Bank-Balanced Sparsity Pattern (BBS) Sparse Matrix Computation and Format for BBS BBS FPGA Accelerator Evaluation Model Accuracy Hardware Efficiency Conclusion

  22. Sparse MV Multiplication (SpMxV) Inter-row parallelism: multiple PEs Vector Matrix PE 0 PE 1 PE 2 PE 3 PE 4 PE 5

  23. Sparse MV Multiplication (SpMxV) Intra-row (inter-bank) parallelism: Vector Matrix V0 V1 V2 V3 V4 V5 V6 V7 V8 V9 V10 V11 A 0 B C D 0 0 E F G 0 H PE 0 PE 1 PE 2 PE 3 PE 4 PE 5

  24. Sparse MV Multiplication (SpMxV) Intra-row (inter-bank) parallelism: BSB matrix row Dense vector 0 B D 0 0 F 0 H A C E G V0 V1 V2 V3 V4 V5 V6 V7 V8 V9 V10 V11 Bank 0 Bank 1 Bank 2 Bank 3 A C E G

  25. Sparse MV Multiplication (SpMxV) Intra-row (inter-bank) parallelism: BSB matrix row 0 B D 0 0 F 0 H A C E G Bank 0 Bank 1 Bank 2 Bank 3 Dense vector V0 V1 V2 A C E G Bank 0 V3 V4 V5 Bank 1 V6 V7 V8 Bank 2 V9 V10 V11 Bank 3

  26. Sparse MV Multiplication (SpMxV) Intra-row (inter-bank) parallelism: BSB matrix row 0 B D 0 0 F 0 H A C E G Bank 0 Bank 1 Bank 2 Bank 3 Dense vector V0 V1 V2 A C E G V0 V3 V7 V9 Bank 0 V3 V4 V5 Partial dot product: V0 A+V3 C+V7 E+V9 G Bank 1 S1 V6 V7 V8 Accumulate Bank 2 V9 V10 V11 Bank 3

  27. Sparse MV Multiplication (SpMxV) Intra-row (inter-bank) parallelism: BSB matrix row A 0 C 0 0 E G 0 B D F H Bank 0 Bank 1 Bank 2 Bank 3 Dense vector V0 V1 V2 B D F H V2 V4 V8 V11 Bank 0 V3 V4 V5 Partial dot product: V2 B+V4 D+V8 F+V11 H Bank 1 S2 V6 V7 V8 Accumulate Bank 2 V9 V10 V11 S1+S2 Bank 3

  28. Sparse MV Multiplication (SpMxV) Both inter-row and inter-bank parallelism Load balancing across rows and banks Bank 3 Bank 2 Bank 0 Bank 1 A 0 B C D 0 0 E F G 0 H Row 0 I J 0 K 0 L M N 0 O P 0 Row 1 Dense vector V0 V1 V2 Bank 0 Conflict-free vector accesses V3 V4 V5 Bank 1 V6 V7 V8 Bank 2 V9 V10 V11 Bank 3

  29. CSR (Compressed Sparse Rows)

  30. CSR (Compressed Sparse Rows) Decoding overhead in BBS Rearrange the order

  31. CSR (Compressed Sparse Rows) Decoding overhead in BBS Rearrange the order Compute the index in bank

  32. Our CSB (Compressed Sparse Banks) 0 A 0 1 C 0 2 E 0 3 G 1 4 B 2 5 D 2 6 F 3 7 H 2 8 I 0 9 K 0 10 11 12 13 14 15 M O J 1 3 1 CSB L 2 N 3 P 1 VALUES BANK INTERNAL INDICES Specifically designed for BBS to eliminate decoding overheads

  33. Our CSB (Compressed Sparse Banks) Data rearrangement for inter-bank parallelization 0 A 0 1 C 0 2 E 0 3 G 1 4 B 2 5 D 2 6 F 3 7 H 2 8 I 0 9 K 0 10 11 12 13 14 15 M O J 1 3 1 CSB L 2 N 3 P 1 VALUES BANK INTERNAL INDICES Specifically designed for BBS to eliminate decoding overheads

  34. Our CSB (Compressed Sparse Banks) 0 1 2 3 0 1 2 3 0 1 2 3 0 1 2 3 Data rearrangement for inter-bank parallelization 0 A 0 1 C 0 2 E 0 3 G 1 4 B 2 5 D 2 6 F 3 7 H 2 8 I 0 9 K 0 10 11 12 13 14 15 M O J 1 3 1 CSB L 2 N 3 P 1 VALUES BANK INTERNAL INDICES Physical BRAM addresses Specifically designed for BBS to eliminate decoding overheads

  35. Outline Motivation Design Bank-Balanced Sparsity Pattern (Pruning Method) Sparse Matrix Computation and Format for BBS BBS FPGA Accelerator Evaluation Model Accuracy Hardware Efficiency Conclusion

  36. Accelerator Overview FPGA Controller Instruction Buffer Host Server PCIe Cntlr SpMxV PE EWOP **+ ACT + + Values Matrix Memory * + * * D M A Output Private Vector Buffer + ... Indices Off-chip DRAM DRAM Cntlr Vector Memory

  37. Accelerator Overview FPGA Controller Instruction Buffer Host Server PCIe Cntlr SpMxV PE EWOP **+ ACT + + Values Matrix Memory * + * * D M A Output Private Vector Buffer + ... Indices Off-chip DRAM DRAM Cntlr Vector Memory

  38. Accelerator Overview FPGA Controller Instruction Buffer Host Server PCIe Cntlr SpMxV PE EWOP **+ ACT + + Values Matrix Memory * + * * D M A Output Private Vector Buffer + ... Indices Off-chip DRAM DRAM Cntlr Vector Memory

  39. Accelerator Overview FPGA Controller Instruction Buffer Host Server PCIe Cntlr SpMxV PE EWOP **+ ACT + + Values Matrix Memory * + * * D M A Output Private Vector Buffer + ... Indices Off-chip DRAM DRAM Cntlr Vector Memory

  40. Outline Motivation Design Bank-Balanced Sparsity Pattern (BBS) Sparse Matrix Computation and Format for BBS BBS FPGA Accelerator Evaluation Model Accuracy Hardware Efficiency Conclusion

  41. Model Accuracy Language model PTB dataset Speech Recognition on TIMIT dataset

  42. Model Accuracy Very close Language model PTB dataset Speech Recognition on TIMIT dataset

  43. Model Accuracy Very close Language model PTB dataset Speech Recognition on TIMIT dataset

  44. Sensitivity to Bank Size LSTM model on PTB dataset Comparisons Different bank size in BBS Different block size in Block sparsity Accuracy drop Almost the same

  45. Hardware Efficiency FPGA platform Catapult[1] with Intel Arria 10 Architecture setting M = 64 (64 PEs in the SpMxV unit) N = 64 (each PE has 64 multipliers) 16-bit data precision Model and Dataset LSTM on TIMIT dataset [1] Caulfield, Adrian M., et al. A loud-Scale Acceleration Architecture, MICRO 16.

  46. Hardware Efficiency FPGA platform Catapult[1] with Intel Arria 10 Architecture setting M = 64 (64 PEs in the SpMxV unit) N = 64 (each PE has 64 multipliers) 16-bit data precision Model and Dataset LSTM on TIMIT dataset Comparisons ESE[2] : improves throughput through batching C-LSTM[3] : block-circulant matrices Delta-RNN[4] : skip dispensable neuron activations [1] Caulfield, Adrian M., et al. A loud-Scale Acceleration Architecture, MICRO 16. [2] Han, Song, et al. ESE: Efficient Speech Recognition Engine with Sparse LSTM on FPGA, FPGA 17 [3] Gao, Chang, et al. DeltaRNN: A Power-Efficient Recurrent Neural Network Accelerator, FPGA 18. [4] Wang, Shuo, et al. C-LSTM: Enabling Efficient LSTM using Structured Compression Techniques on FPGAs, FPGA 18.

  47. Hardware Efficiency

  48. Hardware Efficiency

  49. Hardware Efficiency ~34x ~7x

  50. Hardware Efficiency Much better single batch performance because Enabling extra inter-bank parallelism Addressing the irregular memory access in SpMxV

Related


State Bank of India

State Bank of India

OBCRIGHTS
Recurrent Neural Networks (RNN) and Long Short-Term Memory (LSTM)

Recurrent Neural Networks (RNN) and Long Short-Term Memory (LSTM)

imaan
Recent Advances in RNN and CNN Models: CS886 Lecture Highlights

Recent Advances in RNN and CNN Models: CS886 Lecture Highlights

theseus
Advanced FPGA Design: Ensuring Safety, Reliability, and Predictability

Advanced FPGA Design: Ensuring Safety, Reliability, and Predictability

zoltan
Enhancing FPGA/SoC Projects with Gitlab CI: A Comprehensive Overview

Enhancing FPGA/SoC Projects with Gitlab CI: A Comprehensive Overview

zoya
FPGA Centric Dataflow Buffer Management Overview

FPGA Centric Dataflow Buffer Management Overview

thomasin
Real-Time Network Traffic Prediction Using LSTM Neural Network

Real-Time Network Traffic Prediction Using LSTM Neural Network

hawwa
Sparse vs. Dense Vector Representations in Natural Language Processing

Sparse vs. Dense Vector Representations in Natural Language Processing

zachariah
Overview of Sparse Linear Solvers and Gaussian Elimination

Overview of Sparse Linear Solvers and Gaussian Elimination

haws894
Sparse Millimeter-Wave Imaging Using Compressed Sensing and Point Spread Function Calibration

Sparse Millimeter-Wave Imaging Using Compressed Sensing and Point Spread Function Calibration

turn_310
Constant-Time Algorithms for Sparsity Matroids

Constant-Time Algorithms for Sparsity Matroids

tave_641
The Latest Trends in FPGA Technology and Applications

The Latest Trends in FPGA Technology and Applications

dlum
Pedestal Subtraction and FIR Technique Implementation in FPGA

Pedestal Subtraction and FIR Technique Implementation in FPGA

brennerb
FPGA Data Ingest Processing for NARA Electronic Records

FPGA Data Ingest Processing for NARA Electronic Records

sai_opp
Overview of ALICE ITS UPGRADE System Scrubbing and FPGA Programming

Overview of ALICE ITS UPGRADE System Scrubbing and FPGA Programming

shia678
Recurrent Neural Networks (RNNs) and LSTM Variants

Recurrent Neural Networks (RNNs) and LSTM Variants

jary812
Efficient Training of Dense Linear Models on FPGA with Low-Precision Data

Efficient Training of Dense Linear Models on FPGA with Low-Precision Data

fvier
Testbench Development in FPGA System Design Using VHDL: An Overview

Testbench Development in FPGA System Design Using VHDL: An Overview

byro_yn
Neural Image Caption Generation: Show and Tell with NIC Model Architecture

Neural Image Caption Generation: Show and Tell with NIC Model Architecture

kono_le
Overview of JetStream: A High-Performance PCIe Streaming Library for FPGA Communication

Overview of JetStream: A High-Performance PCIe Streaming Library for FPGA Communication

vela_l
FPGA for Underwater Communication Project Overview

FPGA for Underwater Communication Project Overview

bornemann_l
The Inner Workings of FPGA and Complex Programming Logic Devices

The Inner Workings of FPGA and Complex Programming Logic Devices

gsui

More Related Content