Multiple-base Logarithmic Quantization and Application in Reduced Precision AI Computations
DOI:
https://doi.org/10.55630/dipp.2024.14.5Keywords:
not definedAbstract
The power of logarithmic quantizations and computations has been recognized as a useful tool in optimizing the performance of large ML models. There are plenty of applications of ML techniques in digital preservation. The accuracy of computations may play a crucial role in the corresponding algorithms. In this article, we provide results that demonstrate significantly better quantization signal-to-noise ratio performance thanks to multiple-base logarithmic number systems (MDLNS) in comparison with the floating point quantizations that use the same number of bits. On a hardware level, we present details about our Xilinx VCU-128 FPGA design for dot product and matrix vector computations. The MDLNS matrix-vector design significantly outperforms equivalent fixed-point binary designs in terms of area (A) and time (T) complexity and power consumption as evidenced by a 4 × scaling of AT 2 metric for VLSI performance, and 57% increase in computational throughput per watt compared to fixed-point arithmetic.References
Alam, S. A., Garland, J., & Gregg, D. (2021). Low-precision logarithmic number systems: beyond base-2. ACM Transactions on Architecture and Code Optimization (TACO), 18 (4), Article 47, 1-25. https://doi.org/10.1145/3461699
Arnold, M., Chester, E., Cowles, J., & Johnson, C. (2019, October). Optimizing Mitchell's Method for Approximate Logarithmic Addition via Base Selection with Application to Back-Propagation. In 2019 IEEE Nordic Circuits and Systems Conference (NORCAS): NORCHIP and International Symposium of System-on-Chip (SoC), Helsinki, Finland (pp. 1-6). IEEE. https://doi.org/10.1109/NORCHIP.2019.8906904
Darvish Rouhani, B., Zhao, R., Elango, V., Shafipour, R., Hall, M., Mesmakhosroshahi, M., More A, Melnick L, Golub M, Varatkar G, Shao L., Kolhe, G., Melts, D., Klar, J., L'Heureux, R., Perry, M., Burger, D., Chung, E., Deng, Z., Naghshineh, S., Park, J., & Naumov, M. (2023, June). With shared microexponents, a little shifting goes a long way. In Proceedings of the 50th Annual International Symposium on Computer Architecture (ISCA '23), (Article 83, pp. 1-13). Association for Computing Machinery. https://doi.org/10.1145/3579371.3589351
Dimitrov, V. S., Eskritt, J., Imbert, L., Jullien, G. A., & Miller, W. C. (2001, June). The use of the multi-dimensional logarithmic number system in DSP applications. In Proceedings 15th IEEE Symposium on Computer Arithmetic. ARITH-15 2001 (pp. 247-254). IEEE. https://doi.org/10.1109/ARITH.2001.930126
Gholami, A., Kim, S., Dong, Z., Yao, Z., Mahoney, M. W., & Keutzer, K. (2022). A survey of quantization methods for efficient neural network inference. In Low-Power Computer Vision (1st ed.) (pp. 291-326). Chapman and Hall/CRC.
Johnson, J. (2020, June). Efficient, arbitrarily high precision hardware logarithmic arithmetic for linear algebra. In 2020 IEEE 27th Symposium on Computer Arithmetic (ARITH) (pp. 25-32). IEEE. https://doi.org/10.1109/ARITH48897.2020.00013
LeCun, Y. (2019). 1.1 Deep Learning Hardware: Past, Present, and Future. In 2019 IEEE International Solid-State Circuits Conference - (ISSCC), San Francisco, CA, USA (pp. 12-19). IEEE. https://doi.org/10.1109/ISSCC.2019.8662396.
Miyashita, D., Lee, E. H., & Murmann, B. (2016). Convolutional neural networks using logarithmic data representation, arXiv:1603.01025. https://doi.org/10.48550/arXiv.1603.01025
Niu, Z., Zhang, T., Jiang, H., Cockburn, B. F., Liu, L., & Han, J. (2024, January). Hardware-Efficient Logarithmic Floating-Point Multipliers for Error-Tolerant Applications. IEEE Transactions on Circuits and Systems I: Regular Papers, 71 (1), 209-222. https://doi.org/10.1109/TCSI.2023.3326329
Sicuranza, G. (1982, May). On the accuracy of 2-D digital filter realizations using logarithmic number systems. In ICASSP'82. IEEE International Conference on Acoustics, Speech, and Signal Processing (Vol. 7, pp. 48-51). IEEE. https://doi.org/10.1109/ICASSP.1982.1171386
Vogel, S., Liang, M., Guntoro, A., Stechele, W., & Ascheid. G. (2018, November). Efficient Hardware Acceleration of CNNs using Logarithmic Data Representation with Arbitrary log-base. In 2018 IEEE/ACM International Conference on Computer- Aided Design (ICCAD) (pp. 1–8). https://doi.org/10.1145/3240765.3240803
Zhao, J., Dai, S., Venkatesan, R., Zimmer, B., Ali, M., Liu, M. Y., Khailany, B., Dally, W.J. & Anandkumar, A. (2022, December). Lns-madam: Low-precision training in logarithmic number system using multiplicative weight update. IEEE Transactions on Computers, 71 (12), 3179-3190. https://doi.org/10.1109/TC.2022.3202747
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2024 Digital Presentation and Preservation of Cultural and Scientific Heritage
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
