G-circulant matrix and its matrix-vector multiplication algorithm on transformer neural network

Euis Asriani, S.Si., M.Si., - and Intan Muchtadi-Alamsyah, - and Ayu Purwarianti, - (2024) G-circulant matrix and its matrix-vector multiplication algorithm on transformer neural network. AIP Conference Proceedings, 3201 (1). ISSN 1551-7616

Text ((FULL TEXT)) g-Circulant Transformer.pdf Download (613kB)

Abstract

A g-circulant matrix is an n-dimensional matrix Cn,g that can be expressed as ⁠. This study will investigate the use of g-circulant matrices as weight matrices on transformer neural networks for machine translation. On transformer model tested, the g-circulant weight matrix undergoes multiplication with any vector and involves the DCT-DST algorithm. This algorithm is proposed as a substitute for the commonly used FFT algorithm. For this reason, an orthogonal matrix Un,g was defined by multiplying the Un matrix and appropriate permutation matrix, where Un matrix is a matrix that is spanned by the combination of the real and imaginary parts of eigenvectors of the real 1-circulant matrix. Then we form a Schur decomposition of the g-circulant matrix, composes the Qn,g matrix, and using the DCT-DST to formulate the multiplication of Qn,gUn,g. We discovered that 2-circulant matrices are outperform compared to the other g-circulant matrices and the original model of transformer in terms of accuracy and loss training.

Item Type:	Article
Subjects:	Q Sains > QA Mathematics
Divisions:	KARYA TULIS DOSEN
Depositing User:	UPT Perpustakaan UBB
Date Deposited:	05 Feb 2026 06:52
Last Modified:	05 Feb 2026 06:52
URI:	https://repository.ubb.ac.id/id/eprint/13170

Actions (login required)

View Item