I am using large matrices in my research. The size of matrix vary from 3600 by 3600 to 57600 to 57600. The matrix are sparse in nature.
But, despite using sparse matrix from scipy the computation of eigen vector takes very much time.
Is there any efficient method to do store large matrix?
57.6 k x 57.6 k = 330 M is not very large, according to current standards. So you could try to solve your problem using dense matrix routines, in particular for the smaller matrices. Are you sure you really need all 57.6 k eigenvectors? There is a great potential for speedup if you can reduce that number significantly, in particular for the iterative solvers. SciPy has several options for storage of sparse matrices, in addition to the LinearOperator option of no matrix storage at all, applicable to the sparse (iterative) solvers. But that in general tends to buy increased matrix size with use of additional time, while you seem to be interested in the other way around.
I have successfully solved eigenvalue problems for (sparse) matrix size 2**26 x 2**26 (2**9 x 2**9 x 2**8 lattice) on my 3+ year old laptop, but only for the few (< 100) lowest eigenvalues.
use conjugate gradient descent etc. to solve matrix inversion type problems iteratively, and you can use a sparse structure to store matrices (basically a list with has 2D locations and values, if it is a sparse matrix)
Actually, I am using this kind of matrix for spectral clustering. In sci-kit, it has an inbuilt function for 'SpectralClustering' to which I have to give this sparse matrix as input. But, it is working only for small matrices and not for mentioned matrix.
- Badges
- Science topic
- Similar topics
- Computer Science and Engineering
- Programming Languages
- Python