I am using large matrices in my research. The size of matrix vary from 3600 by 3600 to 57600 to 57600. The matrix are sparse in nature.
But, despite using sparse matrix from scipy the computation of eigen vector takes very much time.
Is there any efficient method to do store large matrix?
57.6 k x 57.6 k = 330 M is not very large, according to current standards. So you could try to solve your problem using dense matrix routines, in particular for the smaller matrices. Are you sure you really need all 57.6 k eigenvectors? There is a great potential for speedup if you can reduce that number significantly, in particular for the iterative solvers. SciPy has several options for storage of sparse matrices, in addition to the LinearOperator option of no matrix storage at all, applicable to the sparse (iterative) solvers. But that in general tends to buy increased matrix size with use of additional time, while you seem to be interested in the other way around.
I have successfully solved eigenvalue problems for (sparse) matrix size 2**26 x 2**26 (2**9 x 2**9 x 2**8 lattice) on my 3+ year old laptop, but only for the few (< 100) lowest eigenvalues.
use conjugate gradient descent etc. to solve matrix inversion type problems iteratively, and you can use a sparse structure to store matrices (basically a list with has 2D locations and values, if it is a sparse matrix)
Actually, I am using this kind of matrix for spectral clustering. In sci-kit, it has an inbuilt function for 'SpectralClustering' to which I have to give this sparse matrix as input. But, it is working only for small matrices and not for mentioned matrix.
Scipy.sparse offers a number of sparseness structures, e.g. csr, coo, lil, etc.. and though each can nearly offer, in principle, the same functionality: the same function in a particular structure like lil can be much faster than e.g. csr depending on the structure of the problem you're dealing with. [ I wish there was a Python book that explains because this is hard-earned specialized knowledge from experience.] Mind you, lil takes more memory than csr in many cases. There is a tradeoff to consider...
- Badges
- Science topic
- Similar topics
- Computer Science and Engineering
- Programming Languages
- Python