View source for Sparse matrix

{| class=wikitable align=right width=240px style="margin: 3px 0 5px 14px;"
| <center>'''Example of sparse matrix'''</center>
<center><math>\left(\begin{smallmatrix}
11 & 22 & 0 & 0 & 0 & 0 & 0 \\
0 & 33 & 44 & 0 & 0 & 0 & 0 \\
0 & 0 & 55 & 66 & 77 & 0 & 0 \\
0 & 0 & 0 & 0 & 0 & 88 & 0 \\
0 & 0 & 0 & 0 & 0 & 0 & 99 \\
\end{smallmatrix}\right)</math></center>
|-
| <center>{{midsize|The above sparse matrix contains only 9 nonzero elements, with 26 zero elements. Its sparsity is 74%, and its density is 26%.}}</center>
|}
[[Image:Finite element sparse matrix.png|right|thumb|A sparse matrix obtained when solving a [[finite element method|finite element problem]] in two dimensions. The non-zero elements are shown in black.]]

In [[numerical analysis]] and [[scientific computing]], a '''sparse matrix''' or '''sparse array''' is a [[matrix (mathematics)|matrix]] in which most of the elements are zero. By contrast, if most of the elements are nonzero, then the matrix is considered '''dense'''. The number of zero-valued elements divided by the total number of elements (e.g., m × n for an m × n matrix) is called the '''sparsity''' of the matrix (which is equal to 1 minus the '''density''' of the matrix). Using those definitions, a matrix will be sparse when its sparsity is greater than 0.5.

Conceptually, sparsity corresponds to systems with few pairwise interactions. Consider a line of balls connected by springs from one to the next: this is a sparse system as only adjacent balls are coupled. By contrast, if the same line of balls had springs connecting each ball to all other balls, the system would correspond to a dense matrix. The concept of sparsity is useful in [[combinatorics]] and application areas such as [[network theory]], which have a low density of significant data or connections.

Large sparse matrices often appear in [[scientific]] or [[engineering]] applications when solving [[partial differential equation]]s.

When storing and manipulating sparse matrices on a [[computer]], it is beneficial and often necessary to use specialized [[algorithm]]s and [[data structure]]s that take advantage of the sparse structure of the matrix. Operations using standard dense-matrix structures and algorithms are slow and inefficient when applied to large sparse matrices as processing and [[Computer memory|memory]] are wasted on the zeroes. Sparse data is by nature more easily [[data compression|compressed]] and thus requires significantly less [[computer data storage|storage]]. Some very large sparse matrices are infeasible to manipulate using standard dense-matrix algorithms.

=={{anchor|storage}} Storing a sparse matrix==

A matrix is typically stored as a two-dimensional array. Each entry in the array represents an element {{math|''a''<sub>''i'',''j''</sub>}} of the matrix and is accessed by the two [[Array data structure|indices]] {{math|''i''}} and {{math|''j''}}. Conventionally, {{math|''i''}} is the row index, numbered from top to bottom, and {{math|''j''}} is the column index, numbered from left to right. For an {{math|''m'' × ''n''}} matrix, the amount of memory required to store the matrix in this format is proportional to {{math|''m'' × ''n''}} (disregarding the fact that the dimensions of the matrix also need to be stored).

In the case of a sparse matrix, substantial memory requirement reductions can be realized by storing only the non-zero entries. Depending on the number and distribution of the non-zero entries, different data structures can be used and yield huge savings in memory when compared to the basic approach. The trade-off is that accessing the individual elements becomes more complex and additional structures are needed to be able to recover the original matrix unambiguously.

Formats can be divided into two groups:

* Those that support efficient modification, such as DOK (Dictionary of keys), LIL (List of lists), or COO (Coordinate list). These are typically used to construct the matrices.
* Those that support efficient access and matrix operations, such as CSR (Compressed Sparse Row) or CSC (Compressed Sparse Column).

===Dictionary of keys (DOK)===

DOK consists of a [[associative array|dictionary]] that maps {{math|(row, column)}}-[[ordered pair|pairs]] to the value of the elements. Elements that are missing from the dictionary are taken to be zero. The format is good for incrementally constructing a sparse matrix in random order, but poor for iterating over non-zero values in lexicographical order. One typically constructs a matrix in this format and then converts to another more efficient format for processing.<ref>See [http://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.dok_matrix.html <code>scipy.sparse.dok_matrix</code>]</ref>

===List of lists (LIL)===

LIL stores one list per row, with each entry containing the column index and the value. Typically, these entries are kept sorted by column index for faster lookup. This is another format good for incremental matrix construction.<ref>See [http://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.lil_matrix.html <code>scipy.sparse.lil_matrix</code>]</ref>

===Coordinate list (COO)===

COO stores a list of {{math|(row, column, value)}} tuples. Ideally, the entries are sorted first by row index and then by column index, to improve random access times. This is another format that is good for incremental matrix construction.<ref>See [http://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.coo_matrix.html <code>scipy.sparse.coo_matrix</code>]</ref>

===Compressed sparse row (CSR, CRS or Yale format)===

The ''compressed sparse row'' (CSR) or ''compressed row storage'' (CRS) or Yale format represents a matrix {{math|'''M'''}} by three (one-dimensional) arrays, that respectively contain nonzero values, the extents of rows, and column indices. It is similar to COO, but compresses the row indices, hence the name. This format allows fast row access and matrix-vector multiplications ({{math|'''M'''''x''}}). The CSR format has been in use since at least the mid-1960s, with the first complete description appearing in 1967.<ref>{{cite conference |first1=Aydın |last1=Buluç |first2=Jeremy T. |last2=Fineman |first3=Matteo |last3=Frigo |first4=John R. |last4=Gilbert |first5=Charles E. |last5=Leiserson |authorlink5=Charles E. Leiserson |title=Parallel sparse matrix-vector and matrix-transpose-vector multiplication using compressed sparse blocks |year=2009 |conference=ACM Symp. on Parallelism in Algorithms and Architectures |url=http://gauss.cs.ucsb.edu/~aydin/csb2009.pdf |citeseerx=10.1.1.211.5256}}</ref>

The CSR format stores a sparse {{math|''m'' × ''n''}} matrix {{math|'''M'''}} in row form using three (one-dimensional) arrays {{math|(A, IA, JA)}}. Let {{math|NNZ}} denote the number of nonzero entries in {{math|'''M'''}}. (Note that [[Zero-based numbering|zero-based indices]] shall be used here.)
[[File:Row_and_column_major_order.svg|thumb|upright|Illustration of row-major order compared to column-major order]]
* The array {{math|A}} is of length {{math|NNZ}} and holds all the nonzero entries of {{math|'''M'''}} in left-to-right top-to-bottom ("row-major") order.
* The array {{math|IA}} is of length {{math|''m'' + 1}}. It is defined by this recursive definition:
**{{math|IA[0] {{=}} 0}}
** {{math|IA[''i''] {{=}} IA[''i'' − 1] +}} (number of nonzero elements on the ({{math|''i''-1}})-th row in the original matrix)
*** Thus, the {{math|''i''}}-th entry in {{math|IA}} stores the index of {{math|A}} at which the non-zero elements of the {{math|''i''}}-th row of {{math|'''M'''}} begin. Therefore, the values of the {{math|''i''}}-th row of the original matrix are read from the elements {{math|A[IA[''i''<nowiki>]]</nowiki>}} to {{math|A[IA[''i'' + 1] − 1]}} (inclusive on both ends).<ref>[http://netlib.org/linalg/html_templates/node91.html netlib.org]</ref>
***Additionally, the last element {{math|IA[''m+1'']}} stores {{math|NNZ}}, the number of elements in {{math|A}}, which can be also thought of as the index in {{math|A}} of first element of a phantom row just beyond the end of the matrix {{math|'''M'''}}.
* The third array, {{math|JA}}, contains the column index in {{math|'''M'''}} of each element of {{math|A}} and hence is of length {{math|NNZ}} as well.

For example, the matrix
:: <math>\begin{pmatrix}
0 & 0 & 0 & 0 \\
5 & 8 & 0 & 0 \\
0 & 0 & 3 & 0 \\
0 & 6 & 0 & 0 \\
\end{pmatrix}</math>

is a {{math|4 × 4}} matrix with 4 nonzero elements, hence

    A  = [ 5 8 3 6 ]
    IA = [ 0 0 2 3 4 ]
    JA = [ 0 1 2 1 ]

So, in array {{math|JA}}, the element "{{math|5}}" from {{math|A}} has column index {{math|0}}, "{{math|8}}" and "{{math|6}}" have index {{math|1}}, and element "{{math|3}}" has index {{math|2}}.

In this case the CSR representation contains 13 entries, compared to 16 in the original matrix. The CSR format saves on memory only when {{math|NNZ < (''m'' (''n'' − 1) − 1) / 2}}.
Another example, the matrix

:<math>\begin{pmatrix}
10 & 20 &  0 &  0 &  0 &  0 \\
 0 & 30 &  0 & 40 &  0 &  0 \\
 0 &  0 & 50 & 60 & 70 &  0 \\
 0 &  0 &  0 &  0 &  0 & 80 \\
\end{pmatrix}</math>

is a {{math|4 × 6}} matrix (24 entries) with 8 nonzero elements, so

     A = [ 10 20 30 40 50 60 70 80 ]
    IA = [  0  2  4  7  8 ]
    JA = [  0  1  1  3  2  3  4  5 ]   

The whole is stored as 21 entries.

* {{math|IA}} splits the array {{math|A}} into rows: <code>(10, 20) (30, 40) (50, 60, 70) (80)</code>;
* {{math|JA}} aligns values in columns: <code>(10, 20, ...) (0, 30, 0, 40, ...)(0, 0, 50, 60, 70, 0) (0, 0, 0, 0, 0, 80)</code>.

Note that in this format, the first value of {{math|IA}} is always zero and the last is always {{math|NNZ}}, so they are in some sense redundant (although in programming languages where the array length needs to be explicitly stored, {{math|NNZ}} would not be redundant). Nonetheless, this does avoid the need to handle an exceptional case when computing the length of each row, as it guarantees the formula {{math|IA[''i'' + 1] − IA[''i'']}} works for any row {{math|''i''}}. Moreover, the memory cost of this redundant storage is likely insignificant for a sufficiently large matrix.

The (old and new) Yale sparse matrix formats are instances of the CSR scheme. The old Yale format works exactly as described above, with three arrays; the new format combines {{math|IA}} and {{math|JA}} into a single array and handles the diagonal of the matrix separately.<ref>{{citation |title=Sparse Matrix Multiplication Package (SMMP) |first1=Randolph E. |last1=Bank |first2=Craig C. |last2=Douglas |journal=Advances in Computational Mathematics |volume=1 |year=1993 |url=http://www.mgnet.org/~douglas/Preprints/pub0034.pdf}}</ref>

For [[Logical matrix | logical]] [[Adjacency matrix | adjacency matrices]], the data array can be omitted, as the existence of an entry in the row array is sufficient to model a binary adjacency relation.

It is likely known as the Yale format because it was proposed in the 1977 Yale Sparse Matrix Package report from Department of Computer Science at Yale University.<ref>{{cite web |url=https://apps.dtic.mil/dtic/tr/fulltext/u2/a047724.pdf |title=Yale Sparse Matrix Package |last=Eisenstat  |first=S. C. |last2=Gursky |first2=M. C. |last3=Schultz |first3=M. H. |last4=Sherman |first4=A. H. |date=April 1977 |language=English |access-date=6 April 2019 }}</ref>

===Compressed sparse column (CSC or CCS)===

[http://netlib.org/linalg/html_templates/node92.html#SECTION00931200000000000000 CSC] is similar to CSR except that values are read first by column, a row index is stored for each value, and column pointers are stored. For example, CSC is {{math|(val, row_ind, col_ptr)}}, where {{math|val}} is an array of the (top-to-bottom, then left-to-right) non-zero values of the matrix; {{math|row_ind}} is the row indices corresponding to the values; and, {{math|col_ptr}} is the list of {{math|val}} indexes where each column starts. The name is based on the fact that column index information is compressed relative to the COO format. One typically uses another format (LIL, DOK, COO) for construction. This format is efficient for arithmetic operations, column slicing, and matrix-vector products.  See [http://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.csc_matrix.html scipy.sparse.csc_matrix].
This is the traditional format for specifying a sparse matrix in MATLAB (via the [https://www.mathworks.com/help/matlab/ref/sparse.html <code>sparse</code>] function).

==Special structure==

===Banded===
{{main article|Band matrix}}
An important special type of sparse matrices is [[band matrix]], defined as follows. The [[lower bandwidth of a matrix]] {{math|'''A'''}} is the smallest number {{math|''p''}} such that the entry {{math|''a''<sub>''i'',''j''</sub>}} vanishes whenever {{math|''i'' > ''j'' + ''p''}}. Similarly, the [[Band matrix#upper bandwidth|upper bandwidth]] is the smallest number {{math|''p''}} such that {{math|''a''<sub>''i'',''j''</sub> {{=}} 0}} whenever {{math|''i'' < ''j'' − ''p''}} {{harv|Golub|Van Loan|1996|loc=§1.2.1}}. For example, a [[tridiagonal matrix]] has lower bandwidth {{math|1}} and upper bandwidth {{math|1}}. As another example, the following sparse matrix has lower and upper bandwidth both equal to 3. Notice that zeros are represented with dots for clarity.
::<math>
\left(
\begin{smallmatrix}
  X   &   X   &   X   & \cdot & \cdot & \cdot & \cdot & \\
  X   &   X   & \cdot &   X   &   X   & \cdot & \cdot & \\
  X   & \cdot &   X   & \cdot &   X   & \cdot & \cdot & \\
\cdot &   X   & \cdot &   X   & \cdot &   X   & \cdot & \\
\cdot &   X   &   X   & \cdot &   X   &   X   &   X   & \\
\cdot & \cdot & \cdot &   X   &   X   &   X   & \cdot & \\      
\cdot & \cdot & \cdot & \cdot &   X   & \cdot &   X   & \\              
\end{smallmatrix}
\right)
</math>

Matrices with reasonably small upper and lower bandwidth are known as band matrices and often lend themselves to simpler algorithms than general sparse matrices; or one can sometimes apply dense matrix algorithms and gain efficiency simply by looping over a reduced number of indices.

By rearranging the rows and columns of a matrix {{math|'''A'''}} it may be possible to obtain a matrix {{math|'''A'''′}} with a lower bandwidth. A number of algorithms are designed for [[Graph bandwidth|bandwidth minimization]].

===Diagonal===
A very efficient structure for an extreme case of band matrices, the [[diagonal matrix]], is to store just the entries in the [[main diagonal]] as a [[one-dimensional array]], so a diagonal {{math|''n'' × ''n''}} matrix requires only {{math|''n''}} entries.

===Symmetric===
A symmetric sparse matrix arises as the [[adjacency matrix]] of an [[undirected graph]]; it can be stored efficiently as an [[adjacency list]].

==Reducing fill-in==

The ''fill-in'' of a matrix are those entries that change from an initial zero to a non-zero value during the execution of an algorithm. To reduce the memory requirements and the number of arithmetic operations used during an algorithm, it is useful to minimize the fill-in by switching rows and columns in the matrix. The [[symbolic Cholesky decomposition]] can be used to calculate the worst possible fill-in before doing the actual [[Cholesky decomposition]].

There are other methods than the [[Cholesky decomposition]] in use. Orthogonalization methods (such as QR factorization) are common, for example, when solving problems by least squares methods. While the theoretical fill-in is still the same, in practical terms the "false non-zeros" can be different for different methods. And symbolic versions of those algorithms can be used in the same manner as the symbolic Cholesky to compute worst case fill-in.

==Solving sparse matrix equations==

Both [[Iterative method|iterative]] and direct methods exist for sparse matrix solving.

Iterative methods, such as [[conjugate gradient]] method and [[GMRES]] utilize fast computations of matrix-vector products <math>Ax_i</math>, where matrix <math>A</math> is sparse. The use of [[preconditioner]]s can significantly accelerate convergence of such iterative methods.

==Software==

Several software libraries support sparse matrices, and provide solvers for sparse matrix equations. The following are open-source:
* [http://faculty.cse.tamu.edu/davis/suitesparse.html SuiteSparse], a suite of sparse matrix algorithms
* [[ALGLIB]] is a C++ and C# library with sparse linear algebra support
* [[Portable, Extensible Toolkit for Scientific Computation|PETSc]], a huge C library, contains many different matrix solvers.
* [[Eigen (C++ library)|Eigen3]] is a C++ library that contains several sparse matrix solvers. However, none of them are [[Parallel computing|parallelized]].
* [[MUMPS (software)|MUMPS]] ('''MU'''ltifrontal '''M'''assively '''P'''arallel sparse direct '''S'''olver), written in Fortran90, is a [[frontal solver]]
* [http://pastix.gforge.inria.fr/ PaStix]
* [http://crd-legacy.lbl.gov/~xiaoye/SuperLU/ SuperLU]
* [[Armadillo (C++ library)|Armadillo]] provides a user-friendly C++ wrapper for [http://crd-legacy.lbl.gov/~xiaoye/SuperLU/ SuperLU]
*[[SciPy]] provides support for several sparse matrix formats, linear algebra, and solvers
* [https://cran.r-project.org/web/packages/spam/index.html SPArse Matrix (spam)] R package for sparse matrices

==History==

The term ''sparse matrix'' was possibly coined by [[Harry Markowitz]] who triggered some pioneering work but then left the field.<ref>pp. 9,10 in [http://purl.umn.edu/107467 Oral history interview with Harry M. Markowitz]</ref>

==See also==
{{columns-list|colwidth=22em|
* [[Matrix representation]]
* [[Pareto principle]]
* [[Ragged matrix]]
* [[Skyline matrix]]
* [[Sparse graph code]]
* [[Sparse file]]
* [[Harwell-Boeing file format]]
* [[Matrix Market exchange formats]]
}}

==Notes==
{{Reflist}}

==References==
* {{Cite book | first1=Gene H. | last1=Golub | author1-link=Gene H. Golub | first2=Charles F. | last2=Van Loan | author2-link=Charles F. Van Loan | year=1996 | title=Matrix Computations | edition=3rd | publisher=Johns Hopkins | place=Baltimore | isbn=978-0-8018-5414-9 | ref=harv}}
* {{Cite book | last1=Stoer | first1=Josef | last2=Bulirsch | first2=Roland | title=Introduction to Numerical Analysis | publisher=[[Springer-Verlag]] | location=Berlin, New York | edition=3rd | isbn=978-0-387-95452-3 | year=2002}}
* {{Cite book | last=Tewarson| first=Reginald P.|title=Sparse Matrices (Part of the Mathematics in Science & Engineering series)|publisher= Academic Press Inc.|date=May 1973}} (This book, by a professor at the State University of New York at Stony Book, was the first book exclusively dedicated to Sparse Matrices.  Graduate courses using this as a textbook were offered at that University in the early 1980s).
* {{Cite web |title=Sparse Matrix Multiplication Package|first1= Randolph E.|last1= Bank|first2= Craig C.|last2= Douglas |url=http://www.mgnet.org/~douglas/Preprints/pub0034.pdf}}
* {{Cite book |last=Pissanetzky|first= Sergio|year= 1984|title=Sparse Matrix Technology|publisher= Academic Press}}
*{{cite journal|doi=10.1007/BF02521587|title=Reducing the profile of sparse symmetric matrices|year=1976|last1=Snay|first1=Richard A.|journal=Bulletin Géodésique|volume=50|issue=4|pages=341}} Also NOAA Technical Memorandum NOS NGS-4, National Geodetic Survey, Rockville, MD.

==Further reading==
* {{cite journal | title = A comparison of several bandwidth and profile reduction algorithms | journal = ACM Transactions on Mathematical Software | year = 1976 | volume = 2 | issue = 4 | pages = 322–330 | url = http://portal.acm.org/citation.cfm?id=355707 | doi = 10.1145/355705.355707 | last1 = Gibbs | first1 = Norman E. | last2 = Poole | first2 = William G. | last3 = Stockmeyer | first3 = Paul K.  }}
* {{cite journal | title = Sparse matrices in MATLAB: Design and Implementation | journal = SIAM Journal on Matrix Analysis and Applications | year = 1992 | volume = 13 | issue = 1 | pages = 333–356 | url = http://citeseer.ist.psu.edu/gilbert91sparse.html | doi = 10.1137/0613024 | last1 = Gilbert | first1 = John R. | last2 = Moler | first2 = Cleve | last3 = Schreiber | first3 = Robert | citeseerx = 10.1.1.470.1054 }}
* [http://www.cise.ufl.edu/research/sparse Sparse Matrix Algorithms Research] at the University of Florida, containing the UF sparse matrix collection.
* [http://www.small-project.eu SMALL project] A EU-funded project on sparse models, algorithms and dictionary learning for large-scale data.

{{Data structures}}
{{Numerical linear algebra}}
[[Category:Sparse matrices| ]]