Differences between a Matrix and a Tensor

"Matrix" and "Tensor" may seem similar but they serve different purposes and possess distinct characteristics.

In this article, we’ll explore matrices and tensors.

Matrix: A Structured 2-D Array

A matrix is a two-dimensional array of numbers arranged in rows and columns. Here’s an example of a 4 \times 4 matrix:

\begin{bmatrix} a_{11} & a_{12} & a_{13} & a_{14} \\ a_{21} & a_{22} & a_{23} & a_{24} \\ a_{31} & a_{32} & a_{33} & a_{34} \\ a_{41} & a_{42} & a_{43} & a_{44} \end{bmatrix}

In this matrix, each entry a_{ij}​ represents the element in the i-th row and j-th column. The size, written as n \times m, indicates the matrix has n rows and m columns. A matrix can be either square (when n = m) or rectangular.

Matrices allow us to perform various mathematical operations, such as:

• Addition and Subtraction: Possible only when matrices are of the same size.
• Multiplication: Requires the matrices to satisfy specific conditions. For two matrices A (of size n \times m) and (of size m \times p), the multiplication result will be an n \times p matrix.

Example of Matrix Multiplication

Given two matrices A and B:

A = \begin{bmatrix} 1 & 2 & 3 \\ 4 & 5 & 6 \\ 7 & 8 & 9 \end{bmatrix}, \quad B = \begin{bmatrix} 1 & 4 & 7 \\ 2 & 5 & 8 \\ 3 & 6 & 9 \end{bmatrix}

Their product A \times B is:

A \times B = \begin{bmatrix} 14 & 32 & 50 \\ 32 & 77 & 122 \\ 50 & 122 & 194 \end{bmatrix}

Matrices are useful for representing transformations, data grids, and more but are limited to two indices, representing rows and columns. This limitation is where tensors come into play.

Characteristics of a Matrix

• Dimensionality: Always 2D.
• Order: Second-order structure.
• Storage: Each element is accessed by two indices (row and column).
• Usage: Common in linear algebra for solving systems of linear equations, transformations, and data representation (e.g., images, where pixel values are organized in a grid).

Tensor: A Multi-Dimensional Array

Tensors generalize matrices to higher dimensions, allowing us to represent complex structures in multi-dimensional space. Tensors have a rank (or order) indicating their dimensionality:

• 0th Rank: Scalar (a single number)
• 1st Rank: Vector (a 1-D array)
• 2nd Rank: Matrix (a 2-D array)
• nth Rank: n-dimensional array

A tensor, unlike a matrix, adapts to changes in the coordinate system. This adaptability makes tensors crucial for fields like physics and machine learning, where transformations are common. When coordinates shift, tensors transform accordingly to maintain the same representation in a new system, while matrices cannot automatically adapt to such transformations.

Dynamism and Adaptability of Tensors

Unlike a matrix, a tensor adapts to changes in the coordinate system, making it crucial in fields like physics and machine learning, where transformations are common. When coordinates shift, tensors transform accordingly to maintain the same representation in a new system, while matrices cannot automatically adapt.

For example, consider a system with matrices limited to a 2 \times 2 structure (corresponding to two directions in the coordinate system). If a new direction is added, requiring a shift to 3 \times 3, it would be impossible to modify all the existing matrices without reconstruction. In contrast, a tensor-based representation adapts smoothly, allowing all tensors to shift to 3 \times 3 by changing just one tensor in the system. This flexibility is unique to tensors.

Additionally, a matrix has only 2 indices (represented as M[n][m]), whereas a tensor can have any number of indices (T[i_1][i_2][i_3]\ldots) and may even exist as a single number without any index. Therefore, while all rank-2 tensors are matrices, not all matrices qualify as tensors.

Example of a Rank-1 Tensor Transformation

Consider a tensor in a standard Euclidean basis. To switch to a basis of 2, we apply a transformation rule, scaling by the inverse of a scaling matrix. If our scaling matrix S is defined as:

S = \begin{bmatrix} s_x & 0 \\ 0 & s_y \end{bmatrix}

then its inverse S^{-1} is:

S^{-1} = \begin{bmatrix} \frac{1}{s_x} & 0 \\ 0 & \frac{1}{s_y} \end{bmatrix}

When scaled, the tensor maintains its properties in the new system. This operation cannot be achieved with matrices alone due to their fixed form.

Tensors Within Tensors

In a matrix, each entry is a single number. For example:

\begin{bmatrix} 1 & 2 & 3 \\ 4 & 3 & 2 \\ 1 & 2 & 4 \end{bmatrix}

But in a tensor, each entry can itself be another tensor:

\begin{bmatrix} [1,2,3] & [3,2,1] & [1,2,3] \\ [3,3,3] & [2,2,2] & [6,6,6] \\ [5,5,5] & [6,5,4] & [4,5,6] \end{bmatrix}​

We can extend this to even higher ranks. For example, an RGB image can be represented by a 3-dimensional tensor, with three layers of 2D matrices corresponding to the red, green, and blue color channels.

Thus, a tensor can contain matrices, but a matrix cannot contain tensors, as matrices are inherently limited to two dimensions. Tensors are therefore often described as multi-way extensions of matrices.

Tensor Multiplication Example

To see tensor multiplication in action, consider two tensors, T1 and T2:

T1 = \begin{bmatrix} [1,3] & [2,3] \\ [3,2] & [2,2] \end{bmatrix}, \quad T2 = \begin{bmatrix} 5 \\ 6 \end{bmatrix}

The product T1 * T2 is computed as follows:

T1 * T2 = \begin{bmatrix} 5 \times [1,3] + 6 \times [2,3] \\ 5 \times [3,2] + 6 \times [2,2] \end{bmatrix}

Breaking this down:

1. 5 \times [1,3] = [5,15] and 6 \times [2,3] = [12,18]
2. Summing the results: [5,15] + [12,18] = [17,33]

For the second row:

1. 5 \times [3,2] = [15,10] and 6 \times [2,2] = [12,12]
2. Summing these: [15,10] + [12,12] = [27,22]

Thus, the product is:

T1 * T2 = \begin{bmatrix} [17,33] \\ [27,22] \end{bmatrix}

This example shows how tensors enable complex multi-dimensional operations.

Characteristics of a Tensor

• Dimensionality: Can be 1D, 2D, 3D, or higher.
• Order: Defined by the number of dimensions or indices required to access an element. A third-order tensor has three indices, a fourth-order tensor has four, and so on.
• Storage: Each element is accessed by multiple indices, corresponding to the tensor's order.
• Usage: Tensors are widely used in physics (to describe phenomena like stress and strain in materials), deep learning (to store data inputs, weights, and biases), and computer vision (where 3D tensors can represent colored images with RGB values across pixel grids).

Key Differences Between a Matrix and a Tensor

Conclusion

Matrices and tensors are powerful tools, but tensors go beyond matrices by adapting to changes in coordinate systems and allowing for complex, multi-dimensional data representations. All rank-2 tensors are matrices, yet not all matrices are tensors, as tensors offer a flexibility that matrices cannot. This flexibility makes tensors essential in fields that require dynamic, adaptable representations, from machine learning to physics.