Linear combination involves combining a set of vectors by multiplying each vector by a scalar (a real number) and then adding the results together. For example, if you have vectors v1 and v2 and scalars a and b, the expression a × v1 + b × v2 is a linear combination of those vectors.
This concept is not limited to just vectors. Linear combinations can also be applied to functions, polynomials and other mathematical entities.
Linear combinations are used in data science and data analysis in the following ways:
- In prediction models, results are calculated by multiplying features with weights and adding them.
- In techniques like Principal Component Analysis, new variables are created by combining old variables.
- In feature engineering, existing data columns are combined to make better inputs for models.
Mathematical Definition
Given a set of vectors v1, v2, . . . ,vn in a vector space, a linear combination of these vectors is an expression of the form:
w = c_1v_1 + c_2v_2 + . . . + c_nv_n
Where c1, c2, . . . , cn are scalars (real numbers, complex numbers, etc.).
Example of Linear Combination
Consider two vectors in R2:
A linear combination of v1 and v2 would be:
\mathbf{w} = c_1 \mathbf{v}_1 + c_2 \mathbf{v}_2 = c_1 \begin{pmatrix} 1 \\ 2 \end{pmatrix} + c_2 \begin{pmatrix} 3 \\ 4 \end{pmatrix} = \begin{pmatrix} c_1 + 3c_2 \\ 2c_1 + 4c_2 \end{pmatrix}
Properties of Linear Combinations
Some of the common properties of linear combinations are:
- Linearity Property
- Commutative Property
- Associative Property
Let's discuss these properties in detail as follow:
Linearity Property
Scalar multiplication distributes over addition:
a(u+v) = au + av
(a+b)u=au+bu
This ensures scaling and addition behave consistently.
Commutative Property
Order of addition does not matter: v1 + v2 = v2 +v 1
In linear combinations: c1 v1 + c2 v2 = c2 v2 + c1 v1
Associative Property
Grouping does not matter: (v1 + v2) + v3 = v1 + ( v2 + v3)
In linear combinations: (c1 v1 + c2 v2) + c3 v3 = c1 v1 + (c2 v2 + c3 v3)
Real-World Example in Data Science & Data Analysis
Example: House Price Prediction (Linear Regression)
In Linear Regression, the predicted price is a linear combination of features.
\text{Price} = w_1(\text{Area}) + w_2(\text{Bedrooms}) + w_3(\text{Age})
Here:
- Area, Bedrooms, Age -> vectors (features)
- w1, w2, w3 -> scalars (model weights)
This is exactly a linear combination.
So whenever we build a regression model, we are creating weighted sums of input features.
This is the foundation of:
- Multiple Linear Regression
- Neural Networks (weighted sums)
- Feature engineering
How to Form a Linear Combination
To form a linear combination we can use both vectors and matrices. Let's discuss these methods in detail.
Using Vectors
To form a linear combination using vectors, follow these steps:
- Identify the Vectors: Determine the vectors you want to combine. Let's denote them as v1, v2, . . . , vn.
- Choose Scalars: Select the scalars (coefficients) that will multiply each vector. Denote these scalars as c1, c2, . . . , cn.
- Multiply and Add: Multiply each vector by its corresponding scalar and then add the results together.
w = c1 v1 + c2 v2 + . . . + cn vn
Let's consider an example for better understanding:
Given
\mathbf{v}_1 = \begin{pmatrix} 1 \\ 2 \end{pmatrix} ,\mathbf{v}_2 = \begin{pmatrix} 3 \\ 4 \end{pmatrix} and scalars c1 = 2 and c2 = -1The linear combination is:
w = 2 ( 1, 2) + (−1) ( 3, 4)
= (2, 4) + (−3, −4)
= (−1 , 0)
Using Matrices
To form a linear combination using matrices, the process is similar to that of vectors but involves matrix addition and scalar multiplication.
- Identify the Matrices: Determine the matrices you want to combine. Let's denote them as A1, A2 ,…, An.
- Choose Scalars: Select the scalars (coefficients) that will multiply each matrix. Denote these scalars as c1, c2,…, cn.
- Multiply and Add: Multiply each matrix by its corresponding scalar and then add the results together.
A = c1 A1 + c2 A2 + . . . + cn An
Example: Find the linear combination of matrices
Solution:
Given:
A_1 = \begin{pmatrix} 1 & 2 \\ 3 & 4 \end{pmatrix} andA_2 = \begin{pmatrix} 5 & 6 \\ 7 & 8 \end{pmatrix} ,Scalers: c1 = 3 and c2 = −2
Thus,
A = 3 \begin{pmatrix} 1 & 2 \\ 3 & 4 \end{pmatrix} + (-2) \begin{pmatrix} 5 & 6 \\ 7 & 8 \end{pmatrix} First, perform the scalar multiplications:
3 \begin{pmatrix} 1 & 2 \\ 3 & 4 \end{pmatrix} = \begin{pmatrix} 3 & 6 \\ 9 & 12 \end{pmatrix}
-2 \begin{pmatrix} 5 & 6 \\ 7 & 8 \end{pmatrix} = \begin{pmatrix} -10 & -12 \\ -14 & -16 \end{pmatrix} Next, add the resulting matrices:
A = \begin{pmatrix} 3 & 6 \\ 9 & 12 \end{pmatrix} + \begin{pmatrix} -10 & -12 \\ -14 & -16 \end{pmatrix} = \begin{pmatrix} -7 & -6 \\ -5 & -4 \end{pmatrix}