In algebra, you’ve studied linear systems of equations before. A huge part of linear algebra involves the study of linear systems, so let’s review them.

A linear equation is an equation of the form \(a_1 x_1 + a_2 x_2 + \cdots + a_n x_n = b\), where \(a_1\) through \(a_n\) are constant coefficients and \(b\) is a constant. A linear system of equations is a set of linear equations.

When we solve a linear equation, that means finding the values of \(x_1\), \(x_2\), ..., \(x_n\) that makes every equation in a system true at the same time.

The set of all solutions to a linear system of equations is known as its solution set. There are three possibilities for the solution set of a linear system of equations.

Linear systems with one solution

Here’s an example of this type of system:

The only solution to this system of equations is \(x_1 = 4\) and \(x_2 = -1\).

We can visually represent this system of equations as two lines: the first equation can be represented by the line \(2x + 3y = 5\) and the second equation by the line \(x - y = 5\). These two lines intersect at exactly one point, which is our solution.

The lines intersect at one point, so the system has one solution.

Linear systems with infinitely many solutions

Here’s an example of this type of system:

Notice how the second equation is just the first equation multiplied by 2, so these two equations are really asking for the same thing! Therefore, there are infinitely many pairs \((x_1, x_2)\) which satisfy both equations.

Visually, we can represent this system of equations with the lines \(x + 3y = 2\) and \(2x + 6y = 4\). These lines are the exact same, so they have infinitely many intersection points!

The lines overlap, so the system has infinitely many solutions.

Linear systems with no solutions

Here’s an example of this type of system:

There are no possible values of \(x_1\) and \(x_2\) that will make both of these equations simultaneously true.

Visually, we can represent this as two lines \(x + 3y = 2\) and \(2x + 6y = 5\). These lines are parallel to each other but not the same line, so there are no intersection points.

The lines are parallel to each other and never overlap, so the system has no solutions.

Equation operations

When we have a system of equations, there are three operations we can perform on them without changing the solution set.

1. Swapping the order of two equations
2. Multiplying an equation by a nonzero constant
3. Adding a constant multiple of one equation to another equation

Here’s an example with this system of equations. We’re going to perform each equation operation once on this system.

An example of the first operation is swapping the order of the first and second equations to get:

We have swapped the order of the highlighted equations.

An example of the second operation is multiplying the third equation by 2 to get:

We have multiplied the highlighted equation by 2.

An example of the third operation is adding 3 times the second equation to the third equation to get:

We have added 3 times the blue equation to the red equation.

In the section after this one, we will learn how to represent systems of linear equations using vectors and matrices. But let’s first talk about what vectors and matrices even are.

Vectors

A vector is a list of numbers. They can be represented in multiple ways: they can be written out like coordinates (e.g. \((1, 2, 3)\)), or as a column vector, where the numbers are stacked vertically:

Vectors are usually denoted by lowercase letters in bold.

There is a special type of vector known as a zero vector, and it’s a vector that only contains zeros. Here’s an example of a zero vector:

The zero vector is normally denoted by the digit 0 in bold.

I will use the notation \([\mathbf{v}]_i\) to refer to the \(i\)th entry of \(\mathbf{v}\). For example, if \(\mathbf{v} = (2, 3, 4)\), then \([\mathbf{v}]_1 = 2\), \([\mathbf{v}]_2 = 3\), and \([\mathbf{v}]_3 = 4\).

Matrices

A matrix is a 2-dimensional grid of numbers. You can think of them as a group of column vectors stacked side by side. Here’s an example:

A matrix with \(m\) rows and \(n\) columns is known as an \(m \times n\) matrix. For this example, because our matrix has 3 rows and 3 columns, it is a \(3 \times 3\) matrix.

Matrices are usually denoted by uppercase letters.

I will use the notation \([A]_{i, j}\) to refer to the entry at the \(i\)th row and \(j\)th column of \(A\). For example, for the above matrix \(A\), \([A]_{1, 1} = 1\), \([A]_{2, 3} = 6\), and \([A]_{3, 1} = 7\).

In future sections, we will learn how we can use vectors and matrices to describe systems of equations and discover their properties. We will also eventually learn about the operations we can perform on vectors and matrices.

We can represent a system of linear equations using a matrix.

The coefficient matrix

The coefficient matrix is a way to represent the coefficients of a linear system of equations. Each row in the coefficient matrix represents the coefficients in one equation. For example, consider the following system of equations:

The coefficients of each equation are highlighted in red. If we put these coefficients into a matrix, we get the coefficient matrix. The coefficient matrix \(A\) for this system of equations is:

Don’t forget about the signs of each coefficient!

The vector of constants

The vector of constants holds the constants that each linear expression in our system of equations equals. Typically, these are the constants on the right-hand side of each equation. Let’s go back to our system of equations:

This time, I’ve highlighted the constants in this system. Putting these vectors into a column vector gives us the vector of constants. The vector of constants \(\mathbf{b}\) for this system is:

The augmented matrix

If we add another column to the right-hand side of the coefficient matrix and fill it up with the vector of constants, we get the augmented matrix for a system of equations. In this example, the augmented matrix is:

The benefit of using a coefficient matrix is that we no longer have to worry about what our variables are called; it provides a compact way to record all of the information about a system of equations in one neat package.

Now that we know how to represent systems of equations with matrices, how can we use this knowledge to actually solve them? To do this, we need to simplify our systems down into a form that’s simpler. One way to do this is to convert a system’s augmented matrix into a simpler form known as reduced row-echelon form.

A matrix is in reduced row-echelon form (abbreviated RREF) when it meets these conditions:

1. If a row only contains zeros (this is known as a zero row), it is below all rows that aren’t zero rows.
2. The leftmost nonzero number of every row is a 1 (unless the row is a zero row); this 1 is known as a leading 1.
3. If a column has a leading 1, it is the only nonzero number in that column.
4. Consider any two leading 1s in the matrix. If one of the leading 1s is in a lower row than the other, then it must be to the right of the other leading 1.
   • In symbols: Let’s say the first leading 1 is in row \(r_1\) and column \(c_1\) and the second leading 1 is in row \(r_2\) and column \(c_2\). It must always be true that if \(r_2 \gt r_1\), then \(c_2 \gt c_1\).

Here’s a matrix that is not in reduced row-echelon form because it violates condition 1:

The highlighted row is not below all other rows with nonzero terms.

Here’s a matrix that violates condition 2:

The highlighted entry is a leading nonzero term of the first row but is not 1.

Here’s a matrix that violates condition 3:

The leading 1 in the second row is not the only nonzero entry in its column (as shown by the highlighted entries).

And finally, here’s a matrix that violates condition 4:

The blue leading 1 is below the red leading 1, but the blue leading 1 is to the left of the red leading 1. (The row number of the blue leading 1 is greater than the row number of the red leading 1, but the column number of the blue leading 1 is less than the row number of the red leading 1.)

Here are some examples of matrices in reduced row-echelon form:

Notice the staircase-like pattern that the leading 1s make in matrices that are in reduced row-echelon form.

Reduced row-echelon form is useful because once we get an augmented matrix into reduced row-echelon form (you will learn how to do this in the next section), it’s easy to find the solutions to the corresponding system of equations.

For example, the corresponding system of equations to the above matrix is:

The last line simplifies to \(0 = 0\), so it is always true no matter what the values of \(x_1\) and \(x_2\) are. Therefore, we can disregard that equation. The other two lines directly give us the values of \(x_1\) and \(x_2\): \(x_1 = 3\) and \(x_2 = 2\).

For a matrix in reduced row-echelon form, a column with a leading 1 is known as a pivot column. In this example, column 1 and column 2 are pivot columns.

A column with a leading 1 is a pivot column.

I will sometimes use the term “row-reducing” to refer to analyzing the reduced row-echelon form of a matrix (without actually changing the original matrix).

From now on, I will box the leading 1s in every row-reduced matrix for clarity. Here’s an example of what that might look like:

Gauss-Jordan elimination is a systematic way to turn a matrix into reduced row-echelon form. The basic idea is to go through our matrix column by column and perform row operations to turn this matrix into reduced row-echelon form.

Row operations

There are three row operations we can perform on a matrix without changing the solution set of the corresponding system of equations:

1. Swap the order of two rows
2. Multiply every entry in a row by a nonzero constant
3. Add a constant multiple of one row to another row (i.e. multiply every entry of a row by a constant multiple and add it to the entries of another row without changing the original row)

Notice the similarities between the row operations and equation operations mentioned in Intro to Systems of Linear Equations. When one matrix can be transformed into another matrix through these row operations, the matrices are known as row-equivalent.

Let’s go through the process of Gauss-Jordan elimination with the following matrix:

We first need to define some variables to keep track of where we are in the process. We will define the variables \(j\) and \(r\) and set them both to 0. \(j\) will serve as a counter to keep track of what column we’re on. In addition, we’ll define \(m\) as the number of rows in the matrix \(A\) and \(n\) as the number of columns (i.e. \(A\) is an \(m \times n\) matrix).

The first column (\(j = 1\))

We start off by increasing \(j\) by 1. The variable \(j\) is now 1, meaning that we’re working on the first column.

Now we look at the entries of \(A\) in this column (in this case the first column). If all of the entries in this column from row \(r + 1\) to \(m\) are zero, then we skip this column. \(r + 1\) is currently 1 in this case, so we need to look at all of the entries in this column. These entries are not all zero, so we proceed.

Our goal now is to convert column 1 to all zeros, except for one entry which will end up being a 1 (this will be a leading 1 in our final row-reduced matrix). Here’s how we’ll do that:

We choose a row from rows \(r + 1\) to \(m\) such that the entry in column \(j\) is nonzero. We’ll call the index of this row \(i\). In this case, we can choose any of the rows, so I’ll choose row 1.

This is row \(i = 1\), the row we’re focusing on.

Now we increase \(r\) by 1 (after incrementing, \(r\) is 1 now). If \(i\) and \(r\) are different, we swap rows \(i\) and \(r\) of the matrix. In this case, because \(i\) and \(r\) are both 1, we don’t need to do anything here.

Now, we multiply row \(r\) by a constant to make the entry at column \(j\) 1. In this case, the entry at row 1 and column 1 is already 1, so we skip this step.

Because the entry at row \(r\) and column \(j\) is already 1, we don’t do anything. Otherwise, we would multiply row \(r\) by a constant to turn this entry into a 1.

We then add constant multiples of row \(r\) to all other rows to make all other entries of column \(j\) (in this case, the first column) zero.

Adding -2 times row 1 to row 2:

Adding 4 times row 1 to row 3:

After completing each column, I will box that column’s leading 1 if it exists for clarity.

Now we can move on to the next column.

The second column (\(j = 2\))

We add 1 to \(j\), so \(j\) is currently 2. This means that we’re focusing on the second column.

Now we need to choose a row from row \(r + 1\) (which is currently 2) to \(m\) (which is 3) with a nonzero entry in this column (i.e. column \(j = 2\)). I’ll choose row 2 for this example, so \(i = 2\).

Now we increase \(r\) by 1, so it’s currently 2. Because we chose row 2 and \(r = 2\), we don’t need to swap any rows.

Now we need to set the entry at row \(r\) and column \(j\) to a 1 by multiplying row \(r\) by a constant. To do this, we multiply row 2 by \(-1/5\) to turn the entry at row 2 and column 2 to a 1.

The purpose of this row operation is to turn the entry at row \(r\) and column \(j\) into a 1.

Now we have to zero out the other entries of column 2 by adding constant multiples of row 2 to every other row. Let’s start by adding -1 times row 2 to row 1:

Now let’s add -4 times row 2 to row 3:

Now we move on to column 3.

The third column (\(j = 3\))

We need to choose a row from \(r + 1 = 3\) to \(m = 3\). Our only choice in this case is the 3rd row. We then increase \(r\) by 1, so \(r = 3\) now. Therefore, we don’t need to swap any rows.

Now we want the entry at row 3 and column 3 to be a 1, so we multiply row 3 by \(5/41\).

Now we just have to zero out the other entries of column 3. Adding \(-1/5\) times the 3rd row to the 2nd row:

Adding \(-4/5\) times row 3 to row 1:

Our matrix is now in reduced row-echelon form! In this form, we can easily read out the solutions to the corresponding system of equations. Let’s translate this matrix into its corresponding system of equations now:

In this form, we can easily tell that the solution is \(x_1 = 1\), \(x_2 = 3\), and \(x_3 = -2\).

Note that every matrix has only one row-equivalent matrix in reduced row-echelon form.

Another example

Here’s another example of Gauss-Jordan elimination with this matrix:

We start off by initializing our variables: \(r = 0\) and \(j = 0\).

The first column (\(j = 1\))

We need to look at the entries in column \(j = 1\) from \(r + 1\) to \(m\) (the number of rows). Because not all of the entries are zero, we don’t skip this column. We choose a row from rows \(r + 1\) to \(m\) with a nonzero entry in column \(j = 1\). I’ll choose row 1.

Now we add 1 to \(r\). We don’t need to swap rows because the row we picked is row \(r\).

We also don’t need to multiply row 1 by a constant because the entry at row \(r = 1\) and column \(j = 1\) is already 1. Now we can zero out the other entries of column \(j = 1\).

Adding -2 times row 1 to row 2:

Adding -3 times row 1 to row 3:

The second column (\(j = 2\))

We now look at the entries in column 2 from rows \(r + 1 = 2\) to \(m = 3\).

Our only choice here is row 3, so we set the variable \(i\) to 3.

We need to add 1 to \(r\) now, so \(r = 2\). Because \(r\) is not equal to \(i\), we need to swap rows \(r\) and \(i\):

The matrix after swapping rows \(r = 2\) and \(i = 3\).

Now we convert the entry at row \(r = 2\) and column \(j = 2\) to a 1 by multiplying row 2 by -1:

Finally, we zero out the entry at row 1 and column 2 by adding -2 times row 2 to row 1:

This matrix is in reduced row-echelon form now! Here’s what it looks like when we translate this matrix back into a system of equations:

After some simplification, this system becomes:

We can disregard the \(0 = 0\) equation because it will always be true. Let’s rearrange the other two equations to solve for \(x_1\) and \(x_2\):

Therefore, any solution to our original system of equations is of the form \((x_1, x_2, x_3) = (6x_3, -5x_3, x_3)\).

Some systems of equations have no solutions, and some systems of equations have one or infinitely many solutions. When a system of equations has at least one solution, we call it a consistent system. A system of equations with no solutions is an inconsistent system.

A simple example of a consistent system is the following:

This system has a solution \(x_1 = 2\) and \(x_2 = 1\). (Note: Consistent systems can also have infinitely many solutions.)

A simple example of an inconsistent system is:

This system has no solution, so it is inconsistent.

An example of a system with infinitely many solutions

Consider the following augmented matrix:

When we convert this matrix into reduced row-echelon form, we get:

The corresponding system of equations is:

We can write this system more simply as:

The last equation \(0 = 0\) is always true, so we can disregard it. But notice what happens with the other two equations. We can rewrite them as follows:

Notice how when we write the solutions in this way, \(x_3\) is free to take on any value, and the values of \(x_1\) and \(x_2\) depend on \(x_3\). Because \(x_3\) can take on any value, there are infinitely many solutions.

We can describe the solution set of this system as all ordered pairs of the form \((x_1, x_2, x_3) = (2 - x_3, x_3, x_3)\) where \(x_3\) is any real number (or even any complex number). Typically, I will write the solution set as a column vector, like this:

Dependent and free variables

If we have \(A\), the augmented matrix of a system of equations, and \(B\), a row-equivalent matrix in reduced row-echelon form, then if column \(j\) of \(B\) is a pivot column, then the variable \(x_j\) is known as a dependent variable. All other variables are known as free variables.

Let’s look back at our previous example. Here is the matrix in reduced row-echelon form:

Notice that columns 1 and 2 are pivot columns. This means that in the corresponding system of equations, the variables \(x_1\) and \(x_2\) are dependent, and \(x_3\) is free. This is related to how \(x_3\) was able to take on any value in the solution of our system of equations, while the values of \(x_1\) and \(x_2\) depended on the value of \(x_3\).

Determining consistency of systems

We can tell if a system is consistent by looking at the reduced row-echelon form of its corresponding augmented matrix. If this row-reduced matrix has a pivot column at column \(n + 1\), where \(n\) is the number of variables in the system, then the system is inconsistent (i.e. the system is inconsistent if the last column of the row-reduced matrix is a pivot column). Otherwise it is consistent.

For example, consider this system:

The augmented matrix of this system row-reduces to:

Because column \(n + 1 = 4\) of this matrix is a pivot column (as indicated by the leading 1 highlighted in red), this system is inconsistent. (Notice how it is impossible for the equations \(x_1 + x_2 + x_3 = 1\) and \(2x_1 + 2x_2 + 2x_3 = 3\) to be true at the same time! In addition, notice how the last row of the row-reduced matrix translates to \(0 = 1\), a false statement.)

Determining the number of solutions of a system of equations

If the row-reduced matrix of a consistent system has \(r\) pivot columns and \(n\) variables, it is guaranteed that \(r \le n\) (i.e. a system cannot have more pivot columns than variables). If \(r \lt n\), then the system has infinitely many solutions, and if \(r = n\), then the system has exactly one solution.

A consistent system with at least one free variable has infinitely many solutions (because there are infinitely many choices for the free variables), and a consistent system with no free variables has exactly one solution.

If a consistent system has more variables than it has equations, then the system has infinitely many solutions. Here’s an example:

We can tell this system has infinitely many solutions because:

1. This system is consistent: \(x_1 = 0\), \(x_2 = 0\), and \(x_3 = 0\) is a solution.
2. There are more variables than equations, since this system has 3 variables and 2 equations.

In conclusion, there are three possibilities for a linear system with \(n\) variables with augmented matrix \(A\):

• If column \(n + 1\) (the right-most column) of the reduced row-echelon form of \(A\) is a pivot column, the system is inconsistent and has no solutions.
• Otherwise:
  • If the reduced row-echelon form of \(A\) has the same number of pivot columns as it has variables (i.e. \(r = n\)), the system has one solution.
  • If the reduced row-echelon form of \(A\) has fewer pivot columns than it has variables (i.e. \(r \lt n\)), the system has infinitely many solutions.

Counting free variables

We can tell how many free variables a consistent system has by looking at its corresponding matrix in reduced row-echelon form. If this system has \(n\) variables and the matrix in reduced row-echelon form has \(r\) nonzero rows (rows that don’t only contain zeros), then we can describe the solutions of the system with \(n - r\) free variables. Note that \(r\) is also the number of pivot columns and the number of leading 1s.

Going back to our first example, the matrix in reduced row-echelon form has two nonzero rows, so \(r = 2\). The system of equations has 3 variables \(x_1\), \(x_2\), and \(x_3\), so \(n = 3\). Therefore, the system of equations has \(n - r = 3 - 2 = 1\) free variable.

This matrix in reduced row-echelon form has 2 pivot columns (as indicated by the two leading 1s). Since the system has 3 variables, there is \(3 - 2 = 1\) free variable in the system.

To contrast this example, let’s say a system has augmented matrix \(A\) and the reduced row-echelon form of \(A\) is:

In this case, the system has 3 variables while the reduced matrix has 3 pivot columns, so there are \(3 - 3 = 0\) free variables.

A homogeneous system of equations is a special type of linear system of equations where all of the constants are zero.

Here’s an example of a homogeneous system:

Notice how the constants (highlighted in red) are zero.

Homogeneous systems are always consistent: you can always find a solution to a homogeneous system by setting all of the variables to zero (this is known as the trivial solution).

In this case, setting \(x_1\), \(x_2\), and \(x_3\) all to zero results in all three equations becoming \(0 = 0\).

Are there any other solutions to this system? We can find that out by row-reducing the augmented matrix to get:

Converting this back into a system of equations, we have:

Therefore, the trivial solution to this system is also the only solution.

If a homogeneous system has more variables than equations, it has infinitely many solutions (this is because homogeneous systems are always consistent and a consistent system with more variables than equations has infinitely many solutions).

Null spaces of matrices

The null space of a matrix \(A\) is the solution set of the system of equations with \(A\) as its coefficient matrix and the zero vector as its vector of constants (i.e. all of the constants are zero).

Let’s look at the coefficient matrix for our previous system of equations:

The null space of this matrix is the solution set to our previous system of equations. In this case, the null space only contains the zero vector, since that’s the only solution to our system of equations.

Now let’s look at another matrix:

The full augmented matrix for the homogeneous system of equations for this matrix is:

Row-reducing this matrix results in:

Because there are 2 pivot columns and 3 variables, there is \(3 - 2 = 1\) free variable. Therefore, there are infinitely many solutions to the system. As a result, the null space of our original matrix \(A\) has infinitely many elements, with each element being a solution to the system.

Before we talk about singular matrices, let’s define some special types of matrices.

Square matrices

A square matrix is a matrix with the same number of rows and columns.

Here’s an example of a square matrix:

This matrix has 3 rows and 3 columns, so it is square.

Here’s an example of a non-square matrix:

This matrix has 3 rows and 4 columns, so it is not square.

Identity matrices

An identity matrix is a square matrix with all 1s on the main diagonal and 0s everywhere else. More formally, the \(n \times n\) identity matrix, denoted by \(I_n\), is defined by:

In simple words, the entry at row \(i\) and column \(j\) is 1 if \(i\) and \(j\) are equal and 0 otherwise.

Here are some examples of identity matrices:

Singular matrices

A square matrix \(A\) is singular if the system of equations with \(A\) as its coefficient matrix and all-zero constants has infinitely many solutions (equivalently, the system has non-trivial solutions). If the system of equations has only the trivial solution, the matrix \(A\) is nonsingular.

In the previous example, we looked at this matrix:

We found that the only solution to the corresponding homogeneous system of equations was the trivial solution where all of the variables were set to zero. Therefore, this matrix is nonsingular.

This is the corresponding homogeneous system. The system only has the trivial solution, so the coefficient matrix is nonsingular.

We also looked at this matrix:

We found that the corresponding homogeneous system had infinitely many solutions, so this matrix is singular.

This is the corresponding homogeneous system. The system has infinitely many solutions, so the coefficient matrix is singular.

An interesting fact about nonsingular matrices is that reducing any nonsingular matrix to reduced row-echelon form always results in an identity matrix. More formally, a matrix is nonsingular if and only if the matrix in reduced row-echelon form is an identity matrix.

Here are the reduced row-echelon forms of the previous two matrices:

This matrix is nonsingular, so it row-reduces to an identity matrix.

This matrix is singular, so it does not row-reduce to an identity matrix.

The null space of a nonsingular matrix contains just one element: the zero vector. A square matrix \(A\) is nonsingular if and only if its null space only contains the zero vector.

In addition, a square matrix \(A\) is nonsingular if and only if the system of equations with coefficient matrix \(A\) and vector of constants \(\mathbf{b}\) has a single unique solution for any possible choice for \(\mathbf{b}\).

To summarize, for any matrix \(A\), the following properties are equivalent (meaning for every property, all other properties are true if and only if that property is true):

1. \(A\) is a nonsingular matrix.
2. The reduced row-echelon form of \(A\) is equal to the \(n \times n\) identity matrix.
3. The null space of \(A\) contains only the zero vector.
4. The linear system of equations with coefficient matrix \(A\) and vector of constants \(\mathbf{b}\) has a single unique solution for every possible choice of \(\mathbf{b}\).

The concept of a nonsingular matrix is one that will appear many times throughout linear algebra. So far, we have collected a few conditions that are equivalent to a matrix being nonsingular.

If \(A\) is an \(n \times n\) square matrix, these conditions are equivalent:

1. \(A\) is a nonsingular matrix.
2. The reduced row-echelon form of \(A\) is equal to the \(n \times n\) identity matrix.
3. The null space of \(A\) contains only the zero vector.
4. The linear system of equations with coefficient matrix \(A\) and vector of constants \(\mathbf{b}\) has a single unique solution for every possible choice of \(\mathbf{b}\).

In future units, I will add more equivalences to this list.

Now we will study vectors in more detail. To start, here are some of the operations we can perform involving vectors. (Specifically, these are properties for column vectors. We will study other types of “vectors” later on.)

Vector equality

Two vectors of the same size \(\mathbf{u}\) and \(\mathbf{v}\) are equal if each entry in \(\mathbf{u}\) equals its corresponding entry in \(\mathbf{v}\). More formally, if \(\mathbf{u}\) and \(\mathbf{v}\) have size \(m\), then \(\mathbf{u} = \mathbf{v}\) if:

This definition of vector equality allows us to compactly describe systems of equations as an equality of two vectors. For example, consider the following system:

We can write this system as follows:

Vector addition/subtraction

We can add two vectors of the same size by adding the corresponding entries. If \(\mathbf{u}\) and \(\mathbf{v}\) are of size \(m\):

Similarly, we can subtract two vectors by subtracting their corresponding entries. If \(\mathbf{u}\) and \(\mathbf{v}\) are of size \(m\):

Scalar-vector multiplication

We can multiply a vector by a scalar by multiplying each of the vector’s components by the scalar. If \(\alpha\) is a scalar and \(\mathbf{v}\) is a vector of size \(m\):

Examples of column vector operations

Column vector properties

Here are some properties of column vectors.

• If \(\mathbf{u}\) and \(\mathbf{v}\) are column vectors of size \(m\), then \(\mathbf{u} + \mathbf{v}\) is a column vector of size \(m\).
• If \(\alpha\) is a scalar (any complex number) and \(\mathbf{u}\) is a column vector, then \(\alpha \mathbf{u}\) is a column vector.
• If \(\mathbf{u}\) and \(\mathbf{v}\) are column vectors, then \(\mathbf{u} + \mathbf{v} = \mathbf{v} + \mathbf{u}\).
• If \(\mathbf{u}\), \(\mathbf{v}\), and \(\mathbf{w}\) are column vectors, then \(\mathbf{u} + (\mathbf{v} + \mathbf{w}) = (\mathbf{u} + \mathbf{v}) + \mathbf{w}\).
• There exists a vector \(\mathbf{0}\) known as the zero vector such that \(\mathbf{u} + \mathbf{0} = \mathbf{u}\) for all possible column vectors \(\mathbf{u}\).
• For every possible column vector \(\mathbf{u}\), there exists a vector \(-\mathbf{u}\) such that \(\mathbf{u} + (-\mathbf{u}) = \mathbf{0}\).
• If \(\alpha\) and \(\beta\) are scalars and \(\mathbf{u}\) is a column vector, then \(\alpha(\beta \mathbf{u}) = (\alpha \beta) \mathbf{u}\).
• If \(\alpha\) is a scalar and \(\mathbf{u}\) and \(\mathbf{v}\) are column vectors, then \(\alpha(\mathbf{u} + \mathbf{v}) = \alpha \mathbf{u} + \alpha \mathbf{v}\).
• If \(\alpha\) and \(\beta\) are scalars and \(\mathbf{u}\) is a column vector, then \((\alpha + \beta)\mathbf{u} = \alpha \mathbf{u} + \beta \mathbf{u}\).
• For all column vectors \(\mathbf{u}\), \(1\mathbf{u} = \mathbf{u}\).

Before we continue talking about vectors, let’s take a moment to review sets, which we will be using a lot in linear algebra.

A set is a collection of objects, which are called elements. Typically these objects will be numbers, but there’s no restriction on the type of objects that a set can contain (they can contain vectors, matrices, or anything else you can think of). Sets can contain a finite or infinite number of elements.

The notation for sets involves curly brackets. For example, the set containing the numbers 1 and 2 can be written as \(\{1, 2\}\), and the set containing the letters A, B, and C can be written as \(\{\text{A}, \text{B}, \text{C}\}\).

Sets are unordered, meaning the order of elements does not matter, e.g. \(\{1, 2, 3\}\) and \(\{3, 1, 2\}\) are the same set.

Sets are typically denoted using capital letters, like \(S\). If an object \(x\) is in the set \(S\), we write \(x \in S\), and if \(x\) is not in \(S\), we write \(x \notin S\).

Subsets and set equality

A subset \(T\) of a set \(S\) is a set such that every element in \(T\) is also in \(S\). That is, if \(x \in T\), then \(x \in S\). This relationship is denoted \(T \subseteq S\).

Note that the definition of a subset allows \(S\) and \(T\) to be equal to each other. A proper subset \(T\) of a set \(S\) is a subset of \(S\) that is not equal to \(S\), denoted by \(T \subset S\). That is, if \(T \subseteq S\) and \(T \neq S\), then \(T \subset S\).

We still haven’t formally defined set equality yet. Two sets \(S\) and \(T\) are equal, denoted by \(S = T\), if \(S\) and \(T\) are subsets of each other; that is, \(S \subseteq T\) and \(T \subseteq S\).

Common sets of numbers

Some sets of numbers are used so often that they get their own symbols:

• \(\mathbb{Z}\): set of integers
• \(\mathbb{Q}\): set of rational numbers
• \(\mathbb{R}\): set of real numbers
• \(\mathbb{C}\): set of complex numbers

Defining sets

We can define a set by listing its elements, e.g. \(S = \{2, 4, 8, 16\}\), but we also could define a set by specifying a condition for an element to be in the set. For example:

The vertical bar means “such that”, so \(S\) is the set of all even integers. Sometimes a colon instead of a vertical bar is used to mean “such that”.

\(T\) contains all column vectors of size 2 whose entries sum to 1.

Now that we’ve defined vector operations, we can create linear combinations of vectors. A linear combination of the scalars \(\alpha_1, \alpha_2, ..., \alpha_n\) and the vectors \(\mathbf{u}_1, \mathbf{u}_2, ..., \mathbf{u}_n\) is:

We can represent systems of linear equations as linear combinations. For example, consider this system:

We can use vector equality to write these three equations as a single equation:

We can write the left-hand side as a sum of vectors:

And we can do some factoring:

Now we have written our system of equations as an equality of a linear combination of the scalars \(x_1\), \(x_2\), and \(x_3\) and the columns of the system’s coefficient matrix.

We previously found that a solution to this system of equations is \(x_1 = 1\), \(x_2 = 3\), and \(x_3 = -2\). We can check this by plugging in these values:

The importance of writing systems as linear combinations is that it gives us a different perspective of what it means to solve a linear system of equations.

Solutions to linear systems as linear combinations

Consider a system of equations with an \(m \times n \) coefficient matrix \(A\) and vector of constants \(\mathbf{b}\). If \(\mathbf{A}_1, \mathbf{A}_2, ..., \mathbf{A}_n\) are the columns of \(A\), then the vector \(\mathbf{x}\) is a solution the the system if and only if:

We can use this to write all of the solutions of a linear system of equations as a linear combination. For example, consider this system:

Let’s write the corresponding augmented matrix in reduced row-echelon form:

Converting this matrix back into a system of equations, we get:

We can disregard the \(0 = 0\) equation, and solve for \(x_1\) and \(x_2\) in terms of the free variables \(x_3\) and \(x_4\) to get:

We can write this in vector form as:

This is one way we can write the solutions to this system. We can also decompose the right-hand side as follows:

This is a compact way to describe all of the solutions of this linear system. We can choose any values of \(x_3\) and \(x_4\) and this equation will give us a solution. For example, here’s what we get when \(x_3 = 1\) and \(x_4 = 1\):

And here’s what we get when \(x_3 = 5\) and \(x_4 = -10\):

With the power of linear combinations, we can use another technique to find solutions to linear systems. If we know one of the solutions of a consistent system, we can find all of the other solutions. Here’s how:

If \(\mathbf{w}\) is the solution to a system of equations with coefficient matrix \(A\) and vector of constants \(\mathbf{b}\), then \(\mathbf{w} + \mathbf{z}\) is also a solution if and only if \(\mathbf{z}\) is in the null space of \(A\).

Let’s say I want to find all of the solutions to the following system of equations:

And let’s say I gave you one of the solutions to this system already: \(x_1 = -1\), \(x_2 = 5\), and \(x_3 = 0\).

We can use this theorem to find the other solutions. To do so, we first find the null space of the coefficient matrix. The first step is to row-reduce the coefficient matrix:

Now let’s create a homogeneous system using this reduced matrix.

From here, we can write all of the solutions of this system as linear combinations:

Writing this as a vector equality:

Every element of the null space of the coefficient matrix is of this form. Therefore, we can write every solution of our initial system of equations as:

Now that we can write solution sets as linear combinations, I’m going to introduce another concept: the span of a set of vectors.

The span \(\langle S \rangle\) of the \(p\) vectors \(S = \{\mathbf{u}_1, \mathbf{u}_2, ..., \mathbf{u}_p \}\) is the set of possible linear combinations of the \(\mathbf{u}_1, \mathbf{u}_2, ..., \mathbf{u}_p\). More formally, for every possible choice of scalars \(\alpha_1, \alpha_2, ..., \alpha_p\), the span contains the following linear combination:

For example, consider this set of vectors:

The span of \(S\) consists of all possible linear combinations of these vectors. Here are some of the elements of \(S\):

How can we test whether a specific element is in this span? For example, let’s say we wanted to know if these vectors are in the span:

Let’s focus on \(\mathbf{u}\) first. If \(\mathbf{u}\) is in the span, this means that there are scalars \(x_1\), \(x_2\), and \(x_3\) such that:

We want to find a set of values for \(x_1\), \(x_2\), and \(x_3\) that will satisfy this equality. Let’s see what happens when we expand the left-hand side:

It turns out that this is just a system of equations in disguise! So now we just need to see if there are solutions to this system of equations. We can do this by row-reducing the augmented matrix:

Because the last column is not a pivot column, this system is consistent. Therefore, our original vector \(\mathbf{u}\) is in the span of \(S\).

Now let’s test if \(\mathbf{v}\) is in the span of \(S\). As a reminder, this is what \(\mathbf{v}\) looks like:

Asking whether \(\mathbf{v}\) is in the span is equivalent to asking whether the following system of equations has a solution:

To test if there are solutions, we row-reduce the augmented matrix:

Because the last column is a pivot column, the system is inconsistent, and so \(\mathbf{v}\) is not part of the span of \(S\).

Writing null spaces as spans

We can write the null spaces of matrices as spans. For example, consider the following matrix:

We want to consider the system of equations with \(A\) as its coefficient matrix and the vector of constants \(\mathbf{0}\). The solutions to this system make up the null space of \(A\).

To find these solutions, we first row-reduce the augmented matrix:

Writing this reduced matrix as a system of equations gives us:

Rearranging to solve for the dependent variables \(x_1\) and \(x_2\) in terms of the free variables \(x_3\) and \(x_4\):

We can write the solutions in vector form as:

We can write every solution as a linear combination of these two vectors. Therefore, the solution set (which is the null space of \(A\)) can also be described as the span of these two vectors. The null space of \(A\) is thus:

Let’s say we’re finding the solutions to a homogeneous system of equations with coefficient matrix \(A\). If we label the columns of \(A\) as \(\mathbf{u}_1, \mathbf{u}_2, ..., \mathbf{u}_n\), then solving this system of equations is equivalent to finding coefficients \(\alpha_1, \alpha_2, ..., \alpha_n\) such that:

A true equation of this form is known as a relation of linear dependence on the set of vectors \(\{\mathbf{u}_1, \mathbf{u}_2, ..., \mathbf{u}_n\}\).

Here’s an example of a relation of linear dependence:

Note that for any set of vectors \(\{\mathbf{u}_1, \mathbf{u}_2, ..., \mathbf{u}_n\}\), the above equation can be trivially be made true by setting the coefficients \(\alpha_1, \alpha_2, ..., \alpha_n\) all to zero. This is known as a trivial relation of linear dependence.

Linearly independent sets of vectors

A set of vectors \(S = \{\mathbf{u}_1, \mathbf{u}_2, ..., \mathbf{u}_n\}\) is linearly independent if the only relation of linear dependence on \(S\) is trivial. If there are non-trivial relations of linear dependence on \(S\), then \(S\) is linearly dependent.

Example 1

For example, consider this set of vectors:

Is this set of vectors linearly independent? To find out, we need to determine if there is a non-trivial relation of linear dependence on \(S\). This is equivalent to finding values of \(\alpha_1\), \(\alpha_2\), and \(\alpha_3\) such that:

Notice that this is the same as solving the following system of equations:

To solve this system, we row-reduce the augmented matrix:

This row-reduced matrix tells us that the only solution to the system is \(\alpha_1 = 0\), \(\alpha_2 = 0\), and \(\alpha_3 = 0\). Therefore, the only relation of linear dependence on \(S\) is the trivial one, and so \(S\) is linearly independent.

Notice how the columns of the augmented matrix we used here are the vectors of \(S\) with a zero vector added on as the rightmost column.

In addition, notice that the rightmost column of zeros does not change when row-reducing the matrix. Therefore, we don’t actually need to keep track of this column.

Using these observations, we can create a faster way to determine if a set of vectors are linearly independent.

Example 2

Let’s say we want to determine if these vectors are linearly independent:

We first create a matrix with the vectors of \(S\) as columns:

Now we row-reduce this matrix:

Because this row-reduced matrix has less pivot columns (2 columns in this case) than its corresponding homogeneous system has variables (3 variables in this case, one for each column in the matrix), the system has infinitely many solutions.

Therefore, because there can only be at most one trivial relation of linear dependence on \(S\), there must be non-trivial relations of linear dependence on \(S\), meaning that \(S\) is linearly dependent.

To summarize, if \(S\) is a set of vectors and \(A\) is a matrix with columns that are the vectors in \(S\), then \(S\) is linearly independent if and only if the homogeneous system of equations with coefficient matrix \(A\) has a unique solution (the trivial solution).

As a consequence, the columns of a square matrix \(A\) are linearly independent if and only if \(A\) is nonsingular.

Looking back at the row-reduced matrix, we were able to deduce that there are infinitely many solutions because the number of pivot columns was less than the number of total columns.

This gives us a simple test to tell if a set of vectors \(S\) is linearly dependent: first create a matrix, with each column being one of the vectors in \(S\). Then convert this matrix to reduced row-echelon form, and count the number of pivot columns.

• If there are fewer pivot columns than total columns, then \(S\) is linearly dependent (because the homogeneous system corresponding to the matrix has infinitely many solutions).
• If there are the same number of pivot columns as total columns, then \(S\) is linearly independent (because the homogeneous system corresponding to the matrix has a unique solution).

If we have a set \(S\) of vectors and there are more vectors in the set than entries in each vector, then the set is linearly dependent. This is because when we create the matrix from the vectors in \(S\), there will be more columns than rows, and so there must be less pivot columns than total columns (as each row can only have one leading 1).

Example 3

For example, consider this set of vectors:

Without having to do any row-reducing, we can notice that \(S\) contains 4 vectors and each vector has 3 entries. Because the number of vectors is greater than the number of entries in each vector, we can immediately tell that \(S\) is linearly dependent.

Here’s what the matrix formed from the vectors in \(S\) looks like:

If we were to row-reduce the matrix, because this matrix has 3 rows, there can be at most 3 pivot columns. However, there are 4 columns in total, so the number of pivot columns must be less than the number of total columns, meaning that the set \(S\) must be linearly dependent.

An interesting property of linearly dependent sets of vectors is that in any linearly dependent set, at least one of the vectors can be written as a linear combination of some of the others.

We can use this fact to describe spans more efficiently. If we have a span described with a linearly dependent set of vectors, we can rewrite the same span using one less vector.

For example, let’s look at this set of vectors:

We found previously that this set is linearly dependent. This means there is a relation of the form:

We can find a solution by row-reducing the augmented matrix for this system:

This reduced matrix corresponds to:

By choosing \(x_3 = 1\), we get the solution \(x_1 = -1\), \(x_2 = -1\), and \(x_3 = 1\), meaning that we can write the following relation of linear dependence:

By solving for \(\mathbf{v}_3\), we can write \(\mathbf{v}_3\) as a linear combination of \(\mathbf{v}_1\) and \(\mathbf{v}_2\):

By the definition of a span, any member \(\mathbf{u}\) of the span of \(S\) can be written as a linear combination as follows:

Using our equation for \(\mathbf{v}_3\), any linear combination of this form can be rewritten as:

We have successfully written an arbitrary vector \(\mathbf{u}\) in the span of \(S\) as a linear combination of the two vectors \(\mathbf{v}_1\) and \(\mathbf{v}_2\). Therefore, because any vector in the span of \(S\) can be written as a linear combination of \(\mathbf{v}_1\) and \(\mathbf{v}_2\), the span of \(S\) can be written with just two vectors:

The reason we were able to reduce this spanning set down from three to two vectors is because our original spanning set \(S\) was linearly dependent. Because of this, we were able to write one of the vectors \(\mathbf{v}_3\) as a linear combination of the other two vectors, allowing us to reduce the spanning set.

A quicker way to reduce spanning sets

There is a more systematic way to reduce spanning sets. Let’s start with this span:

We will put these vectors into a matrix and row-reduce:

The pivot columns of the row-reduced matrix are the 1st and 3rd column. So let’s take the 1st and 3rd columns of the original matrix:

The span of this set is the same as the span of the set we started off with!

Note: This section is incomplete.

Complex conjugates of vectors

The complex conjugate of a complex number \(\alpha\) is the number obtained by swapping the sign of the imaginary part of \(\alpha\). For example, the conjugate of \(5 + 2i\) is \(5 - 2i\). The conjugate of \(\alpha\) is denoted by \(\overline{\alpha}\).

We can find the complex conjugate of a vector by conjugating every entry. More formally, the conjugate of a vector \(\mathbf{u}\), denoted by \(\overline{\mathbf{u}}\), is defined by \([\overline{\mathbf{u}}]_i = \overline{[\mathbf{u}]_i}\) for \(1 \le i \le m\), where \(m\) is the size of \(\mathbf{u}\).

Properties of complex conjugates

• If \(\mathbf{x}\) and \(\mathbf{y}\) are vectors, then \(\overline{\mathbf{x} + \mathbf{y}} = \overline{\mathbf{x}} + \overline{\mathbf{y}}\).
• If \(\alpha\) is a scalar and \(\mathbf{x}\) is a vector, then \(\overline{\alpha\mathbf{x}} = (\overline{\alpha})(\overline{\mathbf{x}})\).

The inner product

The inner product of two vectors \(\mathbf{u}\) and \(\mathbf{v}\) of size \(m\) is defined as:

Here’s an example of an inner product:

Note that when both vectors have only real entries, the inner product is equivalent to the dot product.

Properties of inner products

• \(\langle \mathbf{u} + \mathbf{v}, \mathbf{w} \rangle = \langle \mathbf{u}, \mathbf{w} \rangle + \langle \mathbf{v}, \mathbf{w} \rangle\)
• \(\langle \mathbf{u}, \mathbf{v} + \mathbf{w} \rangle = \langle \mathbf{u}, \mathbf{v} \rangle + \langle \mathbf{u}, \mathbf{w} \rangle\)
• \(\langle \alpha\mathbf{u}, \mathbf{v} \rangle = \overline{\alpha} \langle \mathbf{u}, \mathbf{v} \rangle\)
• \(\langle \mathbf{u}, \alpha\mathbf{v} \rangle = \alpha \langle \mathbf{u}, \mathbf{v} \rangle\)
• \(\langle \mathbf{u}, \alpha\mathbf{v} \rangle = \overline{\langle \mathbf{v}, \mathbf{u} \rangle}\)

Norms of vectors

For a vector \(\mathbf{u}\) of size \(m\), its norm is calculated as follows:

If \(\mathbf{u}\) only contains real number entries, the norm of \(\mathbf{u}\) is equal to its magnitude: the length of the vector when plotted on the coordinate plane.

Note that because the entries of \(\mathbf{u}\) can be complex, we need to take the absolute value of each entry in order to assure that the norm is never negative.

One interesting property about the norm is that \(||\mathbf{u}||^2 = \langle \mathbf{u}, \mathbf{u} \rangle\). In addition, the inner product \(\langle \mathbf{u}, \mathbf{u} \rangle\) is always greater than or equal to zero, and is equal to zero if and only if \(\mathbf{u}\) is the zero vector.

Orthogonal vectors

Two vectors \(\mathbf{u}\) and \(\mathbf{v}\) are orthogonal if the inner product \(\langle \mathbf{u}, \mathbf{v} \rangle\) equals 0.

An orthogonal set of vectors is a set of vectors where any two distinct vectors from that set will always be orthogonal. More formally, a set \(S = \{\mathbf{u}_1, \mathbf{u}_2, ..., \mathbf{u}_n\}\) is orthogonal if \(\langle \mathbf{u}_i, \mathbf{u}_j \rangle = 0\) whenever \(i \ne j\).

With our discussion of linearly independent sets, we can add another equivalence to our list of nonsingular matrix equivalences.

If \(A\) is an \(n \times n\) square matrix, these conditions are equivalent:

1. \(A\) is a nonsingular matrix.
2. The reduced row-echelon form of \(A\) is equal to the \(n \times n\) identity matrix.
3. The null space of \(A\) contains only the zero vector.
4. The linear system of equations with coefficient matrix \(A\) and vector of constants \(\mathbf{b}\) has a single unique solution for every possible choice of \(\mathbf{b}\).
5. The columns of \(A\) are linearly independent.

There are some operations we can perform on matrices.

Matrix equality

Two matrices are equal if they are the same size and if every corresponding entry is equal. More formally, the \(m \times n\) matrices \(A\) and \(B\) are equal if \([A]_{i,j} = [B]_{i,j}\) for \(1 \le i \le m\) and \(1 \le j \le n\).

Matrix addition

We can add two matrices by adding their corresponding entries. For example:

Here’s the formal definition of the sum of two \(m \times n \) matrices \(A\) and \(B\):

Multiplying matrices by scalars

We can multiply a matrix by a scalar by multiplying the individual entries by that scalar. For example:

Here’s the formal definition of scalar multiplication for an \(m \times n\) matrix \(A\):

Matrix properties

• If \(A\) and \(B\) are \(m \times n\) matrices, then \(A+B\) is an \(m \times n\) matrix.
• If \(\alpha\) is a scalar and \(A\) is an \(m \times n\) matrix, then \(\alpha A\) is an \(m \times n\) matrix.
• If \(A\) and \(B\) are matrices, then \(A + B = B + A\).
• If \(A\), \(B\), and \(C\) are matrices, then \(A + (B + C) = (A + B) + C\).
• There exists a matrix \(\mathrm{o}\) known as the zero matrix such that \(A + \mathrm{o} = A\) for all matrices \(A\).
• For every possible matrix \(\mathrm{A}\), there exists a matrix \(-A\) such that \(A + (-A) = \mathrm{o}\).
• If \(\alpha\) and \(\beta\) are scalars and \(A\) is a matrix, then \(\alpha(\beta A) = (\alpha \beta)A\).
• If \(\alpha\) is a scalar and \(A\) and \(B\) are matrices, then \(\alpha(A + B) = \alpha A + \alpha B\).
• If \(\alpha\) and \(\beta\) are scalars and \(A\) is a matrix, then \((\alpha + \beta)A = \alpha A + \beta A\).
• For all matrices \(A\), \(1A = A\).

Notice the similarities between these matrix properties and the vector properties mentioned in Vector Operations.

In addition, the zero matrix \(\mathrm{o}\) is simply a matrix filled with only zeros.

Matrix transposes

The transpose of a matrix \(A\), denoted by \(A^t\), is defined as the matrix formed by swapping \(A\)’s rows and columns. More formally, if \(A\) is an \(m \times n\) matrix:

Here’s an example:

You can also view transposing a matrix as flipping the entries across the main diagonal (the diagonal that starts in the upper-left corner and goes towards the lower-right corner).

Symmetric matrices

A matrix is known as symmetric if it equals its transpose. Here’s an example of a symmetric matrix:

Note that symmetric matrices must also be square matrices. A non-square matrix cannot be symmetric because transposing it will change its dimensions (for example, the transpose of a \(4 \times 3\) matrix is a \(3 \times 4\) matrix).

Properties of matrix transposes

• \((A+B)^t = A^t + B^t\) for any matrices \(A\) and \(B\)
• \((\alpha A)^t = \alpha A^t\) for any scalar \(\alpha\) and matrix \(A\)
• \((A^t)^t = A\) for any matrix \(A\)

Complex conjugates

As a reminder, the complex conjugate of a complex number \(\alpha\), denoted by \(\overline{\alpha}\), is the number obtained by switching the sign of the imaginary part. For example, the complex conjugate of \(2 + 4i\) is \(2 - 4i\).

We can find the complex conjugate of a matrix \(A\), denoted by \(\overline{A}\), by calculating the complex conjugate of every entry. More formally, for an \(m \times n\) matrix \(A\):

Here’s an example:

Matrix conjugation properties

• \(\overline{A+B} = \overline{A}+\overline{B}\) for any two matrices \(A\) and \(B\)
• \(\overline{\alpha A} = \overline{\alpha}\overline{A}\) for any matrix \(A\) and scalar \(\alpha\)
• \(\overline{\left(\overline{A}\right)} = A\) for any matrix \(A\)
• \(\overline{(A^t)} = \left(\overline{A}\right)^t\) for any matrix \(A\)

Adjoints of matrices

The adjoint of a matrix \(A\), denoted by \(A^*\), is the transpose of the conjugate of \(A\).

Here’s an example:

Matrix adjoint properties

• \((A+B)^* = A^* + B^*\) for any two matrices \(A\) and \(B\)
• \((\alpha A)^* = \overline{\alpha}A^*\) for any matrix \(A\) and scalar \(\alpha\)
• \((A^*)^* = A\) for any matrix \(A\)

Matrix-vector products

Now we’re going to look at how to multiply a matrix with a vector. This definition may seem strange at first, but it does have some nice properties that we can takek advantage of later on.

The product of an \(m \times n\) matrix \(A\) with columns \(\mathbf{A}_1, \mathbf{A}_2, ..., \mathbf{A}_n\) and a vector \(\mathbf{u}\) of size \(n\) is:

In other words, to multiply \(A\) by \(\mathbf{u}\), we calculate the linear combination of the entries of \(\mathbf{u}\) and the columns of \(A\).

Here’s an example:

The nice thing about this definition of matrix-vector multiplication is that it allows us to represent an entire system of equations in just one very short equation.

Remember that if \(\mathbf{A}_1, \mathbf{A}_2, ..., \mathbf{A}_n\) are the columns of an \(m \times n\) matrix \(A\), a solution \(\mathbf{x}\) to the system of equations with coefficient matrix \(A\) and vector of constants \(\mathbf{b}\) can be represented by the solutions to:

Using our definition of matrix-vector multiplication, this can be simplified all the way down to:

The solutions to this system of equations are simply the vectors \(\mathbf{x}\) that make the equation \(A\mathbf{x} = \mathbf{b}\) true.

Matrix multiplication

We can extend this definition further to allow us to multiply two matrices together. When we multiply an \(m \times n\) matrix \(A\) by an \(n \times p\) matrix \(B\), we get an \(m \times p\) matrix \(AB\). For \(1 \le i \le p\), column \(i\) of \(AB\) is equal to the matrix-vector product \(A\mathbf{B}_i\), where \(\mathbf{B}_i\) is column \(i\) of the matrix \(B\).

The notation \([\mathbf{v}_1|\mathbf{v}_2|...|\mathbf{v}_n]\) means creating a matrix with columns \(\mathbf{v}_1, \mathbf{v}_2, ..., \mathbf{v}_n\).

In order to multiply two matrices, the number of columns in the first matrix must equal the number of rows in the second matrix.

Here’s an example of matrix multiplication. Let’s say we have these two matrices:

To multiply these matrices, we need to find \(A\mathbf{B}_1\) and \(A\mathbf{B}_2\) first. Here are the calculations for these two products:

We can put these vectors together to get the overall matrix product:

There is another equivalent way to define matrix multiplication: for an \(m \times n\) matrix \(A\) and an \(n \times p\) matrix \(B\), each entry of the \(m \times p\) matrix \(AB\) is given by:

Let’s work out the previous example this way. As a reminder, here are the two matrices we multiplied previously:

The top-left entry of \(AB\) is given by:

The entry at row 1 and column 2 is given by:

We can calculate the rest of the entries this way.

Properties of matrix multiplication

• \(A\mathrm{o}_{n \times p} = \mathrm{o}_{m \times p}\) for any \(m \times n\) matrix \(A\)
• \(\mathrm{o}_{p \times m}A = \mathrm{o}_{p \times n}\) for any \(m \times n\) matrix \(A\)
• \(AI_n = I_m A = A\) for any \(m \times n\) matrix \(A\)
• \(A(B+C) = AB+AC\) for any \(m \times n\) matrix A and \(n \times p\) matrices \(B\) and \(C\)
• \((B+C)D = BD+CD\) for any \(n \times p\) matrices \(B\) and \(C\) and \(p \times s\) matrix \(D\)
• \(\alpha(AB) = (\alpha A)B = A(\alpha B)\) for any scalar \(\alpha\), \(m \times n\) matrix \(A\), and \(n \times p\) matrix \(B\)
• \(A(BD) = (AB)D\) for any \(m \times n\) matrix \(A\), \(n \times p\) matrix \(B\), and \(p \times s\) matrix \(D\). In other words, matrix multiplication is associative.
• Matrix multiplication is not commutative: \(AB\) does not necessarily equal \(BA\).
• \(\overline{A}\overline{B} = \overline{AB}\) for any \(m \times n\) matrix \(A\) and \(n \times p\) matrix \(B\)
• \((AB)^t = B^t A^t\) for any \(m \times n\) matrix \(A\) and \(n \times p\) matrix \(B\)
• \((AB)^* = B^* A^*\) for any \(m \times n\) matrix \(A\) and \(n \times p\) matrix \(B\)

Matrix multiplication allows us to define an interesting operation on matrices.

In the real numbers, we can define the reciprocal of a number \(x\) as the number that when multiplied by \(x\) gives 1. For example, \(1/4\) is the reciprocal of 4 because \(4 \times 1/4 = 1\).

We can define a similar idea for matrices. We define a matrix \(B\) to be the inverse of an \(n \times n\) matrix \(A\) if \(AB = BA = I_n\), where \(I_n\) is the \(n \times n\) identity matrix. (You can think of \(I_n\) as being the matrix version of the number 1, as multiplying a matrix by the identity matrix does not change it.)

We can use this idea of a matrix inverse to solve some systems of linear equations. For example, consider this system:

We can represent this system as the equation \(A\mathbf{x} = \mathbf{b}\), where \(A\) is the coefficient matrix and \(\mathbf{b}\) is the vector of constants. Here are the values of \(A\) and \(\mathbf{b}\):

If we had a matrix \(B\) that was the inverse of the matrix \(A\), then we could write \(\mathbf{x}\) as follows:

In this case, the inverse of \(A\) is:

I will discuss how to calculate inverses like this one later. For now, we can use this inverse \(B\) to find the solution \(\mathbf{x}\) to the system of equations:

Looking back at this example, if we multiply \(A\) and \(B\) together, we get:

Now let’s find the product \(BA\):

Both of these products are the identity matrix, which is what we should always get when we multiply a matrix by its inverse or vice versa.

Finding matrix inverses

Now let’s go through a procedure on how we can find matrix inverses.

To find the inverse of a matrix \(A\), we need to find another matrix \(B\) such that \(AB = I_n\). We can use the definition of matrix multiplication to expand this:

The vectors \(\mathbf{e}_1, \mathbf{e_2}, ..., \mathbf{e}_n\) are the columns of the identity matrix \(I_n\).

Now we can set the columns of both matrices equal to each other, to get the equations \(A\mathbf{B}_1 = \mathbf{e}_1\), \(A\mathbf{B}_2 = \mathbf{e}_2\), ..., \(A\mathbf{B}_n = \mathbf{e}_n\).

Let’s go over an example. We will define \(A\) as the matrix we used previously:

Now we solve for the columns of \(B\): \(\mathbf{B}_1\), \(\mathbf{B}_2\), and \(\mathbf{B}_3\). We can do this by using the equations I previously mentioned: \(A\mathbf{B}_1 = \mathbf{e}_1\), \(A\mathbf{B}_2 = \mathbf{e}_2\), and \(A\mathbf{B}_3 = \mathbf{e}_3\). Let’s start with the first one:

This is the same as solving the system of equations with coefficient matrix \(A\) and vector of constants \(\mathbf{e}_1\). We can do this by row-reducing the augmented matrix:

From here, we can read out that the solution is \(x_1 = -3\), \(x_2 = 1\), and \(x_3 = 1\). Therefore:

The next equation to solve for is \(A\mathbf{B}_2 = \mathbf{e}_2 = \begin{bmatrix} 0 \\ 1 \\ 0 \end{bmatrix}\). Here’s the row-reduction for this equation:

And finally, we have \(A\mathbf{B}_3 = \mathbf{e}_3 = \begin{bmatrix} 0 \\ 0 \\ 1\end{bmatrix}\). Here’s the row-reduction for this equation:

Using these three vectors \(\mathbf{B}_1\), \(\mathbf{B}_2\), and \(\mathbf{B}_3\), we can construct the inverse matrix \(B\):

A faster way to perform this process is to create a new matrix formed by combining the columns of \(A\) with the identity matrix \(I_n\). Here’s what that would look like in this case:

The columns of matrix \(A\) are highlighted in red, and the columns of the identity matrix \(I_3\) are highlighted in blue.

Then, we row-reduce this matrix:

The inverse of \(A\) is then obtained by reading out the rightmost \(n\) columns of this matrix. In this case, the rightmost 3 columns are:

Inverse of a \(2\times 2\) matrix

For \(2 \times 2\) matrices specifically, there is a simple formula for the inverse of \(A\):

Note that this formula doesn’t work if \(ad-bc = 0\).

Invertible matrices

The general notation for an inverse of a matrix \(A\) is \(A^{-1}\). This is similar to how the reciprocal of a real number \(x\) is \(x^{-1}\).

Just like how not every real number has a reciprocal (zero doesn’t have a reciprocal), only some matrices have an inverse. A matrix that has an inverse is known as invertible. Specifically, only nonsingular matrices have an inverse, so the properties “nonsingular” and “invertible” are equivalent.

If \(A\) is nonsingular, the system of equations with coefficient matrix \(A\) and vector of constants \(\mathbf{b}\) has a unique solution given by \(A^{-1}\mathbf{b}\).

Consider a \(2 \times 2\) matrix \(A\) of this form:

The matrix \(A\) is only invertible if \(ad - bc \ne 0\).

Properties of matrix inverses

• Any invertible matrix \(A\) has a unique inverse.
• If \(A\) and \(B\) are invertible, then \(AB\) is invertible and \((AB)^{-1} = A^{-1}B^{-1}\).
• For any invertible matrix \(A\), \(A^{-1}\) is invertible and \((A^{-1})^{-1} = A\).
• For any invertible matrix \(A\), \(A^t\) is invertible and \((A^t)^{-1} = (A^{-1})^t\).
• For any invertible matrix \(A\) and scalar \(\alpha\), \(\alpha A\) is invertible and \((\alpha A)^{-1} = \frac{1}{\alpha}A^{-1}\).

Column spaces

The column space of a matrix is the span of the columns of the matrix (that is, the set of all possible linear combinations of the columns of a matrix).

For example, consider this matrix:

The column space of this matrix is the following span:

How can we determine what vectors are in the column space of \(A\)? Is every possible vector of size 3 in \(A\)’s column space?

For example, let’s try to determine if the following vector is in \(A\)’s column space:

This is the same as asking if \(\mathbf{b}_1\) can be written as a linear combination of \(A\)’s columns. In other words, we are looking for scalars \(x_1\), \(x_2\), and \(x_3\) such that:

This, in turn, is the same as solving the following system of equations:

We can solve this system by row-reducing the following matrix, as we usually would:

Because the last column isn’t a pivot column, there is a solution to our system of equations (in fact, there are infinitely many of them). This means that the vector \(\mathbf{b}_1\) is a member of the column space of \(A\).

Let’s do another example, this time with the following vector:

To find if this vector is in the column space of \(A\), we row-reduce the augmented matrix \([A|\mathbf{b}_2]\):

Because the last column is a pivot column, the corresponding system is inconsistent, so \(\mathbf{b}_2\) is not in the column space of \(A\).

To summarize this process, the vector \(\mathbf{b}\) is in the column space of \(A\) if the system of equations with coefficient matrix \(A\) and vector of constants \(\mathbf{b}\) is consistent (i.e. if \(A\mathbf{x} = \mathbf{b}\) is consistent).

In the last example, \(\mathbf{b}_1\) was in the column space of \(A\), but not \(\mathbf{b}_2\). This also means that not every possible vector of size 3 is in the column space of \(A\). Is there a \(3 \times 3\) matrix with a column space that contains every possible vector of size 3?

Let’s take our matrix \(A\) and make a small adjustment to create a new matrix \(B\):

Let’s bring back our vectors \(\mathbf{b}_1\) and \(\mathbf{b}_2\) from before and check if they’re in \(B\)’s column space.

Both \(\mathbf{b}_1\) and \(\mathbf{b}_2\) are in \(B\)’s column space. But it turns out, any possible vector of size 3 is in \(B\)’s column space! To find out why, let’s look closer at \(B\). What happens if we row-reduce \(B\) by itself?

It turns out that \(B\) is a nonsingular matrix (since it row-reduces to the identity matrix). And what this means is that the system \(B\mathbf{x} = \mathbf{b}\) will always have a single solution no matter what \(\mathbf{b}\) is, so every possible vector of size 3 is in the column space of \(B\).

In general, if \(A\) is an \(n \times n\) nonsingular matrix, then its column space will consist of all possible vectors of size \(n\).

Simplifying a column space

Let’s go back to the matrix \(A\) and look at its column space:

We could describe the column space as the span of \(A\)’s columns:

I’m using \(C(A)\) to denote the column space of \(A\).

However, since these vectors aren’t linearly independent, we can simplify our description of the span. Let’s try reducing this spanning set by row-reducing the matrix \(A\):

Because the 1st and 2nd columns are pivot columns, we can describe this span using only the first two columns from \(A\):

Row spaces

The row space of a matrix \(A\) is the column space of its transpose \(A^t\). The row space of \(A\) can also be seen as the span of the rows of \(A\).

Here’s the row space of our matrix from before:

I’m using \(R(A)\) to denote the row space of \(A\).

Row spaces are interesting because performing a row operation on a matrix does not have any effect on its row space. What this means is that if we row-reduce a matrix, the row space stays the same, allowing us to get a simpler spanning set.

Let’s row-reduce the matrix \(A\) to demonstrate this:

Taking the rows of this reduced matrix and writing them as column vectors, we get that the row space of \(A\) is:

Note that we don’t need to include the zero vector, since including the zero vector doesn’t change the span at all.

Now let’s do the same for \(B\):

This also give us another way to find the column space of a matrix.

Column space from row operations

We can first transpose the matrix, then find its row space by row-reducing the matrix.

Elementary matrices are matrices that are slight variations of the identity matrix. Specifically, they are the result of applying a row operation to the identity matrix. Here are the three types of elementary matrices:

1. For \(i \ne j\), \(E_{i, j}\) is the identity matrix with rows \(i\) and \(j\) swapped.
2. For \(\alpha \ne 0\), \(E_i(\alpha)\) is the identity matrix with the entry at row \(i\) and column \(i\) replaced with \(\alpha\).
3. For \(i \ne j\), \(E_{i, j}(\alpha)\) is the identity matrix with the entry at row \(j\) and column \(i\) replaced with \(\alpha\).

Here are some examples of \(3 \times 3\) elementary matrices:

Elementary matrices are special because multiplying an elementary matrix by a matrix \(A\) has the same effect as performing a row operation on \(A\). Specifically:

• \(E_{i,j}A\) is equal to the matrix obtained by swapping row \(i\) and \(j\) of \(A\).
• \(E_i(\alpha) A\) is equal to the matrix obtained by multiplying row \(i\) of \(A\) by \(\alpha\).
• \(E_{i,j}(\alpha) A\) is equal to the matrix obtained by adding \(\alpha\) times row \(i\) to row \(j\) of \(A\).

In addition, elementary matrices are nonsingular, and any nonsingular matrix can be written as a product of elementary matrices.

The determinant is a special property of a square matrix, and it is a scalar. As we will see in a bit, its main use is to determine if a square matrix is singular or nonsingular. It is calculated as follows:

If \(A\) is a \(1 \times 1\) matrix, then the determinant of \(A\), denoted \(\det(A)\) or \(|A|\), is the value of the only entry of \(A\) (i.e. \([A]_{1, 1}\)).

If \(A\) is a \(2 \times 2\) matrix as follows:

then \(\det(A) = ad-bc\).

If \(A\) is larger, then the calculation of a determinant is more complicated. Let’s go through an example for the following \(3 \times 3\) matrix:

We start by removing the first row and first column to get a smaller matrix, known as a submatrix:

Then we multiply the determinant of this matrix by the first entry of the first row of \(A\) (i.e. \([A]_{1, 1}\)). By doing this, we get:

Now, we remove the first row and second column from \(A\), and multiply the determinant by the negative of the second entry of the first row of \(A\):

Now we remove the first row and third column from \(A\), and multiply the determinant by the third entry of the first row of \(A\).

To find the overall determinant of \(A\), we add these three values together.

Notice how we took the negative of \([A]_{1, 2}\). In general, if the column number \(j\) is even, we will need to take the negative of the matrix entry \([A]_{1, j}\).

To describe this process in a formula, we first need to discuss submatrices.

Submatrices

The submatrix \(A(i|j)\) is the matrix obtained by removing row \(i\) and column \(j\) from \(A\).

For example, consider this matrix:

Here are some examples of submatrices:

Determinant formula

With this submatrix notation, we can write down the formula for a determinant of a \(2 \times 2\) or larger square matrix:

This is known as the expansion about row 1. The reason is that we can actually perform this type of expansion about any row in the matrix, and we will still get the same determinant! In general, the expansion about row \(i\) is:

We can also perform this expansion about any column of the matrix:

Using this information, we can cleverly choose certain rows or columns to expand about in order to calculuate determinants faster. Let’s go back to our previous matrix:

This time, we’re going to calculate the determinant of \(A\) by expanding about the second row:

The advantage of calculating the determinant this way is that two of the entries in the second row are 0, meaning we can skip over calculating two \(2 \times 2\) determinants.

Properties of determinants

• A square matrix with a row or column that only contains zeros has a determinant of 0.
• A square matrix with two equal rows or two equal columns has a determinant of 0.
• If \(B\) is a matrix obtained by swapping two rows or columns of the square matrix \(A\), then \(\det(B) = -\det(A)\).
• If \(B\) is a matrix obtained by multiplying a row or column of the square matrix \(A\) by a scalar \(\alpha\), then \(\det(B) = \alpha\det(A)\).
• If \(B\) is a matrix obtained by adding \(\alpha\) times a row of the square matrix \(A\) to another row of \(A\), or adding \(\alpha\) times a column of \(A\) to another column of \(A\), then \(\det(B) = \det(A)\).
• If \(A\) and \(B\) are square matrices of the same size, then \(\det(AB) = \det(A)\det(B)\).
• The square matrix \(A\) is a singular matrix if and only if \(\det(A) = 0\).

The last property is a large part of why determinants are useful. We now have another way to check if a matrix is singular: using its determinant. If the determinant of a square matrix \(A\) is zero, then \(A\) is singular.

Determinants of elementary matrices

• \(\det(I_n) = 1\)

The following can be derived from the properties of determinants and the determinant of an identity matrix:

• \(\det(E_{i,j}) = -1\)
• \(\det(E_i(\alpha)) = \alpha\)
• \(\det(E_{i,j}(\alpha)) = 1\)

Now that we’ve gone over inverse matrices, column spaces, determinants, we can add three more nonsingularity conditions to our list.

If \(A\) is an \(n \times n\) square matrix, these conditions are equivalent:

1. \(A\) is a nonsingular matrix.
2. The reduced row-echelon form of \(A\) is equal to the \(n \times n\) identity matrix.
3. The null space of \(A\) contains only the zero vector (i.e. the system \(A\mathbf{x} = \mathbf{0}\) has a single solution, the zero vector).
4. The linear system of equations with coefficient matrix \(A\) and vector of constants \(\mathbf{b}\) (i.e. the system \(A\mathbf{x} = \mathbf{b}\)) has a single unique solution for every possible choice of \(\mathbf{b}\).
5. The columns of \(A\) are linearly independent.
6. \(A\) is invertible (i.e. \(A\) has an inverse).
7. The column space of \(A\) consists of all possible vectors of size \(n\).
8. The determinant of \(A\) is nonzero (i.e. \(\det(A) \ne 0\)).

Did you notice that the basic properties of column vectors and matrices are very similar? (See the Vector Operations and Matrix Operations sections to review them.) It turns out that many other mathematical objects also share these properties, so we can use the power of linear algebra with them!

Mathematicians have a special name for sets of objects that behave like how we would expect vectors to behave: vector spaces.

A vector space is made of a set \(V\) of objects along with definitions of these two operations: vector addition (denoted by a plus sign) and scalar multiplication (denoted by juxtaposition, i.e. writing a scalar and a vector next to each other). \(V\) is a vector space if these ten properties (known as axioms) hold:

1. Additive Closure: If \(\mathbf{u}\) and \(\mathbf{v}\) are in \(V\), then \(\mathbf{u} + \mathbf{v}\) is in \(V\).
2. Scalar Closure: If \(\alpha\) is a scalar and \(\mathbf{u}\) is in \(V\), then \(\alpha\mathbf{u}\) is in \(V\).
3. Commutativity: If \(\mathbf{u}\) and \(\mathbf{v}\) are in \(V\), then \(\mathbf{u} + \mathbf{v} = \mathbf{v} + \mathbf{u}\).
4. Additive Associativity: If \(\mathbf{u}\), \(\mathbf{v}\), and \(\mathbf{w}\) are in \(V\), then \((\mathbf{u} + \mathbf{v}) + \mathbf{w} = \mathbf{u} + (\mathbf{v} + \mathbf{w})\)
5. Zero Vector: There is a zero vector, i.e. a vector \(\mathbf{0}\) such that \(\mathbf{u} + \mathbf{0} = \mathbf{u}\) for all \(\mathbf{u}\) in \(V\).
6. Additive Inverses: For every \(\mathbf{u}\) in \(V\), there is a vector \(-\mathbf{u}\) in \(V\) such that \(\mathbf{u} + (-\mathbf{u}) = \mathbf{V}\).
7. Scalar Multiplication Associativity: If \(\alpha\) and \(\beta\) are scalars and \(\mathbf{u}\) is in \(V\), then \(\alpha(\beta \mathbf{u}) = (\alpha\beta)\mathbf{u}\).
8. Distributivity Across Vector Addition: If \(\alpha\) is a scalar and \(\mathbf{u}\) and \(\mathbf{v}\) are in \(V\), then \(\alpha(\mathbf{u} + \mathbf{v}) = \alpha\mathbf{u} + \alpha\mathbf{v}\).
9. Distributivity Across Scalar Addition: If \(\alpha\) and \(\beta\) are scalars and \(\mathbf{u}\) is in \(V\), then \((\alpha + \beta)\mathbf{u} = \alpha\mathbf{u} + \beta\mathbf{u}\).
10. Scalar Multiplication By One: If \(\mathbf{u}\) is in \(V\), then \(1\mathbf{u} = \mathbf{u}\).

The objects in \(V\) are known as vectors. Note that in the context of vector spaces, the term “vector” can refer to any type of object, not just the column vectors we’re used to using.

In addition, vector addition and scalar multiplication can be defined however we want as long as these 10 axioms hold. This means that depending on your definitions, vector addition and scalar multiplication might not behave as you would expect!

Keep in mind that if even one of these axioms doesn’t hold, then the set and its operations is not a vector space.

Here are some examples of vector spaces:

The vector space of column vectors

The set of all column vectors of size \(m\), denoted \(\mathbb{C}^m\) (or \(\mathbb{R}^m\) if you’re only considering column vectors with real entries), is a vector space. As we’ve shown in the Vector Operations section, the ten properties hold when we’re using the standard definitions of vector addition and scalar multiplication.

The vector space of matrices

The set of all matrices of size \(m \times n\), also denoted \(M_{mn}\), is also a vector space. The ten properties required are met with the standard definitions of matrix addition and scalar multiplication, as shown by the properties in the Matrix Operations section.

The vector space of polynomials

The set of all polynomials of degree \(n\) or less, denoted \(P_n\), is a vector space if these properties are defined as such:

• Vector equality: Two polynomials \(a_0 + a_1 x + a_2 x^2 + \cdots + a_n x^n\) and \(b_0 + b_1 x + b_2 x^2 + \cdots + b_n x^n\) are equal if \(a_i = b_i\) for \(0 \le i \le n\) (i.e. the coefficients and constants of both polynomials are equal).
• Vector addition: The sum of two polynomials \(\mathbf{u} = a_0 + a_1 x + a_2 x^2 + \cdots + a_n x^n\) and \(\mathbf{v} = b_0 + b_1 x + b_2 x^2 + \cdots + b_n x^n\) is \(\mathbf{u} + \mathbf{v} = (a_0 + b_0) + (a_1 + b_1)x + \cdots + (a_n + b_n)x^n\).
• Scalar multiplication: If \(\mathbf{u} = a_0 + a_1 x + a_2 x^2 + \cdots + a_n x^n\), then \(\alpha \mathbf{u} = (\alpha a_0) + (\alpha a_1)x + \cdots + (\alpha a_n)x^n\).

Additional examples of vector spaces include the vector space of continuous functions and the vector space of infinite sequences.

A vector space with nonstandard operations

I previously said that we could define the operations of vector addition and scalar multiplication however we wanted as long as the vector space axioms are satisfied. Here’s an example of a vector space with some strange operations.

The vectors consist of all possible ordered pairs \((x_1, x_2)\). The two operations are defined as follows:

• Vector addition: \((x_1, x_2) + (y_1, y_2) = (x_1 + y_1 + 1, x_2 + y_2 + 1)\)
• Scalar multiplication: \(\alpha(x_1, x_2) = (\alpha x_1 + \alpha - 1, \alpha x_2 + \alpha - 1)\)

Surprisingly enough, this forms a vector space! It turns out that all 10 vector space axioms are satisfied even with these operations.

Strangely, the zero vector in this vector space is not \((0, 0)\), but instead \(\mathbf{0} = (-1, -1)\). To see why, let’s add the zero vector to a general vector:

Remember that the zero vector is defined as the vector such that \(\mathbf{u} + \mathbf{0} = \mathbf{u}\) for all vectors \(\mathbf{u}\) in the vector space. Because of this, \((-1, -1)\) is the zero vector in this vector space, since the sum of the zero vector and another vector \(\mathbf{u}\) is just \(\mathbf{u}\).

Properties of vector spaces

This is a list of properties that apply to vectors in vector spaces:

• The zero vector \(\mathbf{0}\) is unique (i.e. there can only be one zero vector in a vector space).
• Additive inverses are unique, i.e. for every vector \(\mathbf{u}\) in a vector space \(V\), there is only one additive inverse \(-\mathbf{u}\).
• \(0\mathbf{u} = \mathbf{0}\) for every \(\mathbf{u}\) in a vector space \(V\).
• \(\alpha\mathbf{0} = \mathbf{0}\) for every scalar \(\alpha\).
• \(-\mathbf{u} = (-1)\mathbf{u}\) for every \(\mathbf{u}\) in a vector space \(V\).
  • Note: \(-\mathbf{u}\) denotes the additive inverse of \(\mathbf{u}\), while \((-1)\mathbf{u}\) is the vector \(\mathbf{u}\) multiplied by the scalar -1. These are different concepts, but this property ensures that they will always be equal to each other.
• If \(\alpha \mathbf{u} = \mathbf{0}\), then either \(\alpha = 0\) or \(\mathbf{u} = \mathbf{0}\).

A subspace \(W\) of a vector space \(V\) is a subset of \(V\) that is also a vector space and has the same definitions of vector addition and scalar multiplication as \(W\).

For example, consider the following subset of \(\mathbb{C}^2\), the set of column vectors of size 2:

This is the set of all column vectors of size 2 such that \(x_1 - 3x_2 = 0\). For example, these column vectors are in \(W\):

And these vectors are not in \(W\):

Is \(W\) a subspace? That is, using the definitions of vector addition and scalar multiplication in \(\mathbb{C}^2\), is \(W\) a vector space? To test if this subset is a subspace, we need to verify that the ten properties (axioms) of vector spaces hold.

First up, additive closure. If \(\mathbf{u}\) and \(\mathbf{v}\) are in \(W\), does that mean \(\mathbf{u} + \mathbf{v}\) is in \(W\)?

Let’s write these two vectors as follows:

We can write the sum of these two vectors as:

Now we need to know if this column vector satisfies the requirement to be in \(W\), \(x_1 - 3x_2 = 0\). For this vector, the first entry \(x_1\) is \(u_1 + v_1\), and the second entry is \(x_2 = u_2 + v_2\). Therefore, we must test if \((u_1 + v_1) - 3(u_2 + v_2) = 0\). Note that because \(\mathbf{u}\) and \(\mathbf{v}\) are in \(W\), we can ensure that \(u_1 - 3u_2 = 0\) and \(v_1 - 3v_2 = 0\).

Therefore, if \(\mathbf{u}\) and \(\mathbf{v}\) are in \(W\), then \(\mathbf{u}+\mathbf{v}\) also meets the requirement to be in \(W\).

Now let’s test scalar closure. If \(\mathbf{u}\) is in \(W\), then is \(\alpha\mathbf{u}\) guaranteed to be in \(W\)? We can write \(\alpha\mathbf{u}\) as:

To see if this vector \(\alpha\mathbf{u}\) is in \(W\), we need to test if \((\alpha u_1) - 3(\alpha u_2) = 0\). Once again, we know for sure that \(u_1 - 3u_2 = 0\), as \(\mathbf{u}\) is in \(W\).

Therefore, if \(\mathbf{u}\) is in \(W\), then \(\alpha\mathbf{u}\) is in \(W\).

Now let’s check if every vector \(\mathbf{u}\) in \(W\) has an inverse \(-\mathbf{u}\) that is in \(W\). The inverse of \(\mathbf{u}\) must be the following:

This time, we need to check if \((-u_1) - 3(-u_2) = 0\):

Therefore, every vector \(\mathbf{u}\) in \(W\) has an inverse that is also in \(W\).

Is there a zero vector in \(W\)? The only candidate is the zero vector in \(\mathbb{C}^2\), and it meets the requirement to be in \(W\), as \(0 - 3(0) = 0\).

The other six properties (commutativity, associativity, etc.) hold for all vectors in \(W\), because any subset of column vectors from \(\mathbb{C}^2\) will retain these properties. Therefore, we don’t need to check them.

Because all ten properties of vector spaces hold, we can say that \(W\) is a vector space.

Testing if a subset is a subspace

There is a simpler way to check if a subset of a vector space is itself a vector space. A subset \(W\) of a vector space \(V\) is a subspace if it meets the following conditions:

1. \(W\) is nonempty.
2. If \(\mathbf{u}\) and \(\mathbf{v}\) are in \(W\), then \(\mathbf{u} + \mathbf{v}\) is in \(W\).
3. If \(\mathbf{u}\) is in \(W\) and \(\alpha\) is a scalar, then \(\alpha\mathbf{u}\) is in \(W\).

Specific subsets that are always subspaces

Given any vector space \(V\), the subsets \(V\) and \(\{\mathbf{0}\}\) will always be subspaces. These are known as the trivial subspaces of \(V\).

Null spaces and subspaces

The null space of any \(m \times n\) matrix \(A\) is a subspace of \(\mathbb{C}^n\).

For example, consider this subset:

This set is really just the solutions to the following homogeneous system of equations:

The solution set to this system is the null space of the following matrix:

Because the set \(W\) is a null space of a matrix, we immediately know that \(W\) is a subspace of \(\mathbb{C}^3\).

We can define linear combinations and spans for vector spaces in general the same way we did for column vectors. A linear combination of \(n\) vectors \(\mathbf{u}_1, \mathbf{u}_2, ..., \mathbf{u}_n\) and \(n\) scalars \(\alpha_1, \alpha_2, ..., \alpha_n\) is:

The span of a set of vectors \(S = \{\mathbf{u}_1, \mathbf{u}_2, ..., \mathbf{u}_n\}\), where \(\mathbf{u}_1, \mathbf{u}_2, ..., \mathbf{u}_n\) are vectors from a vector space \(V\), is the set of all possible linear combinations using these vectors.

Given a set of vectors \(S\) from a vector space \(V\), the span of \(S\) will always be a subspace.

As an example, let’s look at the following span in the vector space of \(2 \times 2\) matrices:

Is the matrix \(A = \begin{bmatrix}4 & -2 \\ 1 & 3 \end{bmatrix}\) in this span? If this matrix is in the span, then there are scalars \(\alpha_1\), \(\alpha_2\), \(\alpha_3\), and \(\alpha_4\) such that:

By equating matrix entries, we can turn this question into a system of linear equations:

As usual, we can solve this system by row-reducing the augmented matrix:

By doing this, we find a solution: \(\alpha_1 = -10\), \(\alpha_2 = -8\), \(\alpha_3 = 7\), and \(\alpha_4 = 16\). Therefore, the matrix \(A\) is in our span.

Notice how we were able to convert this question of whether the matrix \(A\) is in a span into a question of whether a system of linear equations is consistent. This idea of turning questions about vector spaces into questions about linear systems is an important technique that we will continue to encounter in the future.

Just like with column vectors, we can also define what it means for a set of vectors from a vector space to be linearly independent.

Given scalars \(\alpha_1, \alpha_2, ..., \alpha_n\) and vectors \(\{\mathbf{u}_1, \mathbf{u}_2, ..., \mathbf{u}_n\}\) from a vector space \(V\), a relation of linear dependence is a true equation of the form:

If all of the scalars are zero, then this is a trivial relation of linear dependence.

A set of vectors \(S = \{\mathbf{u}_1, \mathbf{u}_2, ..., \mathbf{u}_n\}\) is linearly independent if the only relation of linear dependence involving these vectors is the trivial one. Otherwise, \(S\) is linearly dependent.

Spanning Sets

A set \(S\) is known as a spanning set of a vector space \(V\) if \(S\) is a subset of \(V\) and the span of \(S\) is equal to \(V\) (i.e. \(\langle S \rangle = V\)). In this case, \(S\) is said to span \(V\).

For example, let’s consider the following span in \(P_2\), the vector space of polynomials of degree 2 or less:

Does the set \(S_1\) span \(P_2\)? Let’s first check if some polynomials are in the span or not. For example, is \(1 + 2x + 17x^2\) in the span? This is equivalent to asking whether a linear combination of this form exists:

By equating coefficients of the polynomial (i.e. setting the constant terms on both sides equal, as well as the \(x\) and \(x^2\) coefficients on both sides), we get the following system of equations:

Row-reducing the augmented matrix will tell us if there is a solution or not.

Note: Notice how each column in the first three columns of our original augmented matrix represents a polynomial in \(S_1\). For example, the first column represents the polynomial \(3 - 2x + 5x^2\). The last column represents the polynomial we are checking, which is \(1 + 2x + 17x^2\).

Because the last column is not a pivot column, there is a solution to the system of equations, meaning \(1 + 2x + 17x^2\) is in the span of \(S_1\).

Now let’s try doing the same thing for the polynomial \(1 + x + x^2\). We get the following system of equations and we want to know if it has a solution:

Let’s row-reduce the augmented matrix now:

Because the last column is a pivot column, the system is inconsistent, so \(1 + x + x^2\) is not in the span of \(S_1\).

Now that we’ve found a polynomial in \(P_2\) that isn’t in the span of \(S\), we can conclude that \(S\) does not span \(P_2\), since not every polynomial in \(P_2\) can be written as a linear combination of the polynomials in \(S_1\).

Now let’s look at this span:

Let’s once again test if a specific polynomial is in the span, such as \(1 + 6x + 2x^2\).

When testing if this polynomial is in the span, we end up with the following system:

By row-reducing the augmented matrix, we get:

The system is consistent, and so \(1 + 6x + x^2\) is part of the span of \(S_2\).

As it turns out, no matter what polynomial in \(P_2\) we choose, we will always be able to find the scalars \(\alpha_1\), \(\alpha_2\), and \(\alpha_3\) necessary to write the polynomial as a linear combination of the three polynomials in \(S_2\). To see why, let’s look at the coefficient matrix we’ve been using:

By row-reducing this matrix or finding its determinant (which is 11), we can conclude that this matrix is nonsingular, meaning that no matter what the vector of constants \(\mathbf{b}\) is, we will be able to find a solution to the system \(A\mathbf{x} = \mathbf{b}\). Therefore, every polynomial in \(P_2\) is part of the span of \(S_2\), and so \(S_2\) spans \(P_2\).

In this section, we’ll be looking at a special type of spanning set known as a basis. A basis of a vector space \(V\) is a set of vectors that is linearly independent and spans \(V\).

Remember that when we have a spanning set that is not linearly independent, we can write at least one of the vectors as a linear combination of the others, allowing us to create an equivalent span with fewer vectors. (This was shown in the Linear Dependence/Independence and Spans section.)

However, a linearly independent spanning set cannot be reduced any further. Therefore, a basis can be thought of as a spanning set that contains the least vectors possible.

For example, consider the span of the following set of vectors:

Let’s check to see if this set is linearly indpendent by row-reducing the matrix formed by combining these column vectors:

By taking the columns of the original matrix that correspond to the pivot columns of the row-reduced matrix, we can find that this spanning set can be written more simply as:

At this point, this set of vectors is linearly independent and spans \(\mathbb{C}^3\), so we have a basis of \(\mathbb{C}^3\).

The standard basis

The simplest example of a basis in \(\mathbb{C}^3\) is the following:

These are known as the standard unit vectors for \(\mathbb{C}^3\). Any vector in \(\mathbb{C}^3\) can be easily written as a combination of these three vectors, so the set spans \(\mathbb{C}^3\). Here are a few examples:

The set is also linearly independent since none of the vectors can be written as a linear combination of the other two vectors.

A similar standard basis can be made for the vector space \(\mathbb{C}^n\) by creating \(n\) vectors such that the \(i\)th standard unit vector has a 1 as the \(i\)th entry and zeros for all other entries. For example, the standard unit vectors for \(\mathbb{C}^4\) are:

Notice that these vectors are just the columns of the \(4 \times 4\) identity matrix. In general, the \(i\)th standard basis vector of \(\mathbb{C}^n\) is the \(i\)th column of the \(n \times n\) identity matrix \(I_n\).

Because the columns of an \(n \times n\) nonsingular matrix are linearly independent and also span \(\mathbb{C}^n\), the columns of any nonsingular matrix form a basis for \(\mathbb{C}^n\).

It turns out that all possible bases of a vector space have the same number of vectors. This means that the size of any basis of a vector space is a property of that vector space.

For example, since the vector space \(\mathbb{C}^3\) has a basis with 3 vectors (the standard basis), \(\mathbb{C}^3\) has a dimension of 3. In addition, all other possible bases of \(\mathbb{C}^3\) have 3 vectors.

In general, the vector space \(\mathbb{C}^n\) is \(n\), since the columns of the \(n \times n\) identity matrix (i.e. the \(n\) standard basis vectors) form a basis for \(\mathbb{C}^n\).

The vector space of all polynomials of degree \(n\) or less, \(P_n\), has dimension \(n + 1\), since the following set of polynomials is linearly independent and spans \(P_n\):

The vector space of \(m \times n\) matrices, \(M_{mn}\) has dimension \(mn\). For example, the vector space of \(2 \times 2\) matrices has the following basis of size 4:

In general, for any vector space \(M_{mn}\), you can create a basis with \(mn\) matrices, each with one entry equal to 1 and the rest of the entries equal to 0.

One thing to keep in mind is that a vector space that only contains the zero vector has dimension 0.

Rank and nullity of a matrix

The dimensions of the column space and null space of a matrix are given special names: the rank and nullity respectively.

For example, consider the following matrix that we looked at in the Vector Spaces: Bases section:

What is the rank and nullity of this matrix? First, we need to determine the column space of this matrix. In the previous section, we did this by row-reducing the matrix:

By taking columns of the original matrix that correspond to the pivot columns, we get that the column space is the span of three vectors:

Therefore, the rank of \(A\) is 3. Now let’s find the null space of \(A\). We can do that by row-reducing the augmented matrix for \(A\)’s homogeneous system:

Translating the row-reduced matrix back into equations:

Writing the solution set as a span:

Because the null space of \(A\) can be represented as the span of one vector (and a set with one nonzero vector is linearly independent), the nullity of \(A\) is 1.

The sum of a matrix’s rank and nullity will always equal the number of columns of the matrix. In this case, the rank of \(A\), which is 3, plus the nullity of \(A\), which is 1, equals the number of columns that \(A\) has, which is 4.

Since nonsingular matrices row-reduce to the identity matrix (which has \(n\) pivot columns, where \(n\) is the number of columns of the original matrix), a nonsingular matrix will always have a rank equal to the number of columns it has.

In addition, a nonsingular matrix will always have a nullity of zero, since for a matrix with \(n\) columns, the rank (which is \(n\) for a nonsingular matrix) plus nullity must equal \(n\).

Another way to show this is that the null space of a nonsingular matrix only contains the zero vector, meaning that its nullity must be zero.

Let’s update our nonsingular matrix equivalences list again using what we know about vector spaces.

1. \(A\) is a nonsingular matrix.
2. The reduced row-echelon form of \(A\) is equal to the \(n \times n\) identity matrix.
3. The null space of \(A\) contains only the zero vector (i.e. the system \(A\mathbf{x} = \mathbf{0}\) has a single solution, the zero vector).
4. The linear system of equations with coefficient matrix \(A\) and vector of constants \(\mathbf{b}\) (i.e. the system \(A\mathbf{x} = \mathbf{b}\)) has a single unique solution for every possible choice of \(\mathbf{b}\).
5. The columns of \(A\) are linearly independent.
6. \(A\) is invertible (i.e. \(A\) has an inverse).
7. The column space of \(A\) is equal to \(\mathbb{C}^n\), the set of all possible vectors of size \(n\).
8. The determinant of \(A\) is nonzero (i.e. \(\det(A) \ne 0\)).
9. The columns of \(A\) form a basis for \(\mathbb{C}^n\).
10. The rank of \(A\) is \(n\).
11. The nullity of \(A\) is zero.

(If we are only considering vectors and matrices with real number entries, replace all occurrences of \(\mathbb{C}^n\) with \(\mathbb{R}^n\).)

Sometimes when we multiply a matrix by a vector, it has the same effect as multiplying that vector by a scalar. For example, consider this matrix and vector:

Notice how \(A\mathbf{v}\) is equal to 8 times \(\mathbf{v}\). In this case, we call \(\mathbf{v}\) an eigenvector of \(A\) with eigenvalue 8.

In general, if \(A\mathbf{x} = \lambda \mathbf{x}\), then \(\mathbf{x}\) is known as an eigenvector of \(A\) with eigenvalue \(\lambda\). (Note that an eigenvector \(\mathbf{x}\) cannot be the zero vector, because that would trivially satisfy the equation \(A\mathbf{x} = \lambda \mathbf{x}\) no matter what \(A\) and \(\lambda\) are.)

How can we find the eigenvalues and eigenvectors of a matrix? We can first consider the equation \(A\mathbf{x} = \lambda \mathbf{x}\):

There are two cases here:

1. \(A - \lambda I_n\) is nonsingular, meaning that the system \((A - \lambda I_n)\mathbf{x} = \mathbf{0}\) has a single solution (the trivial one). This means that \(\mathbf{x} = \mathbf{0}\), but we are looking for values of \(\mathbf{x}\) that are nonzero (since eigenvectors cannot be the zero vector).
2. \(A - \lambda I_n\) is singular, meaning that the system \((A - \lambda I_n)\mathbf{x} = \mathbf{0}\) has infinitely many solutions. This means that there are nontrivial (nonzero) values of \(\mathbf{x}\) that satisfy the system, and these are the eigenvectors of \(A\) with eigenvalue \(\lambda\).

We want \(A - \lambda I_n\) to be singular, so we want to find the values of \(\lambda\) such that \(\det(A - \lambda I_n) = 0\).

The value of \(\det(A - \lambda I_n)\) is known as the characteristic polynomial, and the eigenvalues of \(A\) are the roots of this polynomial.

Going back to our previous example, the characteristic polynomial of \(A\) is:

Now let’s find the roots of this polynomial:

These are the eigenvalues of \(A\).

Now let’s find the eigenvectors associated with each eigenvalue. To do this, we need to solve the equation \((A - \lambda I_n)\mathbf{x} = \mathbf{0}\) for each eigenvalue.

Let’s start with \(\lambda = 4\):

Now let’s do the same with \(\lambda = 8\):

The set of eigenvectors for an matrix \(A\) and an eigenvalue \(\lambda\) is known as the eigenspace of \(A\) for \(\lambda\).

The algebraic multiplicity of an eigenvalue is the highest power of \(x - \lambda\) that appears in the factored form of the characteristic polynomial.

The geometric multiplicity of an eigenvalue is the dimension of the eigenspace for \(\lambda\).

In our example, the algebraic and geometric multiplicity of \(\lambda = 4\) and \(\lambda = 8\) is 1.

Let’s see what actually happens when we multiply a matrix \(A\) by a vector \(\mathbf{x}\). Here’s an example:

This is what we get when we multiply \(A\) by an arbitrary vector \(\mathbf{x}\) with components \(x_1\), \(x_2\), and \(x_3\):

Notice how the entries of the final result “feel linear”, that is, they look like linear expressions (they purely involve addition and multiplication without division or exponents).

There is a name for functions that behave this way: linear transformations. A linear transformation (also known as a linear map) is a function \(T(\mathbf{x})\) that meets these two requirements:

• \(T(\mathbf{u} + \mathbf{v}) = T(\mathbf{u}) + T(\mathbf{v})\)
• \(T(\alpha\mathbf{u}) = \alpha T(\mathbf{u})\)

The first property requires that \(T\) behaves linearly with vector addition. Applying \(T\) to two vectors \(\mathbf{x}\) and \(\mathbf{y}\) individually then adding the results gives the same result as adding the vectors first then applying \(T\) to the sum.

The second property requires that \(T\) behavies linearly with scalar multiplication. Multiplying \(\mathbf{u}\) by a scalar \(\alpha\) then applying \(T\) is the same as applying \(T\) to \(\mathbf{u}\) then multiplying by \(\alpha\).

Consider the following function:

Is \(T\) a linear transformation? We can test the first property:

Because \(T\) doesn’t meet the requirements, \(T\) is not a linear transformation.

One property of linear transformations is that they must take the zero vector to the zero vector, i.e. \(T(\mathbf{0}) = \mathbf{0}\). (Think about it: If \(T(\mathbf{0}) \ne \mathbf{0}\), how would the requirements for a linear transformation be violated?)

A function that multiplies a matrix by a vector always ends up being a linear transformation.

Some linear transformations have special properties that I will discuss here.

Function notation

In this section, I will be using a special notation for functions. The notation \(f: A \to B\) represents a function \(f\) with domain \(A\) and codomain \(B\).

The domain of a function is the set of all possible inputs of a function. The codomain of a function is a set that includes all possible outputs of the function. Note that this is different than the function’s range, which is the set of all values that can actually come out of the function. The range of a function is always a subset of the codomain.

For example, consider the function \(f: \mathbb{R} \to \mathbb{R}\) defined by \(f(x) = x^2\).

• The domain of \(f\) is \(\mathbb{R}\), meaning that we are only considering real number inputs.
• The codomain of this function is \(\mathbb{R}\) (all real numbers). (We can specify the codomain to be any set we want as long as it contains all possible outputs of \(f\).)
• The range of the function is all non-negative real numbers (since it’s impossible to get a negative number as an output of this function). Note that this is not the same as the codomain!

I will be using the same notation for linear transformations. For example, \(T: U \to V\) represents a linear transformation \(T\) with domain \(U\) and codomain \(V\).

Injective and surjective functions

An injective function is a function \(f\) such that if \(f(x) = f(y)\), then \(x = y\). In other words, no two different inputs correspond to the same output of \(f\).

• The function \(f: \mathbb{R} \to \mathbb{R}\) defined by \(f(x) = x^3\) is injective because every output has only one input.
• The function \(f: \mathbb{R} \to \mathbb{R}\) defined by \(f(x) = x^2\) is not injective because multiple inputs can correspond to the same output. For example, \(f(2) = f(-2) = 4\).

A surjective function is a function \(f\) such that for every \(y\) in the codomain of \(f\), there is an \(x\) such that \(f(x) = y\).

• The function \(f: \mathbb{R} \to \mathbb{R}\) defined by \(f(x) = x^3\) is surjective because every element in the codomain (i.e. every real number) is a possible output of \(f\).
• The function \(f: \mathbb{R} \to \mathbb{R}\) defined by \(f(x) = x^2\) is not surjective because negative numbers are not part of the possible outputs of \(f\). For example, there is no \(x\) in the domain such that \(f(x) = -1\).
• The function \(f: \mathbb{R} \to [0, \infty)\) defined by \(f(x) = x^2\) is surjective because element in the codomain (i.e. every non-negative number) is a possible output of \(f\). (Notice that we’ve excluded negative numbers from the codomain of \(f\).)

An injective (or one-to-one) linear transformation is a transformation \(T: U \to V\) such that if \(T(\mathbf{x}) = T(\mathbf{y})\), then \(\mathbf{x} = \mathbf{y}\). In other words, no two distinct input vectors result in the same output vector when fed into \(T\).

A surjective (or onto) linear transformation is a transformation \(T: U \to V\) such that for every \(\mathbf{y}\) in \(V\), there is an \(\mathbf{x}\) in \(U\) such that \(f(\mathbf{x}) = \mathbf{y}\). In other words, every element of the codomain is a possible output of \(T\).

The kernel and range of linear transformations

The kernel of a linear transformation \(T: U \to V\) is the set of vectors that \(T\) takes to the zero vector, that is, the set of vectors \(\mathbf{u}\) in \(U\) such that \(T(\mathbf{u}) = \mathbf{0}\). In set notation, for a linear transformation \(T: U \to V\):

The range of a linear transformation \(T: U \to V\) is the set of all possible outputs of \(T\), that is, a vector \(\mathbf{v}\) in \(V\) is in the range of \(T\) if there is a \(\mathbf{u}\) in \(U\) such that \(T(\mathbf{u}) = \mathbf{v}\). In set notation:

Injective/surjective linear transformations and their kernels/ranges

A linear transformation is injective if and only if its kernel consists only of the zero vector.

A linear transformation is surjective if and only if its range is equal to its codomain.