Matrices as Operators

$Ax$ can be read two ways: row by row as a list of dot products, or as a recipe for mixing the columns of $A$ using the entries of $x$ as amounts. The arithmetic is identical and the second reading is the one everything later depends on. A basis is a set of columns, and a reduced cost prices one.

The arithmetic is mechanical. The reading carries the content: almost every later argument in this subject is a question about columns.

The canonical text sets out both readings and says the column one carries the subject. Here is where it is actually spent.

Feasibility. “Does $Ax=b$ have a solution?” is “is $b$ in the span of the columns?” A linear program with no feasible point is one where the constraint columns cannot reach the right-hand side, and that is visible without solving anything.

Bases. A basis is a choice of $m$ columns that are independent, so they reach every point of ${\mathbb{R}}^m$ exactly one way. That uniqueness is what lets a basis name a single point, and it is why a dependent choice leaves the basic solution undefined rather than merely awkward.

Pricing. A reduced cost asks what happens to the objective if a nonbasic column is mixed in a little. The question only makes sense under the column reading: the row reading computes the answer but says nothing about which column to try next.

The row reading remains how the arithmetic is done. The column reading is how the decisions are made, and every later structural argument in this subject is stated in it.

The canonical statement fixes $Ax$ as a combination of columns. Two consequences are worth drawing out, because the later material uses them constantly and neither is stated above.

The column reading answers reachability; the entry reading answers arithmetic. $(Ax{)}_i=\sum _j{a}_{ij}{x}_j$ tells you how to compute a single component. $Ax=\sum _j{x}_j{A}_j$ tells you which vectors are reachable at all: precisely the span of the columns. Asking whether $Ax=b$ has a solution is asking whether $b$ lies in that span, and no amount of entrywise computation makes that question easier to see.

The product is composition, which is why it is not commutative. $(AB)x=A(Bx)$ says $AB$ is “do $B$, then do $A$”. Two actions applied in the other order are a different action, so $AB\ne BA$ in general, and the shapes need not even permit both. The transpose reverses for the same reason: $(AB{)}^{\mathsf{T}}=B^{\mathsf{T}}{A}^{\mathsf{T}}$ undoes the composition from the outside in.

Invertibility is a statement about the columns. $A^{-1}$ exists exactly when the columns are independent and span the whole of ${\mathbb{R}}^m$ — when every target is reachable, and by exactly one combination. That is the same condition the simplex method checks when it asks whether a chosen set of columns forms a basis, which is why a singular $A_B$ makes a basic solution undefined rather than merely hard to compute.

Take

By rows. First entry: $(2)(3)+(1)(2)+(0)(1)=8$. Second entry: $(1)(3)+(3)(2)+(4)(1)=13$. So $Ax=(8,13)$.

By columns. The columns are $A_1=(2,1)$, $A_2=(1,3)$, $A_3=(0,4)$, and

Same answer, as it must be.

What each shows. The row computation produces the number. The column computation shows that $(8,13)$ is reachable by mixing $A$'s columns with weights $3,2,1$, and therefore that this particular $b$ has at least one solution to $Ax=b$. That is a fact about existence, and the row reading does not surface it.

Note the shapes. $A$ is $2\times 3$ and $x$ is in ${\mathbb{R}}^3$, so $Ax$ lands in ${\mathbb{R}}^2$. The action moved between spaces.

Problem. Let

Compute $AB$ and $BA$, and explain the difference in terms of what each matrix does.

Goal. Show that order matters, and say why rather than only that.

Relevant principle. $AB$ means do $B$, then $A$. Composing two actions in the other order is a different action.

Step 1: compute $AB$. Apply $A$ to each column of $B$. First column of $B$ is $(1,1)$, and $A(1,1)=(1+1,1)=(2,1)$. Second column is $(0,1)$, and $A(0,1)=(0+1,1)=(1,1)$.

Reason: each column of the product is $A$ applied to the corresponding column of $B$.

Step 2: compute $BA$. First column of $A$ is $(1,0)$, and $B(1,0)=(1,1)$. Second column is $(1,1)$, and $B(1,1)=(1,2)$.

Step 3: read the difference. $A$ adds the second coordinate onto the first; $B$ adds the first onto the second. Doing those in different orders gives different results because the second operation acts on what the first produced.

Reason: composition is not symmetric. Putting on socks then shoes is not putting on shoes then socks.

Result. $AB\ne BA$, and both are defined here only because both matrices are square and the same size.

Check. Apply each to $(1,0)$. $AB$ gives $(2,1)$; $BA$ gives $(1,1)$. Different, as the matrices predict.

Interpretation. With non-square matrices the asymmetry is starker still: $A$ of size $2\times 3$ and $B$ of size $3\times 4$ make $AB$ defined and $BA$ meaningless. Checking the inner dimensions is not a formality. It is asking whether the composition exists at all.

Multiplying mismatched shapes. A $2\times 3$ matrix times a $2\times 2$ matrix is undefined: the inner dimensions, $3$ and $2$, disagree. The product is not zero and not approximate. There is nothing to compute.

Assuming $AB=BA$. It generally fails, and for non-square matrices one side may not even exist.

Transposing without reversing. $(AB{)}^{\mathsf{T}}=B^{\mathsf{T}}{A}^{\mathsf{T}}$, not $A^{\mathsf{T}}{B}^{\mathsf{T}}$. The order flips, and derivations that forget it go quietly wrong.

Expecting every square matrix to be invertible. has no inverse: its second column is twice the first, so its columns cannot reach everything. Whether an inverse exists is settled by the independence of the columns.

Cancelling matrices. From $AB=AC$ it does not follow that $B=C$, unless $A$ is invertible. Matrices are not numbers, and division is not an available move.

Reading $Ax$ as a scaling of $x$. It is a combination of $A$'s columns, and it generally lives in a different space from $x$ altogether.

Both are correct. They are the same sum, grouped differently. Neither is an approximation of the other.

Why the column reading earns its place. Solvability, bases, pivots and degeneracy are all questions about which vectors the columns can reach. Phrased in rows, each becomes a statement about a system of equations having a solution, true, but it hides the object doing the work.

A concrete case. If $A$'s columns are $(1,2)$ and $(2,4)$, the column reading shows immediately that everything reachable lies along a single line, so $Ax=b$ is solvable only for $b$ on that line. The row reading arrives at the same conclusion after elimination, having said nothing about why.

When to reach for rows. When you need the number. The two readings cost the same arithmetic; only one of them tells you something while you do it.

1: fully structured. Let and $x=(4,1)$.

(a) Compute $Ax$ by rows. (b) Write $Ax$ as a combination of $A$'s columns. (c) What space does $Ax$ live in?

Check: (a) $(1)(4)+(2)(1)=6$ and $(0)(4)+(3)(1)=3$, so $Ax=(6,3)$; (b) $4(1,0)+1(2,3)=(4,0)+(2,3)=(6,3)$; (c) ${\mathbb{R}}^2$, since $A$ is $2\times 2$.

2: partly structured. $A$ is $3\times 4$ and $B$ is $4\times 2$.

(a) Is $AB$ defined, and what is its shape? (b) Is $BA$ defined? (c) What does $(AB{)}^{\mathsf{T}}$ equal, and what shape is it?

Check: (a) yes, inner dimensions are both $4$, and $AB$ is $3\times 2$; (b) no: $B$ is $4\times 2$ and $A$ is $3\times 4$, so the inner dimensions $2$ and $3$ disagree; (c) $(AB{)}^{\mathsf{T}}=B^{\mathsf{T}}{A}^{\mathsf{T}}$, of shape $2\times 3$.

3: unstructured. An analyst is simplifying a derivation and writes: "Since $M=AB$ and $N=BA$ use the same two matrices, $M$ and $N$ have the same entries, so I can substitute one for the other and cancel $B$ from both sides of $AB=CB$ to conclude $A=C$."

Identify every error and say what would have to be true for the final step to hold.

Check: three errors. First, $AB$ and $BA$ are generally different matrices even when both are defined, composition is order-dependent, and for non-square shapes one may not exist at all. Second, "the same entries" confuses the inputs with the result: the same two matrices combined in different orders give different products. Third, cancelling $B$ from $AB=CB$ is division, which matrices do not support; the step is valid only if $B$ is square and invertible, in which case right-multiplying both sides by $B^{-1}$ gives $A=C$ legitimately. Without invertibility the conclusion does not follow, for instance with $B$ a matrix of zeros, $AB=CB$ holds for every $A$ and $C$.

A matrix acts. $Ax=\sum _j{x}_j{A}_j$. A linear combination of the columns, weighted by the entries of $x$.

Two readings, one answer. Rows give the arithmetic; columns give the structure. Solvability, span and bases are all column questions.

Shapes must meet. $AB$ needs $A$ to be $m\times n$ and $B$ to be $n\times p$. Mismatched inner dimensions make the product undefined, not zero.

Multiplication composes. $(AB)x=A(Bx)$, do $B$, then $A$. That is why order matters.

$AB\ne BA$ in general, and for non-square matrices one side may not exist.

Transpose reverses. $(AB{)}^{\mathsf{T}}=B^{\mathsf{T}}{A}^{\mathsf{T}}$.

Inverses are conditional. Only square matrices can have one, and only when their columns are independent.

No cancelling. $AB=AC$ gives $B=C$ only when $A$ is invertible.

The recurring error. Treating matrix multiplication as commutative.