Matrix Multiplication | AI Pathshala

Definition

Matrix multiplication combines two matrices into one: each entry of the result is the dot product of a row from the first and a column from the second. The transpose flips a matrix over its diagonal, turning rows into columns. The identity matrix has 1s on the diagonal and 0s elsewhere, and leaves anything it multiplies unchanged.

In simple words

A shop has a table of how many items each customer bought, and a table of prices. Matrix multiplication gives each customer's bill in one step. Each bill is a row of quantities dotted with a column of prices.

Why it matters

A layer in a neural network is a matrix multiplication followed by a non-linear function. GPUs are valuable because they do matrix multiplication extremely fast. Writing data as matrices also replaces slow loops with one line of code.

How it works

The shape rule

You can multiply A × B only when A's columns equal B's rows. The inner numbers must match, and the outer numbers give the result's shape: (m × n) × (n × p) = (m × p).

Property	True?
A(BC) = (AB)C	Yes, order of grouping doesn't matter
A(B + C) = AB + AC	Yes
AB = BA	No. Usually false, and this catches many people.
(AB)ᵀ = BᵀAᵀ	Yes, and the order reverses

The maths

Matrix multiplication

(A B)_{ij} = k = 1 \sum n A_{ik} B_{kj}

$(A B)_{ij}$: The result's entry at row i, column j
$A_{ik}$: Row i of A
$B_{kj}$: Column j of B

Transpose

(A^{⊤})_{ij} = A_{ji}

Identity matrix

A I = I A = A, I = [1001]

A = [1 2]    B = [5 6]
    [3 4]        [7 8]

AB row 1: [1×5 + 2×7,  1×6 + 2×8] = [19, 22]
AB row 2: [3×5 + 4×7,  3×6 + 4×8] = [43, 50]

BA row 1: [5×1 + 6×3,  5×2 + 6×4] = [23, 34]
BA row 2: [7×1 + 8×3,  7×2 + 8×4] = [31, 46]

AB and BA are different, which shows why the order matters.

In code

matrices.py

import numpy as np

A = np.array([[1, 2], [3, 4]])
B = np.array([[5, 6], [7, 8]])

print("A x B =\n", A @ B)
print("B x A =\n", B @ A)
print("Transpose of A =\n", A.T)
print("A x identity =\n", A @ np.eye(2, dtype=int))

quantities = np.array([[2, 1], [0, 3]])   # customers x items
prices = np.array([10, 40])               # price per item
print("Bills:", quantities @ prices)

Output

A x B =
 [[19 22]
 [43 50]]
B x A =
 [[23 34]
 [31 46]]
Transpose of A =
 [[1 3]
 [2 4]]
A x identity =
 [[1 2]
 [3 4]]
Bills: [ 60 120]

Common mistakes

Using * for matrix multiplication in NumPy. * multiplies element by element. Use @.
Assuming AB = BA.
Ignoring the shape rule. Most runtime errors in deep learning are shape mismatches.

Check yourself

1. A is 3×4 and B is 4×2. What is the shape of AB?

2. A is 2×3 and B is 2×3. Can you compute AB?

3. Why are GPUs so useful for AI?

Summary

Each entry of AB is a row of A dotted with a column of B.
Inner shapes must match: (m×n)(n×p) = (m×p).
Order matters: AB is usually not BA.

Finished reading?

Sources: Deisenroth, Faisal & Ong, Mathematics for Machine Learning