Hadamard Product

Contrast multiplying matching matrix entries with composing two transformations, and test when positivity is preserved.

Hadamard Product vs Ordinary Product

The Hadamard product of two matrices A and B is their entrywise product, where each element C_ij = A_ij × B_ij. This differs fundamentally from the ordinary matrix product, which composes linear transformations using dot products of rows and columns. While less common in abstract coordinate-free algebra, the Hadamard product appears frequently in statistics, loss functions, and hardware-accelerated elementwise operations.

Worked example: positive semi-definiteness

A symmetric 2×2 matrix is Positive Semi-Definite (PSD) if its diagonal entries are non-negative and its determinant is non-negative. Let's consider an example:

Matrix A = [[2, -1], [-1, 3]].
Matrix B = [[1, 2], [2, 4]].

Check if A is PSD: diagonals are 2 ≥ 0 and 3 ≥ 0. Its determinant is 2 × 3 - (-1)(-1) = 6 - 1 = 5 ≥ 0. So A is PSD.
Check if B is PSD: diagonals are 1 ≥ 0 and 4 ≥ 0. Its determinant is 1 × 4 - (2)(2) = 4 - 4 = 0 ≥ 0. So B is PSD.
Compute the Hadamard product C = A ∘ B:
C₁₁ = 2 × 1 = 2
C₁₂ = -1 × 2 = -2
C₂₁ = -1 × 2 = -2
C₂₂ = 3 × 4 = 12
Resulting in C = [[2, -2], [-2, 12]].
Check if C is PSD: diagonals are 2 ≥ 0 and 12 ≥ 0. Its determinant is 2 × 12 - (-2)(-2) = 24 - 4 = 20 ≥ 0. So C is also PSD.
In contrast, compute the ordinary product AB:
(AB)₁₁ = 2(1) + (-1)(2) = 0
(AB)₁₂ = 2(2) + (-1)(4) = 0
(AB)₂₁ = (-1)(1) + 3(2) = 5
(AB)₂₂ = (-1)(2) + 3(4) = 10
Resulting in AB = [[0, 0], [5, 10]].

This demonstrates the Schur Product Theorem: the Hadamard product of two PSD matrices is itself a PSD matrix. This holds for our 2×2 non-negative principal minors, where a₁₁a₂₂b₁₁b₂₂ - (a₁₂b₁₂)² ≥ 0.

Experiment: testing matrix operations

Open the Hadamard Product module to test how changing matrix entries affects both products and their properties.

Set the sliders for Matrix A to A₁₁ = 2, A₁₂ = 1, A₂₂ = 2. This gives A = [[2, 1], [1, 2]].
Set the sliders for Matrix B to B₁₁ = 3, B₁₂ = -1, B₂₂ = 3. This gives B = [[3, -1], [-1, 3]].
Observe the Hadamard product C. It computes to [[6, -1], [-1, 6]]. The ordinary product AB computes to [[5, 1], [1, 5]].
Check the "Analysis: PSD Preservation" panel. Both A and B are PSD, so the Schur Product Theorem applies, guaranteeing C is PSD. Confirm this matches the Positive Semi-Definite indicator for the Hadamard Product.
Now, adjust A₁₂ to 3, making A = [[2, 3], [3, 2]]. A is no longer PSD because its determinant is 2×2 - 3×3 = 4 - 9 = -5. The theorem now gives no guarantee about the product being PSD.

Limitations and common confusions

The Schur Product Theorem only applies if both matrices are PSD. If either matrix is not PSD, the theorem simply gives no guarantee; it does not necessarily mean the Hadamard product will fail to be PSD.

The all-ones matrix J is the identity for the Hadamard product: A ∘ J = A. An entrywise reciprocal is defined only when every entry is nonzero; it is not the ordinary matrix inverse. In particular, ordinary inverse identities cannot be transferred to entrywise multiplication.

Furthermore, be careful not to confuse the Hadamard product (elementwise multiplication) with the Hadamard transform, which is a specific type of Fourier transform operating on vectors using matrices of 1s and -1s.

References

The elementwise multiply operation is a standard part of numerical array libraries. See the official documentation: numpy.multiply, which supports the entrywise operation and details how it contrasts with ordinary matrix multiplication.