What a Convolution Assumes

Suppose the same two-pixel patch should score +1 in the upper left of an image and −1 in the lower right. The pixels under the detector are identical; only their absolute position differs. A shared local kernel cannot represent that rule. This limitation makes the assumptions behind convolution precise.

The rule that is shared

Let $X$ be a single-channel image and $K$ an $h\times w$ kernel. At output position $(i,j)$ , a stride-one layer computes

\left(C_KX\right)_{i,j} =\sum_{a=0}^{h-1}\sum_{b=0}^{w-1}K_{a,b}X_{i+a,j+b}

The short ranges of $a$ and $b$ make the operation local. Sharing appears in what $K$ lacks: its weights have no output-position indices $(i,j)$ . The same coefficients are therefore used at every valid location.

A single-channel $3\times3$ kernel has nine weights whether the image is 32 pixels wide or 1,000. That economy comes directly from preventing each location from choosing its own local rule.

Locality and sharing are separate choices. A locally connected layer can still use a small window while assigning a different kernel to every output position. The counterexample below concerns the additional restriction imposed by sharing: identical patches are sent through identical coefficients.

Following deep-learning convention, the displayed operation does not flip the kernel and is mathematically cross-correlation. For a learned kernel, flipping the parameterization does not change the argument here.

Same evidence, different answer

Take a kernel that matches a two-pixel diagonal:

K=\begin{bmatrix}1&0\\0&1\end{bmatrix}

Same patch, opposite targets

Upper left

Target:+1

Shared response: 2 · row 1, column 1

Lower right

Target:−1

Shared response: 2 · row 4, column 4

The local patch is identical, and the desired target changes only with position. The shared kernel still returns 2 in both places. Gray cells are partial overlaps; blank cells are zero. Rows and columns are numbered from 1.

The kernel returns 2 at both positions. If the task instead asks for +1 in the upper left and −1 in the lower right, fitting both examples would require

\langle K,P\rangle=+1 \quad\text{and}\quad \langle K,P\rangle=-1

The value 2 is incidental. Replace $K$ with any other shared weights and call the resulting score $s$ . Because the local patch $P$ is identical in both examples, both scores must still equal $s$ .

More training data cannot remove the conflict. With squared error, the two examples give

\begin{aligned} L(s) &= (s-1)^2+(s+1)^2 \\ &= 2s^2+2 \end{aligned}

The best shared score is $s=0$ , leaving a loss of 2. This is not an optimization failure. The layer maps the two cases to the same value before the loss is evaluated.

Changing a shared kernel changes both responses in the same way. The layer has access to the patch, not the coordinate needed to distinguish the targets.

Giving each position its own kernel $K^{(i,j)}$ would make the two targets representable, but it would also remove spatial sharing and make the parameter count grow with the number of output locations.

This is a claim about one shared local layer, not an entire network. A larger system may receive position or infer it from information outside the patch. The counterexample isolates what this layer alone cannot express.

The useful consequence: equivariance

Write $T_\delta$ for a translation by $\delta$ . With stride 1, on an infinite grid, or at finite-image locations unaffected by the boundary rule, the shared local layer obeys

C_K(T_\delta X)=T_\delta(C_KX)

Translating the input moves each patch under the same $K$ , so the response moves by the same amount. It does not remain unchanged. This is translation equivariance, not invariance: the layer preserves where the match occurred.

Boundaries complicate the equality because padding defines values beyond the image. A stride larger than one samples only part of the response grid. Shifts divisible by the stride preserve the simple alignment; other shifts generally do not.

What training changes

Training changes the entries of $K$ . It does not decide whether a conventional layer shares those entries across space. At the output of this layer, should identical local patches receive the same score regardless of position? If yes, sharing removes unnecessary parameters and gives equivariance. If no, the missing position must enter the calculation.