AI Algorithms
IsoHC: Fixed-Vector Isometric Residual Mixing
IsoHC constrains residual stream mixing to a fixed-vector orthogonal manifold, preserving both stream mean and residual energy in deep architectures.
IsoHC studies a narrow stability problem in stream based residual architectures: how to mix multiple residual streams without destroying the invariants that make deep residual networks trainable. The method constrains the residual mixing operator to preserve both the global stream mean and Euclidean energy. This article summarizes the technical mechanism without disclosing unpublished experimental metrics or run level details. 1. Motivation Residual architectures are stable partly because they keep an identity path through depth: x {l+1} x l+F(x l, theta l). Hyper Connection style models generalize this idea by expanding the residual stream into multiple parallel streams. A simplified residual transport step can be written as: X {l+1} H l^{ mathrm{res}}X l+ mathcal{A} l(X l), where X l in mathbb{R}^{n times d} contains n residual streams and H l^{ mathrm{res}} in mathbb{R}^{n times n} m...