TagsDeep Delta Learning (Jan 2026)
- Title: Deep Delta Learning (Jan 2026)
- Link: http://arxiv.org/abs/2601.00417v1
- Date: January 2026
Abstract
This paper introduces Deep Delta Learning (DDL), a novel architecture that generalizes standard residual connections to overcome the strictly additive inductive bias of ResNets. DDL employs a “Delta Operator”, a learnable rank-1 perturbation of the identity matrix based on the Householder transformation. Controlled by a data-dependent scalar gate and a reflection vector, this operator allows the network to dynamically interpolate between identity mapping, orthogonal projection (erasure), and geometric reflection. This flexibility enables the modeling of complex, non-monotonic state transitions and negative eigenvalues while unifying the processes of feature forgetting and writing within a stable training framework.
Key Topics:
- Deep Delta Learning
- Residual Networks
- Householder Transformation
- Delta Operator
- Neural ODEs
- Geometric Linear Algebra
- State Transitions