Running the default experiment…
Per-step loss on the batch actually used (binned), and the loss on the full training set at each checkpoint. The dotted line marks the selected checkpoint.
Top eigenvalues of the curvature matrix. For gradient descent on a quadratic, λ_max above 2/η is unstable; with mini-batches and Adam the relevant threshold differs.
λ_min on a symmetric log axis, every optimiser. Negative values mean the iterate still sits near a saddle direction.
‖∇L‖ at each checkpoint, computed on the same samples as the curvature.
Share of negative eigenvalues by count (solid) and by absolute spectral mass (dashed).
\(\Delta^{(i)}(t_w,t_w+t)=\frac1P\min\{\|v^{(i)}(t_w)\mp v^{(i)}(t_w+t)\|^2\}\), one curve per waiting time \(t_w\). Curves that collapse are stationary; curves that relax more slowly as \(t_w\) grows show ageing. Reaching the random baseline τ means the eigenvector has fully decorrelated.
\(|\cos(v^{(i)}(t),v^{(i)}(t_{\rm final}))|\). Leading eigenvectors usually settle before the lower ones.
How much of the weight change after \(t_w\) lies along the leading eigenvector at \(t_w\).
Every eigenvector at the selected checkpoint, against its eigenvalue: alignment with the gradient, and with the weight change since the previous checkpoint (the AL field of the reference script). Gradient energy concentrated on the top of the spectrum is the dominant-subspace effect.
\(a_{\rm grad}^{(i)}=|v^{(i)}\cdot\nabla L|/\|\nabla L\|\) against \(a_{\rm upd}^{(i)}=|v^{(i)}\cdot\Delta\theta|/\|\Delta\theta\|\) for the next step. SGD sits on the diagonal; Adam's per-coordinate scaling and momentum pull the update off the leading directions.
\(\sum_{i\le k}(a^{(i)}_{\rm grad})^2\): the share of the squared gradient norm lying in the tracked top-k subspace.
\({\rm IPR}(v)=\sum_j v_j^4\in[1/P,1]\). Near 1/P the vector spreads over all parameters; IPR ≈ 1/k means it lives on about k of them.
Every eigenvector at the selected checkpoint. Bulk eigenvectors near zero are often the most localised, on dead or saturated units.
All optimisers from the same initialisation, the same data and, for first-order methods, the same mini-batch sequence.
Each marker is one eigenvector index, averaged over checkpoints; dashed is the diagonal.
What is computed
A multilayer perceptron with parameters \(\theta\in\mathbb R^P\) is trained on a loss \(L(\theta)\). At every checkpoint the page forms the curvature matrix \(C\) on a fixed set of samples and diagonalises it exactly, \(C=\sum_i\lambda_i v^{(i)}v^{(i)\top}\) with \(\lambda_1\ge\dots\ge\lambda_P\). Eigenvectors are tracked by index, and their sign is fixed so that \(v^{(i)}\cdot\nabla L\le0\), as in the reference script.
Curvature
Hessian. Column \(j\) is a central difference of the exact (reverse-mode) gradient, $$H_{:,j}=\frac{\nabla L(\theta+\varepsilon e_j)-\nabla L(\theta-\varepsilon e_j)}{2\varepsilon},$$ then symmetrised. That costs 2P gradients per checkpoint.
Gauss–Newton. For a least-squares loss \(L=\|r(\theta)\|^2\) with Jacobian \(J=\nabla_\theta r\), \(\nabla^2L=2J^\top J+2\sum_i r_i\nabla^2 r_i\). The Gauss–Newton matrix \(2J^\top J\) drops the second term, is positive semidefinite and coincides with the Hessian at an interpolating solution.
Both matrices are diagonalised by Householder tridiagonalisation followed by the implicit QL algorithm, in double precision.
Eigenvector statistics
Two-time displacement, sign-invariant: $$\Delta^{(i)}(t_w,t_w+t)=\tfrac1P\min\big\{\|v^{(i)}(t_w)-v^{(i)}(t_w+t)\|^2,\ \|v^{(i)}(t_w)+v^{(i)}(t_w+t)\|^2\big\}=\tfrac2P\big(1-|\cos|\big).$$ The random baseline \(\tau^{(i)}\) is the mean displacement between the \(i\)-th eigenvectors of independently initialised networks.
Localisation: \({\rm IPR}(v)=\sum_j v_j^4\), between \(1/P\) (spread evenly) and 1 (one coordinate).
Alignment with the gradient and with the update: $$a^{(i)}_{\rm grad}=\frac{|v^{(i)}\cdot\nabla L|}{\|\nabla L\|},\qquad a^{(i)}_{\rm upd}=\frac{|v^{(i)}\cdot(\theta_{t+1}-\theta_t)|}{\|\theta_{t+1}-\theta_t\|}.$$ Adam rescales each coordinate and mixes past gradients into the step, so its update can leave the sharp directions almost untouched even when the current gradient points along them.
Layer-resolved curvature
With \(P_\ell\) parameters in layer \(\ell\) (weights and biases), the block trace, energy share and curvature density are $$\tau_\ell=\sum_{j\in\ell}C_{jj},\qquad E_\ell=\frac{\tau_\ell}{\sum_{\ell'}\tau_{\ell'}},\qquad \rho_\ell=\frac{E_\ell}{P_\ell/P}.$$ \(\rho_\ell>1\) means the layer carries more than its share of the curvature. LANCE uses this to choose which diagonal blocks of the Gauss–Newton matrix to treat exactly.
Optimisers
- SGD: \(\theta\leftarrow\theta-\eta\nabla L_B\).
- Adam: \(\beta_1=0.9,\ \beta_2=0.999,\ \epsilon=10^{-8}\), with bias correction.
- SAM: gradient taken at \(\theta+\rho\,\nabla L_B/\|\nabla L_B\|\), then an SGD step.
- GN-CG: \(K\) conjugate-gradient steps on \((J^\top J+\lambda I)d=\nabla L\), Armijo backtracking from the unit step, damping ×0.7 after a full step and ×1.5 otherwise.
- LANCE: GN-CG preconditioned by \(M={\rm blkdiag}\big([J_\ell^\top J_\ell+\lambda I]_{\ell\in S},\ [(\mu_\ell+\lambda)I]_{\ell\notin S}\big)\), with \(S=\{\ell:\rho_\ell>1,\ P_\ell\le512\}\) and \(\mu_\ell\) the layer's mean diagonal. Here the residual Jacobian is small enough to form explicitly; the method itself only needs Jacobian–vector products.
PINN residuals
Input derivatives are propagated forward through the network: for a layer \(z=Wh+b\), \(h'=\sigma'(z)\odot z'\) and \(h''=\sigma''(z)\odot z'^2+\sigma'(z)\odot z''\). With the hard constraint \(u=x(1-x)\,{\rm NN}(x)\) the residual \(r_i=\big(u''(x_i)+k^2u(x_i)-f(x_i)\big)/(\|f\|_{\rm rms}\sqrt N)\) is an ordinary differentiable function of \(\theta\), so its gradient, Jacobian and Hessian follow from reverse mode.
Limits of this page
- Full matrices cost \(O(P^2)\) memory and \(O(P^3)\) time, so the page stops at P = 700. The paper's MLP has about 90,000 parameters and uses Lanczos for the top eigenpairs instead.
- Finite-difference Hessians are accurate to roughly \(\varepsilon^2\); with ReLU the kinks add noise.
- Runs are single seeds; repeat with a few seeds before reading much into one curve.
References
- M. Marjankowska, V. Modugno, P. Barucca, Characterizing optimizer-dependent training dynamics through Hessian eigenvector displacement and localization, arXiv:2606.30226.
- LANCE: exact matrix-free Gauss–Newton training for physics-informed neural networks (NeurIPS 2026 OPT workshop submission).