Constrained Training Manifolds

1. **Specify target properties:** Decide whether the goal is stability, efficiency, alignment, or all three.
2. **Choose manifold parameterization:** Define how weights map onto the manifold (e.g., parameterize orthogonal matrices via Cayley transforms).
3. **Modify optimization steps:** After computing gradients, apply projection operators to ensure updates stay on the manifold.
4. **Monitor constraint satisfaction:** Track deviation metrics (distance from manifold) and enforce corrections when thresholds are exceeded.

Projection techniques#

• Geodesic updates: Move along shortest paths on the manifold to maintain constraints naturally.
• Retraction operators: Approximate geodesic steps with computationally cheaper mappings back onto the manifold.
• Penalty methods: Add regularization terms that penalize departures from the manifold, then tighten penalties over time.