In which space is the object high-dimensional?#

Fix an architecture. Parameters are $\theta\in\Theta\subseteq\mathbb R^D$, inputs belong to $X$, and the output function is $f_\theta:X\to\mathbb R^k$. Here $D$ is parameter dimension and $k$ output dimension. An input can be an encoded sequence rather than a scalar. Write the representation at layer $\ell$ as $z_{\ell,\theta}:X\to\mathbb R^{d_\ell}$. Parameter space, function space, and each representation space now have distinct elements and coordinates.

Suppose training forms a trajectory $\theta_t$. Write $f_t=f_{\theta_t}$ and $z_{\ell,t}=z_{\ell,\theta_t}$. Initially, $t$ means continuous training time; a stepwise algorithm requires its own discrete step count. Layer index $\ell$ tells us where a forward computation is, rather than how long training has lasted. Autoregressive position tells us how many tokens have been generated or supplied. Calling all three “iterations” obscures the object compared by a curve.

If the input population has law $P_X$ and the representation map is measurable, we can study the pushforward

This is the probability that a representation of an input falls in $B$. Changing the population can change $\mu$ with parameters fixed; changing parameters can change it with the population fixed. Separating these sources lets us analyze training and population effects on a scatterplot respectively. When both change, comparison must specify which conditions it holds fixed. Without a specified population, a finite-sample plot observes those samples.

Distances have different addresses too. $\|\theta-\theta'\|_2$ measures parameter-coordinate differences. For fixed input, we can measure $\|z_{\ell,\theta}(x)-z_{\ell,\theta'}(x)\|_2$. Function comparisons can concern sample error, population mean error, or maximum error on an input domain. For square-integrable outputs, for example, $\|f-g\|_{L_2(P_X)}^2=\int\|f(x)-g(x)\|_2^2\,dP_X(x)$. Small population error can coexist with large differences in a low-probability region; maximum error includes that region.

The third essay’s hidden-unit permutation gave an exact comparison. Rearranging adjacent weights together preserves all of $f$, rearranges representation coordinates, and usually changes the parameter array. Comparing representations coordinate by coordinate reveals a change; aligning the permitted permutation first may give zero difference. Both readings have mathematical meaning, selected by purpose. Function equality also did not establish equality of continued training, since updates may use those coordinates.

We can consequently make “a high-dimensional model is complex” more precise. Parameter and realization size matter directly for computational cost. Functions and layer operators are closer to input-perturbation questions. Feature formation requires representations and their population relations rather than a terminal loss alone. Choosing spaces gives an ambitious inquiry objects that can actually be compared.

How transformations retain, lose, and add content#

First distinguish changes along objects already developed. An invertible coordinate change carries structure to another presentation. The third essay’s $V=Ah$, for fixed $A>0$, converted evolution, initial conditions, and thresholds together, permitting recovery in both directions. Changing the variable but keeping an old initial condition does not perform that correspondence. Invertibility and preservation jointly support further reasoning; numerical resemblance does not prove them.

A quotient groups objects already declared equivalent. Hidden-unit permutation can remove coordinate redundancy; absolute-value observation identifies $x$ and $-x$. Continuing an operation after identification requires independence from the representative. Absolute values determine the absolute value of a product but not a sum. Learning raises the same obligation: if parameters realizing the same function induce different next functions, training has not become a single-valued update on that function quotient.

Coarse-graining or projection retains some information. A fine object generally determines its projection, while the projection need not recover the fine object uniquely. A vector of predictions on fixed samples projects a function. Agreement there does not establish agreement elsewhere. Whether lost information matters depends on subsequent queries. New inputs or compositions may require a richer observation or enough added state to close the coarse object.

A lift adds needed structure. Sometimes the original topology is too weak to support a response we already want to study. For integers $n\ge1$, take

The paths converge uniformly to zero because their maximum amplitude is $n^{-1/2}$. Yet $dx_n^2=\sqrt n\cos(ns)\,ds$, so $\int_0^{2\pi}x_n^1\,dx_n^2=\int_0^{2\pi}\cos^2(ns)\,ds=\pi$. The zero path has zero integral. Uniform convergence of path amplitude has not preserved this response. If the problem uses it, richer objects including the relevant iterated integrals are needed. The first essay connected this entrance to rough-path construction.

Approximation needs common observation and error. For the earlier scalar storage law $\dot V=-aV$, $a>0$, exact sampling has factor $e^{-a\Delta}$; Euler has $1-a\Delta$. For nonnegative initial values, Euler preserves nonnegativity when $0<a\Delta\le1$ and decays to zero when $0<a\Delta<2$. In $1<a\Delta<2$, values decay but alternate in sign. Terminal decay and nonnegativity throughout are different observations.

There is also a finite-interval error relation. For $0<a\Delta<1$ and $j\Delta\le H$, let $\delta=-\log(1-a\Delta)-a\Delta$. Using $\delta=\int_0^{a\Delta}u/(1-u)\,du\le(a\Delta)^2/[2(1-a\Delta)]$ and $1-e^{-j\delta}\le j\delta$ gives

For fixed $H$, the right side vanishes. It compares grid values with the same initial condition rather than recovering all continuous-interval properties. This storage law also differs from the second essay’s orifice-outflow law. Model, computation, and query retain their own addresses. The example supplies a short bridge from finite approximation to the network questions below.

Taking a limit requires a sequence of differently sized objects to enter a common space. Width changes parameter dimension, so arrays do not immediately have a shared fixed-coordinate distance. Function values, kernels, empirical measures, or normalized statistics can provide comparison. The object of convergence, its topology, and whether time or input ranges are fixed affect the conclusion. A limit is not synonymous with “sufficiently large.” Proximity of a finite object requires another error relation.

Operations can be combined, but their retained relations must connect. Quotienting before training requires an update that descends. Projecting before a limit requires the relevant continuity. Approximating before a long-time judgment requires long-time error control. Changes in mathematical language now have concrete content: some re-coordinate an object, some give a question richer structure, and some change the research object itself.

How parameter training induces function change#

Fix $m$ training inputs $x_1,\ldots,x_m$ and first take scalar outputs. Define the prediction vector $F(\theta)=(f_\theta(x_1),\ldots,f_\theta(x_m))\in\mathbb R^m$. This observes the current samples, while $f_\theta$ remains defined on all of $X$. Use a differentiable loss $\ell:\mathbb R^m\to\mathbb R$, giving objective $\mathcal L(\theta)=\ell(F(\theta))$. Labels and samples are fixed here, entering $\ell$ and $F$.

Assume differentiability along the trajectory and Euclidean parameter gradient flow. Let $J_F(\theta)\in\mathbb R^{m\times D}$ be the prediction Jacobian, with row $i$ equal to $\nabla_\theta f_\theta(x_i)^T$. The chain rule gives parameter velocity, then prediction velocity:

All three identities hold for the current finite network. They require no infinite width. $K_\theta$ is an $m\times m$ Gram matrix, with entry $(i,j)$ the inner product of parameter gradients at two inputs. It describes how a loss direction moves sample predictions through parameter space. If the direction induced by the first sample’s error changes the second prediction, the cross term carries that influence. The process contains more than independent diagonal adjustments.

The matrix is positive semidefinite: for every $u\in\mathbb R^m$, $u^TK_\theta u=\|J_F^Tu\|_2^2\ge0$. Along the trajectory, $d\ell(F)/dt=-\nabla\ell(F)^TK_\theta\nabla\ell(F)\le0$. Loss is nonincreasing, but convergence to arbitrary labels does not follow. Some prediction directions may belong to a nullspace, and the matrix may change during training. Descent and attainable directions are distinct questions we can now investigate.

For fixed input $x$ outside the training samples, the chain rule gives

The sample matrix $K$ restricts this whole-input training kernel to sample pairs. Knowing $K$ supports sample-prediction analysis. Evolution at a new input also needs its cross-kernel values with the training inputs. A finite Gram matrix does not uniquely determine the kernel on all inputs or establish generalization error on a target population.

We temporarily use $\ell(F)=\frac12\|F-y\|_2^2$, without a $1/m$ averaging factor. An averaged loss also changes gradients and time scale; different source conventions require conversion. Multiple outputs can be flattened across samples and coordinates, yielding a Gram matrix that retains cross-coordinate coupling. Rearrangement changes its presentation, while corresponding rearrangement of observation and loss keeps the inference aligned.

Euclidean flow is one choice. With a positive-definite preconditioner $M(\theta)$, velocity becomes $-M\nabla\mathcal L$ and the prediction matrix becomes $J_FMJ_F^T$. Momentum adds state, stochastic gradients add a random process, and discrete step size adds a one-step approximation. Each can have its own semantics and observation. Treating them all as the same $K$ would erase structure supplied by the algorithm.

The finite identity first shows how parameters carry loss directions into function change, rather than simplifying a model immediately to a fixed kernel. We can now ask whether the kernel freezes, has a limit, or retains feature learning. The third essay’s equal functions with unequal training have a concise address: equal $F$ may have different $J_FJ_F^T$, hence different prediction velocities.

Fixed-kernel evolution and comparison with a changing kernel#

If $K_\theta$ is exactly fixed at $K_0$ under specified conditions, squared-loss residual $r=F-y$ satisfies $\dot r=-K_0r$, giving $r(t)=e^{-K_0t}r(0)$. A symmetric positive-semidefinite matrix has an orthogonal eigendecomposition. Each component of the initial residual decays as $e^{-\lambda_it}$; nullspace components remain. A fixed kernel thus connects learnable directions and rates explicitly.

A positive minimum eigenvalue gives exponential decay of every sample-residual component. A small eigenvalue gives a slow direction; a zero eigenvalue retains the initial nullspace residual. These judgments concern the current samples and loss. Changing samples changes the Gram matrix; changing purpose changes the error that matters. Fast training-error decay does not determine risk outside the sample set alone.

A finite network generally supplies $K(t)$. To approximate it by a fixed kernel, specify the error. Let both prediction processes start at the same $F(0)$, with original residual $r(t)$ and frozen-kernel residual $r_0(t)$. Assume $\|K(t)-K_0\|_{\mathrm{op}}\le\varepsilon_T$ on $[0,T]$. Their difference $e=r-r_0$ satisfies

Since $K_0$ is positive semidefinite, $\|e^{-K_0u}\|_{\mathrm{op}}\le1$. Original loss descent gives $\|r(s)\|_2\le\|r(0)\|_2$. Therefore $\|e(t)\|_2\le t\varepsilon_T\|r(0)\|_2$. This bound needs the stated conditions rather than learnability of every original-kernel direction. Small kernel variation becomes a finite-time prediction bound; extending time requires another estimate.

Approximation by another kernel $\widehat K_0$, or different initial predictions, introduces additional terms in the difference equation. A bound assuming identical initial conditions cannot be retained unchanged. New-input analysis also needs changes in its cross-kernel values. Freezing now has a visible benefit and cost: training becomes a linear residual process, and changing geometry enters an error that must be controlled.

Finite points can also reach a larger observation through input geometry. Suppose the common input domain $U\subseteq\mathbb R^p$ is compact and test points form a radius-$h$ covering net. If the two functions are $L_f,L_g$ Lipschitz on $U$ and net-point error is at most $\delta$, choose a point within $h$ of any $x\in U$. The triangle inequality gives $|f(x)-g(x)|\le\delta+(L_f+L_g)h$. Coverage and uniform continuity together carry finite comparisons to the whole domain. High-dimensional domains may need many points; smaller data support changes coverage. These conditions must come from the current object. Sampling from a population does not itself cover all legal inputs.

This gives a useful empirical question. Compare $K(t)$ on selected samples and check whether fixed-kernel approximation covers the needed time and predictions. If feature formation is the target, also observe representations and readout relations rather than declaring equal mechanisms from similar predictions. An approximation can serve current predictions without retaining every internal question. The required content should guide adoption.

We have not yet established that greater width makes every network a kernel. We have a finite identity, a conditional fixed-kernel solution, and a checkable error bound. How width reduces variation depends on parametrization, initialization, learning rate, activation, and training range. The next section puts those conditions into a comparison of two scales.

How width choices retain feature formation#

Take a shallow network we can calculate directly, with scalar input $x\in[-R,R]$, activation $\sigma=\tanh$, and hidden width $N$. First use

Both $a_i,w_i$ are trainable; write $f_N(t,x)$ when training time must be explicit. Parameters use ordinary Euclidean gradient flow and loss remains a squared-error sum on fixed $m$ samples. Width $N$ differs from sample count $m$. To make the scaling statement a complete finite construction, take even $N$, with paired units initialized at equal $w$ and opposite $a$, and uniform initial bounds on $|a_i|,|w_i|$. Then $f_N(0,x)=0$ on all inputs and initial loss does not grow with width.

This deterministic pairing removes initialization fluctuations for the construction, rather than describing every training initialization. Let $r_j=f_N(x_j)-y_j$. Calculation gives

Loss descent uniformly bounds $\|r(t)\|_2$ in $N$. Since $|\sigma|\le1$, $|\sigma'|\le1$, and inputs and sample count are fixed, $|\dot a_i|\le C/\sqrt N$. On fixed $[0,T]$, all $a_i$ remain uniformly bounded, giving $|\dot w_i|\le C_T/\sqrt N$. Integration bounds both parameter changes by $O_T(N^{-1/2})$ and representation changes $\sigma(w_ix)$ by the same order uniformly on $|x|\le R$.

The factor $N^{-1/2}$ comes from output parametrization. Loss and boundedness conditions then extend it over a finite time interval. This is a calculation rather than the intuition that many parameters each change little. Time growing with $N$ may change constants and accumulated effects. A width-dependent diverging initial loss or another learning-rate scale would likewise require a new argument.

The corresponding training kernel is

The first and second derivatives of $\sigma$ are bounded on the current range. Each summand changes by $O_T(N^{-1/2})$, and averaging keeps that order. On fixed samples, $K_N(t)-K_N(0)$ has an analogous operator bound, giving frozen-kernel prediction error through the preceding section. If initial empirical statistics also have a definite limit, we can study convergence of $K_N(0)$. Feature freezing, kernel approximation, and the initial-kernel limit are three connected steps.

Now use a different scale:

Output becomes an average and training speed is multiplied by $N$. Each unit has $\dot a_i=-\sum_j r_j\sigma(w_ix_j)$ and $\dot w_i=-a_i\sum_j r_j\sigma'(w_ix_j)x_j$. Width factors cancel, so the previous $N^{-1/2}$ bound no longer forces feature changes to vanish. They may still vanish for particular loss, data, or initialization. A scale admitting finite change is different from every task necessarily producing it.

The same pairing construction exhibits the possibility. Initialize every $w_i(0)=1$, half the $a_i(0)=1$, and half $-1$, with sole sample $x=1$ and label one. Both networks initially represent the zero function. The second scale has residual minus one and $\dot w_i(0)=a_i(0)\sigma'(1)$. Positive and negative groups separate at rates that do not shrink with $N$. The first scale adds a factor $N^{-1/2}$. Equal initial functions do not impose the same scale of feature evolution.

Collect parameter pairs into $\lambda_t^N=N^{-1}\sum_i\delta_{(a_i(t),w_i(t))}$. The average network becomes $\widetilde f_N(x)=\int a\sigma(wx)\,d\lambda_t^N(a,w)$. Each particle’s velocity depends on shared sample residuals. Feature learning gains another object: moving parameter particles change their distribution, which changes the function. This relation already holds at finite $N$. A general-initial-measure limit additionally needs existence, stability, and finite-error arguments; writing an integral does not establish them.

Tensor Programs IV gives a broader classification for fixed-depth MLPs and abc parametrizations. Its classification uses specified $\tanh$ or smooth activations and stable, nontrivial parametrizations, with explicit training-routine and feature-change definitions.¹ Under those conditions it separates feature-learning and kernel regimes; linear activations outside them cannot simply inherit the classification. Admitting feature learning requires some routine, fixed step, and input with nonvanishing feature change in the width limit. The kernel regime requires one fixed positive-semidefinite kernel organizing updates for every admissible routine. These are existential and universal judgments respectively.

Our two finite-flow calculations show how scale enters change. The source classification concerns its defined discrete SGD limits. They connect through the mechanism question while retaining different proof objects. If feature formation is the target, choose a model retaining it or specify finite corrections. If prediction with fixed features is the target, kernel dynamics can be useful. Expressivity, training reachability, population risk, and feature mechanism each have an object; simplification at one level does not answer them all.

Why changing parameter coordinates also requires a metric#

The third essay’s $f_{a,b}(x)=abx$ showed different Euclidean training velocities for one function. A coordinate change reveals the relation directly. Let $\theta=h(\eta)$ be smoothly invertible, with invertible Jacobian $H=Dh(\eta)$. Both coordinates represent the same function and loss. But adopting Euclidean gradients anew in $\eta$ gives $\dot\eta=-H^T\nabla_\theta\mathcal L$, hence

This generally differs from the original $-\nabla_\theta\mathcal L$. Loss-value correspondence has been preserved, while geometry of parameter motion changes. Orthogonal transformations satisfy $HH^T=I$ and preserve ordinary Euclidean gradients; arbitrary scaling does not. Hidden permutations in the third essay supplied an orthogonal transformation with a provable preservation relation.

To retain original Euclidean geometry in new coordinates, give tangent vectors the pullback metric $G=H^TH$. Then $\dot\eta=-G^{-1}H^T\nabla_\theta\mathcal L$, and multiplying by $H$ yields $\dot\theta=-\nabla_\theta\mathcal L$. Coordinates and metric now move together, giving corresponding trajectories. A language change can reveal the geometry of a process; a variable change followed by a default metric may instead define another process.

This also connects learning rates and units. Scaling a coordinate by ten changes gradient scaling in the reverse direction. Keeping one numerical step size on each coordinate then gives different actual motion. Metric and preconditioner organize direction and magnitude together. Saying only that loss decreases retains a scalar result without specifying how the process moves in parameter space.

Quotients have their own difficulty. If symmetry preserves function and metric, and updates commute with the group action, training may define a good process on the quotient. If equal-function parameters induce different $J_FJ_F^T$, function values alone do not determine the next velocity. Retain more state or alter training to make it close. Quotienting removes proved redundancy, while variables affecting the future need their own treatment.

The metric here is first a mathematical choice. Suitability for optimization requires further grounds. It may improve conditioning, change implicit preferences, or change reachable processes; usefulness must be checked along loss, data, realization, and purpose. Self-attention will give a concrete later example: changing a metric retains the vector field while changing its gradient interpretation. Both cases require identifying objects before calculating preservation.

Giving effective geometry calculable content#

Parameter dimension $D$ may be large while one observation involves fewer directions. To give effective complexity content, specify a direction set $S\subseteq\mathbb R^D$ and norm. For standard Gaussian $g\sim N(0,I_D)$, define Gaussian width $w(S)=\mathbb E\sup_{u\in S}\langle g,u\rangle$. It measures the largest random linear observation across the set, rather than its cardinality or manifold dimension.

Take a $k$-dimensional linear subspace $E$ and its unit sphere $S$, with $1\le k\le D$. Projection $\Pi_Eg$ is standard Gaussian within that subspace. Selecting the best unit direction gives $\sup_{u\in S}\langle g,u\rangle=\|\Pi_Eg\|_2$, so

Jensen’s inequality and $\mathbb E\|g_k\|_2^2=k$ give the upper bound. For the lower bound, $\|g_k\|_2\ge\|g_k\|_1/\sqrt k$, while each standard Gaussian coordinate has expected absolute value $\sqrt{2/\pi}$. Width grows as $\sqrt k$; unused ambient coordinates do not enlarge this set’s width. The structure is a specified subspace, rather than one supplied automatically by a network’s size.

Next take $M$ fixed unit directions $u_1,\ldots,u_M$. Their observations may be correlated, yet all can be controlled. For $s>0$, $e^{s\max_i|\langle g,u_i\rangle|}\le\sum_i(e^{s\langle g,u_i\rangle}+e^{-s\langle g,u_i\rangle})$. Gaussian moment-generating functions bound the right side’s expectation by $2Me^{s^2/2}$. Taking logarithms, applying Jensen, and optimizing $s$ gives

For $r\ge0$, applying one-dimensional Gaussian tails and a union bound similarly gives $\Pr(\max_i|\langle g,u_i\rangle|\ge r)\le2Me^{-r^2/2}$. Independence across directions is unnecessary. The directions must be fixed or independent of the Gaussian draw being tested. Uniform control of many queries incurs a logarithmic dependence on $M$ beyond one-direction scaling.

Selecting the direction from the same noise can defeat the bound. Let the sole direction be $u(g)=g/\|g\|_2$, defined arbitrarily at zero. Then $\langle g,u(g)\rangle=\|g\|_2$, with expectation of order $\sqrt D$. Treating it as a preselected direction would incorrectly give a dimension-independent bound. Dependence between learned directions and test randomness is therefore a condition to establish. Sample splitting, conditioning, or a new complexity estimate may supply the connection.

Small intrinsic dimension also requires a structural choice. A planar circle is a smooth one-dimensional manifold with a two-dimensional linear span. A finite set can have local manifold dimension zero while containing many distant directions. For hypercube vertices $S=\{(\pm1,\ldots,\pm1)/\sqrt D\}$, for example,

A local zero-dimensional label has not made global directional complexity small. This does not contradict the subspace calculation; the structures differ. Discrete mixtures, low rank, smooth manifolds, and branching sets can resemble one another in a two-dimensional projection while requiring different coverage for global perturbations and uniform queries.

For representations, we might study population support, tangent directions near data, or changes reachable during training. Each choice requires identifying how the set is obtained, its distance, and the comparison scale. Low covariance rank states a second-order linear property without recovering a smooth manifold alone. Information readable from a variable does not alone establish that the output uses it. Geometry sharpens the question; mechanisms reconnect through the appropriate intervention and generation relations.

Why propagation through layers requires operators#

Fix trained parameters and let forward layers satisfy $z_{\ell+1}=T_\ell(z_\ell)$. At differentiable points, first-order input changes propagate through Jacobians: $\delta z_L\approx J_{L-1}\cdots J_0\delta z_0$. Products are ordered and directions change between layers. Collecting one eigenvalue per layer does not specify that effect; exchanging order may change the product itself.

Finite perturbations require uniform Lipschitz conditions on relevant connecting paths or visited regions. A derivative at one point describes a local first-order response. Spectral norms give $\|J_{L-1}\cdots J_0\|_{\mathrm{op}}\le\prod_\ell\|J_\ell\|_{\mathrm{op}}$. This is useful but can be loose when directions do not align. Studying the product, reachable directions, or task observation can yield a comparison closer to the actual question.

Stable eigenvalues can coexist with substantial transient amplification. For $M>0$, take

Both eigenvalues are minus one, and every fixed initial response eventually vanishes. Starting from $e_2=(0,1)^T$, however, the norm at $s=1$ is $e^{-1}\sqrt{1+M^2}\ge M/e$. Increasing $M$ makes amplification at this finite time arbitrarily large. Here $s$ is evolution time in the operator construction, rather than training time or an actual network’s layer index.

The matrix is nonnormal, so an orthogonal eigenbasis cannot decouple all directions. Asymptotic spectrum describes eventual decay; singular values and induced norms describe maximal amplification at a specified time. Even diagonalizable matrices require carrying the coordinate change’s condition number back to the original norm. Listing eigenvalues has not performed that return.

This construction establishes the need for operator comparisons, rather than reporting the same mechanism in an actual Transformer. Studying a trained network needs its corresponding Jacobians, norms, and input range, followed by the required task observation. Normalization, residual connections, activation, and attention coupling all enter those operators. Derivatives observed on a few samples address those samples and their relevant neighborhoods.

Operators connect geometry and scale too. A large full-space norm may mainly involve directions absent from current data; a small average may miss rare directions central to a task. Restriction to a specified direction set can help. If training, intervention, or context moves outside that set, its bound must be checked again. Effective geometry chooses the comparison, and operators specify how change passes through it.

Two ways contributions accumulate with depth#

Width changes coordinate scale within layers; depth adds connected layers. Write a residual network as $z_{\ell+1}=z_\ell+s_Lg_\ell(z_\ell)$, with total depth $L$ and branch multiplier $s_L$. Random initialization and training-induced updates need not accumulate alike. The former can partly cancel; the latter can align through shared loss. Dividing each layer by depth does not yet specify which process is retained.

First inspect initialization through an explicit scalar construction. Take independent $\xi_1,\ldots,\xi_L\sim N(0,\sigma^2)$ and $S_L=L^{-\alpha}\sum_{j=1}^L\xi_j$. Its law is exactly $N(0,\sigma^2L^{1-2\alpha})$. At $\alpha=1/2$ its distribution is unchanged with $L$; above that it tends to zero in mean square; below it variance grows and probability in any fixed bounded interval vanishes.

Between layers separated by roughly $\varepsilon L$, variance at $\alpha=1/2$ tends to $\sigma^2\varepsilon$, giving root-mean-square change $\sigma\sqrt\varepsilon$. This differs from a smooth deterministic depth curve with $O(\varepsilon)$ change over the same normalized interval. Keeping terminal output bounded does not distinguish these relations when the question concerns diversity across layers.

Independence, zero mean, and Gaussian law belong to this construction. Real branches depend on previous states, and training can correlate them. They cannot all be treated as these independent summands automatically. Nonzero means accumulate as $L^{1-\alpha}$; identical summands have variance of order $L^{2-2\alpha}$. Means and correlation change the required scale beyond branch amplitude alone.

In general, $\operatorname{Var}(\sum_jY_j)=\sum_j\operatorname{Var}(Y_j)+2\sum_{i<j}\operatorname{Cov}(Y_i,Y_j)$. Roughly $L^2$ covariance terms can change the total order even when individual terms are small. Training updates respond to shared labels or loss, making correlated accumulation relevant. If every layer moves by $L^{-1}v$ in one shared direction, total change is $v$; movement by $L^{-1/2}v$ gives $\sqrt Lv$. A scale accommodating initialization does not automatically accommodate effective training updates.

Linear residual networks permit another exact recursion. With width $N$, independently centered entries of $W_\ell$ having variance $1/N$, independent of preceding state, let $z_{\ell+1}=(I+L^{-\alpha}W_\ell)z_\ell$. Conditional on $z_\ell$, the cross term has zero expectation and $\mathbb E\|W_\ell z_\ell\|_2^2=\|z_\ell\|_2^2$. Therefore

This is a finite-network initialization identity. Its multiplier tends to $e$ at $\alpha=1/2$, to one above that, and diverges below. The second-order statistic retains an energy scale rather than uniform guarantees for every coordinate, sample path, or high-probability event. Keeping the conclusion at its address permits further appropriate controls.

After training, $W_\ell$ no longer retains its original independence from forward and backward states. An update is formed from layer input and output-propagated gradient, then changes later inputs. The initialization cross term cannot still be set to zero without justification. A model redrawing independent weights at every step changes the training object. It may serve another mechanism but cannot replace the original process solely under the label random network.

Tensor Programs VI studies joint scaling of branch multipliers, learning rates, and gradient processing. Its general setup uses widthwise $\mu$P before the depth limit and conditions including mean subtraction. Stability, nontriviality, nonlinearity-input scale, and feature diversity enter separately.² Bounded output is one requirement; a trainable limit can still make nearby layers too similar.

In its scale notation, branches use $L^{-\alpha}$ and updates after unified gradient processing use $L^{-\gamma}$. The choice $\alpha=\gamma=1/2$ is Depth-$\mu$P for its single-layer residual blocks. Raw SGD and coordinate-normalized optimizers require different learning-rate scales; transferring a numerical rate directly does not preserve effective updates. The next section calculates this distinction in one exact step and separates local from global linearization.

Why small changes in every layer can still change the whole network#

Take a scalar residual network with input one and $L$ factors $1+L^{-\alpha}w_\ell$, giving

Initialize all $w_\ell(0)=0$, with target zero and loss $f^2/2$. Initial output is one and every loss gradient is exactly $L^{-\alpha}$. One ordinary SGD step with raw rate $\eta_L=cL^{2\alpha-1}$, $c>0$, gives $w_\ell(1)=-cL^{\alpha-1}$. Each layer’s multiplier becomes $1-c/L$, so

For $\alpha<1$, every parameter update vanishes, as does every multiplier change, while total output retains a finite change. Small changes multiply across increasingly many layers. Vanishing one-layer updates do not establish global freezing. The zero initialization is deliberately chosen for this construction; it does not satisfy every preceding random-network condition or classify general residual networks.

Linearizing the entire output once in parameters at initialization gives $f_{\mathrm{lin},1}=1+\sum_\ell L^{-\alpha}(-cL^{\alpha-1})=1-c$, generally different from $e^{-c}$. Each layer was already linear in its own parameter, while multiplication between layers retained higher-order interactions. Layerwise linearization and global parameter-function linearization are separated by a complete construction.

The product expansion explains the retained order. Roughly $L^2/2$ second-order terms each have size $c^2/L^2$, so their total need not vanish. Higher orders accumulate with their corresponding combinatorial counts. Inspecting each small term without its count or alignment loses the total change. A global fixed-kernel process needs its own conditions beyond small layer updates.

Optimizers change rate scaling further. Ideal coordinatewise sign updates replace each positive gradient by one. At the same initial point, rate $\eta_L=cL^{\alpha-1}$ gives the same multiplier $1-c/L$. For $\alpha=1/2$, raw ordinary-SGD rate is constant-order, whereas sign-update rate is $L^{-1/2}$. Comparable global changes require different raw step scales.

The sign update has no momentum or stabilizing constant and is not complete Adam. Actual normalization can contain $g/(|g|+\epsilon)$, $\epsilon>0$. For gradients much smaller than $\epsilon$, this approaches $g/\epsilon$ rather than sign. Scaling must check whether gradient-processing inputs stay in the original regime. Tensor Programs VI retains gradient prescaling and effective processing inputs separately; omitting them changes the limit.

Within-block interaction introduces another obligation. A two-layer linear branch has $(W_2+\Delta W_2)(W_1+\Delta W_1)$: an old product, two first-order terms, and $\Delta W_2\Delta W_1$. Assume uniformly bounded original operator norms, $O(L^{-1/2})$ update norms, and outside branch factor $L^{-1/2}$. The cross term is $O(L^{-3/2})$, totaling only $O(L^{-1/2})$ under simple addition over $L$ blocks.

That calculation has not included arbitrary amplification between blocks. Without uniform bounds on preceding and subsequent operator products, small terms can amplify, as the nonnormal example showed. Under the required propagation conditions, within-block cross terms can disappear while first-order updates continue combining across blocks. A limit retains some interactions and removes others. Its suitability depends on which interactions carry the mechanism under investigation.

Tensor Programs VI studies the distinction, with different proof statuses. Section 4 and appendices develop linear-network sequential limits and classification; depth-convergence estimates in the chosen version include $L=2^k$. Section 7 explicitly labels general nonlinear classification as claims, with heuristic proofs and additional technical conditions. Section 9 and experiments address interaction limits in multilayer residual blocks. General derivations, finite-model experiments, and rigorous linear results consequently support different judgments.

Scale selection can organize finite updates, trained features, and diversity across layers. It also asks whether the choice has removed the mechanism of interest. A mathematically convenient scale, or hyperparameter transfer in specified experiments, still needs connections to another finite architecture, training duration, and optimizer. Existence of a limit, existence of feature learning, and transfer of optimal hyperparameters are related but distinct judgments.

From attention updates to spherical dynamics#

Now fix trained parameters and examine one sequence across layers. Let token representations be $z_1,\ldots,z_n\in\mathbb R^d$. Single-head attention turns query-key inner products into weights and aggregates values. Set $A=Q^TK$ and inverse-temperature parameter $\beta>0$:

Rows are nonnegative and sum to one. They weight values inside the current sequence rather than give vocabulary-class probabilities. Each $a_{ij}$ depends on $z_i,z_j$ and other tokens through the denominator. Changing one token can change its value and other rows’ aggregation. Sequence length $n$, hidden width, and vocabulary size have different positions in the model.

First specify a simplification permitting evolution analysis. Representations lie on $S^{d-1}$ and normalization projects to the sphere; $Q,K,V$ are fixed and shared across layers. Multihead attention, feed-forward sublayers, and masks are temporarily removed. This entrance follows Geshkovski and colleagues’ mathematical model.³ It retains nonlinear attention coupling. The omitted structures still require reconnection for a complete architecture.

Suppose a layer uses a small residual step $\Delta>0$: $z_i^+=\bigl(z_i+\Delta u_i(Z)\bigr)/\|z_i+\Delta u_i(Z)\|_2$. For unit $z_i$, its denominator is $1+\Delta\langle z_i,u_i\rangle+O(\Delta^2)$. Uniformly bounded $u_i$ gives

Projection $P_z$ puts velocity in the spherical tangent space. The first-order continuous-depth model is $dz_i/d\tau=P_{z_i}u_i(Z)$, where $\tau$ is continuous layer depth. Since $z_i^TP_{z_i}=0$, norms remain spherical. This does not turn training into a continuous process; matrices stay fixed throughout forward evolution.

The discrete-continuous connection also has a finite relation. With a Lipschitz field on the current compact domain and uniformly bounded one-step remainder $C\Delta^2$, errors propagate by the Lipschitz factor. Discrete Gronwall gives $O(\Delta)$ global error on fixed depth intervals. Arbitrarily varying matrices, nonshrinking steps, or uncontrolled remainders do not supply that relation. A network with a given layer count alone does not specify a family with $\Delta\to0$.

Spherical normalization retains its own identity. Actual LayerNorm can include mean, variance, learned scale, and shift; reduction to unit norm is a model choice. It makes the domain compact, uniformly bounds exponential weights, and enables tangent and geometric analysis. Restoring other parameters requires checking changes in domain and vector field before applying spherical proofs to that normalization.

Without positions or masks, a shared update permutes outputs when tokens are permuted. Each row proves this: rearrangement moves the exponential numerators and denominator sum together, rearranging attention rows and columns; value aggregation and projection follow. This is permutation equivariance on labeled sequences rather than equality of outputs at unchanged positions. Passing further to an unlabeled set or measure removes order.

Positions need another object. Concatenating content with distinguishable position labels can preserve position in each measure atom. Adding content and position vectors requires injectivity on the allowed content domain to recover both; addition itself does not supply it. A causal mask lets a position receive only earlier positions, so labels and available information must enter the kernel or history rather than remain an unrestricted symmetric sum.

The simplification obtains a defined question: how token representations interact across depth under fixed training conditions. It permits calculations of clustering, stability, and measures with explicit return endpoints. Formation of trained matrices, advancing generation positions, and reading task outputs need other connections. The next section distinguishes the three measures already present.

Why three measures cannot replace one another#

Original relationship diagram. Training time $t$, layer index $\ell$, and continuous depth $\tau$ are distinct. The body develops the measures’ spaces and sampling units; arrows assemble mappings already specified.

Earlier, $\mu_{\ell,t}=(z_{\ell,t})_\#P_X$ described a chosen representation across inputs. Shallow feature learning used $\lambda_t^N$ for parameter particles. Within one sequence, define

The sampling unit for $\nu$ is a token; for $\mu$, an input; for $\lambda$, a hidden unit’s parameter pair. All use probability-measure language with different underlying spaces, size indices, and sources of change. Calling them all representation distributions without further specification merges distinct limits into one curve.

A sequence’s tokens can be highly correlated. Order, repetition, reference, or masking relates a position to others. An equal-weight empirical sum specifies $1/n$ per atom without proving independent sampling. Independent uniform spherical initialization must be stated where a mathematical construction uses it. Actual learned representations need their own distributional grounds.

Population representations also require choosing what to retain. One can push forward a sentence’s final vector, a fixed position’s token, or a token obtained by first sampling a sentence and then sampling a position under a rule. The last is a joint sampling process affected by positional weighting and length distribution. It differs from one fixed sequence’s unlabeled $\nu$. Changing population sequence lengths also brings normalization into comparison.

A simple construction shows population averages losing within-sequence relations. For length two with binary tokens, population A chooses $(0,0)$ or $(1,1)$ equally, while B chooses $(0,1)$ or $(1,0)$ equally. Randomly choosing a position gives $\frac12\delta_0+\frac12\delta_1$ in both. Yet A always has equal tokens within a sequence and B always different ones. One-token population laws do not retain the joint relation.

Attention using relations between the two tokens can therefore induce different updates. Shared marginals do not determine the sequence function. We can retain ordered input vectors, or study a distribution of measures, treating each sequence’s $\nu$ as another random object. Order and joint information retained by each choice still need specification. Mathematical language here selects content rather than changes a letter for one quantity.

An unlabeled $\nu$ suits the symmetric particle model: each point’s velocity is a function of itself and the common measure, without extra mechanisms attached to its index. Projection then connects particle and measure equations exactly. Different token capabilities, matrices, or positional effects require retaining labels or measures of different types. Sufficient information can close a coarse object; a failure identifies what to add.

Vocabulary probabilities inhabit another space. A final representation passes through readout and softmax to probabilities on vocabulary symbols. Atoms of $\nu$ are hidden-space points, not those probabilities. Closer token points may affect readout, but shape alone does not establish entropy, accuracy, or reasoning ability. Readout, target population, and task loss must connect geometry to those judgments.

How finite tokens enter a weak equation exactly#

For unlabeled shared attention, take $V=I$ and express velocity as a function of a point and measure:

Substituting $\nu^n=n^{-1}\sum_i\delta_{z_i}$ cancels the $1/n$ factors and recovers the finite particle equation. No limit in $n$ has been taken. Integration gives the existing sum a shared organization; calling this step mean-field approximation would understate the exact relation.

Take a smooth spherical test function $\varphi$, such as a coordinate, smoothed-region statistic, or other differentiable observation. Differentiate the finite sum directly:

Here $\langle\nu,\varphi\rangle$ denotes integration and $\nabla_S$ the spherical gradient. An ambient extension gives the same inner product with tangent $b$. The identity retains each atom’s contribution to observation changes without requiring a smooth measure density.

This is the weak form of $\partial_\tau\nu+\operatorname{div}_S(b\nu)=0$. For smooth time-dependent $\varphi(\tau,z)$ vanishing at $T$, integrate the chain rule:

The weak equation moves mass along a vector field without requiring a differentiable density at each location. Atomic measures satisfy it exactly; general measure solutions can be constructed through characteristic pushforwards. Particle and measure observations now connect. Differential-equation language studies both without merging their initial-data identities.

Unequal fixed atomic weights also satisfy the weak equation if velocities use the corresponding measure. With fixed position labels $p_i$, retain atoms in joint space $(p,z)$, evolving only $z$ while the interaction kernel reads both positions. Causal masks, capabilities, or types can require this enlargement. Projecting away labels may leave velocity undetermined by the unlabeled $\nu$ alone.

Changing particle count differs again. Adding a generated token changes the atom collection and $n^{-1}$ weights, including old atoms’ weights. This adds mass and redistributes weights rather than advancing the fixed-$n$ continuity equation. Generation position and depth separate once more: mass conservation inside one forward pass cannot replace a growing generation history.

Layer-varying matrices allow the weak identity with explicitly time-dependent $b_\tau$, while energy conclusions must include those changes. Training-varying matrices give new conditions for each forward pass. Similar equations allow connection without granting identical invariance. The chosen time and object determine test functions, fields, and errors.

The result is strong and bounded: finite particles in the unlabeled symmetric model enter the weak equation exactly. General initial measures need existence and uniqueness; approximation of particles by those measures needs stability; sampling rates for actual sequences need distributional conditions. Each additional obligation follows from the preceding result and is calculated next.

Removing the denominator versus changing the metric#

The attention denominator normalizes each row to unit total weight. Replacing it by $n$ gives unnormalized velocity:

Both objects can be studied, but their vector fields differ. USA’s total row weight is $Z_i/n$, depending on configuration; SA divides it away. Equal denominators across particles permit a common time change along a configuration path. General denominators differ by row, leaving no single multiplier for the whole system. The operation then changes more than time units.

Take two spherical points symmetric about an axis with angle $2\vartheta$, $0<\vartheta<\pi/2$, and $A=V=I$. Self-interaction projects to zero; the other point contributes tangent magnitude $\sin(2\vartheta)$. Thus

Both velocities are negative on the open interval, eventually clustering on the symmetry axis. Their angles at the same depth usually differ. Both clustering and preservation of evolution are distinct judgments. This symmetric case permits a special time correspondence without proving one for arbitrary many-particle configurations.

Original theoretical curves numerically integrating the body’s two-point equations: $\beta=1$, initial half-angle one radian, and fixed identity matrices. They compare geometry at the same depth rather than report an actual network experiment or task performance.

Energy gives another route. Fix symmetric $A$ and $V=A$. This coincides with the preceding $V=I$ instance when $A=I$. For this symmetric object on $(S^{d-1})^n$, define negative interaction energy

Symmetry combines the two derivative contributions involving $z_i$, yielding

We use negative energy and descent; the source also uses positive interaction energy and ascent for the same field. Retaining sign and normalization permits comparison without assuming identical metric or time merely from the label gradient flow.

For tangent vectors $\xi=(\xi_i)$ and $\eta=(\eta_i)$, first use average product metric $g_0(\xi,\eta)=n^{-1}\sum_i\langle\xi_i,\eta_i\rangle$. By the gradient definition, component $i$ of $\operatorname{grad}_{g_0}E$ is $n\nabla_{S,i}E$, whose negative equals USA velocity. USA is therefore energy descent under $g_0$.

To retain original SA, instead use the configuration-dependent metric

Every $Z_i$ is positive, giving a positive-definite metric on each tangent space. Gradient components become $(n^2/Z_i)\nabla_{S,i}E$, whose negatives are exactly $b_i^{\mathrm{SA}}$. Dynamics and energy stay fixed while geometry of the gradient changes. The differential $dE[\xi]$ remains the same; a metric turns it into a velocity vector.

Along SA, dissipation is $dE/d\tau=-n^{-2}\sum_i Z_i\|b_i^{\mathrm{SA}}\|_2^2\le0$; along USA it is $-n^{-1}\sum_i\|b_i^{\mathrm{USA}}\|_2^2\le0$. Under the symmetry conditions, equilibria agree while rates differ. Energy monotonicity alone does not prove one limiting cluster. Critical configurations, stability, and initialization still matter; a clustering proof follows after finite-error analysis.

The matrix conditions carry this interpretation. Nonsymmetric $A$ introduces transpose contributions in differentiation; $V\ne A$ changes value directions. The field then need not equal this energy gradient. The conditions match the source’s finite-particle module without extending to arbitrary trained matrices. Its weighted-Wasserstein interpretation for measures is a formal geometric construction at another level, rather than a complete theory established by finite-dimensional calculation alone.

Changing metric thus preserves the original dynamics while connecting it to energy as a gradient, enabling further geometry and stability analysis. Removing the denominator may also aid analysis but yields another dynamics. Both routes can serve a purpose; choose whether the research requires the original process or accepts a substitute with related properties.

Carrying finite-measure error into time and dimension#

Return to $V=I$ and fixed $A$, possibly nonsymmetric. Particle and general measure dynamics use the same normalized field, giving common comparison endpoints. Equip the sphere with ambient chord distance and define

Couplings $\Pi(\nu,\mu)$ have marginals $\nu,\mu$, specifying how their masses pair. The distance minimizes mean transport length. It compares atomic and continuous measures on a common space and scale. Another task observation requires its own conversion rather than inheriting an error merely under the label distributional proximity.

Compactness gives explicit exponential-kernel bounds. Set $B=\beta\|A\|_{\mathrm{op}}$. Then $e^{-B}\le e^{\beta z^TAy}\le e^B$, and the denominator is at least $e^{-B}$. Each position variable has kernel Lipschitz constant at most $Be^B$; the kernel times $y$ has $y$-Lipschitz constant at most $e^B(1+B)$. This counts kernel and value changes separately.

Write numerator and denominator as $N(z,\nu)$ and $Z(z,\nu)$. For fixed $z$, coupling gives $\|N(z,\nu)-N(z,\mu)\|_2\le e^B(1+B)W_1(\nu,\mu)$ and $|Z(z,\nu)-Z(z,\mu)|\le Be^BW_1(\nu,\mu)$. Since $\|N\|_2\le Z$, both ratio-error terms are controlled by the same denominator lower bound.

For $z,z'$, numerator and denominator each have bound $Be^B\|z-z'\|_2$, and $\|P_z-P_{z'}\|_{\mathrm{op}}\le2\|z-z'\|_2$. One explicit, nonoptimal set of constants is consequently

Particle count does not appear explicitly, while temperature and matrix norm do. Growth of those quantities with dimension or size changes the constants. Also $\|b\|_2\le1$: a convex average of unit values has norm at most one, and projection cannot increase it. Compact domain, a positive denominator bound, and Lipschitz dependence jointly support existence and stability.

For example, prescribe a continuous measure path, solve ordinary differential equations in its field, and push initial mass forward. On short intervals, Lipschitz control gives a contraction estimate for this map. Successive continuation constructs a unique self-consistent path. Finite particles give one such path. This is the same bounded Lipschitz model; growing, unnormalized, or singular interactions require their own well-posedness analysis.

Stability follows through transport directly. Take initial coupling $\pi_0$ and evolve its endpoints along fields driven by $\nu_\tau$ and $\mu_\tau$, producing coupling $\pi_\tau$. Let $D(\tau)=\int\|z_\tau-y_\tau\|_2\,d\pi_0$. At zero separation, use upper-right derivatives or integral inequalities. The field bound and $W_1(\nu_\tau,\mu_\tau)\le D(\tau)$ give $D'\le(C_x+C_\nu)D$. Gronwall yields

Take the infimum over initial couplings to obtain the relation. Initial empirical measures approaching a general measure therefore approach its evolution at fixed finite depth. Particles already entered the weak equation exactly; the new relation compares different initial measures. Its guarantee covers specified finite intervals, while long-time approximation requires controlling exponential growth.

A uniform field error $\delta$ adds $D'\le CD+\delta$, giving $D(\tau)\le e^{C\tau}D(0)+\delta(e^{C\tau}-1)/C$. Initial and dynamical errors enter differently. Changing normalization or matrices requires the latter term; numerical realization adds discretization error. More initial tokens do not automatically reduce either.

Approximating initial data raises dimension next. Fix $d\ge2$ and intrinsic spherical dimension $s=d-1$. Cover the sphere by $M$ radius-$h$ balls and partition it into cells assigned to their centers. Moving both the measure and empirical sample measure to centers costs at most $h$ each. Remaining center-mass transport costs at most $\sum_j|\widehat p_j-p_j|$, since sphere diameter is two.

For $n$ points genuinely IID from $\mu_0$, cell frequencies have variance $p_j(1-p_j)/n$. Cauchy’s inequality gives $\mathbb E\sum_j|\widehat p_j-p_j|\le\sqrt{M/n}$. Therefore

For a fixed sphere, small-ball area scales as $h^s$. A maximal separated set has disjoint radius-$h/2$ balls and covering radius-$h$ balls, giving $M\le C_dh^{-s}$. Taking $h=n^{-1/(s+2)}$ gives the conservative expectation bound $(2+\sqrt{C_d})n^{-1/(s+2)}$. This is not an optimal sampling rate, but its conditions and derivation are explicit. Correlated tokens cannot inherit IID frequency variance directly.

A sampling-independent lower bound also exposes dimension. Let $\mu_0$ be uniform spherical measure and choose any $n$ support points. A radius-$r$ cap has mass at most $K_dr^s$: angular integration with $\sin^{d-2}u\le u^{d-2}$ gives the small-cap power bound. Take $r=(2K_dn)^{-1/s}$, enlarging $K_d$ if needed to ensure $r\le1$. The union of all $n$ caps still covers at most half the mass.

At least half the uniform mass lies at least $r$ from every support point. Any transport pays that distance, so $W_1(\nu_0^n,\mu_0)\ge r/2$, for arbitrary atomic weights. This gives a geometric obstruction of scale $n^{-1/(d-1)}$ at fixed dimension. Larger dimension reduces the exponent. Many samples must be assessed together with the space and observation. A metric retaining fewer statistics can change the difficulty while changing retained content too.

Propagating the conservative sampling bound gives $\mathbb E W_1(\nu_\tau^n,\mu_\tau)\le C_de^{C\tau}n^{-1/(s+2)}$, absorbing prefactors into $C_d$. For $\tau_n=c\log n$ under a fixed model and dimension, $c<1/[C(s+2)]$ suffices for this bound to vanish. Arbitrary long times, growing dimension, and scale-dependent temperature remain beyond it. These conditions locate the joint-scaling question.

The stability relation returns measure language to finite systems. Initial distance, time, temperature, matrices, and dimension enter separately, allowing finite size to be chosen for a target depth. Next comes another relation: why a finite configuration in a hemisphere clusters, and how its rate also depends on scale.

A complete clustering proof inside a hemisphere#

Keep $V=I$, arbitrary fixed $A$, and finite $\beta>0$, so normalized attention coefficients are positive. Suppose a unit $w$ satisfies $\langle w,z_i(0)\rangle>0$ for every initial point. They lie in one open hemisphere. This is checkable for a finite configuration rather than an automatic property of token representations.

Let $r_i=\langle w,z_i\rangle$ and $r_0=\min_i r_i(0)>0$. The spherical equation gives $\dot r_i=\sum_j a_{ij}(r_j-\langle z_i,z_j\rangle r_i)$. At a positive minimum, $r_j\ge r_i$ and $\langle z_i,z_j\rangle\le1$, so every summand is nonnegative. The minimum of finitely many smooth functions is locally Lipschitz, with almost-everywhere derivative selected from minimizing components. Thus the minimum does not decrease.

A first time reaching zero would contradict continuity and the preceding lower bound $r_0$. Hence all $r_i\ge r_0$ for all time. This positive margin makes the next coordinate conversion legitimate and supplies an estimate constant.

Use projective coordinates

The quotient derivative proves the last identity: terms along $z_i$ in $\dot z_i/r_i$ cancel those from $z_i\dot r_i/r_i^2$. Spherical projection becomes consensus dynamics on a shared affine plane, with configuration-dependent coefficients. The proof retains that dependence and uses their positive lower bound.

With $B=\beta\|A\|_{\mathrm{op}}$, we have $a_{ij}\ge e^{-2B}/n$. Also $r_j\ge r_0$ and $r_i\le1$, so $c_{ij}\ge c_0/n$, where $c_0=r_0e^{-2B}>0$. This keeps every point interacting and enables contraction. A mask removing edges would require different connectivity and information conditions.

Fix any unit direction $u$. Set $M_u=\max_i\langle u,y_i\rangle$, $m_u=\min_i\langle u,y_i\rangle$, with mean $\overline y_u$. At a maximizing component, all differences are nonpositive, so decreasing coefficients to $c_0/n$ increases its derivative, giving $M_u'\le c_0(\overline y_u-M_u)$. At a minimum, differences are nonnegative, giving $m_u'\ge c_0(\overline y_u-m_u)$.