Sequential Local Operator Alignment
for Training-Free Model Merging

• Existing training-free mergers combine $\mathbf{W}_Q,\mathbf{W}_K,\mathbf{W}_V,\mathbf{W}_O$ independently — ignoring how attention operation actually operattes on inputs.
• SLOA merges the functional operators $\mathbf{W}_Q\mathbf{W}_K^{\!\top}$ and $\mathbf{W}_V\mathbf{W}_O$ directly.
• Same operator-alignment principle also handles LayerNorm and the MLP sub-blocks ($\mathbf{W}_1,\mathbf{W}_2$) — SLOA covers the whole transformer, not just attention.
• Alignment is performed sequentially on activations of the partially-merged model.
• Re-factorization lets rank act as capacity control.
• State-of-the-art training-free merging on CLIP/ViT vision and RoBERTa NLU benchmarks.

• $T$ task-specific models $\{\bm{\Phi}_1,\ldots,\bm{\Phi}_T\}$, each fine-tuned from the same pretrained $\bm{\Phi}_{\text{pre}}$ on its dataset $D_t$.
• Goal: aggregate them into a single multi-task model without joint retraining on $D_1\cup\cdots\cup D_T$:

• Training-free: a small unlabeled calibration set is allowed; no gradient updates.

Multi-Head Attention (MHA). Sum of $H$ heads:

• QK circuit $\mathbf{M}_{\text{QK}}^{(j)}=\mathbf{W}_Q^{(j)}\mathbf{W}_K^{(j)\top}$ — where attention reads.
• VO circuit $\mathbf{M}_{\text{VO}}^{(j)}=\mathbf{W}_V^{(j)}\mathbf{W}_O^{(j)}$ — what is written.
• Each circuit is a $d{\times}d$ operator with a hidden rank-$d_k$ bottleneck.

MLP block. Two linear projections separated by a nonlinearity $\sigma(\cdot)$:

• SLOA aligns $\mathbf{W}_1$ on the attention output, then $\mathbf{W}_2$ on the merged post-activation $\sigma(\widehat{\mathbf{Z}}\mathbf{W}_1)$ — sequential, closed-form, same template as attention.

Relative functional error along linear interpolations between equivalent factorizations, grows rapidly with the scalar $c$.

• Functional unit. Attention scores depend only on the product $\mathbf{M}_{\mathrm{QK}}=\mathbf{W}_Q\mathbf{W}_K^{\!\top}$ — not on $\mathbf{W}_Q,\mathbf{W}_K$ separately.
• Hidden ambiguity. Any invertible $\mathbf{R}$ gives a different factorization with the same function:
  $$\bigl(\mathbf{W}_Q\mathbf{R}\bigr)\bigl(\mathbf{W}_K\mathbf{R}^{-\top}\bigr)^{\!\top} \;=\;\mathbf{W}_Q\mathbf{W}_K^{\!\top}.$$
• Independent averaging fails. Averaging the two parameterizations element-wise gives $\bar{\mathbf{W}}_Q=\tfrac{1}{2}\mathbf{W}_Q(\mathbf{I}+\mathbf{R})$, $\bar{\mathbf{W}}_K=\tfrac{1}{2}\mathbf{W}_K(\mathbf{I}+\mathbf{R}^{-\top})$, so
  $$\bar{\mathbf{M}}_{\mathrm{QK}}=\bar{\mathbf{W}}_Q\bar{\mathbf{W}}_K^{\!\top} =\tfrac{1}{4}\mathbf{W}_Q(\mathbf{I}+\mathbf{R})(\mathbf{I}+\mathbf{R}^{-1})\mathbf{W}_K^{\!\top} \;\neq\;\mathbf{W}_Q\mathbf{W}_K^{\!\top}.$$
• Unbounded blow-up. Scalar case $\mathbf{R}=c\mathbf{I}$:
  $$\bar{\mathbf{M}}_{\mathrm{QK}}=\dfrac{(1+c)(1+c^{-1})}{4}\,\mathbf{M}_{\mathrm{QK}} \;\;\xrightarrow[c\to\infty]{}\;\;\infty.$$
  Equality holds only at $c=1$; error grows without bound otherwise.
• Consequence. Two task models that are functionally identical at the QK level can produce arbitrarily different merged attention scores under isolated weight averaging.

• Merge at the level of local functional operators, not raw weight tensors.
• Each component's behavior is linear in its operator $\bm{\theta}$ — closed-form least squares.
• Apply alignment sequentially: each merge corrects accumulated upstream error.
• Factorize the merged operator back; the chosen rank controls multi-task capacity.

SLOA aligns the effective local operators (QK, VO, MLP, LayerNorm), merges them sequentially under partially-merged activations, then factorizes each operator back into transformer-compatible weights.

Unified alignment objective

Local operators & data-aware metrics

$\nu(\mathbf{X})$ is the LayerNorm-normalized input; $\mathbf{A}$ is the attention matrix; $\bm{\Lambda}=\alpha\,\mathrm{diag}\bigl(\mathrm{Cov}(\mathbf{X})\bigr)+\epsilon\mathbf{I}$.

• Activation term matches local input–output behavior on calibration data.
• Geometry term is a Mahalanobis distance: penalizes deviation along directions the calibration activations actually use.
• For fixed activations the objective is quadratic in $\bm{\theta}$ — closed-form regularized least squares per component.
• Sequential: downstream operators are aligned on activations from the partially merged model $\widehat{\mathbf{Z}}$.

After merging the QK / VO operator in $\mathbb{R}^{d\times d}$, SVD the operator $\bar{\bm{\theta}}=\mathbf{U}\bm{\Sigma}\mathbf{V}^{\!\top}$ and factorize at chosen rank $r_j$.

QK circuit (with denominator $c_j=\sqrt{r_j}$):

VO circuit:

• $r_j = d_k$ preserves the original head width.
• $r_j > d_k$ retains additional task-specific singular directions.
• The sum of task heads can span a larger subspace than any single rank-$d_k$ head — expansion avoids unnecessary truncation.

1. Collect activations $\{\mathbf{Z}_{t,i}\}$ for each operator from a small calibration set.
2. Order operators topologically $o_1\!\prec\!\cdots\!\prec\!o_M$ (LayerNorm $\to$ QK $\to$ VO $\to$ MLP).
3. For each $o_m$: forward through the partially-merged model to get $\widehat{\mathbf{Z}}_{t,i}$, then solve closed-form regularized LS.
4. Factorize QK/VO via SVD at chosen rank $r_j$ and write back transformer-compatible weights.

No gradients, no labels — one deterministic pass through the operator graph.

Table 1. Average classification accuracy (%) of the merged model under three increasingly difficult settings: 8, 14, and 20 task-specific ViT-B/32 models. Higher is better; training-free methods use 256 calibration examples per task. For SLOA, $r$ is the retained QK/VO factorization rank ($r{=}64$ matches the original head dimension).

• SLOA is the best training-free method on all three benchmarks; the gap to baselines widens with more tasks.
• +6.6 pts over RegMean++ on the hardest 20-task setting.
• Rank expansion $r{=}64{\to}128$ consistently helps — the merged operator's spectrum spans a larger subspace than any single head.

Table 2. Per-task and average Metric of the merged RoBERTa model on seven GLUE tasks (CoLA, SST-2, MRPC, QQP, MNLI, QNLI, RTE). Higher is better; training-free methods use 256 calibration examples per task. SLOA preserves the original head rank; SLOA ($r{=}128$) expands the QK/VO factorization rank.

bold = best in column; underline = second best. SLOA ($r{=}128$) wins 6/7 tasks and the average.

Component ablation (20 tasks)

Decoupling QK and OV ranks

Scaling with calibration & rank (vision)

Sensitivity to regularization $\alpha$

• Sequential alignment alone is already strong.
• QK and OV must be merged jointly — isolating one hurts.
• Rank expansion consistently improves performance.
• Optimal $\alpha$ decreases as more calibration data is available.