SLOA: Training-Free Transformer Merging via Sequential Local Operator Alignment

SLOA aligns the effective local operators ($\mathbf{W}_Q\mathbf{W}_K^\top$, $\mathbf{W}_V\mathbf{W}_O$, LayerNorm affine, MLP projections) rather than raw weight matrices, sequentially under activations of the partially-merged model, then factorizes each operator back into transformer-compatible weights.

Abstract

Training-free model merging aims to combine multiple fine-tuned models into a single model without further optimization on labeled data. Yet, in transformers, independently merging individual layers can affect a shared attention computation because the query–key and value–output operators depend on composed matrices, overlooking the functional structure. Moreover, when merging earlier components, downstream components receive different activations than they do in the original model, thus, the merged and original execution paths no longer match.

In this paper, we introduce Sequential Local Operator Alignment, a training-free method that merges transformers along the execution path of the partially merged model. Our method uses calibration data to estimate the local behavior of each functional component, aligns operators sequentially under the intermediate activation of the partially merged model, and subsequently factorizes the merged operators back into valid transformer parameters. We empirically show that this sequential step reduces error accumulation across layers. Furthermore, the proposed operator factorization step enables rank expansion, providing a principled mechanism for increasing multi-task capacity.

We demonstrate that our approach generalizes across modalities, model scales, and varying numbers of tasks, from CLIP and RoBERTa to billion-parameter LLMs, and further extends naturally to the merging of LoRA-fine-tuned models. The results indicate improvements over strong merging baselines without requiring rank expansion, while optional expansion provides a further accuracy–inference-cost trade-off.

Motivation: Why Isolated Merging Fails

Relative functional error along linear interpolations between equivalent factorizations. Error grows rapidly — equality holds only at $c=1$.

Hidden ambiguity. Any invertible $\mathbf{R}$ gives a different factorization with the same function:

$$(\mathbf{W}_Q\mathbf{R})(\mathbf{W}_K\mathbf{R}^{-\top})^\top = \mathbf{W}_Q\mathbf{W}_K^\top$$

Independent averaging fails. Averaging two such parameterizations yields

$$\bar{\mathbf{M}}_\mathrm{QK} = \tfrac{1}{4}\mathbf{W}_Q(\mathbf{I}+\mathbf{R})(\mathbf{I}+\mathbf{R}^{-1})\mathbf{W}_K^\top \;\neq\; \mathbf{W}_Q\mathbf{W}_K^\top$$

In the scalar case $\mathbf{R}=c\mathbf{I}$, the error grows without bound as $c\to\infty$. Two task models that are functionally identical at the QK level can produce arbitrarily different merged attention scores under isolated weight averaging.

Method

Unified Alignment Objective

For each local operator $\bm{\theta}$, SLOA solves:

$$\bm{\theta}_\mathrm{merge}= \arg\min_{\bm{\theta}}\; \underbrace{\tfrac{1}{T}\!\sum_{t,i}\! \bigl\|\mathcal{A}(\widehat{\mathbf{Z}}_{t,i};\bm{\theta})- \mathcal{A}(\mathbf{Z}_{t,i};\bm{\theta}_t)\bigr\|_F^2}_{\text{activation matching}} \,+\, \underbrace{\tfrac{1}{T}\!\sum_t\!\|\bm{\theta}-\bm{\theta}_t\|^2_{\widehat{\chi}_t}}_{\text{data-aware geometry}}$$

$\widehat{\mathbf{Z}}$ are activations from the partially-merged model; the geometry term uses a Mahalanobis distance that penalizes deviation along directions the calibration activations actually use. For fixed activations the objective is quadratic in $\bm{\theta}$ — closed-form regularized least squares per component.

Local Operators & Metrics

Component	Local operator $\bm{\theta}$	Behavior map $\mathcal{A}(\mathbf{Z};\bm{\theta})$	Metric
LayerNorm	$(\bm{\gamma},\bm{\beta})$	$\nu(\mathbf{X})\,\mathrm{diag}(\bm{\gamma})+\mathbf{1}\bm{\beta}^\top$	$\mathrm{Cov}(\nu(\chi))$
Linear map	$\mathbf{W}$	$\mathbf{X}\mathbf{W}$	$\bm{\Lambda}$
QK component	$\tfrac{1}{\sqrt{d_k}}\mathbf{W}_Q\mathbf{W}_K^\top$	$\mathbf{X}\bm{\theta}\mathbf{X}^\top$	$\bm{\Lambda}\otimes\bm{\Lambda}$
VO component	$\mathbf{W}_V\mathbf{W}_O$	$\mathbf{A}\mathbf{X}\bm{\theta}$	$\bm{\Lambda}$

$\nu(\mathbf{X})$ = LayerNorm-normalized input; $\mathbf{A}$ = attention matrix; $\bm{\Lambda}=\alpha\,\mathrm{diag}(\mathrm{Cov}(\mathbf{X}))+\epsilon\mathbf{I}$.

SLOA in 4 Steps

1. Collect activations $\{\mathbf{Z}_{t,i}\}$ for each operator from a small calibration set (256 samples per task by default).

2. Order operators topologically: LayerNorm $\to$ QK $\to$ VO $\to$ MLP $W_1$ $\to$ MLP $W_2$.

3. For each operator: forward through the partially-merged model to get $\widehat{\mathbf{Z}}_{t,i}$, then solve closed-form regularized LS.

4. Factorize QK/VO operators via SVD at chosen rank $r_j$ and write back transformer-compatible weights. Rank $r_j > d_k$ retains additional task-specific singular directions.

No gradients, no labels — one deterministic pass through the operator graph.

Results

All training-free methods that require calibration, including SLOA, use $N{=}256$ samples per task drawn exclusively from the training split; no test data is used for merging. Results are averaged over three random seeds. For SLOA, $r$ is the retained factorization rank of the QK and VO operators.

Per-task comparison of SLOA ($r{=}128$) with data-free and training-free baselines on ViT-B/32 with 20 vision tasks (left two) and RoBERTa with 7 GLUE tasks (right two). SLOA consistently outperforms both data-free and training-free baselines.

Vision: ViT-B/32 (CLIP fine-tunes)

Average accuracy (%) on 8, 14, and 20 vision classification tasks. $r{=}64$ matches the original ViT-B/32 attention-head dimension.

	Method	Samples	8 tasks	14 tasks	20 tasks
	Fine-tuned	—	90.39	89.26	89.78
Data-Free	Weight Averaging	—	66.29	65.37	61.08
	Task Arithmetic	—	69.49	66.47	60.60
	TIES-Merging	—	71.70	67.58	62.76
	TSV-M	—	83.04	79.11	73.66
	Iso-C	—	82.64	77.94	71.16
	Iso-CTS	—	82.87	79.24	73.41
Training-Free	Fisher Merging	256	69.12	66.26	61.74
	RegMean	256	82.63	77.93	72.20
	RegMean++	256	84.37	80.06	74.51
	SLOA ($r{=}64$)	256	84.89	81.94	80.06
	SLOA ($r{=}128$)	256	85.50	82.61	81.03

At the original rank, SLOA outperforms all baselines in every setting. Its advantage over the strongest baseline, RegMean++, grows with the number of merged tasks: +0.6 pp (8 tasks), +1.9 pp (14 tasks), +5.5 pp (20 tasks). Rank expansion $r{=}64\to128$ improves accuracy everywhere, with larger gains for more tasks.

Vision: ViT-B/16 (CLIP fine-tunes)

Average accuracy (%) on the same 8-, 14-, and 20-task benchmarks with the larger ViT-B/16 backbone.

	Method	Samples	8 tasks	14 tasks	20 tasks
	Fine-tuned	—	92.34	91.31	91.61
Data-Free	Weight Averaging	—	72.28	69.72	64.81
	Task Arithmetic	—	77.17	70.92	64.23
	TIES-Merging	—	80.21	71.48	66.31
	TSV-M	—	86.70	82.09	78.31
	Iso-C	—	88.36	82.39	75.82
	Iso-CTS	—	89.01	84.54	78.50
Training-Free	Fisher Merging	256	73.67	71.00	66.50
	RegMean	256	86.13	81.43	75.53
	RegMean++	256	87.25	82.23	76.64
	SLOA ($r{=}64$)	256	87.16	84.19	82.12
	SLOA ($r{=}128$)	256	88.04	85.16	83.43

The trend mirrors ViT-B/32: on 8 tasks the data-free Iso-CTS is narrowly strongest, by 14 tasks SLOA has overtaken it, and by 20 tasks SLOA leads by nearly 5 pp (83.43 vs. 78.50). Larger backbones are more weight-disentangled, so data-free baselines stay competitive at few tasks; as interference grows with task count, aligning functional operators wins by an increasingly clear margin.

NLU: RoBERTa on 7 GLUE Tasks

Per-task metric and average on seven GLUE tasks with merged RoBERTa-base models.

	Method	Samples	CoLA	SST-2	MRPC	QQP	MNLI	QNLI	RTE	Avg
	Individual	—	0.6018	0.9404	0.8922	0.9141	0.8720	0.9271	0.7906	0.8483
Data-Free	Weight Averaging	—	0.1808	0.8188	0.8116	0.7381	0.4383	0.7106	0.6173	0.6165
	Task Arithmetic	—	0.2909	0.8463	0.8188	0.7869	0.5591	0.7584	0.6462	0.6724
	TIES-Merging	—	0.1951	0.8429	0.8333	0.8351	0.6501	0.7309	0.4260	0.6448
	TSV-M	—	0.3477	0.8739	0.8446	0.8286	0.6163	0.8124	0.6606	0.7120
	Iso-C	—	0.2639	0.8601	0.8130	0.8211	0.7303	0.7882	0.7040	0.7115
	Iso-CTS	—	0.4136	0.7798	0.6385	0.6500	0.5277	0.6590	0.6534	0.6174
Training-Free	Fisher Merging	256	0.1126	0.8719	0.7419	0.7808	0.4227	0.5311	0.5150	0.5680
	RegMean	256	0.2797	0.8933	0.8069	0.7915	0.7251	0.8336	0.6161	0.7066
	RegMean++	256	0.2433	0.9170	0.7740	0.8099	0.7543	0.8299	0.6691	0.7139
	SLOA ($r{=}64$)	256	0.5608	0.9232	0.8814	0.8152	0.7678	0.8755	0.7100	0.7906
	SLOA ($r{=}128$)	256	0.5721	0.9239	0.8810	0.8166	0.7753	0.8824	0.7341	0.7979

bold = best in column; underline = second best. At the original rank SLOA improves the average over the strongest baseline by 7.7 points; $r{=}128$ raises the margin to 8.4 points. SLOA is best on six of seven tasks, trailing only TIES-Merging and TSV-M on QQP.

LLM: Merging LLaMA-2-7B LoRA Experts

Merging three LoRA-adapted LLaMA-2-7B experts (math, code, instruction-following). SLOA forms the effective weight $\mathbf{W}_t=\mathbf{W}_0+\mathbf{B}_t\mathbf{A}_t$ and merges its composed operators, so it is invariant to the LoRA factorization ambiguity $\mathbf{B}_t\mathbf{A}_t=(\mathbf{B}_t\mathbf{R})(\mathbf{R}^{-1}\mathbf{A}_t)$. Rotary position embeddings prevent a single composed QK operator, so only the VO circuit is composed (SLOA-VO) while $\mathbf{W}_Q,\mathbf{W}_K$ are merged as linear operators. Avg. = mean over GSM8K, HumanEval (pass@1), and IFEval.

	Method	GSM8K	HumanEval	IFEval	Avg.
Experts	LLaMA-2-7B Pretrained	4.32	0.00	20.89	8.40
	LLaMA-2-7B IF-Tulu	12.59	1.22	66.17	26.66
	LLaMA-2-7B Code (Magic-Coder)	9.70	32.93	26.99	23.21
	LLaMA-2-7B Math (Meta-Math)	65.50	0.00	15.53	27.01
Data-Free	Weight Averaging	40.11	18.29	40.67	33.02
	Task Arithmetic ($\alpha{=}0.5$)	58.98	15.85	42.70	39.18
	TSV-M ($\alpha{=}0.5$)	58.00	18.29	42.14	39.48
	TIES ($\alpha{=}0.3$)	48.14	20.73	34.57	34.48
	KnOTS-TIES ($\lambda{=}0.85$)	55.27	15.24	40.85	37.12
	Iso-C	38.89	21.34	57.67	39.30
	Iso-CTS	36.09	21.34	55.45	37.63
Training-Free	RegMean	56.03	29.27	48.24	44.51
	RegMean++	55.95	29.27	54.16	46.46
	SLOA-VO ($r_{ov}{=}128$)	58.61	28.05	57.86	48.17
	SLOA-VO ($r_{ov}{=}256$)	59.51	27.44	58.96	48.64
	SLOA-VO ($r_{ov}{=}384$)	59.51	27.44	60.63	49.19

bold/underline = best/second-best merging method per column. At the architecture-preserving rank $r_{ov}{=}128$, SLOA-VO has the highest average of all merging methods (+1.7 over RegMean++). Expanding to $r_{ov}{=}384$ further improves GSM8K and IFEval, with a small HumanEval reduction.

Additional Experiments & Ablations

Merging LoRA-adapted ViT experts

	Method	Samples	Norm. Acc. (%)
	Fine-tuned (LoRA)	—	100.00
Data-Free	Simple Avg	—	63.76
	TA	—	63.70
	TIES	—	63.70
	DARE-TIES	—	63.70
	KnOTS-TIES	—	68.00
	KnOTS-DARE-TIES	—	63.90
	Iso-C	—	73.60
	Iso-CTS	—	73.70
Training-Free	Fisher Merging	256	62.81
	RegMean	256	72.72
	RegMean++	256	81.11
	SLOA ($r{=}64$)	256	82.66
	SLOA ($r{=}128$)	256	88.07

Normalized accuracy (mean over tasks of merged/fine-tuned accuracy $\times$ 100) on ViT-B/32 LoRA experts (KnOTS, rank 16) across 8 vision tasks; the LoRA oracle's raw average accuracy is 84.28%.

SLOA needs no change to its local objective for LoRA experts: it merges the effective functional operators rather than the non-identifiable LoRA factors. SLOA ($r{=}128$) reaches 88.07%, 7.0 points above the strongest baseline, and even $r{=}64$ beats all baselines — the merged operator's subspace can exceed the rank of any single task expert.

Scaling with calibration samples and rank

ViT-B/32, 20 tasks

ViT-B/16, 20 tasks

RoBERTa, 7 GLUE tasks

Across both modalities and all ranks, SLOA improves steadily with more calibration data — larger sets give a better-conditioned system for the merged weights. Even with only 128 samples SLOA outperforms all baselines. Raising $r$ from 64 to 128 and 256 helps further on vision; on NLU, 128 helps clearly while 256 gives diminishing returns.

Diagnosing the source of SLOA's gains

Layer-wise merging error on ViT-B/16, 20 tasks: mean relative $L_2$ error between merged and expert activations at each block. SLOA and RegMean++ track closely for six blocks, then diverge; SLOA's error peaks at 1.65–1.69$\times$ its input-level error vs. 1.87$\times$ (RegMean++) and 2.08$\times$ (RegMean). Rank expansion narrows the gap further.

Accuracy vs. calibration samples on ViT-B/32: RegMean/RegMean++ plateau early once their Gram statistics stabilize, while SLOA keeps improving. Sequential correction alone (no operator composition) already beats both; composing QK/VO and expanding rank adds a further gain at every budget.

Circuit capacity, components, and regularization

Decoupling QK and OV ranks (ViT-B/32, 20 tasks, 1024 samples). Expanding one circuit alone saturates near $r{=}256$; QK expansion beats OV expansion; jointly expanding both gives the best, most sustained gains.

Component ablation (ViT-B/32, 20 tasks). Sequential alignment alone is already strong; adding only QK or only OV hurts, since the two are coupled inside attention; merging both jointly gives the best result.

Sensitivity to $\alpha$ (ViT-B/32, 8 tasks). Larger $\alpha$ helps with few samples; the optimum decreases as calibration data grows, approaching $\alpha\approx0.05$.

Merge cost

Method	Data requirement	Merge time (s)	Accuracy (%)
Task Arithmetic	Data-free	14.63	64.23
TIES-Merging	Data-free	39.64	66.31
TSV-M	Data-free	153.34	78.31
Iso-C	Data-free	17.09	75.82
Iso-CTS	Data-free	167.24	78.50
Fisher Merging	Calibration ($N{=}256$)	112.70	66.50
RegMean	Calibration ($N{=}256$)	84.11	75.53
RegMean++	Calibration ($N{=}256$)	115.08	76.64
SLOA ($r{=}64$)	Calibration ($N{=}256$)	175.95	82.12
SLOA ($r{=}128$)	Calibration ($N{=}256$)	182.44	83.43
SLOA ($r{=}256$)	Calibration ($N{=}256$)	186.74	83.74

Wall-clock merge time on ViT-B/16, 20 tasks. SLOA costs about as much as the SVD-based data-free methods (TSV-M, Iso-CTS) while being ~5 pp more accurate. The cost is paid once, when building the merged checkpoint; afterwards no task identity, routing, or test-time adaptation is needed. Rank expansion $r{=}64 o256$ adds only ~11 s.

Key takeaways:

Sequential alignment limits error accumulation across depth and is the largest single source of gains.
QK and OV must be merged jointly — composing them also unlocks rank expansion for extra multi-task capacity.
SLOA keeps improving with more calibration data, while RegMean-style baselines plateau.
SLOA transfers unchanged to LoRA-adapted experts, in both ViT and LLaMA-2-7B.
The optimal $\alpha$ decreases as calibration data grows; the regularizer matters most in the low-data regime.

BibTeX

@misc{maurya2026trainingfreetransformermergingsequential,
      title={Training-Free Transformer Merging via Sequential Local Operator Alignment},
      author={Akansh Maurya and Ya-Wei Eileen Lin and Stefanie Jegelka and Sebastian U Stich and Rotem Mulayoff},
      year={2026},
      eprint={2610.06415},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2610.06415},
}

Training-Free Transformer Merging viaSequential Local Operator Alignment