SLOA: Sequential Local Operator Alignment for Training-Free Model Merging

SLOA aligns the effective local operators ($\mathbf{W}_Q\mathbf{W}_K^\top$, $\mathbf{W}_V\mathbf{W}_O$, LayerNorm affine, MLP projections) rather than raw weight matrices, sequentially under activations of the partially-merged model, then factorizes each operator back into transformer-compatible weights.

Abstract

Training-free model merging combines multiple task-specialized models into a single multi-task model without joint retraining. Existing methods merge weight matrices $\mathbf{W}_Q, \mathbf{W}_K, \mathbf{W}_V, \mathbf{W}_O$ independently — ignoring the fundamental structure of transformer attention, where behavior depends on the composed operators $\mathbf{W}_Q\mathbf{W}_K^\top$ (which inputs the model attends to) and $\mathbf{W}_V\mathbf{W}_O$ (what information is written to the residual stream).

We propose Sequential Local Operator Alignment (SLOA), which merges these local functional operators directly. Each operator is aligned via a closed-form regularized least-squares objective. Crucially, alignment is performed sequentially in topological order — each step uses activations from the partially-merged model, correcting for accumulated upstream errors. After merging in the ambient $d \times d$ space, each operator is factorized back via SVD at a chosen rank $r$, allowing rank expansion ($r > d_k$) as a capacity-control mechanism that retains multi-task singular directions without any training.

SLOA achieves state-of-the-art training-free merging on CLIP/ViT vision benchmarks (8, 14, and 20 tasks) and RoBERTa NLU benchmarks (7 GLUE tasks), outperforming RegMean++ by +6.6 pp on the hardest 20-task vision setting.

Motivation: Why Isolated Merging Fails

Relative functional error along linear interpolations between equivalent factorizations. Error grows rapidly — equality holds only at $c=1$.

Hidden ambiguity. Any invertible $\mathbf{R}$ gives a different factorization with the same function:

$$(\mathbf{W}_Q\mathbf{R})(\mathbf{W}_K\mathbf{R}^{-\top})^\top = \mathbf{W}_Q\mathbf{W}_K^\top$$

Independent averaging fails. Averaging two such parameterizations yields

$$\bar{\mathbf{M}}_\mathrm{QK} = \tfrac{1}{4}\mathbf{W}_Q(\mathbf{I}+\mathbf{R})(\mathbf{I}+\mathbf{R}^{-1})\mathbf{W}_K^\top \;\neq\; \mathbf{W}_Q\mathbf{W}_K^\top$$

In the scalar case $\mathbf{R}=c\mathbf{I}$, the error grows without bound as $c\to\infty$. Two task models that are functionally identical at the QK level can produce arbitrarily different merged attention scores under isolated weight averaging.

Method

Unified Alignment Objective

For each local operator $\bm{\theta}$, SLOA solves:

$$\bm{\theta}_\mathrm{merge}= \arg\min_{\bm{\theta}}\; \underbrace{\tfrac{1}{T}\!\sum_{t,i}\! \bigl\|\mathcal{A}(\widehat{\mathbf{Z}}_{t,i};\bm{\theta})- \mathcal{A}(\mathbf{Z}_{t,i};\bm{\theta}_t)\bigr\|_F^2}_{\text{activation matching}} \,+\, \underbrace{\tfrac{1}{T}\!\sum_t\!\|\bm{\theta}-\bm{\theta}_t\|^2_{\widehat{\chi}_t}}_{\text{data-aware geometry}}$$

$\widehat{\mathbf{Z}}$ are activations from the partially-merged model; the geometry term uses a Mahalanobis distance that penalizes deviation along directions the calibration activations actually use. For fixed activations the objective is quadratic in $\bm{\theta}$ — closed-form regularized least squares per component.

Local Operators & Metrics

Component	Local operator $\bm{\theta}$	Behavior map $\mathcal{A}(\mathbf{Z};\bm{\theta})$	Metric
LayerNorm	$(\bm{\gamma},\bm{\beta})$	$\nu(\mathbf{X})\,\mathrm{diag}(\bm{\gamma})+\mathbf{1}\bm{\beta}^\top$	$\mathrm{Cov}(\nu(\chi))$
Linear map	$\mathbf{W}$	$\mathbf{X}\mathbf{W}$	$\bm{\Lambda}$
QK component	$\tfrac{1}{\sqrt{d_k}}\mathbf{W}_Q\mathbf{W}_K^\top$	$\mathbf{X}\bm{\theta}\mathbf{X}^\top$	$\bm{\Lambda}\otimes\bm{\Lambda}$
VO component	$\mathbf{W}_V\mathbf{W}_O$	$\mathbf{A}\mathbf{X}\bm{\theta}$	$\bm{\Lambda}$

$\nu(\mathbf{X})$ = LayerNorm-normalized input; $\mathbf{A}$ = attention matrix; $\bm{\Lambda}=\alpha\,\mathrm{diag}(\mathrm{Cov}(\mathbf{X}))+\epsilon\mathbf{I}$.

SLOA in 4 Steps

1. Collect activations $\{\mathbf{Z}_{t,i}\}$ for each operator from a small calibration set (256 samples per task by default).

2. Order operators topologically: LayerNorm $\to$ QK $\to$ VO $\to$ MLP $W_1$ $\to$ MLP $W_2$.

3. For each operator: forward through the partially-merged model to get $\widehat{\mathbf{Z}}_{t,i}$, then solve closed-form regularized LS.

4. Factorize QK/VO operators via SVD at chosen rank $r_j$ and write back transformer-compatible weights. Rank $r_j > d_k$ retains additional task-specific singular directions.

No gradients, no labels — one deterministic pass through the operator graph.

Results

Vision: ViT-B/32 (CLIP fine-tunes)

Average classification accuracy (%) on 8, 14, and 20 vision tasks. Training-free methods use 256 calibration samples per task.

	Method	Samples	8 tasks	14 tasks	20 tasks
	Fine-tuned (oracle)	—	90.39	89.26	89.78
Data-Free	Weight Averaging	—	66.32	65.36	61.10
	Task Arithmetic	—	67.55	52.80	36.30
	TIES-Merging	—	71.90	67.57	55.56
	TSV-M	—	83.07	78.79	73.22
	Iso-C	—	82.51	76.51	66.54
	Iso-CTS	—	82.71	78.32	70.22
Training-Free	Fisher Merging	256	70.35	66.50	62.04
	RegMean	256	82.50	77.75	72.14
	RegMean++	256	83.97	79.52	74.06
	SLOA ($r{=}64$)	256	84.89	81.90	80.08
	SLOA ($r{=}128$)	256	85.52	82.63	81.02
	SLOA ($r{=}256$)	256	85.62	82.79	81.28

SLOA is the best training-free method on all three benchmarks; the gap to baselines widens with more tasks. Rank expansion $r{=}64\to128$ consistently helps. +6.6 pp over RegMean++ on the hardest 20-task setting.

Task-wise comparison: SLOA vs. data-free and training-free baselines on ViT-B/32 (20 tasks) and RoBERTa (7 GLUE tasks). SLOA dominates across tasks in both modalities.

Vision: ViT-B/16 (CLIP fine-tunes)

Average classification accuracy (%) on 8, 14, and 20 vision tasks. Training-free methods use 256 calibration samples per task.

	Method	Samples	8 tasks	14 tasks	20 tasks
	Fine-tuned (oracle)	—	92.34	91.31	91.61
Data-Free	Weight Averaging	—	72.33	69.75	64.82
	Task Arithmetic	—	77.14	60.79	39.89
	TIES-Merging	—	77.60	71.51	60.64
	TSV-M	—	87.10	82.13	78.31
	Iso-C	—	87.86	81.11	73.71
	Iso-CTS	—	88.32	83.88	77.66
Training-Free	Fisher Merging	256	75.59	72.01	67.01
	RegMean	256	86.25	81.55	75.50
	RegMean++	256	87.32	82.36	76.67
	SLOA ($r{=}64$)	256	87.18	84.19	82.14
	SLOA ($r{=}128$)	256	88.01	85.16	83.45
	SLOA ($r{=}256$)	256	88.09	85.30	83.62

SLOA leads on 14 and 20 tasks; on 8 tasks it is within 0.2 pp of Iso-CTS. The margin over the best baseline grows with task count (+6.95 pp over RegMean++ at 20 tasks).

Vision: ViT-L/14 (CLIP fine-tunes)

Average classification accuracy (%) on 8, 14, and 20 vision tasks. Training-free methods use 256 calibration samples per task.

	Method	Samples	8 tasks	14 tasks	20 tasks
	Fine-tuned (oracle)	—	94.34	93.39	93.54
Data-Free	Weight Averaging	—	79.87	77.53	71.15
	Task Arithmetic	—	80.46	63.12	35.99
	TIES-Merging	—	83.83	77.79	62.96
	TSV-M	—	90.57	88.21	83.19
	Iso-C	—	92.15	89.32	82.80
	Iso-CTS	—	92.72	90.01	86.12
Training-Free	Fisher Merging	256	82.41	75.26	69.85
	RegMean	256	90.45	87.82	81.13
	RegMean++	256	90.94	87.80	82.26
	SLOA ($r{=}64$)	256	89.37	87.55	85.67
	SLOA ($r{=}128$)	256	89.94	88.25	86.38
	SLOA ($r{=}256$)	256	90.09	88.37	86.62

On the largest backbone SLOA wins clearly at 20 tasks (+4.36 pp over RegMean++); Iso-CTS edges it on 8/14 tasks, but SLOA is the only method robust as the task count scales up.

NLU: RoBERTa on 7 GLUE Tasks

Per-task metric (Matthews/F1/accuracy) and average on seven GLUE tasks. Training-free methods use 256 calibration samples per task.

	Method	Samples	CoLA	SST-2	MRPC	QQP	MNLI	QNLI	RTE	Avg
	Individual	—	0.602	0.940	0.892	0.914	0.872	0.927	0.791	0.848
Data-Free	Weight Averaging	—	0.181	0.819	0.779	0.796	0.438	0.711	0.617	0.620
	Task Arithmetic	—	0.233	0.866	0.787	0.840	0.637	0.730	0.610	0.672
	TIES-Merging	—	0.250	0.835	0.787	0.852	0.607	0.758	0.422	0.644
	TSV-M	—	0.332	0.872	0.838	0.860	0.590	0.727	0.466	0.669
	Iso-C	—	0.341	0.876	0.811	0.799	0.669	0.721	0.729	0.707
	Iso-CTS	—	0.294	0.820	0.707	0.670	0.459	0.636	0.686	0.610
Training-Free	Fisher Merging	256	0.113	0.872	0.742	0.781	0.423	0.531	0.515	0.568
	RegMean	256	0.280	0.893	0.807	0.792	0.725	0.834	0.616	0.707
	RegMean++	256	0.245	0.914	0.770	0.813	0.760	0.851	0.691	0.721
	SLOA	256	0.493	0.906	0.876	0.795	0.721	0.856	0.681	0.761
	SLOA ($r{=}128$)	256	0.508	0.917	0.879	0.814	0.762	0.881	0.714	0.782

bold = best in column; underline = second best. SLOA ($r{=}128$) wins 6/7 tasks and the average.

LLM: LLaMA-2-7B LoRA Experts

Merging three LLaMA-2-7B LoRA experts (math, code, instruction-following) into one model. RoPE blocks a position-independent $\mathbf{W}_Q\mathbf{W}_K^{\top}$, so SLOA is applied to the VO circuit only ($r_{\mathrm{VO}}$ = factorization rank); training-free methods use 256 calibration samples per task.

	Method	GSM8K	HumanEval	IFEval	Avg.
	LLaMA-2-7B (pretrained)	4.32	0.00	20.89	8.40
Experts	IF-Tulu	12.59	1.22	66.17	26.66
	Code (Magic-Coder)	9.70	32.93	26.99	23.21
	Math (Meta-Math)	65.50	0.00	15.53	27.01
Data-Free	Weight Averaging	40.11	18.29	40.67	33.02
	Task Arithmetic	58.98	15.85	42.70	39.18
	TIES-Merging	48.14	20.73	34.57	34.48
	TSV-M	58.00	18.29	42.14	39.48
	Iso-C	38.89	21.34	57.67	39.30
	Iso-CTS	36.09	21.34	55.45	37.63
Training-Free	RegMean	56.03	29.27	48.24	44.51
	RegMean++	55.95	29.27	54.16	46.46
	SLOA-VO ($r_{\mathrm{VO}}{=}128$)	58.61	28.05	57.86	48.17
	SLOA-VO ($r_{\mathrm{VO}}{=}256$)	59.51	27.44	58.96	48.64
	SLOA-VO ($r_{\mathrm{VO}}{=}384$)	59.51	27.44	60.63	49.19

SLOA-VO is the best training-free method, beating RegMean++ by +2.7 avg. pts and every data-free baseline — operator-level merging extends beyond attention-only heads to a 7B decoder-only LLM and to LoRA-adapted experts. Rank expansion ($r_{\mathrm{VO}}{=}128{\to}384$) keeps improving IFEval/avg., mirroring the vision rank-expansion trend.

Ablations

Component ablation on 20-task ViT-B/32. Sequential alignment alone is already strong; adding QK+VO alignment and LayerNorm correction both help.

Decoupling QK and OV ranks on 20 tasks. QK and OV must be merged jointly — isolating one hurts performance.

Performance vs. calibration samples and rank on 20-task ViT-B/32. Rank expansion consistently improves performance; more calibration data helps too.

Sensitivity to regularization strength $\alpha$ on 8 tasks. Optimal $\alpha$ decreases as more calibration data is available.

NLP calibration scaling (RoBERTa, $r{=}64$, $\alpha{=}0.25$, 3-seed avg). Performance increases monotonically — 1024 samples (avg 0.799) surpasses $r{=}256$ at 256 samples (0.797).

Key takeaways from ablations:

Sequential alignment alone is already strong.
QK and OV must be merged jointly — isolating one hurts.
Rank expansion consistently improves performance.
Optimal $\alpha$ decreases as more calibration data is available.
NLP performance increases monotonically with more calibration data — no saturation observed up to 1024 samples.

BibTeX

Citation coming soon.

Sequential Local Operator Alignmentfor Training-Free Model Merging

Abstract

Motivation: Why Isolated Merging Fails

Method

Unified Alignment Objective

Local Operators & Metrics

SLOA in 4 Steps

Results

Vision: ViT-B/32 (CLIP fine-tunes)

Vision: ViT-B/16 (CLIP fine-tunes)

Vision: ViT-L/14 (CLIP fine-tunes)

NLU: RoBERTa on 7 GLUE Tasks

LLM: LLaMA-2-7B LoRA Experts

Ablations

BibTeX

Sequential Local Operator Alignment
for Training-Free Model Merging