LLM Domain Expertise: Fine-Tuning, Merging & Emergence

The Challenge of Specialization: Adapting AI for Technical Frontiers

Large Language Models (LLMs) have undeniably revolutionized how we interact with information and automate tasks involving natural language. Giants like Llama and Mistral, even in their open-source forms, showcase remarkable fluency in understanding and generating text that often rivals human output. Their prowess spans a vast landscape, from everyday conversation to complex summarization. However, venturing into the specialized, jargon-rich territories of science and engineering—fields like materials science or biomateriomics—presents a unique hurdle.

These technical domains demand more than general knowledge; they require deep, nuanced understanding, the ability to reason over specific principles, and familiarity with specialized terminology and data structures. Standard LLMs, trained on broad web corpora, often falter when faced with these demands. The challenge, therefore, lies in domain adaptation: how can we effectively tailor these powerful generalist models to become expert assistants in highly specific fields?

Simply feeding more specialized data isn’t always the answer, nor is it always feasible. Training these behemoths from scratch is prohibitively expensive, and the original, massive datasets used for their initial pre-training are typically inaccessible. This is particularly true for popular open-source models where, despite some transparency, the full recipe—the exact data mixes and sequences used during pre-training, fine-tuning, and alignment—remains largely proprietary. Researchers and engineers need robust, efficient strategies to imbue existing models with new, specialized knowledge while crucially preserving the vast general capabilities acquired during their initial training. This delicate balancing act is paramount for creating truly useful AI tools for scientific discovery and engineering innovation, such as developing engines capable of multimodal reasoning to explore biological material design inspiration across diverse scales and contexts.

Charting the Training Landscape: From Pre-Training to Preference Optimization

Navigating the path to domain-specific LLM expertise involves exploring a diverse toolkit of fine-tuning strategies. Each approach offers a different way to shape the model’s knowledge and behavior.

• Continued Pre-Training (CPT): This strategy involves extending the initial pre-training phase, but this time using a corpus focused squarely on the target domain—like a collection of materials science research papers. The goal is to immerse the model in the specific language, concepts, and knowledge structures of the field, allowing it to absorb domain-specific information more deeply than is possible with task-specific fine-tuning alone. It lays a foundation of relevant knowledge.

• Supervised Fine-Tuning (SFT): Following CPT or starting from a base model, SFT directly teaches the model how to perform specific tasks. This is achieved using curated datasets of input-output pairs, often formatted as instructions and desired responses, or questions and accurate answers relevant to the domain. SFT hones the model’s ability to follow instructions, answer questions accurately within the specialized context, and adhere to desired output formats.

• Low-Rank Adaptation (LoRA): While not the primary focus here, LoRA represents an efficient alternative or supplement. Instead of retraining the entire model, LoRA introduces small, trainable ‘adapter’ layers. This allows for significant adaptation with much lower computational cost, though it may have limitations in how much fundamentally new knowledge can be integrated compared to CPT.

• Preference-Based Optimization: Moving beyond simple task completion, preference optimization aims to align the model’s outputs more closely with human judgments or specific criteria like helpfulness, harmlessness, and accuracy in reasoning. Instead of relying solely on predefined ‘correct’ answers (as in SFT), these methods learn from comparisons.

  • Direct Preference Optimization (DPO): DPO learns directly from pairs of responses where one is preferred over the other (e.g., by a human evaluator or another AI). It optimizes the model to increase the likelihood of generating preferred responses without needing a separate reward model, simplifying the traditional Reinforcement Learning from Human Feedback (RLHF) pipeline.
  • Odds Ratio Preference Optimization (ORPO): A newer entrant, ORPO modifies the optimization objective, sometimes yielding improved performance or stability compared to DPO, particularly in aligning models towards specific stylistic or reasoning criteria within a domain.

These techniques are not mutually exclusive; they are often employed sequentially or in combination, forming complex training pipelines. A common sequence might involve CPT to build domain knowledge, followed by SFT for task proficiency, and finally DPO or ORPO for alignment and refinement. However, the optimal combination and sequence remain active areas of research, particularly for achieving peak performance in specialized scientific domains.

Beyond Simple Tuning: The Promise of Merging Models

While refining a single model through sequential training stages can yield significant improvements, another intriguing avenue has emerged: model merging. This practice involves taking two or more separately trained models and combining their parameters—their internal ‘weights’—to create a single, new hybrid model.

Why attempt such a fusion? The core idea is to synergistically combine the strengths of the parent models. Imagine one model expertly trained on materials science literature (via CPT and SFT) and another general-purpose ‘instruct’ model highly adept at following complex instructions and engaging in coherent dialogue. Merging them could potentially create a model that possesses both deep domain knowledge and excellent conversational and instruction-following abilities.

Early explorations hinted that this process might be more than simple averaging. Instead of just blending capabilities, merging could potentially unlock entirely new, emergent functionalities—abilities not explicitly present in either parent model. This suggests a highly non-linear interaction between the parameters during the merge, potentially leading to a whole greater than the sum of its parts. If proven effective and controllable, model merging could represent a powerful, transformative tool for pushing the boundaries of LLM capabilities, creating highly adaptable and potent AI systems tailored for complex, real-world scientific and engineering challenges.

Unveiling the Power of SLERP: A Geometric Approach to Merging

The effectiveness of model merging hinges critically on how the parameters of the parent models are combined. A simple linear averaging (often called Linear Interpolation or LERP) might seem intuitive, but it often leads to suboptimal results or even degrades performance. This is likely because the high-dimensional parameter space of LLMs is not flat; it possesses a complex, curved geometry. Linear interpolation risks traversing ‘dead zones’ or high-loss regions within this space, effectively scrambling the carefully learned representations of the parent models.

Enter Spherical Linear Interpolation (SLERP). Originally developed for smooth animation of rotations in computer graphics, SLERP offers a geometrically sophisticated way to interpolate between two points (in this case, the parameter vectors of two models) by following the shortest path along the surface of a hypersphere.

Imagine the parameter sets of the two parent models as two points on the surface of a giant sphere.

• LERP would draw a straight line through the sphere connecting the points. This path might not stay on the surface and could pass through regions representing poorly performing models.
• SLERP, conversely, travels along the curved surface of the sphere itself. This path inherently respects the underlying geometric structure of the parameter space.

Why is this spherical path potentially superior for merging LLMs?

1. Structure Preservation: By staying ‘on the sphere,’ SLERP maintains the geometric relationships between parameters, preserving the learned structures within each parent model more effectively than a linear path.
2. Avoiding High-Loss Regions: The curved path is less likely to intersect regions of the parameter space associated with high prediction errors (loss).
3. Non-Linear Combination: The interpolation formula for SLERP is inherently non-linear. This allows for complex, synergistic interactions between the parameters from the parent models, potentially unlocking combinations that represent novel capabilities. A merged parameter might activate features in a way neither parent could alone.
4. Smooth Transitions: SLERP provides a mathematically smooth transition between the parent models’ states, potentially leading to better generalization in the merged model.

Because SLERP respects the model’s intrinsic geometry and facilitates non-linear parameter interactions, it holds the potential to not just average capabilities but to genuinely blend them in a way that fosters emergent properties. This makes it a particularly promising candidate for merging models aimed at complex domains like materials science, where subtle interactions and nuanced understanding are key.

Putting Theories to the Test: Llama and Mistral Experiments

To rigorously investigate these fine-tuning and merging strategies, a systematic series of experiments was conducted using popular open-source model families: Llama 3.1 (8 billion parameters) and Mistral (7 billion parameters). The goal was to compare different training pipelines and assess the impact of SLERP merging.

The experimental design involved several key steps:

1. Base Models: Experiments started with both the foundational ‘base’ models (pre-trained but not instruction-tuned) and the ‘instruct’ versions (already fine-tuned for chat and instruction following) for both Llama and Mistral families.
2. Domain Corpus: A specialized corpus focused on materials science was compiled from scientific publications and processed data.
3. Training Pipelines: Various combinations of training techniques were applied:
   • CPT only
   • CPT followed by SFT (CPT-SFT)
   • CPT-SFT followed by ORPO (CPT-SFT-ORPO)
   • CPT-SFT followed by DPO (CPT-SFT-DPO)
   • Some variations starting directly from the Instruct model (e.g., Instruct-CPT-SFT-DPO).
4. Model Merging: For many of the fine-tuned models, SLERP merging was performed, typically combining the domain-adapted model with the corresponding general-purpose ‘instruct’ model from the same family (e.g., a CPT-SFT-DPO Llama model merged with the standard Llama 3.1 Instruct model).
5. Evaluation: The performance of all resulting models (both merged and non-merged) was assessed across a suite of relevant benchmarks designed to test domain knowledge, reasoning, and instruction following.

Key Findings Across Llama and Mistral:

• SLERP Merging Consistently Boosts Performance: Across both model families and various training pipelines, the models enhanced via SLERP merging generally achieved the highest accuracy on the evaluation benchmarks. This strongly supports the hypothesis that SLERP is an effective technique for combining model strengths.
• Synergistic Effects Confirmed: The performance of the SLERP-merged models frequently exceeded a simple average of the performances of the two parent models. Plotting the actual achieved score against this expected average revealed a significant positive deviation, confirming that the merging process often unlocks synergistic gains and emergent capabilities. The merged entity was demonstrably more capable than just the sum of its parts.
• Preference Optimization Adds Value: Incorporating preference optimization stages (DPO or ORPO) often provided an additional performance lift, particularly when combined with SLERP merging. Strategies like CPT-SFT-DPO-SLERP or CPT-SFT-ORPO-SLERP were frequently among the top performers.
• Optimal Non-Merged Strategy Varies: Without merging, the best-performing strategy differed slightly between model families. For Llama 3.1, Instruct-CPT-SFT-DPO showed strong results, while for Mistral, Base-CPT-SFT performed comparably well to its Instruct counterpart.
• Impact of CPT Duration: Further analysis on Mistral models showed that performance generally improved with more epochs of Continued Pre-Training (up to the five tested), especially when starting from the Instruct model, reinforcing the value of sufficient domain exposure during CPT.

These results paint a clear picture: while sequential fine-tuning is valuable, strategic model merging using SLERP offers a powerful pathway to significantly enhance LLM performance, particularly for specialized domains, often yielding capabilities beyond simple aggregation.

Deeper Dive: What Makes Merging Work?

The consistent success of SLERP merging prompts a closer look at the underlying mechanics and influencing factors. Why does this geometric approach yield such potent results, and what conditions optimize its effectiveness?

• Non-Linear Interactions: As theorized, SLERP’s non-linear path through the parameter space appears crucial. It allows the merged model to explore combinations of parameters that linear averaging would miss. These combinations can represent novel interactions between learned features, leading to emergent reasoning or problem-solving abilities tailored to the domain. Imagine combining parameters that, individually, represent understanding ‘material strength’ and ‘biological structures’ – SLERP might find a combination that effectively represents ‘bio-inspired high-strength materials’ in a way neither parent model explicitly did.

• The Role of Diversity: How different should the parent models be? Analysis suggested complex relationships. While extreme diversity might seem beneficial, some correlations indicated that in certain contexts (like Llama models), higher performance diversity between parents might slightly reduce the reliance on subsequent SFT, perhaps because merging already captures a broader capability set. The interplay is subtle and likely depends on the specific fine-tuning methods used for the parents.

• Base vs. Instruct Starting Point: The choice of starting model matters. For the Llama experiments, the top-performing merged model originated from the Instruct version. Conversely, for Mistral, a top performer was derived from the Base model before undergoing CPT, SFT, and merging. This suggests architectural differences or variations in the initial pre-training makeups of the Llama and Mistral families influence how they respond to specific fine-tuning and merging pipelines. There isn’t a single universal ‘best’ starting point; it requires empirical testing.

• Data Quality in CPT: The foundation laid during Continued Pre-Training is critical. Experiments using a largerbut ‘noisier’ CPT dataset (containing more formatting errors or artifacts from optical character recognition) resulted in decreased performance compared to using a smaller, cleaner dataset. This underscores the importance of high-quality, well-processed domain-specific data for the CPT stage to be effective. Garbage in, garbage out still applies.

• Fine-Tuning SLERP Parameters: SLERP itself has parameters, notably the interpolation coefficient (often denoted as ‘t’, ranging from 0 to 1) determining how much weight is given to each parent model. Furthermore, merging doesn’t have to be uniform across all model layers. Experiments explored varying the interpolation factor differently for self-attention layers versus multilayer perceptron (MLP) layers, or even varying it progressively through the model’s depth. Results showed that specific non-uniform weighting schemes could outperform the standard uniform approach, suggesting further optimization potential by carefully tailoring the merge process across the network’s architecture. A simple linear progression of weights across layers proved effective in one Llama case.

• Regularization Effect: SLERP might also act as a form of regularization. By finding a smooth path between two potentially specialized models, it might discourage overfitting to the idiosyncrasies of either parent’s training data, leading to better generalization on unseen domain-specific problems. It might also help mitigate ‘catastrophic forgetting,’ where fine-tuning on one task erases knowledge from a previous one.

In essence, SLERP’s effectiveness stems from its ability to navigate the complex geometry of the LLM parameter space intelligently, fostering beneficial non-linear interactions while preserving learned knowledge structures. However, optimizing its use requires careful consideration of parent model choice, training history, data quality, and potentially even the fine-grained details of the merge itself.

Does Size Matter? Exploring Scaling Effects with Smaller Models

The impressive synergistic effects observed with 7-billion and 8-billion parameter models raise a natural question: do these emergent capabilities unlocked by SLERP merging also manifest in much smaller language models? Or is there a scale threshold below which the magic fades?

To investigate this, similar experiments were conducted using the SmolLM model series, specifically a variant with only 1.7 billion parameters. This model is significantly smaller, making it suitable for resource-constrained environments like mobile devices or edge computing, but potentially lacking the parameter richness of its larger cousins.

The SmolLM models underwent the same pipeline: CPT with the materials science corpus, followed by SFT and DPO (which proved more effective than ORPO for this smaller architecture). SLERP merging was then applied, combining the fine-tuned SmolLM with its base version or other variants.

The Findings with SmolLM:

• Fine-tuning Still Helps: The CPT-SFT-DPO pipeline did improve the SmolLM model’s performance on domain tasks relative to its original state. The fine-tuning process itself was beneficial, enhancing its specialized knowledge.
• Emergence Largely Absent: However, unlike the Llama and Mistral experiments, the SLERP-merged SmolLM models generally did not exhibit significant synergistic effects. Their performance typically landed close to a simple average of the parent models, or only slightly above. The dramatic performance leaps and clear signs of emergent capabilities seen in the 7B/8B models were missing.

Implications:

This contrast suggests that model scale is likely a key factor in realizing the full potential of SLERP merging for generating emergent properties. Smaller models, with their less complex and lower-dimensional parameter spaces, might lack the representational capacity or richness required for these potent non-linear interactions to occur during merging. The ‘room’ for discovering novel, beneficial parameter combinations seems significantly constrained compared to larger models.

These results align with broader observations about scaling laws in deep learning, where certain qualitative capabilities often only emerge once models reach a certain size threshold. It appears that the synergistic power of SLERP merging might be one such capability that depends critically on sufficient model scale and complexity.

Quantifying the Gains: A Closer Look at Performance Lift from Merging

While benchmarks show merged models often perform best overall, it’s useful to quantify precisely how much better they are compared to their parents. Specifically, does the merged model consistently outperform even the stronger of the two models used to create it?

To analyze this, the performance deviation was calculated for each SLERP-merged model. This deviation was defined as:

Performance Deviation = Performance(Merged Model) - Max(Performance(Parent 1), Performance(Parent 2))

• A positive deviation (visualized in shades of blue) means the SLERP model performed better than the best of its parents – clear evidence of synergy.
• A negative deviation (visualized in red) means the SLERP model performed worse than at least one of its parents, indicating the merge was detrimental or at best, averaging.

The Analysis Revealed:

Across the majority of experiments involving the Llama 3.1 (8B) and Mistral (7B) models, the performance deviations were predominantly positive. In many cases, especially for the well-optimized pipelines (e.g., those involving CPT, SFT, preference optimization, and SLERP), the merged models showed substantial positive deviations, indicating they significantly surpassed the capabilities of even their strongest parent.

There were instances, particularly with less optimized parent models or perhaps suboptimal merging parameters, where the deviation was slightly negative or near zero. However, the overarching trend was clear: strategic SLERP merging frequently provides a genuine performance lift beyond what either parent model could achieve alone. This reinforces the idea that merging is not just averaging, but a process capable of synthesizing superior capabilities. The SmolLM (1.7B) results, in contrast, would show much smaller or negative deviations, consistent with the lack of strong emergent effects at that scale.

From Benchmarks to Brainstorming: Interactive Applications in Material Design

Beyond quantitative benchmarks, the true value of these domain-adapted models lies in their ability to assist with real-world tasks, such as scientific reasoning and creative design. To assess this qualitative aspect, interactive chat sessions were conducted with several of the top-performing models (including both merged and non-merged variants).

The setup involved providing a consistent system prompt instructing the model to act as a materials science expert, followed by a user prompt designed to test creative, cross-domain reasoning. A typical task involved asking the model to:

1. Consider two seemingly disparate biological concepts (e.g., the structure of collagen and the venation patterns of leaves).
2. Brainstorm novel material designs inspired by combining principles from both concepts.
3. Explain the reasoning behind the proposed designs.
4. Output the suggestions in a structured format (like JSON) for potential downstream processing.

Qualitative Observations:

• Strong Domain Understanding: All the fine-tuned models demonstrated a solid grasp of the underlying biological and materials science concepts, using appropriate terminology and referencing relevant principles. The CPT and SFT stages clearly imparted significant domain knowledge.
• Creative Synthesis: The models were generally capable of bridging the conceptual gap between the disparate inputs (like collagen and leaves) to propose innovative material architectures or functionalities. This showcased their ability to perform analogical reasoning within the specialized domain.
• Structured Output: Models successfully adhered to instructions requesting structured output (JSON), indicating good instruction-following capabilities, particularly for those refined with SFT and preference optimization or originating from Instruct bases.
• Varying Depth and Clarity: While all performed the core task, differences emerged in the depth of the reasoning provided, the novelty and practicality of the proposed designs, and the overall clarity and coherence of the explanation. Models that underwent more comprehensive training pipelines, especially those including preference optimization and SLERP merging, often provided richer, more insightful, and more creative responses.
• Influence of Merging: Merged models often exhibited a good balance between domain-specific accuracy and conversational fluency/creativity, seemingly integrating the knowledge from the domain-tuned parent with the interaction skills of the general-purpose instruct parent.

These interactive sessions provided valuable qualitative evidence that the fine-tuning and merging strategies translate into tangible improvements in practical, open-ended tasks requiring domain-specific reasoning and creativity. They demonstrated the potential of these tailored LLMs to act as valuable collaborators in scientific exploration and design ideation within fields like materials science.