SteerMusic: Enhanced Musical Consistency for Zero-shot Text-guided and Personalized Music Editing

Earlier music generation work focused on low-level control signals with strict temporal alignment, such as lyrics (65; 12) and MIDI (64) conditioning. Recently, high-level semantic prompts have gained popularity (39; 40; 33; 24; 8; 1; 31; 23; 35; 36; 57). More recent studies further explore melody prompts from reference music (10; 51; 52; 22; 7; 32), enabling more precise, user-driven generation.

Music editing transforms existing audio according to target conditions while preserving the original content. Earlier approaches train models from scratch (61; 1; 9; 22; 46) or fine-tune pretrained models (53; 70; 60; 17; 42). Recent work explores zero-shot methods (71; 45; 41) using pretrained TTA generator. However, these pipelines rely on a forward-backward diffusion process, which can introduce inversion errors as in Fig. 3 (a).

DreamSound (54) explores the possibility of capturing musical concepts from the given reference music using a personalization diffusion model (56; 13; 32). This method allows users to generate new music samples by incorporating the captured personalized musical concept token into the text prompts. In addition to personalized music generation (54; 7), DreamSound further extends their method to personalized music editing. However, we noticed that personalized music editing is still immature. Existing methods (54) struggle to maintain the musical consistency while editing into the desired music concept captured in the reference audio, which use the same music editing pipeline illustrated in Fig. 3 (a).

We introduce SteerMusic, a zero-shot text-guided music editing method that performs editing in the data space. In this setting, our goal is to edit a source music signal $x^{\text{src}}$ by modifying its corresponding text prompt $y^{\text{src}}$, which $y^{\text{src}}$ is a brief description of the source music that includes the specific musical attribute intended for modification. The modified text prompt $y^{\text{tgt}}$ acts as the target prompt to guide the editing. To obtain desirable results, the musical content shared by $y^{\text{tgt}}$ and $y^{\text{src}}$ should be preserved, changing only the content that is distinct in $y^{\text{tgt}}$. For example, if the only change in $y^{\text{tgt}}$ compared to $y^{\text{src}}$ is replacing only the word “piano” with “guitar” while keeping the rest of the sentence unchanged, the edited music should preserve the melody and tempo, while modifying only the musical instrument.

To achieve this goal, we adopt the DDS, which has been previously explored only in the image domain. To the best of our knowledge, we are the first to investigate the potential application in music editing. Following DDS, we define $x=g(\theta)$ rendering a source music signal $x^{\text{src}}$. We picked $g(\theta)=\theta$ as the differentiable function in Eq. 1, where we initialize $\theta=x_{0}^{\text{src}}$. i.e. $x=x_{0}^{\text{src}}$. In SteerMusic, we set $\epsilon_{\phi}$ as a pretrained TTA or text-to-music DPMs.

Similar to 19, the delta score in Eq. 1 steers the optimization process toward the target prompt while reducing the noisy editing direction commonly associated with vanilla SDS, leading to enhanced edit fidelity. SteerMusic method enables flexible editing using text instructions alone; however, it is limited to coarse-grained editing, as text prompts often lack the precision information to capture fine-grained musical details.

As mentioned before, text-guided editing lacks customization for precise music editing, such as transferring music to a specific style. To enable fine-grained personalized editing, we propose SteerMusic+, an extension of SteerMusic. In this setting, we have a set of source music and prompt pair $\{x^{\text{src}},y^{\text{src}}\}$. The target prompt $y^{\text{tgt}}$ is constructed by modifying $y^{\text{src}}$ to manipulate a user-defined concept token $[S]$, representing the desired customization (see example in Fig. 4 (b)). SteerMusic+ uses two pretrained diffusion models:

• •

  A pretrained diffusion probabilistic model (DPM), denoted as $\epsilon_{\phi}$, which serves as a reference for maintaining consistency with the original music content; and

• •

  A personalized diffusion model (PDM), denoted as $\epsilon_{\phi^{\prime}}$, is fine-tuned on a small set of data containing reference music to capture the user-defined concept $[S]$ and guide the editing process toward the desired direction.

We define successful personalized editing by the criteria:

• •

  The instruction-irrelevant part (i.e., {$y^{\text{tgt}}\cap y^{\text{src}}$}) in the source music should be maintained.

• •

  The edited musical attributes should perceptually align with the intended personalized musical concept $[S]$.

We now present the SteerMusic+ method that enables personalized music editing with enhanced music consistency.

Personalized Diffusion Model (PDM) serves a foundational part in SteerMusic+. Since training a PDM has been extensively studied in 54; 56; 13; 32, we assume the availability of a pretrained text-to-music PDM in SteerMusic+, as SteerMusic+ is a plug-in pipeline compatible with existing PDMs. The text-to-music PDM $\epsilon_{\phi^{\prime}}$ captures the user-defined musical concept $S$ by fine-tuning a pretrained DPM $\epsilon_{\phi}$ under a small set of reference music $\mathcal{D}^{\text{ref}}=\{(x^{\text{ref}},y^{\text{ref}})_{n}\}^{N}_{n=1}$, which $N$ can be as few as $1$ (54). The fine-tuning is achieved via optimizing the objective

where $\hat{\phi}$ is initialized with a pretrained DPM weights $\phi$. As illustrated in Fig. 4 (a), the prompt $y^{\text{ref}}$ takes form of “a recording of a [$S$]”, where the placeholder [$S$] corresponds to a defined new concept word embedding. During inference, the PDM can generate music with the newly learned concept (e.g., “A disco song with a [$S$]”). The dataset $\mathcal{D}^{\text{ref}}$ consists of reference audio clips that encapsulate the concept users aim to extract. According to 54, the concept represents a musical style, which can be either an instance of instrument sounds or a specific genre that cannot be yielded even with the most detailed textual description. For instance, if the objective is to capture the user’s guitar playing style, the reference audio should feature performances on the user’s guitar playing. Conversely, if the goal is to capture the concept of jazz, the reference audio should consist of recordings that exemplify the same jazz style.

While the PDM can generate new music based on concepts from $\mathcal{D}^{\text{ref}}$, it cannot directly edit existing music using $y^{\text{tgt}}$ that contains the concepts. Next, we introduce key components of SteerMusic+ that enable personalized editing with PDMs.

Personalized Delta Score (PDS) is an extension of Eq. 1 to enable personalized music editing. We define the PDS loss as $\mathcal{L}_{\text{PDS}}(\phi^{\prime},\phi,x=g(\theta),y^{\text{tgt}},x^{\text{src}},y^{\text{src}})=\mathbb{E}_{t,\epsilon}[w(t)\|\epsilon_{\phi^{\prime}}(x_{t},y^{\text{tgt}},t)-\epsilon_{\phi}(x^{\text{src}}_{t},y^{\text{src}},t)\|^{2}_{2}]$. By omitting the UNet Jacobian, the gradient over $\theta$ is given by $\nabla_{\theta}\mathcal{L}_{\text{PDS}}=\mathbb{E}_{t,\epsilon}[w(t)(\epsilon_{\phi^{\prime}}(x_{t},y^{\text{tgt}},t)-\epsilon_{\phi}(x^{\text{src}}_{t},y^{\text{src}},t))\frac{\partial x}{\partial\theta}]$, where $x_{t}$ and $x_{t}^{\text{src}}$ share the same sampled noise $\epsilon$ at the time step $t$. This modified delta score can be decomposed into two components: the score of $\epsilon_{\phi^{\prime}}$ provides the desired direction to guide the editing to match the target prompt with the concept $[S]$. The score of $\epsilon_{\phi}$ reduces the noisy direction of unintended modification areas. The delta score between the PDM $\epsilon_{\phi^{\prime}}$ and the DPM $\epsilon_{\phi}$ may not produce an effective direction toward $y^{\text{tgt}}$, as $\epsilon_{\phi^{\prime}}$ shifted to the reference distribution $\mathcal{D}^{\text{ref}}$. We introduce an additional component to compensate for the distribution shift induced by the score of $\epsilon_{\phi^{\prime}}$.

Distribution Shift Regularization. To bridge the distribution gap between $\epsilon_{\phi^{\prime}}$ and $\epsilon_{\phi}$, we introduce a regularization term to regularize the edited score to the personalized diffusion model $\epsilon_{\phi^{\prime}}$. We wish to minimize the distribution shift between two diffusion models by adding a constraint as

where $\zeta$ is a small amount of constant; the regularization term is $\mathcal{L}_{\text{shift}}(\phi^{\prime},\phi,x=g(\theta),y^{\text{tgt}})=\mathbb{E}_{t,\epsilon}[w(t)\|\epsilon_{\phi^{\prime}}(x_{t},y^{\text{tgt}},t)-\epsilon_{\phi}(x_{t},y^{\text{tgt}},t)\|^{2}_{2}]$. The gradient in respect to $\theta$ is given by $\nabla_{\theta}\mathcal{L}_{\text{shift}}=\mathbb{E}_{t,\epsilon}[w(t)(\epsilon_{\phi^{\prime}}(x_{t},y^{\text{tgt}},t)-\epsilon_{\phi}(x_{t},y^{\text{tgt}},t))\frac{\partial x}{\partial\theta}]$. Therefore, the overall gradient through $\theta$ is

where $\lambda$ is a constant that adjusts regularization strength.

Personalized Contrastive (PCon) Loss. Eq. 4 formulates a regularized reference guided editing direction, where the regularized delta score encourages alignment with the target prompt while mitigating the distribution shift. However, it does not explicitly enforce the fidelity to the concept, which may lead to suboptimal editing quality. To further enhance the fidelity of the edit, we incorporate a PCon loss between temporal features, which is modified from a patch-wise contrastive loss (48). PCon loss extracts intermediate features $h^{\text{src}}_{l}$ and $h_{l}$ that pass through the residual block and the self-attention block from $\epsilon_{\phi}$ conditioned on $y^{\text{src}}$ and $\epsilon_{\phi^{\prime}}$ conditioned on $y^{\text{tgt}}$ at the $l$-th self-attention layer, respectively. The features are then reshaped to size $\mathbb{R}^{T_{l}\times F_{l}\times C_{l}}$, where $T_{l},F_{l}$, and $C_{l}$ represent the size of the temporal, spatial, and channel dimensions in the $l$-th layer, respectively. The patch corresponding to the temporal location on the feature map $h^{\text{src}}_{l}$ is designated as ‘positive’, and vice versa. The PCon loss is defined as

where $t^{\prime}\in\{1,...,T_{l}\}$ represents the temporal location query patch, the positive patch as $h_{l}^{src,t^{\prime}}$ while the other patches as $h_{l}^{src,T_{l}\backslash t^{\prime}}$. $\text{exp}(h\cdot h^{+}/\tau)$ is a a positive sample with the same temporal location, $\text{exp}(h\cdot h^{-}/\tau)$ is a negative sample with a mismatched temporal location in the self-attention features, $\tau$ is a temperature parameter as $\tau>0$.

The gradient of $\mathcal{L}_{\text{PCon}}(x,x^{\text{src}})$ will propagate to the hidden state of self-attention layers $h$ in personalized diffusion $\epsilon_{\phi^{\prime}}$. Given that the personalized diffusion $\epsilon_{\phi^{\prime}}$ has a distribution shift over the reference dataset $\mathcal{D}^{\text{ref}}$, the $\mathcal{L}_{\text{PCon}}$ explicitly encourages feature similarity at the frequency domain in self-attentions, particularly for attributes that distinguish the target concept. This reinforcement leads the model to prioritize concept consistency over strict temporal alignment with the source music. Consequently, $\mathcal{L}_{\text{PCon}}$ amplifies the distinctive characteristics of the target concept of $\epsilon_{\phi^{\prime}}$ in SteerMusic+, ensuring that the edited music maintains stronger fidelity to the desired style while allowing structural variations.

We evaluate music editing objectively based on two aspects: musical consistency (content preservation before and after editing) and editing fidelity. We follow 45 and calculate the following objective metrics for measurement. To evaluate musical consistency, we use:

• •

  Fréchet Audio Distance (FAD) (30) which measures the distributional difference between source and edited music (lower the better). We calculate FAD based on both VGGish (18) and clap-laion-music (63) embeddings, denoted as FAD${}_{\text{Vggish}}$ and FAD${}_{\text{CLAP}}$ respectively.

• •

  LPAPS (26), an audio version of LPIPS (68), which quantifies the consistency of the edited audio relative to the source audio (lower the better).

• •

  Top-1 Constant-Q Transform Pearson Correlation Coefficient (CQT1-PCC), which measures melody consistency between the source and edited music (higher the better). The CQT1-PCC (5) extracts the main melody of the music audio and has been shown to outperform traditional chroma-based features in representing melodic characteristics (22). While existing metrics such as FAD and LPAPS provide insight into audio quality and perceptual similarity, they fall short in comprehensively capturing melodic structure. Moreover, existing transcription models (11; 2; 14; 6; 43) effectively extract melodies from real music, they are unreliable for synthesized audio. To address these limitations, we introduce CQT1-PCC as a supplementary objective metric, specifically designed to quantify melodic consistency in generative music editing. This metric enables a more targeted evaluation of whether the core melodic structure of the source audio is retained in the edited result.

To evaluate editing fidelity, we use:

• •

  CLAP Score (63) which measures the alignment between edited music and the target prompt in text-guided music editing (higher is better).

• •

  CDPAM (44), a perceptual audio metric that leverages deep learning representations to measure perceptual distance between audios such as music and speech (27; 16; 15). We use CDPAM to evaluate audio perceptual similarity between reference music and the edited result in personalized music editing (lower the better).

Subjective evaluation. We designed a mean opinion score (MOS) study to evaluate target editing fidelity (MOS-T) and source content preservation (MOS-P) by asking participants to rate the results from 1-Bad to 5-Excellent (58) for randomly selected edited samples. We provide more experimental details in our extended version.

In this part, we evaluate our SteerMusic method on the zero-shot text-guided music editing task.

Dataset. We use the ZoME-Bench dataset (41) which includes 1,000 10-second audio samples from MusicCaps (1), each paired with source and target text prompts. We evaluate our models on four well-defined editing tasks that require modifying a specific aspect of the audio while preserving the original melody: change instrument (131 clips), change genre (134), change mood (100), and change background (95). To assess long-form editing, we use the MusicDelta subset of MedleyDB (3) with ranging from 20 seconds to 5 minutes, comprising 34 excerpts of varying styles and lengths, with prompts from 45.

Baseline. We compare SteerMusic with zero-shot text-guided music editing methods plug-in the same pretrained AudioLDM2 (40), including SDEdit (47), DDIM (59), ZETA (45), and MusicMagus (71). To ensure statistical reliability, experiments are conducted using multiple random seeds. We are unable to include MelodyFlow (35) and MEDIC (41) due to the lack of source code. We exclude AudioEditor (28) and AudioMorphX (37), which are designed for general sound editing rather than music.

Experimental results. Tab.1 compares SteerMusic with zero-shot baselines across various style transfer tasks. SteerMusic achieves higher source consistency, shown by improved CQT1-PCC, lower LPAPS, and FAD. While DDIM attains a slightly higher CLAP score (+5e-3), its low CQT1-PCC and high LPAPS indicate poor preservation of source content due to lack of further source consistency constraints during denoising. In contrast, SteerMusic effectively balances source consistency and edit fidelity, fulfilling the core objective of music editing. Furthermore, our method attains the highest MOS scores and yields statistically significant improvements over all baseline models. An ANOVA test shows MOS-P $=2.92$ with p-value $=7.37\times 10^{-27}$ and MOS-T $=2.5$ with p-value $=3.24\times 10^{-7}$. These results demonstrate that SteerMusic provides substantially better editing performance in terms of both source-music consistency and editing fidelity as perceived by human listeners.

To assess real-world applicability, we further evaluate our method on MusicDelta dataset in Tab. 3, which consists of varying lengths of music clips. MusicMagus fails in this experiment as it was designed for 5-second editing and doesn’t support long-form music, and hence is excluded. SteerMusic consistently outperforms other baselines in terms of edit fidelity and source consistency, demonstrating its robustness and effectiveness in handling longer and more complex music editing.

User preference study. We evaluate SteerMusic with the user preference study following the design in 45. To reduce cognitive load and improve reliability, we compare SteerMusic with the top-performing baselines, MusicMagus and ZETA. In this study, users were asked to answer a sequence of 20 questions, each question contains original music, an editing instruction, and two edited results. Users were instructed to select the edited result that better matches the instruction while preserving the main content of the original music. We collected 25 full responses, which the participants having a minimum of 1 and average of 5 years of music training. As shown in Fig. 5, our method was clearly preferred over all competing methods.

In this part, we evaluate our SteerMusic+ method, which is designed for personalized music editing task.

Dataset. To cover both common and exotic concepts, we selected eight representative musical concepts from the 32 defined in 54: four instruments style (Guitar, Bouzouki, Ocarina, Sitar) and four genres (Morricone, Reggae, Hiphop, Sarabande). Each concept includes a placeholder instruction and five 10-second reference clips from YouTube and FreeSound. We use the “change instrument” and “change genre” tasks from ZoME-Bench (41), replacing the original target prompt with the selected concept token. Due to the lack of detailed instructions in MusicDelta, we use the standardized prompt as “A recording of a [style] song.”, where [style] is either the source style or the target style concept [S].

Baseline. We set two existing personalized music editing methods proposed by 54 as the baselines, Textual inversion and DreamSound. Textual inversion optimizes a concept embedding, whereas DreamSound fine-tunes an AudioLDM2 with rare-token identifiers (56). Both methods perform personalized music editing by manipulating the concept token during denoising process. We follow the official codes provided by 54 to obtain text-to-music PDM. We reproduce the personalized music editing methods by calculating noisy latent representations $x_{t}$ from $x_{0}^{\text{src}}$ of a DPM conditioned on the source prompt with a predefined shallow time step $t$ using DDIM inversion (59), where $t=30$. We denoise $x_{t}$ on the PDM ¹¹ 1 Textual inversion optimizes only the concept token embedding rather than fine-tuning a PDM, we denoise $x_{t}$ using the DPM employed during DDIM inversion. Consequently, textual inversion is incompatible with SteerMusic+ pipeline, which relies on a PDM. conditioned on the target prompt linked to the learned concept to obtain $x_{0}^{\text{tgt}}$. To ensure statistical reliability, experiments are conducted using multiple random seeds. We do not compare with Jen-1 DreamStyler (7), which is a personalized music generation method.

Experimental results. We plug SteerMusic+ into the PDM used in DreamSound¹. Tab. 2 shows that SteerMusic+ achieves superior musical consistency compared to the baselines. It indicates that the edited outputs of SteerMusic+ successfully preserve instruction-irrelevant music content in the source music. Furthermore, SteerMusic+ performs accurate edits that align well with the concepts captured from references, as indicated by the low CDPAM compared to the baseline methods. These objective evaluation results aligned with the subjective metrics, with SteerMusic+ obtaining significantly higher MOS-P and MOS-T than the baselines as measured by an ANOVA test (MOS-P with p-value $1.38\times 10^{-13}$ and MOS-T with p-value $1\times 10^{-3}$). In long-form music editing (Tab. 3), SteerMusic+ still outperforms the baseline. We exclude Textual Inv. from comparison as it fails to produce meaningful results on MusicDelta, likely due to the limitation of its base model with complex inputs. Fig. 6 shows a personalized instrument style transfer example, where SteerMusic+ effectively preserves instruction-irrelevant content from the source music.

User preference study. We evaluate SteerMusic+ by a user preference study compared to DreamSound and Textual Inv. Users were asked to select the edited result that best matches the reference style while preserving the main content of the source music. We collected 24 full responses, excluding responses that are partially finished. Participants have a minimum of 1 and an average of 5 years of music training experience. As shown in Fig. 7, SteerMusic+ is clearly preferred by participants compared to baselines.

Ablation study. We conduct an ablation study to understand the effects of different components in SteerMusic+. We used the concept [bouzouki] as it is an uncommon musical instrument that typically requires the use of personalized models. The regularization weight $\lambda$, which acts as the Lagrange multiplier for the constraint in Eq. 4, was tested within [-1,1] to balance the constraint enforcement and the optimization stability. Yellow dots indicate results obtained by $\mathcal{L}_{\text{PDS-O}}$ in Eq. 4 with varying $\lambda$ values. From the zoomed-in view in Fig. 8, as the CDPAM value decreases with $\lambda$, the loss of $\mathcal{L}_{\text{shift}}$ within $\mathcal{L}_{\text{PDS-O}}$ helps steer the editing toward the concept. However, the effect of $\mathcal{L}_{\text{shift}}$ is minor, suggesting that the gradient of $\mathcal{L}_{\text{PDS}}$ alone does not sufficiently capture the target concept during editing. As a result, the edited music struggles to align with the reference musical characteristics. In contrast, SteerMusic+ (with $\mathcal{L}_{\text{PCon}}$ in Eq. 5) leads to a significant decrease in the CDPAM value, which indicates better alignment with the target concept. Since $\mathcal{L}_{\text{PCon}}$ was proposed to explicitly enhance the characteristics of the target concept on the editing, we observe that $\mathcal{L}_{\text{shift}}$ becomes more effective in this setting. Specifically, when $\lambda$ is negative, the edit preserves the original content, while positive $\lambda$ pushes the edit toward the target concept.

This highlights a fundamental trade-off between the source music consistency and the adherence to the target style, consistent with the findings of 45. In personalized music editing, where target characteristics are derived from reference music, shifting the output towards the concept often disrupts the original content. This differs from text-guided editing, which follows abstract textual cues rather than concrete musical references. The key challenge in personalized editing is balancing the retention of the original music content, while integrating the distinctive attributes of the reference music. For practical use, we recommend keeping a smaller $\lambda$ value (e.g., $\lambda=0.05$) to avoid over-editing.

Limitation. As shown in Fig. 9, the number of fine-tuning steps in the PDM significantly affects the performance of DreamSound and SteerMusic+. A small number of steps (e.g., 50) leads to poor concept learning and low edit fidelity, while a large number of steps (e.g., 200) causes overfitting and source music structural loss. These findings highlight the importance of balancing PDM fine-tuning steps to achieve both high edit fidelity and content preservation in personalized music editing settings. The editing fidelity of SteerMusic+ is fundamentally limited by the ability of the PDMs to capture the concept attributes. Consequently, failure cases can often be attributed to inherent shortcomings in the PDM’s representational capacity. We refer readers to 54, which investigated the capacity of PDMs to capture musical attributes and offers insights into improving this capacity.