Towards Blind Data Cleaning: A Case Study in Music Source Separation

We propose a data cleaning method based on data attribution, motivated by the hypothesis that noisy samples contribute less to the trained model. Inspired by a recent unlearning-based data attribution method [10, 13], we use unlearning to attribute the influence of each training sample. To make influence estimation efficient, we adopt the mirrored influence hypothesis [9, 10], which states that train-to-test influence is highly correlated with test-to-train influence. Accordingly, instead of unlearning $N$ training samples, we unlearn a small set of $M$ trusted clean samples and measure the resulting impact on each training sample’s loss. This approach is computationally efficient since $M\ll N$. A low impact suggests that the training sample does not align well with the characteristics of clean data, making it a candidate for removal.

As shown in Figure 1, our unlearning-based data cleaning approach proceeds as follows. First, we train a baseline model on the raw dataset. Next, we unlearn a small set of trusted clean evaluation samples as if they were part of the training data, and compute the average attribution of each training sample with respect to these clean samples. Training samples with the lowest attribution scores are considered for removal, and a specified ratio of them is filtered out. Filtering can be done in either a unified or per-target manner. Finally, the model is retrained on the remaining samples.

However, this naive implementation is prone to catastrophic forgetting, where the model not only forgets the characteristics of a single data point, but also loses generalization ability on the rest of the dataset [14, 15]. To mitigate this, we adopt the regularization strategy of elastic weight consolidation (EWC), which preserves pretraining knowledge through the Fisher Information Matrix (FIM) [15, 10]. Similar to [10], the final update rule can be written as

where $\mathbf{F}$ corresponds to the pre-calculated FIM, and $N$ is the number of original training samples.

The FIM quantifies the importance of each model parameter to the inference task. Parameters with high $\mathbf{F}$ values are crucial, and modifying them would degrade model performance. Thus, EWC reduces changes to these crucial parameters during unlearning to avoid catastrophic forgetting [10, 15]. The FIM is computed with a diagonal approximation, which can be written as

where $\mathbf{F}_{ii}$ is the $i$-th element on the diagonal of the FIM, $D$ represents the training dataset, $N=|D|$ is its total number of samples, and $\frac{\partial\mathcal{L}}{\partial\theta_{i}}$ is the gradient of the loss with respect to a single model parameter $\theta_{i}$.

To compute data attribution using this formulation, we first compute the reference training loss for each training song $x_{i}$ ($i=1,\dots,N$) on the baseline model $\theta$, which produces $N$ baseline losses, each denoted as $\mathcal{L}_{i}$. Then, we unlearn a set of $M$ clean samples $y_{j}$ ($j=1,\dots,M$), forming $M$ unlearned models, each denoted as $\theta^{\prime}_{j}$. Finally, we compute the attribution by measuring the change in loss of each training sample $x_{i}$ after unlearning, by evaluating $x_{i}$ on $\theta^{\prime}_{j}$, which gives us a total of $N\times M$ losses $\mathcal{L}^{\prime}_{i,j}$. We subtract the final loss from the initial loss to measure the loss change $\Delta\mathcal{L}_{i,j}=\mathcal{L}^{\prime}_{i,j}-\mathcal{L}_{i}$. For architectures such as Open-Unmix [16], where each target is an independent model, the procedure above is repeated for each target $t\in\{\text{vocals},\text{bass},\text{drums},\text{other}\}$.

In the per-target approach, we rank training samples separately for each target $t$ based on their mean attribution for that target:

where $M_{t}$ is the number of clean tracks corresponding to target $t$. Then, we create an independent cleaned dataset for each target by removing the bottom $1-r$ proportion of the samples with the lowest attribution scores for that target. Each target model is then retrained on its respective cleaned dataset.

We also propose a data cleaning method based on distributional metrics. Unlike sample-level metrics such as mean squared error, distributional metrics compare across sample populations, allowing us to quantify the similarity more accurately between datasets of different sizes. In this work, we use the Fréchet Audio Distance (FAD) [11, 12], a widely used distributional metric for evaluating audio quality. FAD leverages a pretrained model to capture high-level audio features. Inspired by its effectiveness in assessing audio quality, we propose a straightforward filtering method using FAD to identify and remove noisy samples from a training dataset.

Our FAD-based data cleaning method begins by quantifying the perceptual dissimilarity of each training song to a small and clean reference set. Using the per-song FAD method detailed in [12], we compare the set of embedding time-frames from one song to all time-frames across all songs in the reference set. We use MERT [17] and CLAP [18] as the underlying embeddings for this calculation since they were found to be the most effective in capturing musical differences [12]. After computing a score for each song, we use the same approach to rank, filter, and retrain as detailed in the unified unlearning approach.

To evaluate our two data cleaning methods, we created a semi-synthetic dataset named “Mixed23” to simulate contamination with multiple types of errors found in real recordings. To ensure the performance gap was clearly observable, the prevalence and intensity of artifacts were made more pronounced than what is typical in studio recordings. The dataset comprises $N=200$ songs: 100 clean samples from the MUSDB18-Training set [19], 50 samples with label noise from SDXDB23_LabelNoise [2], and 50 samples with audio bleeding from SDXDB23_Bleeding [2]. The $M=50$ samples from MUSDB18-Test is used as the clean reference in unlearning, FAD, and MLP.

The Open-Unmix model architecture [16] was selected for experimentation due to its simplicity and training efficiency. To ensure a robust evaluation of our methods, we trained each model with three different seeds for 500 epochs without early stopping, and picked the epoch with the best validation loss on the MUSDB18-Test set. We evaluated them with the SDR metric [20] on the hidden evaluation dataset of the Music Demixing Challenge 2021, which consists of 27 out of 30 tracks in MDXDB21 [21]. We take the median SDR across test songs and report the average performance across three seeds.

We compare our proposed methods with several baselines, including Open-Unmix models trained on the contaminated Mixed23 dataset and the original MUSDB18 [19] training dataset containing 100 clean samples. In addition, inspired by Koo et al. [7], we developed a classifier-based baseline method leveraging prior knowledge about label noise and bleeding. Specifically, we use a 4-class multi-layer perceptron (MLP) instrument classification model trained on 50 clean songs from MUSDB18-Test. The MLP model is designed to classify individual 768-dimensional audio frame embeddings extracted using the MERT-v1-95M model [17]. The MLP has an input layer that projects these embeddings into a 256-dimensional hidden space, followed by a single hidden layer with a ReLU activation and a dropout rate of 0.5. A final linear layer maps the hidden representation to one of the four output classes: vocals, bass, drums, or other. The model then predicts and averages the class probabilities over all frames of a given song. Songs are then sorted based on the average predicted probability for their correct instrument class (i.e., for a vocal stem, the probability of the ‘vocals’ class). Similar to our unlearning filtering method, a percentage of the songs with the lowest probability is removed. We experimented with different ratios $r$ for this baseline MLP approach.

In our experiments, we discovered that the ratio $r$ for filtering out low-attribution samples is dependent on the specific characteristics of the dataset and the model architecture. Therefore, independently for each considered method, we performed the filtering process with multiple ratios, ranging from removing 5% to 50% of the data. The results are shown in Figure 2. The best $r$ for cleaning the Mixed23 dataset on the Open-Unmix model is determined to be $r=0.75$ (where 25% of data is removed), achieving a mean SDR of 4.91 dB. We applied the same method to find the optimal ratio for FAD and MLP as well, with $r=0.5$ found to be optimal for both approaches.