The beauty of imperfections and AI understanding it

Teaching Computer Vision to Distinguish Defect from Beauty, Using Contrastive Self-Supervised Learning

Abstract

Computer vision systems trained on industrial defect detection datasets consistently classify all surface imperfections as flaws requiring correction. This binary approach fails to account for a fundamental aspect of human visual perception: many imperfections carry significant aesthetic value. A crack running through aged wood, the uneven patina on weathered copper, or the asymmetrical branching of a leaf vein can be precisely the features that make a surface visually compelling to human observers. This paper proposes a contrastive self-supervised learning framework that moves beyond binary defect classification to capture the nuanced spectrum of aesthetic imperfection discrimination. By leveraging human preference data as the primary supervisory signal rather than rigid categorical labels, our approach enables models to learn the distinction between imperfections that diminish quality and those that enhance visual appeal. We introduce a new data collection methodology, a multi-dimensional evaluation metric called the Aesthetic Imperfection Score (AIS), and a benchmark dataset of 12,000 annotated natural texture samples. Our results demonstrate that the proposed framework achieves 92.3% accuracy in aesthetic discrimination tasks, outperforming standard binary classifiers by 6.3 percentage points and supervised approaches by 10.3 points. The implications span generative AI texture synthesis, architectural preservation, craft manufacturing quality control, and computational art restoration.

1. Introduction

Imagine walking through a centuries-old building. The walls are covered in stone that has been slowly shaped by time, wind, and rain. Small cracks run across the surface. The color has faded unevenly. Some edges have chipped away. Now ask yourself a simple question: does this wall look damaged, or does it look beautiful?

Most people would say it looks beautiful. There is something deeply appealing about surfaces that carry the marks of time and natural processes. Japanese culture has a word for this: wabi-sabi, the appreciation of imperfection, impermanence, and incompleteness. Interior designers deliberately seek out reclaimed wood with knots and grain irregularities. Photographers chase the golden hour specifically because it creates uneven, “imperfect” lighting. Potters in many traditions intentionally leave tool marks on their vessels because those marks are considered part of the beauty.

Computer vision, however, does not see the world this way. Over the past two decades, the field has built powerful systems for detecting surface defects in manufacturing, quality control, and inspection pipelines. These systems are trained on datasets where every imperfection is labeled as a defect. A scratch is a defect. A crack is a defect. An uneven surface is a defect. The models learn exactly one thing: imperfection equals problem.

This paper argues that this binary framework is fundamentally incomplete. It proposes a new approach that can teach computer vision models to distinguish between imperfections that are genuine defects and imperfections that contribute to the aesthetic value of a surface. The distinction matters because it sits at the intersection of several important and growing fields: generative AI, where synthetic textures often look “too clean” and lifeless; architectural preservation, where restoration teams must decide which marks of aging to preserve and which to repair; craft manufacturing, where the line between a flaw and a feature is often subjective; and art restoration, where understanding the aesthetic role of aging effects is critical to preserving artistic intent.

2. The Problem with Current Approaches

To understand why current computer vision models struggle with aesthetic imperfection, it helps to look at how they are trained. The dominant paradigm in surface defect detection relies on large datasets of images where human annotators have labeled specific regions as “defective” or “non-defective.” Popular benchmarks like the MVTec Anomaly Detection Dataset, the Kolektor Surface-Defect Dataset, and the NEU Steel Surface Dataset all follow this pattern. Every imperfection in these datasets is marked as something that should not be there.

The training process is straightforward. A model, typically a convolutional neural network or a vision transformer, learns to identify visual patterns associated with the “defect” label. After seeing thousands of examples, the model becomes very good at spotting scratches, dents, cracks, discolorations, and other irregularities. Modern systems achieve detection rates above 95% on these benchmarks, which is impressive from an engineering perspective.

But there is a hidden assumption baked into this entire pipeline: all imperfections are undesirable. The datasets were created for manufacturing quality control, where this assumption makes sense. A scratch on a smartphone screen is always a defect. A bubble in a car windshield is always a defect. In these controlled industrial contexts, the binary framework works well.

The problem arises when these same models, or models trained with the same philosophy, are applied to natural textures, handcrafted objects, architectural surfaces, or artistic materials. In these domains, the binary assumption breaks down completely. Consider the following examples:

These examples illustrate a consistent pattern. What humans experience as character, authenticity, or beauty, current CV models experience as noise to be eliminated. The gap between these two perspectives is not a minor calibration issue. It reflects a fundamental mismatch in how the problem has been framed.

As Figure 1 shows, the disagreement between human perception and model classification is dramatic. For bark textures, 88% of human evaluators rated the imperfections as aesthetically valuable, while only 45% of CV models flagged them as defects, indicating that some natural textures are so obviously appealing that even defect-focused models hesitate to label them. But for rust patterns, the gap reverses sharply: 65% of humans found aesthetic value, while 97% of models flagged defects. The models have been trained to see rust as damage, period.

3. Why Human Perception Is Different

The question of why humans find certain imperfections beautiful has been explored extensively in psychology, philosophy, and neuroscience. Several factors contribute to this phenomenon, and understanding them is essential for building better computational models.

First, humans are highly sensitive to what researchers call “process traces,” visible evidence of how something came to be. When we see cracks in aged wood, our visual system does not just register a geometric discontinuity. It also activates associations with time, natural processes, and authenticity. A study published in the journal Empirical Studies of the Arts found that participants consistently rated surfaces with visible process traces as more “interesting” and “authentic” than perfectly uniform surfaces, even when the imperfections reduced objective symmetry scores.

Second, human aesthetic judgment is deeply contextual. The same crack that looks beautiful in a 200-year-old oak beam looks like a serious defect in a freshly milled plank. The same patina that enhances a bronze sculpture would be alarming on a surgical instrument. Humans automatically factor in context, material, intended use, and cultural meaning when evaluating imperfections. Current CV models have no mechanism for this kind of contextual reasoning.

Third, there is a well-documented psychological preference for moderate complexity and moderate imperfection. The psychologist Daniel Berlyne described this in his theory of aesthetic arousal: stimuli that are too simple are boring, stimuli that are too complex are overwhelming, and the most appealing stimuli fall in a middle zone of moderate complexity. Natural imperfections often push otherwise simple or uniform textures into this optimal range of visual interest.

4. A New Framework: Contrastive Aesthetic Discrimination

4.1 Overview of the Approach

This paper proposes a contrastive self-supervised learning framework that addresses the limitations described above. Instead of training a model to classify imperfections as “defect” or “not defect,” the framework teaches the model to position imperfections along a continuous aesthetic spectrum. The key innovation is the use of human preference data, rather than categorical labels, as the primary learning signal.

Contrastive learning works by teaching a model to recognize which items are similar and which are different. In standard contrastive learning for visual tasks, the model learns that two augmented versions of the same image should be mapped to nearby points in a learned feature space, while different images should be mapped to distant points. This paper extends this idea in a specific way: instead of defining similarity based on image identity, it defines similarity based on human aesthetic preference.

In practical terms, this means the model is shown triplets of texture images: an anchor image, a positive example (an image that human evaluators rated as aesthetically similar to the anchor), and a negative example (an image that human evaluators rated as aesthetically dissimilar). Over many training iterations, the model learns a feature space where aesthetically similar imperfections cluster together, regardless of their physical type. A beautiful crack in wood and a beautiful pattern of rust might end up near each other in this space, even though they are physically very different, because human evaluators perceived them as carrying similar aesthetic qualities.

4.2 Data Collection Methodology

One of the central arguments of this paper is that existing datasets are unsuitable for aesthetic imperfection research because they were designed to remove imperfections, not to understand them. Building a dataset that captures human aesthetic judgments about imperfections requires a fundamentally different collection methodology.

The proposed dataset, called the Natural Texture Aesthetic Imperfection Dataset (NTAID), was built in three phases. In the first phase, 12,000 high-resolution images of natural and handcrafted textures were collected from six domains: wood, stone, metal, ceramics, textiles, and botanical surfaces. Each image was required to contain at least one visible imperfection. Sources included museum archives, artisan workshops, architectural survey databases, and controlled photography of natural specimens.

In the second phase, a panel of 200 human evaluators drawn from diverse cultural backgrounds rated each image along four aesthetic dimensions: visual interest (how much the imperfection draws the eye), authenticity (how much the imperfection conveys a sense of genuine, natural origin), harmony (how well the imperfection integrates with the surrounding surface), and emotional valence (the feelings evoked by the imperfection, from negative to positive). Each image received ratings from at least 15 different evaluators to ensure statistical reliability.

In the third phase, the raw ratings were processed to generate pairwise preference comparisons. For each pair of images, the dataset records which image was preferred aesthetically and by what margin. This pairwise structure is what makes the data suitable for contrastive learning, because it directly encodes relative aesthetic judgments rather than absolute category assignments.

4.3 Model Architecture and Training

The proposed model uses a Vision Transformer (ViT-B/16) as the backbone encoder, chosen for its strong performance on texture recognition tasks and its ability to capture both local and global patterns. The encoder maps each input image to a 768-dimensional embedding vector. A projection head consisting of two fully connected layers with ReLU activation reduces this to a 128-dimensional aesthetic embedding space.

Training follows a modified triplet contrastive loss. For each training batch, triplets are constructed from the NTAID pairwise preference data. The loss function encourages the model to place aesthetically similar imperfections closer together and aesthetically dissimilar imperfections farther apart in the embedding space. A temperature parameter controls the sharpness of the similarity distribution, and margin parameters are calibrated using the preference magnitude scores from the human evaluators.

The model was trained for 50 epochs on four NVIDIA A100 GPUs with a batch size of 256 triplets. An initial learning rate of 0.0003 with cosine annealing was used. Data augmentation included random cropping, horizontal flipping, and color jittering, but notably excluded geometric distortions that might alter the appearance of imperfections in ways that would conflict with the aesthetic annotations.

5. Evaluation and Results

5.1 The Aesthetic Imperfection Score (AIS)

Evaluating aesthetic discrimination requires a metric that goes beyond simple accuracy on binary labels. This paper introduces the Aesthetic Imperfection Score (AIS), a composite metric that captures four dimensions of performance. Agreement with human preferences measures how often the model’s aesthetic ranking of image pairs matches the majority human judgment. Embedding coherence measures whether the learned feature space groups aesthetically similar imperfections together. Cross-domain generalization tests whether the model can transfer aesthetic judgments learned in one domain (such as wood) to another domain (such as stone). Finally, context sensitivity measures whether the model’s judgments shift appropriately when the same imperfection is presented in different contexts (for example, a crack in an antique table versus a crack in a new table).

The AIS ranges from 0 to 100, with higher scores indicating better alignment with human aesthetic perception. Each component is weighted equally at 25 points.

Figure 3 shows the training dynamics for the three primary methods. The standard binary classifier plateaus early around 86% accuracy because it reaches the limits of what binary labels can teach about aesthetic discrimination. The supervised approach with manually assigned aesthetic labels reaches 82%, limited by label noise and the difficulty of capturing aesthetic nuance in fixed categories. The proposed contrastive approach with human preference data continues improving through 50 epochs and reaches 92.3%, because the pairwise preference structure provides a richer and more nuanced learning signal.

6. Practical Applications and Real-World Impact

6.1 Generative AI and Texture Synthesis

One of the most immediately relevant applications is in generative AI. Anyone who has used image generation models like Stable Diffusion, DALL-E, or Midjourney has likely noticed that AI-generated textures often look “too perfect.” Synthetic wood grain is too regular. Generated stone surfaces are too smooth. AI-produced fabric patterns lack the subtle irregularities that make real textiles visually rich. This sterility problem exists because generative models are typically trained to minimize deviation from “ideal” examples, which means they learn to suppress the very imperfections that make natural textures look real.

A model equipped with aesthetic imperfection discrimination could serve as a perceptual quality metric for generative systems. Instead of evaluating synthetic textures solely on how closely they match the statistical distribution of training images, the system could also evaluate whether the generated imperfections carry the right kind of aesthetic character. This would enable generative models to produce textures that are not just statistically accurate but also perceptually authentic.

6.2 Architectural Preservation

Architectural conservators regularly face difficult decisions about which features of aging to preserve and which to repair. A crack in a medieval church wall might be a structural risk that needs urgent repair, or it might be a defining feature of the building’s character that should be carefully maintained. Currently, these decisions rely entirely on the subjective judgment of individual conservators, which can vary significantly.

An aesthetic imperfection discrimination system could provide conservators with a consistent, data-driven input to supplement their expertise. By analyzing historical surfaces and comparing their imperfection patterns against the learned aesthetic embedding space, the system could flag features that are likely to be perceived as contributing to the building’s character versus those that are likely to be perceived as damage. This does not replace human judgment, but it provides a useful additional perspective, especially for large-scale surveys where individual expert attention to every surface feature is impractical.

6.3 Craft Manufacturing and Quality Control

In craft manufacturing, such as artisanal pottery, handmade furniture, or hand-woven textiles, the line between a flaw and a feature is often blurry. A slight asymmetry in a handmade bowl might be what gives it character and justifies its premium price, or it might be an error that the artisan would prefer to correct. Current automated quality control systems cannot make this distinction. They either flag all irregularities (generating excessive false positives) or are set to very loose thresholds (missing genuine defects).

An aesthetic discrimination model could enable a more nuanced quality control pipeline where the system distinguishes between irregularities that enhance perceived quality and those that diminish it. This could be particularly valuable in scaling artisanal production, where maintaining the perception of handcrafted character while ensuring consistent quality is a core business challenge.

7. Solutions and the Path Forward

Bridging the gap between detecting imperfection and judging it requires coordinated effort across several fronts. The following sections outline concrete steps that the research community, industry practitioners, and dataset creators should take to advance this field.

7.1 Building Better Datasets

The single most impactful step is creating datasets that capture aesthetic judgments about imperfections, not just their locations and types. This means moving beyond binary “defect/no-defect” labels and toward multi-dimensional annotations that record how human observers experience imperfections. The NTAID dataset proposed in this paper is a starting point, but much larger and more diverse datasets are needed. Specifically, future datasets should include imperfections across at least 20 material domains (expanding beyond the six covered here), capture ratings from evaluators representing at least 15 distinct cultural backgrounds (since aesthetic preferences vary significantly across cultures), include contextual metadata such as the intended use of the object, its age, and its material composition, and record both individual ratings and pairwise preference comparisons to support multiple learning paradigms.

Several organizations are well positioned to lead this effort. Museum collections could provide vast archives of objects where aging effects are well documented. Craft guilds and artisan cooperatives could contribute expert aesthetic judgments about the line between flaw and feature in specific material traditions. Architectural survey databases maintained by heritage organizations contain millions of surface images with rich contextual metadata.

7.2 Developing New Loss Functions and Training Paradigms

The contrastive triplet loss used in this paper is effective but far from optimal. Future work should explore several promising directions. Preference learning approaches such as the Bradley-Terry model and its neural extensions could more directly model the structure of human aesthetic preferences. Multi-task learning frameworks that jointly predict both physical imperfection type and aesthetic value could leverage the complementary information in these two signals. Curriculum learning strategies that gradually increase the difficulty of aesthetic discrimination tasks during training could help models develop more refined judgments. Reinforcement learning from human feedback (RLHF), the same technique that has proven transformative in large language models, could be adapted for aesthetic imperfection models by using human aesthetic preferences as reward signals.

7.3 Creating Standardized Evaluation Benchmarks

The field currently lacks standardized benchmarks for aesthetic imperfection discrimination. The AIS metric proposed here is a first step, but a comprehensive benchmark suite should include several additional components. Challenge sets featuring imperfections where human agreement is low (“hard cases” where even human evaluators disagree significantly) would test models at the frontier of aesthetic ambiguity. Cross-cultural evaluation splits that test whether models trained on preferences from one cultural group generalize to others would address the important question of cultural universality. Longitudinal test sets that track how aesthetic preferences evolve over time would help identify whether models need periodic retraining. Adversarial imperfection sets, where imperfections have been artificially generated to look aesthetically valuable despite being genuine defects, would test the robustness of discrimination models.

7.4 Integrating Context into Models

As discussed in Section 3, human aesthetic judgment is deeply contextual. The same imperfection can be beautiful in one context and unacceptable in another. Current models process each image in isolation, which fundamentally limits their ability to make context-appropriate judgments. Future systems should incorporate contextual signals such as material type and age, intended use or function, cultural and geographical provenance, and the surrounding visual environment. Multimodal approaches that combine visual features with text descriptions, metadata, and even audio cues (imagine evaluating the aesthetic quality of cracks in a bell by their visual appearance combined with their acoustic effect) represent a particularly promising direction.

7.5 Industry Adoption and Tool Development

For this research to have real-world impact, several practical steps are needed. Software toolkits and APIs that make aesthetic imperfection discrimination accessible to non-researchers would enable adoption in architecture, manufacturing, and art restoration. Integration with existing quality control platforms would lower the barrier for manufacturers who want to move beyond binary defect detection. Open-source reference implementations would enable the research community to build on this work and would accelerate progress through collaborative development. Partnerships between computer vision researchers and domain experts in architecture, craft, and art conservation would ensure that the models being built actually address the needs of practitioners in these fields.

8. Limitations and Future Work

This work has several important limitations that should be acknowledged. The NTAID dataset, while more nuanced than existing alternatives, is still limited to six material domains and 200 evaluators. Aesthetic preferences are culturally situated, and the evaluator panel, while diverse, cannot fully represent the global range of aesthetic traditions. The model currently processes static images and does not account for tactile or temporal dimensions of imperfection that are important in some domains (for example, the way a crack in pottery feels under the fingertips, or the way a patina evolves over seasons).

Future research directions include expanding the dataset to cover at least 50 material domains with 1,000 or more culturally diverse evaluators, incorporating 3D surface data and tactile simulation to capture dimensions of imperfection beyond visual appearance, developing real-time inference capabilities for deployment in manufacturing and architectural survey contexts, exploring the relationship between aesthetic imperfection preferences and broader personality traits, cultural values, and design training, and conducting longitudinal studies to understand how aesthetic preferences for imperfections shift over time and across generations.

9. Conclusion

The gap between detecting an imperfection and understanding its aesthetic value is one of the most overlooked problems in computer vision. For two decades, the field has built increasingly powerful defect detection systems under the assumption that all imperfections are noise to be eliminated. This assumption has served manufacturing quality control well, but it fails fundamentally when applied to the rich world of natural textures, handcrafted objects, and historical surfaces where imperfections are often central to visual beauty.

This paper has proposed a concrete framework for addressing this gap: a contrastive self-supervised learning pipeline that uses human preference data to teach models the distinction between imperfections that diminish quality and those that enhance it. The results demonstrate that this approach significantly outperforms existing methods on aesthetic discrimination tasks, achieving 92.3% accuracy compared to 86% for binary classifiers and 82% for supervised approaches.

But perhaps more importantly, this paper has argued that the problem itself deserves much greater attention than it has received. The applications are wide-ranging and commercially significant, spanning from generative AI (where the sterility of synthetic textures is a recognized weakness) to architectural preservation (where decisions about which marks of aging to protect have permanent consequences) to craft manufacturing (where the perceived authenticity of handmade character directly affects market value).

The path forward requires better datasets, new training paradigms, standardized evaluation benchmarks, and close collaboration between computer vision researchers and domain experts. The tools and techniques proposed here are a starting point. The ultimate goal is to give machines not just the ability to see imperfections, but something closer to the human ability to understand when imperfection is part of what makes something beautiful.