Code-Level Cost Function Generation for Spatial Image Steganography Using RAG-Enhanced Large Language Models

To advance automated cost function design within the minimum-distortion framework, we establish a knowledge-driven evolutionary paradigm. While recent automated frameworks rely on unguided two-stage generation [5], our architecture fundamentally transforms this process by establishing the SE-RAG module as the core engine for explicit domain guidance. The framework evolves over multiple generations, each comprising cost function generation and evaluation stages. As illustrated in Fig. 1, the process is initialized with a foundational handcrafted steganographic algorithm $p_{\text{init}}$, which acts as the initial program.

Stage 1 focuses on algorithm generation and preliminary evaluation. In each iteration, a reference program $p_{\text{ref}}$ is selected from the program database and fed into the forward process of SE-RAG. Crucially, the CSS of $p_{\text{ref}}$ is extracted to perform separate dual-path retrievals from a static literature knowledge base $\mathcal{K}_{\text{sta}}$ and a dynamic experience knowledge base $\mathcal{K}_{\text{dyn}}^{t}$, supplying the LLM with classic academic concepts and historically successful optimization paths. Guided by this retrieved context, the LLM optimizes $p_{\text{ref}}$ to generate new candidate algorithms ($p_{1},p_{2},\dots,p_{n}$). These variants undergo rapid preliminary evaluation via pretrained steganalyzers to filter out non-executable code. We uniformly utilize the minimum average decision error rate $P_{E}$ as the security metric:

$$P_{E}=\min_{P_{\text{FA}}}\frac{1}{2}(P_{\text{FA}}+P_{\text{MD}}(P_{\text{FA}})),$$

(1)

where $P_{\text{FA}}$ and $P_{\text{MD}}$ represent the false alarm and missed detection rates, respectively. Valid programs and their preliminary $P_{E}$ scores are saved back into the program database.

Stage 2 conducts precise evaluation for candidates passing the preliminary filter to provide feedback. A dedicated steganalyzer is trained for each program to provide a rigorous assessment via the $P_{E}$ metric defined in Eq. (1). The results of this accurate evaluation update the pretrained steganalyzer pool in Stage 1, and the optimized programs are added to the seed program pool for subsequent generations. Simultaneously, the evaluation results leverage the backward process of SE-RAG to update the dynamic experience knowledge base $\mathcal{K}_{\text{dyn}}^{t}$ based on the evaluation results, accumulating effective empirical paths for continuous improvement.

The overall workflow is illustrated in Fig. 1, and the internal pipeline of the SE-RAG module is detailed in Fig. 2. Section II-B describes the forward generation process, while Section II-C outlines the backward feedback loop.

The forward generation process transitions the LLM from a blind code generator into a knowledge-informed algorithm optimizer via domain knowledge injection. To achieve effective retrieval, relying directly on the raw reference program $p_{\text{ref}}$ as a query is suboptimal due to the conceptual gap between procedural implementations and steganographic semantics. To bridge this gap, as illustrated in Fig. 2, we introduce the CSS mechanism, where $p_{\text{ref}}$ is parsed by an AST parser. This parser maps specific Python operations to high-level steganographic semantic concepts based on the mapping rules standardized in Table I, sequentially connecting them into an ordered semantic sequence $S(p)$.

Prior to retrieval, literature documents and historically successful rules are partitioned into semantic text chunks $k$. These chunks populate two repositories, namely, the Static Knowledge Base $\mathcal{K}_{\text{sta}}$ indexing fundamental reference papers, and the Dynamic Knowledge Base $\mathcal{K}_{\text{dyn}}^{t}$, which continuously expands at each evolutionary iteration $t$ to store concise optimization summaries. Formally, each repository $\mathcal{K}_{i}\in\{\mathcal{K}_{\text{sta}},\mathcal{K}_{\text{dyn}}^{t}\}$ is defined as a set of constituent chunks: $\mathcal{K}_{i}=\{k_{1},k_{2},\dots,k_{|\mathcal{K}_{i}|}\}$. All chunks are processed by a Bi-Encoder embedding model $M_{\text{BE}}$ and stored alongside their continuous vector representations.

During the retrieval phase, the obtained semantic signature formulates the query $q=S(p_{\text{ref}})$. A parallel two-stage retrieve-and-rerank pipeline is independently applied to each repository. First, $M_{\text{BE}}$ projects $q$ into the latent space for a rapid vector screening, where $q$ is compared against every chunk $k\in\mathcal{K}_{i}$ to calculate the preliminary similarity score $s_{1}(q,k)$ via cosine similarity:

$$s_{1}(q,k)=\frac{M_{\text{BE}}(q)\cdot M_{\text{BE}}(k)}{\|M_{\text{BE}}(q)\|\|M_{\text{BE}}(k)\|}.$$

(2)

This screens the search space to retain a highly relevant matching pool $\mathcal{M}_{i,q}$. Second, the matching pools are forwarded to a Cross-Encoder model $M_{\text{CE}}$ for precise reranking to capture fine-grained cross-attention. The final ranking score is computed as:

$$s_{2}(q,k)=M_{\text{CE}}(q,k),\quad\forall k\in\mathcal{M}_{i,q}.$$

(3)

Finally, a prompt compiler systematically aggregates the acquired information into a highly structured prompt to govern the LLM code synthesis. Instead of a simple concatenation, the compilation process utilizes a sequential integration function $\Phi$ to combine four essential components: First, the expert task guidelines $\mathcal{T}$, which assign a domain expert persona to the language model and explicitly mandate structural algorithmic innovations, such as novel residual sources or nonadditive fusion rules, while strictly prohibiting superficial parameter tuning; Second, the retrieved context $\mathcal{C}_{q}$, which supplies explicit domain guidance and historical optimization strategies; Third, the reference program $p_{\text{ref}}$ undergoing evolution; and Fourth, the rigorous safety constraints $\mathcal{R}$, which enforce deterministic execution, restrict external dependencies to standard numerical packages, and define rigorous mathematical boundaries to satisfy necessary steganographic security constraints. This compilation operation is formally defined as:

$$\mathcal{P}=\Phi(\mathcal{T},\mathcal{C}_{q},p_{\text{ref}},\mathcal{R}).$$

(4)

The resulting knowledge-injected prompt is then delivered to the LLM to generate the new batch of program candidates.

To prevent the evolutionary process from stalling or repeating past optimization errors, the proposed framework incorporates a backward semantic knowledge extraction mechanism driven by a semantic auditor. The core purpose of this component is to transform fortuitous performance improvements into interpretable steganographic rules, which are subsequently leveraged to guide future mutation directions.

As illustrated in the backward process of Fig. 2, during the accurate evaluation phase, a dedicated steganalysis tool evaluates the optimized program $p_{c}$ to obtain its security score $P_{E}(p_{c})$, as defined in Eq. (1). If this score exceeds that of the initial global baseline, i.e., $P_{E}(p_{c})>P_{E}(p_{\text{init}})$, the knowledge extraction mechanism inside the Semantic Auditor is triggered. The framework employs an independent LLM summarizer within the auditor to perform a detailed comparative analysis between $p_{c}$ and $p_{\text{init}}$. By examining the algorithmic structures and semantic variations, the summarizer extracts and summarizes the specific methodological modifications that lead to the enhanced security. The empirical insights generated during the current iteration $t$ form a set:

$$E_{\text{new}}^{t}=\{e^{(1)},e^{(2)},\dots\}.$$

(5)

Consistent with the knowledge representation established in Section II-B, each empirical experience $e\in E_{\text{new}}^{t}$ is partitioned into semantic text chunks, embedded into vector representations, and structurally archived. Let $\mathcal{K}_{\text{new}}^{t}=\{k_{1},k_{2},\dots\}$ denote the set of all semantic chunks derived from $E_{\text{new}}^{t}$. The dynamic knowledge base is updated via a direct set union operation:

$$\mathcal{K}_{\text{dyn}}^{t}=\mathcal{K}_{\text{dyn}}^{t-1}\cup\mathcal{K}_{\text{new}}^{t},$$

(6)

where $\mathcal{K}_{\text{dyn}}^{0}=\emptyset$.

Integrating this structured knowledge back into the loop establishes a sustainable self-evolving mechanism. As evolution progresses, the dynamic knowledge base $\mathcal{K}_{\text{dyn}}^{t}$ continuously accumulates historically successful design rules. Consequently, when the next iteration performs forward retrieval, the retrieval engine draws directional guidance from both the foundational static knowledge base $\mathcal{K}_{\text{sta}}$ and the updated dynamic knowledge base $\mathcal{K}_{\text{dyn}}^{t}$, drastically accelerating convergence toward highly secure steganographic cost functions.

To evaluate the effectiveness of the proposed framework, we compare the steganographic security of the original handcrafted algorithms and the best algorithms generated by our framework with those produced by the existing LLM-based approach [5]. HILL [2], WOW [3], and S-UNIWARD [4] are adopted as the starting algorithms for the evolutionary search process in both frameworks. Security is quantified by the detection error rate $P_{E}$ defined in Eq. (1), which is evaluated using dedicated SRNet [17] steganalyzers trained specifically for each evaluated algorithm at each embedding payload.

As reported in Table II and Table III, while the algorithms produced by both automated paradigms successfully enhance the security of their handcrafted counterparts, those generated by our framework consistently achieve the highest $P_{E}$ across varying payloads on both BOSSBase and BOWS2. By continuously outperforming the baseline across divergent steganographic paradigms, this framework-level comparison systematically demonstrates the superior optimization capacity of our evolutionary framework incorporating the SE-RAG engine, proving that the security gains stem from methodological advancements rather than stochastic fluctuations during the evolutionary search. Consequently, rather than relying solely on the implicit knowledge of the language model, our framework explicitly incorporates domain-specific knowledge into the evolutionary process via the SE-RAG module, enabling a more meaningful exploration of the program space and consistently producing higher-quality steganographic algorithms.

To further evaluate the proposed framework, we analyze its evolutionary efficiency in terms of code executability, defined as the percentage of successfully executed candidate algorithms, and search efficiency, representing the number of generated algorithms required to discover the best-performing solution.

As shown in Table IV, the proposed framework improves the overall code execution rate across all three initial algorithms, increasing the average execution rate from 30.4% to 44.5%, which corresponds to a relative improvement of 46.3%. Although the execution rate enhancement for the highly constrained S-UNIWARD is marginal, the overall search efficiency is consistently optimized. Specifically, our framework requires substantially fewer generated algorithms to discover the best-performing solution across all baselines, reducing the overall search cost by an average of 26.1%. Collectively, despite the inherent challenges in optimizing complex spatial steganographic models like S-UNIWARD, these results demonstrate that our framework enables more reliable candidate generation and highly efficient optimization. This substantially reduces the computational cost of automatic steganographic algorithm design, highlighting its practical applicability.

To explore the effectiveness of our evolutionary system, we present the evolved cost functions derived from HILL, WOW, and S-UNIWARD, detailing the specific modifications made to each algorithm.