Position: Comprehensive Vulnerability Analysis is Necessary for Trustworthy LLM-MAS

While there are various malicious goals to attack LLM-MAS, mathematically, they can be summarized to the following general formula: Denote $G$ as the malicious goal and the attacked LLM-MAS component as $S\in(f_{i},P_{i},T_{i},\mathcal{C}(\mathcal{S},\mathcal{M},\mathcal{T}),\mathcal{E},M,Q)$ (i.e., any possible component in $S_{MA}$), then the attacker aims to solve the following:

where $Evaluator$ is the evaluation function determining whether the attack achieves the specific goal given the LLM-MAS $S_{MA}$, initial query $Q$, and the final target $G$. The notation $\Theta_{S}$ denotes the malicious space.

While Eq. (1) presents a general formulation, through configuring its (1) objective function, (2) optimization variables, and (3) the optimization algorithm, it can be transformed into specific forms given the detailed threat model and malicious goals. For (1) objective, the exact metric $Evaluator(\cdot)$ is determined by the specific malicious goal. For (2) variable, the malicious space $\Theta_{S}$ is determined by the exact component $S$ and the attacker’s capability.

In terms of (3) optimization algorithm, to optimize the above formula, depending on the level of attacker’s access to the LLM-MAS, several scenarios can be considered. Black-box: The attacker acts like a regular user and have no knowledge of the system, including the system configurations, LLM cores, etc. White-box: The attack is assumed to have access to everything of the LLM-MAS. Gray-box: The attack can infer partial knowledge of LLM-MAS. We list two representative cases. (1) The attacker can infer the communication structure of the LLM-MAS based on its functionality, e.g., in a software company LLM-MAS (31), there are certain roles in the system and the workflow is clear. (2) The attacker has the knowledge of some specific agents such as the architecture of LLMs utilized in the agents and access to their inputs, but no knowledge of the rest of the system.

In the following, we categorize common malicious goals that attackers may pursue in LLM-MAS.

Harmful behavior. Since the pre-trained LLMs utilize broad internet data, they may generate malicious outputs such as dangerous answers or insecure codes (12). Consequently, alignment methods have been developed to make LLMs refuse harmful queries (110). In parallel, jailbreak attack research focuses on bypassing these alignment safeguards (12; 111; 52; 7; 38), and corresponding adversarial training methods have been developed (74). In the context of LLM-MAS, harmful outputs can escalate into harmful behaviors. Unlike standalone LLMs, agents in LLM-MAS are equipped with tool-calling capabilities and elevated permissions, which significantly amplify the risks—enabling actions such as executing destructive programs (23), performing unauthorized transactions (19), or carrying out social engineering attacks (70). Given the above, the definition of the evaluation metric in Eq.1 is closely tied to the intended malicious consequence. For example, if the goal $G$ is to generate harmful texts, the $Evaluator$ can be implemented using an LLM-based judge. If the goal is to produce harmful code, evaluation can be conducted by running test cases through an external executor.

Resource exhausting. In traditional system security, resource exhaustion attacks aim to consume excessive resources (e.g., CPU, memory, disk, bandwidth) to impair functionality for legitimate users. Classic examples include Denial of Service (DoS)(88), memory exhaustion(80), and algorithmic complexity attacks (9). In the context of LLM-MAS, attackers can similarly overload computational resources to inflate costs or disrupt availability. For example, attackers may generate progressively longer messages between agents (108), overloading message-processing components. They may also induce tools to generate large data volumes from minimal input, sending them to external servers—causing tool abuse, quota exhaustion, billing spikes, or even service bans (23). Beyond direct resource strain, such attacks can disrupt coordination: overloaded planners may time out, executors may stall, and a single failed agent can compromise overall system functionality (5). To evaluate such attacks under our formulation in Eq. 1, $Evaluator$ metrics can include output token length, memory usage, computation time, and tool-calling frequency, while the goal $G$ can be defined as inducing excessive operational cost.

Performance degradation. In poisoning/evasion attack literature for deep neural networks, a general goal is to craft adversarial samples to worsen prediction performance (e.g., classification accuracy or regression error) (57). Similar performance degradation concepts appear in LLM research. For instance, (30) shows that poisoned demonstration examples in in-context learning can degrade the prediction performance. In Eq. 1, we directly measure the performance specified by the particular task, such as prediction accuracy as $Evaluator$ and a wrong answer as $G$ (either targeted or untargeted).

Privacy leakage. Privacy concerns span systems from operating systems and web applications to deep learning models. LLMs and their applications face similar risks. For example, attacks have been developed to extract sensitive data from retrieval-augmented generation systems (98), recover prompts (64; 73; 37; 94), or leak memory contents in single-agent settings (82; 87). In LLM-MAS, privacy risks are further amplified by inter-agent communication. A compromised agent may extract private information from others or induce them to leak confidential data, even without direct access to sensitive tools or databases (45). To define the malicious goal and $Evaluator$ in Eq. 1, various evaluation metrics can be applied, e.g., ROUGE-L or cosine similarity to compare with $G$.

Guided by the overall structure of LLM-MAS and formal formulation in Eq 1, we analyze vulnerabilities in each system component, especially their feasibility and potential severe consequences. While setting $n=1$ reduces the system to a single-agent system, revealing some shared vulnerabilities, we unveil distinct vulnerabilities for LLM-MAS, particularly for the unique components–communication $\mathcal{C}$ and agents $\{A_{i}\}$. Note that we exclude memory $Mem$ in the discussion because its design is flexible and highly agent-specific, e.g., (47).

Malicious inputs ($Q$). Malicious users can manipulate LLM-MAS through carefully crafted queries designed to induce malicious behaviors. This vulnerability has been extensively studied in single-agent literature (53; 75; 41; 14) and represents one of the most common attack approaches used by individual attackers in real-world scenarios. Besides the documented incidents involving ChatGPT and OpenAI systems mentioned in Section 1, compromises have also occurred with other AI assistants, resulting in unauthorized disclosure of personal data (18) and organizational information (16). The relative simplicity of this attack approach makes it particularly concerning. In our formulation in Eq. 1, various factors can be considered. For instance, one can directly use searching algorithms such as GCG (specific to a white-box scenario) or LLM-based optimization (e.g., TextGrad (97) under a black-box scenario) to search for the best $Q$. Other static designs like direct injection (35), adding escape characters (89), or mislead the agent to a different context (89) can also be applicable.

Individual agent ($A_{i}$). Individual agents are also exposed to significant threats (104; 101; 54; 86; 66). Compared to LLMs, agents contain more functionality, thus expose more potential vulnerabilities. Existing studies point out that the vulnerabilities emerge when an agent’s learned or programmed objectives diverge from intended goals, resulting in undesirable behaviors (36; 62; 48; 47). In the following, we provide vulnerability analysis associated with each sub-component within individual agents: the LLM core ($f_{i}$), agent profile ($P_{i}$), and tools ($T_{i}$).

Vulnerability in LLM core ($f_{i}$). The LLM core can impact the LLM-MAS vulnerability from two perspectives. First, if a compromised or unverified model is deployed, vulnerabilities are directly introduced into the system. For example, a backdoored LLM may execute malicious reasoning or actions when triggered. As agents interact with diverse inputs, such as user queries, retrieved knowledge, and tool feedback, a compromised model can propagate risk throughout the entire system. Second, unlike single-agent systems, each agent in LLM-MAS may use a different model. With varying levels of tuning and safety alignment, the overall system behavior becomes highly dependent on which specific agent’s model is compromised, leading to heterogeneous and potentially unpredictable security failures.

Hijack agent profile ($P_{i}$). Agent profiles significantly guide behaviors, thus compromising them severely impacts the overall system performance. A distinct characteristic of LLM-MAS is that collective profile configuration defines inter-agent collaboration. In systems such as MetaGPT and ChatDev, different agents fulfill specific roles (manager, designer, engineer) to collaboratively develop software requested in the initial query. Therefore, different agent roles can have distinct effects on the system performance, and a comprehensive evaluation on the threats introduced by these roles is necessary. Furthermore, with the rise of Agent-to-Agent (A2A, (22)) protocols and the support for external agent integration, profile-based attacks have become increasingly feasible. This highlights the growing need to identify vulnerabilities in these standard protocols—such as weak authentication of agent profiles (8) and profile poisoning attacks, where fake agent credentials are injected.

Tools ($T_{i}$). Existing benchmarks evaluate single-agent system vulnerabilities when tools return compromised values (68; 99; 109). As documented in Table 1, agent systems demonstrate significant vulnerability to malicious tools, with Attack Success Rates (ASR) ranging from 20% to 87%. In LLM-MAS, with more than one agents in the system, malicious tools can also indirectly impact other agents. For example, in a planner-executor system (43), malicious tools can directly change the output of the executor, while indirectly impacting the behavior of the planner.

Besides directly injecting attacks into local tools, the growing adoption of Model Context Protocol (MCP) introduces more intense threats through multiple perspectives. First, poisoned MCP, such as embedding malicious instructions in the description of tools (42) can induce the agent to do malicious actions. Second, MCP’s ability to dynamically request additional information from client agents—such as through content sampling mechanisms—opens up further attack surfaces, including data leakage or manipulation (69). While some threats are identifies (79), investigations are still required to secure MCP.

Agent communication ($\mathcal{C}$). Communication-based attacks can result in various malicious consequences in LLM-MAS. This component represents a unique vulnerability surface which is not applicable in single-agent architectures.

Hijack communicating messages ($\mathcal{M}$). Similar to traditional distributed systems, Agent-in-the-middle attacks can target LLM-MAS when agents are deployed across different servers (28). Message interception poses severe risks, enabling attackers to steal internal messages and inject malicious instructions or misinformation. Besides, researches demonstrate that different communication structures $\mathcal{S}$ significantly impact the system’s resilience against communication attacks. For example, (28) compares complete, tree, random, chain structures, and observe that tree and random structures are more robust compared to the other two structures. Similar analyses appear in (34), which shows how decentralized communication patterns provide inherent resistance to single-point compromise, and (95), which quantifies security improvements from redundant communication paths.

Trust management ($\mathcal{T}$). As demonstrated by (46; 92), a fundamental vulnerability in LLM-MAS stems from LLMs’ lack of skepticism toward received messages. Unlike human collaborators who evaluate information credibility, LLMs treat all inputs as part of their context window and attempt to continue coherently, regardless of content trustworthiness. Based on (46; 92), this blind trust emerges because agents typically act upon or chain their reasoning from received messages without embedded mechanisms for verifying factuality, consistency, or other trustworthiness aspects. With the rise of A2A and MCP, establishing robust trust management systems becomes increasingly essential. The absence of proper trust verification mechanisms significantly amplifies attack vectors (65; 8).

Environment ($\mathcal{E}$). Agent systems operate in various environments depending on their specific use cases, generally categorized into two types. The first is physical environments, such as those navigated by autonomous vehicles (21) or robot teamwork scenarios (20). These situations necessitate consideration of diverse security factors including safety issues and engineering challenges. Various studies have also studied the impact of the environment on the agents, e.g., (91; 51; 90; 49). Regarding attack feasibility, while many researchers focus on internet environments, physical attacks have been studied extensively in conventional deep learning models. In computer vision and related fields, defending against potential physical attacks—such as snow obscuring stop signs or blurred camera inputs—remains a significant concern (83). The implementation of the attack in $\mathcal{E}$ needs to be tailored specifically for each scenario.

We evaluate LLM-MAS vulnerability using a planner–executor architecture, a basic and common design pattern in agentic systems across multiple domains (17; 71; 85). In this paradigm, the planner receives a user task and generates a high-level plan specifying the required actions, which is then passed to the executor. The executor invokes available tools to carry out the specified actions. This separation reflects a realistic deployment setting where reasoning and execution are handled by distinct agents.

In our experiments, we instantiate the planner using GPT-5-mini, and the executor using either GPT-5-mini or GPT-5-nano, allowing us to examine whether vulnerabilities persist across models of different capacities. We conduct experiments on the banking domain from AgentDojo (11), which includes representative user tasks such as performing transactions, checking account balances, and managing basic financial operations. This domain is particularly suitable for security analysis due to its sensitivity to both privacy leakage and harmful actions.

In terms of the attack, we follow (28) to perform a prompt injection attack in the agent communication, and leverage the injection prompts in (15).

To study adversarial robustness, we perform prompt injection attacks targeting agent communication, following the threat model in (28) and using injection templates adapted from (15). Concretely, we evaluate attacks at three distinct stages of the planner–executor pipeline:
Planner start: Injection occurs when the planner receives the user task, allowing the adversary to influence plan generation.
Executor start: Injection is applied when the executor receives the planner’s output, targeting the interpretation of the plan.
Executor end: Injection is introduced after execution, aiming to manipulate the final response or action outcomes.

Malicious goals: We test with (1) privacy leakage and (2) harmful behavior in the experiment.

Evaluation metrics: We test the utility, defined as the proportion of completed original tasks, and the attack success rate (ASR), defined as the proportion of malicious goals achieved.

The evaluation results are summarized in Table 2. We observe that LLM-MAS exhibit substantial vulnerability to prompt injection attacks across different attack stages, malicious goals, and model variants. In particular, injections at the planner entry point and at the executor output consistently achieve high attack success rates (around 90%), indicating that LLM-MAS vulnerabilities arise from system-level interactions rather than from individual agents or models alone. Moreover, the persistence of these patterns across privacy and harmful objectives, as well as across models of different capacities, suggests that such vulnerabilities are structural and largely goal-agnostic.

Call to Action. Taken together, these findings reinforce our central position that comprehensive vulnerability analysis is essential for building trustworthy LLM-based multi-agent systems. We urge the research community to move beyond isolated, agent-centric evaluations and adopt system-level security analyses that explicitly model malicious goals, attacker capabilities, and the interactions among agents, tools, and execution stages. Benchmark and framework builders should develop evaluation suites specifically tailored to LLM-MAS, enabling structured comparison across architectures, communication patterns, and attack surfaces. In parallel, we encourage exploration of attack vectors unique to multi-agent settings and the integration of trust management mechanisms that can regulate inter-agent interactions under adversarial conditions. Advancing LLM-MAS security requires principled formulations and reproducible analyses that reflect the structural complexity of real-world agentic systems, rather than ad-hoc testing of individual components. We will detail the open challenges and future directions in the following sections.

To systematically understand the vulnerabilities of LLM-MAS, a comprehensive analysis is essential. Although currently there is no benchmark study specifically focused on the security issues in LLM-MAS, some researches work on benchmarking the security in single-agent systems. In Table 1, we summarize existing benchmarks in single-agent systems and a few benchmarks in MAS categorized by vulnerability types. While LLM-MAS shares similar malicious goals with those found in the existing literature, its unique components introduce different levels of vulnerability and distinct attack surfaces compared to single-agent systems. We list more details as follows:

Communication structure ($\mathcal{S}$). While existing literature such as (28) analyzes the influence of $\mathcal{S}$ on LLM-MAS, current analyses lack depth in applying established graph metrics. With fruitful studies in graph-related researches, many metrics can be borrowed and worth investigation in the context of LLM-MAS, such as degree centrality, betweenness centrality, and eigenvector centrality (61). These metrics, commonly employed in social network analysis, offer valuable insights for social simulation studies and facilitate evaluation of distributed systems with agents operating across heterogeneous platforms (10).

Granularity of $Evaluator$. Compared to single evaluation metrics used in LLM attack literature (e.g., ASR for jailbreak attacks), since there are several components in single-agent systems, existing benchmarks in single-agent systems have already considered different granularity of the same evaluation metric. For example, (99) utilizes two versions of ASR considering both (1) whether the malicious program is executed or not, and (2) whether the agent output is valid or not. Similarly, in LLM-MAS, it is also necessary to consider different granularity of the evaluation metrics. Specifically, in addition to the aforementioned ones considered in single-agent systems, it is also possible to refine the evaluation metrics to focus on either individual agents or the overall system. An example is that (4) creates snapshots for each step of the system to comprehensively analyze the failure of the LLM.

Benchmarking protocol performance ($\mathcal{M}$). Evaluating different communication protocols is essential for both practical deployment and vulnerability quantification. Following (93), besides MCP and A2A, researchers have developed other protocols such as the inter-agent protocol (ANP, (1)) and language to protocol generation (Agora, (59)). Protocol benchmarking presents greater challenges than single-agent system evaluation, as tasks become more complex and implementation hurdles increase significantly. Standardized evaluation frameworks that measure protocol resilience against attacks would significantly advance LLM-MAS security research.

In the following, we list some potential attacks inspired from Eq. (1):

Structure inference attack. Developing attacks tailored to infer the structure of LLM-MAS represents a critical research direction, which helps developers better understand the potential risks and protect their intellectual properties. Structure inference attacks may operate through systematic probing of the system, where an attacker sends carefully crafted messages to work on different agents and analyzes response patterns, timing differences, and content variations to infer the underlying structure of the system. To formalize such attacks within Eq. (1), we define ${Evaluator}$ as the similarity between the inferred structure derived from $S_{MA}$ and query $Q$, compared with the actual structure $G$.

System stability attack. Based on (26), agents in LLM-MAS often possess varying levels of computational power and data access, leading to various system instability: (1) Coordination failure: dominant agents may prioritize their objectives, leading to misalignment with the goals of other agents. (2) Resource monopolization: stronger agents might monopolize shared resources. (3) System fragility: the system may fail if the dominant agents fail. An attacker can exploit such a property to design different attack surfaces to impact the system stability. To formalize ”stability” within the Eq. 1 framework, corresponding to the above instability factors, we can define ${Evaluator}$ as (1) the correlation between the final output and the target attacked agent, (2) the resource allocation (measured by proper divergence metrics), and (3) the source of system failure (measured by the distance between the failure summary and the attack).

Practicality of attacks. While Section 3.3 outlines the feasibility of vulnerabilities, developing practical attacks remains challenging (29), particularly regarding the effect of optimization methods.

There are two potential challenges when performing optimization. First, while precise gradient computation enables GCG-based jailbreak attacks against individual LLMs, calculating the actual gradient of ${Evaluator}$ for complex LLM-MAS systems remains computationally infeasible. An existing work (60) attempts to compromise a system with input and output filters via specifically attacking the input filter via jailbreak prefix and attacking the output filter via jailbreak suffix. However, it is still an open question on how to efficiently develop an attack. Second, implementation differences across various multi-agent systems introduce additional complexity, resulting in diverse vulnerability and robustness profiles, making it hard for both the attacker and the defender to implement algorithms with good generalization.

Building on the comprehensive vulnerability analysis, we propose potential defense strategies to systematically enhance the robustness and trustworthiness of LLM-MAS.

Monitor agents for real-time oversights. To enhance the safety and reliability of LLM-MAS, integrating dedicated monitor agents is a promising approach. Similar to human oversight in complex systems, these agents supervise inter-agent communication, detect anomalies, and intervene when necessary, e.g., (6; 81; 56). However, the LLM-powered monitor agents heavily depends on the underlying model’s robustness, reliability, and generalization ability, and may fail given the variety of types of inter-agent communications. Additionally, real-time, per-message monitoring may also introduce latency in the system, and attackers may attempt to evade the monitoring system if they are aware of its mechanism. Thus, although a monitor agent is a feasible solution to enhance the safety of LLM-MAS, developing reliable, low-latency, and resilient monitor agents remains an open challenge.