AI agents: limits and opportunities for your digital marketing

Research overview & empirical testing methodology

Evaluating the performance of AI agents in a business context is a major challenge. While their potential to transform organizations is widely recognized, effectively benchmarking their capabilities has been hindered by the lack of realistic, publicly available enterprise data.

Current evaluation methods often lack fidelity in simulating real-world environments, underlying data, and user-agent interactions, while offering limited coverage of diverse business scenarios.

To bridge these gaps, a new benchmark has been introduced: CRMArena-Pro. Developed by researchers at Salesforce AI Research, this benchmark was specifically designed to enable a holistic and realistic assessment of LLM agents across a variety of professional settings. It provides a comprehensive analytical framework that includes:

• Nineteen expert-validated business tasks.
• Coverage across multiple domains: sales, customer service, and Configure, Price, Quote (CPQ) processes.
• Application to both business-to-business (B2B) and business-to-consumer (B2C) scenarios.

What exactly is an AI agent (or LLM agent)?

It is the operational system that uses one or more LLMs to accomplish a complex task through a sequence of actions within a specific environment. The agent is an architecture that encompasses the LLM and gives it the capacity to act.

In the context of this study, the agent processes the user's request, uses the LLM to "reason," and then decides on an action. That action may consist of:

• Executing a query on a database (via SOQL or SOSL) to retrieve an observation.
• Responding to the user, either to ask clarifying questions or to deliver the final answer and complete the task.

An LLM (Large Language Model) is the foundational model—the "brain." Examples include GPT-4o, Gemini 2.5 Pro, or Llama 3.1 405B. In the study's experiments, each agent is powered by a single model at a time.

The distinction is therefore clear: the LLM is the reasoning engine, whereas the AI agent is the complete entity that autonomously executes a logical sequence of actions to achieve an objective.

What is the most effective operational mode for an AI agent?

Top-performing AI agents succeed in approximately 58% of tasks when executed in a single step (single-turn). However, when they must engage in back-and-forth dialogue to gather information and complete the task (multi-turn), their performance drops to around 35%. This underscores the major challenges associated with multi-turn reasoning when agents are expected to execute diverse tasks autonomously.

Which marketing competencies do AI agents master best?

Among the various competencies tested, AI agents performed remarkably well when following predefined step-by-step checklists to complete a task (referred to as workflow execution). Leading models like Gemini 2.5 Pro achieve a success rate of over 83% on these structured tasks.

Conversely, they perform significantly worse on the following competencies:

• Company policy compliance: the LLM agent's ability to verify whether configurations, proposed solutions, or specific actions adhere to internal company policies, business rules, or established contractual agreements.
• Information retrieval and textual reasoning: the ability to locate, understand, synthesize, and reason over information drawn from unstructured or semi-structured text sources, such as knowledge base articles, emails, or case notes.
• Database querying and numerical computation: formulating precise queries to extract specific records from a structured CRM database, and then performing numerical calculations or aggregations on that data.

Enterprise data privacy & security: is confidentiality maintained?

AI agents exhibit very little built-in awareness of data confidentiality. Even when given explicit system instructions to handle sensitive information with caution, their compliance improves only at the expense of primary task performance.

Furthermore, when privacy guardrails must be maintained across multiple conversational turns, their effectiveness degrades. AI agents powered by open-source LLMs (whose architecture is public, such as Llama or Gemma) struggle significantly more to enforce confidentiality than proprietary models developed by enterprise providers.

Which AI architectures and LLM families deliver peak performance?

Models equipped with reasoning capabilities (Gemini 2.5 Pro and GPT o1) outperform non-reasoning models or lighter variants (Gemini 2.5 Flash and GPT-4o), with performance advantages ranging from 12.2% to 20.8% in single-turn task completion rates. This trend remains consistent across both B2B and B2C scenarios, as well as in single-turn and multi-turn modes.

Performance disparities between B2B and B2C marketing tasks

There are slight performance differences between business-to-business (B2B) and business-to-consumer (B2C) tasks. For instance, an AI agent powered by Gemini 2.5 Pro performs slightly better in B2C scenarios (58.3%) than in B2B scenarios (57.6%) on straightforward tasks.

Conversely, an AI agent using GPT-4o mini shows the opposite behavior (21.3% in B2B vs. 18.6% in B2C). The substantially larger volume of records in B2C environments may hinder smaller AI models that struggle to process large amounts of information simultaneously.

How agent communication styles impact downstream execution

Moving from a single query to a multi-turn conversation poses a major challenge for AI agents. Testing revealed a significant drop in performance. For example, the Gemini 2.5 Pro-powered agent failed in 9 out of 20 cases because it did not manage to gather all the required information during the dialogue. This highlights how difficult it is for AI systems to clarify missing details when initial user prompts are incomplete.

Which LLM delivers the best price-to-performance ratio for agents?

To identify the AI models offering the best value for money, the study compared task performance against inference costs. It demonstrates that Gemini 2.5 Flash and Gemini 2.5 Pro deliver the strongest price-to-performance ratio. Although GPT o1 achieves high accuracy, its operational cost is substantially higher.

Key architectural characteristics of a high-performing agent

Researchers observed a clear correlation between a model's propensity to ask clarifying questions and its overall success rate during multi-turn interactions. The top-performing models tend to request clarification from users more proactively. This suggests that the ability to effectively gather missing context is a critical asset for LLM agents in interactive workflows.

Essential strategic lessons for entrepreneurs and marketing executives

Here are seven actionable recommendations drawn directly from current AI agent capabilities and benchmark findings for business leaders.

1. Upskill teams in agentic prompting & systems architecture

Training your teams in generative AI is vital for your business. Within six months, without proper training, your workforce will be less productive than competitors who leverage AI to automate and streamline their operations.

Within twelve months, untrained teams risk falling behind, unable to understand or effectively orchestrate the newest tools on the market. Within two years, your company could face a severe technological gap that threatens both its competitiveness and its capacity to innovate.

Investing in generative AI training today drives measurable efficiency gains, enhances complex problem-solving, and ensures your team can fully harness these tools to stay relevant and competitive in a fast-moving market.

2. Start with narrow, highly deterministic workflows

Prioritize automating tasks that follow clear steps and fixed rules, such as routing customer service tickets or assigning sales leads. AI agents are already highly reliable for this type of structured execution.

3. Avoid unconstrained autonomy for complex multi-branch decisions

AI agents currently underperform on activities that require extended back-and-forth discussions, nuanced contextual understanding, and complex reasoning. You should therefore maintain human supervision or implement specialized fine-tuning for these scenarios.

4. Enforce strict data isolation & local compute boundaries

If an AI agent must handle sensitive information (customer records, internal strategies), it is imperative to enforce explicit, repeated guardrails regarding data confidentiality and security.

Even with strict instructions, constant vigilance remains necessary, as enforcing privacy constraints can sometimes degrade the AI's ability to complete its primary task. Keep in mind that open-source models currently lag behind proprietary models in privacy compliance.

5. Leverage specialized reasoning models for strategic planning

Models equipped with dedicated "reasoning" capabilities (such as Gemini 2.5 Pro or OpenAI's reasoning series) are significantly more effective for complex business tasks. Even though their per-token cost is higher, investing in these models often yields a superior long-term return on performance.

6. Balance token latency and API unit economics

For day-to-day operational deployment, Gemini 2.5 Flash and Gemini 2.5 Pro offer an excellent balance between performance and cost. Assess your actual operational complexity before defaulting to the most expensive frontier models.

7. Deploy proactive agents capable of asking clarifying questions

In workflows that require multiple exchanges to gather complete information, favor AI agents prompted to ask clarifying questions. This ability to proactively seek clarification is a strong predictor of higher success rates in complex conversations.