AI-Based Attacks: What is the Actual Risk?
AI has moved from a cybersecurity talking point to an operational tool in the attacker’s arsenal, reducing the time, cost and expertise required to conduct reconnaissance, develop convincing social-engineering campaigns, analyze vulnerabilities, modify malicious code and process stolen data. Work that once required several specialists can increasingly be handled by a single operator supported by AI.
The question is no longer whether threat actors are using AI. They are. The more useful question is how much operational advantage it provides and whether existing security programs are prepared to absorb the resulting increase in speed, scale, and capability by threat actors ranging from novice to nation state sponsored.
The answer warrants concern, not panic.
What Qualifies as an AI-Based Attack?
AI-related cyber risk generally falls into two categories.
The first is an AI-assisted attack. In this scenario, an attacker uses AI to improve one or more stages of a conventional operation. The technology might identify potential targets, write personalized phishing messages, translate scams, troubleshoot malicious code, search for vulnerabilities or summarize information taken from a compromised environment.
The second category involves attacks against AI systems. These include data poisoning, model theft, privacy attacks and prompt injection malicious instructions designed to manipulate how an AI system behaves.
The distinction matters. Current evidence suggests that organizations are more likely to encounter established attack techniques augmented by AI. Such as phishing, social engineering, reconnaissance, and content generation than fully autonomous, AI-generated attacks. However, that balance is beginning to shift as AI agents gain access to browsers, software-development tools, cloud environments and business applications.
The Current Threat: Acceleration, Not Magic
The most credible threat intelligence does not support the idea that AI has suddenly made every criminal an elite hacker.
In January 2025, Google’s Threat Intelligence Group reported that government-backed actors were using Gemini for research, translation, content creation and assistance with coding problems. Google found that AI improved efficiency but did not provide those groups with fundamentally new attack capabilities. Its analysis of adversarial AI use characterized the technology primarily as an operational accelerator.
That finding remains broadly accurate. AI does not replace the need for access, infrastructure, technical judgment and an understanding of the target environment. Generated code can fail. Automated decisions can be unreliable. Real networks rarely behave as neatly as training environments.
What has changed is the depth of AI involvement.
Anthropic reported disrupting an extortion operation in which an AI coding agent supported reconnaissance, credential theft and analysis of stolen information. It later disclosed a state-linked espionage campaign in which AI reportedly discovered vulnerabilities, exploited systems, collected credentials and made tactical decisions with limited human intervention.
According to Anthropic, this represented the first documented cyber-espionage campaign conducted largely autonomously at scale. The claim comes from a model provider describing activity observed on its own platform, so it should not be treated as an independent measurement of the entire threat landscape. Even so, the incident demonstrates that agentic attacks are no longer purely theoretical.
A subsequent analysis of 832 accounts banned by Anthropic for malicious cyber activity found that 67.3% had used AI for malware-related work. Smaller groups used it for more advanced post-compromise activity, including moving through breached networks. The sample is not representative of all cybercriminals, but the findings show that attackers are beginning to integrate AI deeper into the attack lifecycle.
Where AI is Creating Material Risk
Material risk is a risk significant enough that, if realized, it could meaningfully affect an organization’s financial condition, operations, strategic objectives, legal or regulatory obligations, reputation, or stakeholder decisions. In some cases, the existence of the exposure alone could have that effect.
Social Engineering and Impersonation
AI is making social engineering more convincing and easier to scale. Phishing messages once stood out because of awkward wording, generic greetings or inconsistent branding. Those clues are less reliable when attackers can generate polished messages that reflect a company’s tone and draw on information about a specific employee or project.
The interaction can also continue beyond a single email. An attacker can use AI to tailor follow-up messages to a person’s responses, creating a conversation that feels familiar and credible. A request to change payment details or share sensitive information may appear to come from someone who understands the organization and its current work.
Voice cloning and deepfake video add another layer of impersonation. A familiar voice or face can reinforce a fraudulent request, especially when someone feels pressure to act quickly. The same concern applies to messages that appear to come from a trusted executive, colleague, customer or vendor.
Organizations should verify sensitive requests through a separate, trusted channel. A recognizable voice, face or writing style is no longer enough to establish who is making the request.
Vulnerability Discovery and Exploitation
AI is also compressing the time required to understand and exploit technical weaknesses.
An attacker can use AI to review vulnerability disclosures, analyze software, locate exposed systems and adapt public exploit code. Skilled operators can use the technology to automate repetitive research while concentrating their attention on decisions that still require experience.
This matters because defenders already operate within a narrow window between the disclosure of a vulnerability and its active exploitation. |
The UK National Cyber Security Centre expects AI to increase the frequency and intensity of cyber intrusions through 2027, largely by improving existing techniques. Its assessment of AI’s impact on cyber threats identifies vulnerability research and exploit development as one of the most consequential areas of change.
The 2026 Verizon Data Breach Investigations Report found that exploitation of software vulnerabilities had become the initial entry point in 31% of breaches. Verizon also identified 15 attack techniques being augmented by generative AI. The DBIR findings reinforce a lesson security teams have understood for years: an unpatched system becomes more dangerous when attackers can find and weaponize its weakness more quickly.
Lower Barriers and Higher Attack Volume
The most significant effect of AI may not be technical sophistication. It may be volume.
A criminal does not need to become an expert for AI to improve the probability of success. The technology only needs to make the criminal faster, more persistent or slightly more capable. |
A novice can use AI to modify an existing tool or troubleshoot a coding error. An established fraud group can automate personalized conversations without employing a large writing and translation team. A technically capable attacker can delegate reconnaissance and data analysis to an agent.
When those incremental improvements are repeated across thousands of attackers, they produce a meaningful change in operational risk.
AI Systems Are Also Expanding the Attack Surface
Organizations are connecting AI assistants to email, cloud storage, customer records, software repositories and internal business applications. Those integrations deliver value, but they also create new pathways into sensitive systems.
Consider an AI agent authorized to review messages, open documents and take actions on behalf of an employee. An attacker could place hidden instructions inside an email or webpage the agent is expected to process. If the system cannot distinguish trusted instructions from untrusted content, it may disclose information or take an unauthorized action.
This is the core problem behind indirect prompt injection.
The risk increases sharply when an AI system can send messages, change files, execute code, approve transactions or access credentials. An AI agent with excessive permissions should be treated with the same concern as an overprivileged employee account—except that the agent may operate continuously and at machine speed.
What Should Organizations Do?
AI does not invalidate established security principles. It increases the cost of failing to apply them consistently.
Organizations should prioritize the following controls:
Use phishing-resistant authentication, such as passkeys or FIDO security keys, for email, remote access and privileged accounts.
Verify payment requests, banking changes and releases of sensitive information through an independent channel.
Require dual approval for financial transactions and other high-impact actions.
Patch internet-facing systems quickly and maintain an accurate inventory of exposed assets.
Apply least privilege to employees, service accounts and AI agents.
Segment critical systems to limit lateral movement after an initial compromise.
Maintain isolated, tested backups.
Monitor unusual account behavior and retain the logs required for investigation.
Test AI applications for prompt injection, data leakage and unsafe tool use.
Require human approval before an AI agent performs a consequential or irreversible action.
Conduct realistic phishing exercises, incident-response exercises and adversarial security testing.
CISA recommends phishing-resistant multifactor authentication for important accounts and identifies text-message verification as one of the weakest commonly used options. Its multifactor authentication guidance provides a useful implementation hierarchy.
Security-awareness programs must also evolve. Training users to look for spelling errors or visual defects in a video is no longer enough. Employees should be trained to evaluate the requested action:
Is the request unusual?
Is it attempting to create urgency or bypass an established process?
Does it involve money, credentials or sensitive data?
Can the requester’s identity be verified through a trusted channel?
Organizations that need help translating these controls into an operating security program can review Socium Security’s cybersecurity advisory, assessment, testing and managed security solutions.
Where Do We Invest Our Resources?
The priority is not simply adding another layer of AI-branded technology. It is strengthening the controls that AI helps attackers exploit: weak identity management, slow vulnerability remediation, excessive privileges, unverified financial processes and untested response plans. Organizations should direct resources toward phishing-resistant authentication, continuous vulnerability management, effective monitoring, resilient backups, incident-response exercises and clear safeguards for AI systems. These investments build resilience against AI-assisted attacks while also addressing the conventional threats still responsible for most breaches.
Socium Security helps organizations turn these priorities into a practical, sustainable security program—from assessing risk and testing defenses to closing control gaps and preparing teams for an incident. Start a conversation with Socium Security to determine where your next cybersecurity investment can reduce the most risk.
