Generative AI has significantly impacted offensive cybersecurity by increasing the sophistication and scale of cyber threats by allowing more complex and varied types of cyber-attacks (Palani et al. 2024). This dual nature of Generative AI highlights its role as a double-edged sword in cybersecurity. LLMs can be a valuable tool for cyber security professionals aiding in tasks such as penetration testing and developing security solutions (Al-Hawawreh et al. 2023). The use of LLMs has been shown to automate cyber-attacks effectively, with language models like ChatGPT able to generate executable attack code and script fragments (Iturbe et al. 2024). Generative AI-powered attacks achieve a 67% higher success rate and a 72% reduction in operational complexity, enhancing the effectiveness of cyber offensive operations by simulating sophisticated attack scenarios (Reddem 2024). Microsoft (Karamthulla et al. 2024) observed that threat actors like Forest Blizzard, Emerald Sleet, Crimson Sandstorm, Charcoal Typhoon, and Salmon Typhoon are increasingly looking to AI, including Generative AI, to enhance their attacks. The analysis revealed that these LLM-enhanced attack campaigns correspond to several tactics described in the MITRE ATT&CK framework. During the reconnaissance phase, threat actors utilized LLMs to scan and collect victim-specific information, thereby tailoring their attacks more effectively. In the resource development phase, LLMs was applied to enhance malware capabilities and develop supporting infrastructure. For initial access, attackers launched targeted spear phishing campaigns, where LLMs generated highly convincing and context-specific content. Defense evasion was observed through the use of AI-generated obfuscation techniques to bypass detection mechanisms on compromised systems. Finally, during the collection phase, adversaries employed generative models to extract information from sensitive data repositories, often targeting high-profile individuals or critical thematic domains.

Traditional cybersecurity offensive testing methods, like red teaming and penetration testing, are often time and resource-intensive, necessitating the adoption of specialized tools and algorithms for improved efficiency. Integrating LLMs into the red team testing process offers new opportunities to enhance efficiency, precision, and cost-effectiveness by automating complex tasks, improving decision-making, and providing real-time insights during engagements (Patil et al. 2024). LLMs can significantly reduce the time required for each testing phase by rapidly processing extensive datasets and proposing tailored actions (Zaydi and Maleh 2024). It allows even inexperienced IT operations staff to execute tests by providing a more efficient and accessible approach to penetration testing, potentially reducing costs and increasing the frequency of security assessments for organizations, particularly small and medium enterprises that may lack the budget for professional security testing services (Valea and Oprişa 2020).

The use of LLMs in the threat intelligence phases of offensive security tests significantly enhances the accuracy and speed of information extraction and analysis. LLMs can automate the extraction and summarization of important information from large datasets, such as historical cyber incident reports, thereby improving the accuracy of threat intelligence and the ability to forecast future threats (Sufi 2024). Overall, LLMs provide deep insights that help offensive cybersecurity professionals respond more efficiently to emerging threats and risks (Hassanin and Moustafa 2024).

LLMs, particularly ChatGPT, have high potential to enhance cyberattacks done by individuals with entry-level skills (Yigit et al. 2024). Using generative AI in cybersecurity, particularly in Capture the Flag (CTF) exercises, has significant potential (Chamberlain and Casey 2024). LLMs can automate the generation of attack scenarios, provide personalized feedback, and simulate real-world threat actors, enhancing the realism and effectiveness of CTF exercises. When LLMs are properly fine-tuned and combined with prompt engineering techniques like Chain of Thought (CoT) and Optimization by PROmpting (OPRO), they can effectively automate threat modelling, involving simulating attacks to identify vulnerabilities (Yang et al. 2024).

LLM agents are valuable tools for the reconnaissance phase of penetration tests (Temara 2023). In the reconnaissance phase, LLM agents generate detailed reports, enhancing initial information gathering (Hilario et al. 2024). That means that they provide insightful information, like technology stack, domain names, SSL/TLS configurations, ports and services used, that can be directly used for planning the next phase of a penetration test, offering meaningful insights that previously required multiple tools to obtain. During scanning, it automated test scenario generation, streamlining vulnerability detection. In exploitation, the LLM quickly responded to vulnerabilities, providing strategic exploitation options (Hilario et al. 2024). LLM-generated phishing and spam emails crafted to be more sophisticated and realistic, often bypassing the keyword-based and heuristic approaches that traditional spam detectors rely on. These spam detectors struggle with zero-shot and few-shot rephrased learning scenarios, where the emails are designed to evade detection by mimicking legitimate communication more closely (Afane et al. 2024).

Through neural machine translation, LLMs can effectively generate syntactically and semantically correct software exploits from natural language descriptions, though minor errors prevent full automation, indicating great potential (Liguori et al. 2021). MITRE ATT&CK top tactics, where LLMs are effective in generating successful executable code fragments, are “initial access” (TA0001), “defense evasion” (TA0005), and “discovery” (TA0007). The tactics “persistence” (TA0003), “privilege escalation” (TA0004), and “exfiltration” (TA0010) also showed satisfactory outcomes (Iturbe et al. 2024).

LLMs can effectively facilitate cyber offensive attacks, specifically generating viruses and polymorphic malwares (Gupta et al. 2023). They can be leveraged to generate code that targets CPU vulnerabilities, such as those that allow viruses to read kernel memory, thereby gaining control over the system. Additionally, LLMs can be leveraged to generate polymorphic malware, which is designed to alter its code with each execution to evade detection by traditional antivirus systems.

There are some early successful examples of LLM applications illustrating the potential of LLM-based automation to transform cybersecurity by reducing manual effort, enhancing accuracy, and enabling comprehensive threat assessment. PTHelper (Gracia and Sánchez-Macián 2024) streamlines the penetration testing process by automating transitions between phases, using modules for scanning, exploiting, natural language processing, and reporting, demonstrating effectiveness in both black-box and controlled environments. PentestGPT (Deng et al. 2023a) simplifies testing by guiding LLMs through micro-steps, reducing reliance on domain expertise, though expert oversight remains essential for accuracy. AutoAttacker (Xu et al. 2024) leverages GPT-4 for automating post-breach cyber-attack stages, excelling in lateral movement and credential gathering with modular components for planning and navigation. GAIL-PT (Chen et al. 2022) uses generative adversarial imitation learning to address the challenges of high-dimensional action spaces, integrating expert knowledge to improve decision-making in penetration testing.

Figure 1 illustrates how LLMs can be integrated into different phases of offensive cybersecurity operations, including penetration testing, red teaming, and threat intelligence-based simulations. The figure emphasizes how LLMs not only streamline traditional workflows but also enable new capabilities, such as automated scenario generation, enhanced reconnaissance, and dynamic exploitation options. This visual supports the argument that LLMs can significantly improve the efficiency, scalability, and accessibility of offensive security testing.

Figure 1.

Key applications of LLMs across phases of cyber offensive security testing.

In the context of cyber defense, LLMs excel in tasks such as threat detection, vulnerability analysis, and automated defense mechanisms (Zhou et al. 2024). They offer adaptive and intelligent technologies that can dynamically create and deploy actionable defense mechanisms, thereby increasing the efficiency of security operations. LLMs enhance cybersecurity by analyzing historical and real-time data to accurately predict future threats and vulnerabilities, enabling organizations to implement proactive security measures and strengthen their defenses (Metta et al. 2024). Furthermore, their capability to automate routine cyber operations tasks like threat analysis and incident response, allowing cybersecurity professionals to dedicate more time to strategic decision-making and complex investigations.

LLMs hold transformative potential in the field of cybersecurity defensive operations, offering significant advancements across various applications including threat intelligence, cybersecurity risk monitoring, vulnerability management, static malware analysis, dynamic debugging, anomaly detection and behavior analysis, web content security, phishing and spam detection, digital forensic, fuzz testing, program repairing, secure code generation, honeypots, and incident response and recovery (Zhang et al. 2025). Additionally, by being trained on frameworks like MITRE ATT&CK and D3FEND, LLMs can provide comprehensive insights into both attack techniques and corresponding defense procedures, facilitating a more robust cybersecurity posture. This dual capability of LLMs not only accelerates the detection and response to cyber threats but also empowers cybersecurity professionals to develop more sophisticated defense strategies (Alotaibi et al. 2024).

A recent comprehensive review by Ding et al. (2025) highlights that the integration of LLMs into cyber defense operations offers significant advancements in managing and enhancing cybersecurity posture. By analyzing extensive datasets, LLMs can extract valuable features and information, providing strategic recommendations to mitigate cyberattacks and effectively detect threats. Their application in security datasets allows for the generation of human-like text, which aids in threat and risk detection, facilitating rapid responses to potential threats. However, the successful deployment of LLMs necessitates access to comprehensive datasets, including security data, network traffic, and log files, which are crucial for accurate threat detection and risk mitigation. Organizations must exercise caution in several areas before implementing LLMs in their cyber operations. Ensuring data privacy and protection is critical, given the large-scale datasets involved. Awareness of the potential vulnerabilities and threats that LLMs might introduce is essential to maintaining a robust cybersecurity posture. Compliance with regulatory requirements is also vital to avoid legal and operational challenges. Furthermore, LLMs should be seamlessly integrated with existing cybersecurity systems to maximize their effectiveness. Lastly, ethical and responsible use of LLMs is crucial to prevent misuse and maintain trust in their outputs, ensuring that these advanced tools contribute positively to cybersecurity efforts (Ding et al. 2025).

In the current cyber security landscape various real-world cybersecurity products are being used in the cybersecurity operations that leverage Generative AI to enhance security measures (Sai et al. 2024). Google Cloud Security AI Workbench and Microsoft Security Copilot are designed to enhance threat detection and response. SentinelOne Purple AI focuses on addressing emerging threats with advanced AI techniques. Talon Enterprise Browser integrates with Microsoft Azure OpenAI Service to provide enterprise-grade access to Generative AI tools like ChatGPT, enhancing data protection and productivity. SlashNext Generative Human AI defends against advanced threats such as business email compromise and financial fraud by mimicking human threat researchers. Recorded Future AI leverages over a decade of threat analysis data to provide real-time threat landscape analysis and improve analyst efficiency. SecurityScorecard integrates with OpenAI’s GPT-4 to enhance its cybersecurity assessments, providing more comprehensive insights into potential vulnerabilities.

Figure 2 illustrates how LLMs and LLM integrated products can support cyber security operations. This figure illustrates also the increasing role of generative AI in operational cyber defense, showing how LLM-integrated tools support threat identification, detection, protection, response and recovery.

Figure 2.

Gen AI on cyber defense operations.

While LLMs have significant potential in cybersecurity, particularly in threat intelligence process, they are not yet perfectly accurate due to hallucination where LLMs generate false information (Patsakis et al. 2024). LLM models can produce biased and unreliable content, and their increased consistency might make such content more credible and potentially more dangerous (Corchado et al. 2023). They may exhibit biases due to their training datasets, potentially leading to inaccurate or skewed recommendations (Zaydi and Maleh 2024). Another common operational concern is data poisoning, where malicious actors can corrupt the training datasets, leading to compromised AI performance and decision-making (Maryam et al. 2024). Attackers can manipulate the training data to cause the algorithm to make incorrect decisions, like misclassified cyber threats, misleading events/alerts and incorrect mitigations. The complexity of integrating AI systems with existing cybersecurity infrastructure poses a significant hurdle, as it requires substantial skills, resources and expertise (Jana et al. 2024).

First major challenge is the potential for adversarial attacks, where malicious actors can exploit LLM models by feeding them deceptive inputs to manipulate their outputs (Jana et al. 2024). Adversarial attacks, particularly prompt injection attacks, pose a significant risk, as they involve manipulating LLM inputs to generate unauthorized or harmful outputs, which can be exploited to simulate adversarial tactics and test system defenses against such manipulations (Taghavi and Feyzi 2024). A well-known example is Morris II, the first worm specifically designed to target Generative AI ecosystems using adversarial self-replicating prompts, highlighting a novel attack vector that exploits the interconnected nature of Generative AI-powered applications (Cohen et al. 2024). This example revealed the potential for adversarial attacks to compromise Generative AI systems, leading to malicious activities such as spamming, data exfiltration, and phishing.

Model theft is another critical risk, where unauthorized entities gain access to and replicate AI models, undermining proprietary technologies and security protocols (Maryam et al. 2024). Model theft may pose significant losses on organizations, reputational risks, including using the stolen model for malicious purposes (Taghavi and Feyzi 2024).

Although many developers have adopted AI technology in their workflows and generally find the code provided by AI to be usable and fairly accurate, there is caution regarding AI-generated code being insecure and inaccurate in coding and scripting practices (Sergeyuk et al. 2024). The use of AI-powered tools in cyber security operations may lead to the production of insecure code, posing significant risks (Oh et al. 2023). This can result in cyber security practitioners unknowingly incorporating insecure code into their systems, potentially compromising the integrity, effectiveness and security of their operations.

Despite the deployment of undisclosed defenses by service providers, LLM agents are vulnerable to jailbreak attacks, where malicious prompts can manipulate these models to bypass their safeguards and generate harmful or sensitive content (Deng et al. 2023b). Users could manipulate the LLM by crafting jailbreak prompts that bypass internal controls, leading the model to generate unauthorized or harmful information.

An important concern regarding the systemic risks of LLMs is about their self-replication capability. AI systems driven by LLMs such as Meta’s Llama31-70B-Instruct and Alibaba’s Qwen25-72B-Instruct were demonstrated to be able to autonomously create separate copies of themselves, which could lead to the uncontrolled proliferation of AI systems, potentially forming independent networks that might act against its usage purpose (Pan et al. 2024). There are also concerns about the ethical aspects and potential misuse of AI-driven cyber offensive applications (Raman et al. 2024) and AI-generated content (Jana et al. 2024), which can be used to create convincing cyber-attacks or deepfakes, complicating the detection of genuine threats. Furthermore, the use of LLMs in cybersecurity processes poses significant data leakage risks due to their need for accessing sensitive system information, which can lead to unauthorized access, especially in cloud-hosted environments (Zaydi and Maleh 2024). Lastly, some regulatory challenges have been defined in using LLMs for cybersecurity (Sai et al. 2024), including the risk of intellectual property violations due to content generation similar to proprietary research, and the need for quality control and standardization to ensure consistent AI-generated advice. Additional challenges include defining data ownership, ensuring continuous monitoring and validation of AI performance, obtaining informed consent from users, maintaining interpretability and transparency of AI decision-making processes, and preventing over-reliance on LLM models, which could diminish human expertise.

The inherent black-box nature of LLMs presents significant challenges in understanding and controlling their operations, which raises critical concerns about transparency and accountability (Barman et al. 2024). Due to their complex and opaque internal workings, it is difficult for users and developers to predict or explain the outputs generated by these models. The lack of explainability in LLMs limits their utility as coding and scripting tools, which could hinder the understanding and mitigation of security risks during cyber operations (Khoury et al. 2023). This lack of explainability and transparency can make it also difficult to assess the reliability and accuracy of the AI’s threat detection and exploit capabilities (Mohammed 2024).

Overreliance on content generated by LLMs poses significant risks, particularly due to the difficulty in detecting incorrect or misleading information produced by these models (Yao et al. 2024). LLMs are capable of generating highly convincing text that can easily be mistaken for accurate cyber intelligence information, leading to incorrect actions and conclusions.

This article aimed to highlight the intersection between generative AI and cybersecurity, focusing on both opportunities and associated risks. Future research could further explore how audit, risk, and internal control functions can enhance the cybersecurity assurance of GenAI systems. This includes examining control frameworks, audit methodologies, and regulatory compliance strategies tailored to the unique characteristics of AI-based technologies.