Look, I’ll be honest with you. When I first heard about AI writing malware, I laughed. “Cool,” I said, “another overhyped vendor slideshow.” Then I watched a junior red‑teamer with zero Python experience use a jailbroken LLM to spit out a fully functional, polymorphic dropper in about eight minutes. He was eating a bagel while it happened. That’s when I stopped laughing. And started drinking. We’re living in the era of weaponized AI. The same large language models that help us write detection rules and summarize alerts are now being used  by attackers—and legitimate red teams—to launch attacks at a scale and  speed we’ve never seen. This isn’t science fiction. It’s Tuesday. So, let’s talk about what’s actually happening out there, how the bad guys (and the “ethical” bad guys) are wielding these tools, and what we, the poor souls stuck defending the castle, can do about it. Spoiler alert: it involves fighting fire with fire, and also maybe a little screaming into a pillow. The Offensive Playbook: When Script Kiddies Become AI Warlords Remember the good old days? To be a dangerous attacker, you needed to know C,  understand assembly, or at least be able to Google your way through a Metasploit tutorial. Now, thanks to the glorious unregulated chaos of the internet, any idiot with a credit card can get their hands on an uncensored AI model. Meet the New Villains: WormGPT and FraudGPT WormGPT FraudGPT You’ve  probably heard the names. WormGPT and FraudGPT were the first widely publicized “dark LLMs”—models specifically trained to be the opposite of helpful. No content filters, no “I can’t help with that” nonsense. You ask for a ransomware builder, you get a ransomware builder. You ask for a perfectly crafted spear‑phishing email impersonating the CEO, it’ll  even throw in a “best regards” with the right corporate font. Now, a lot of these original services have been taken down, shuttered, or driven underground. But here’s the kicker: they didn’t need to survive. They already did their damage by proving the concept. Today, attackers are just using regular LLMs—ChatGPT, Claude, the open‑source models you can run on a laptop—with clever jailbreaks. There’s a whole cat‑and‑mouse game where researchers publish a new jailbreak, the model gets patched, and within hours, someone finds a new one. It’s like  whack‑a‑mole, except every time you whack one, it spawns three more, and one of them steals your identity. Hyper‑Personalized Phishing: The End of the Nigerian Prince The old phishing email was a work of art in its own way, but it was also laughably easy to spot. Bad grammar, weird urgency, and a prince who somehow had your email address. AI changed that overnight. Now, red teams (and real adversaries) can feed an LLM a target’s LinkedIn profile, a few public posts, and maybe a leaked email from some old data breach, and the AI will generate a phishing email that sounds exactly like a colleague. It’ll mention the project they’re working on, the coffee shop they like, even their dog’s name. I’ve seen one that included a fake Slack screenshot to build credibility. A fake Slack screenshot. That’s not phishing; that’s psychological warfare with a side of art direction. fake Slack screenshot And the scale? Forget sending 10,000 emails hoping for a 0.1% click rate. With AI, you can send 10,000 unique emails, each tailored to its recipient. The only thing limiting you is how fast you can hit “send.” unique Reconnaissance at the Speed of Light Attackers used to spend weeks or months footprinting a target. Now, they can dump a company’s entire public GitHub repos, SEC filings, and help‑desk articles into an LLM and ask, “Based on this, what’s the most likely technology stack they’re using? What are their likely VPN endpoints? And  can you generate a plausible internal document naming scheme?” I’ve  seen red teams do this in a single afternoon. One guy literally fed a model 300 pages of a target’s public documentation, and it output a list of potential internal system names, employee email formats, and a rough organizational chart. That’s not recon. That’s cheating, but like, in a way that makes you want to cry. The Defensive Reality: We’re Playing Catch‑Up, But We’re Not Helpless Okay, so the bad guys have rocket launchers. What do we have? Well, if you believe the vendor marketing, we have AI‑powered everything—AI threat hunting, AI incident response, and AI that can apparently make a decent cup of coffee. The reality is messier, but also more interesting. Fighting AI with AI: The Rise of the Little Models One of the dirty secrets of the industry is that you don’t always need a giant, cloud‑hosted LLM to defend against AI attacks. In fact, sometimes, you want the exact opposite. Small, fine‑tuned models that can run  on‑prem, or even on a laptop, are becoming the defensive workhorses. Take phishing detection. Generic email filters are okay, but they weren’t built to catch AI‑generated prose that’s almost indistinguishable from human writing. So, people are fine‑tuning models like Phi‑3, Mistral, or even a well‑tuned BERT variant specifically on datasets of AI‑generated emails. They’re feeding them examples from their own red team exercises, from public corpus, and from the sad, cringey emails that somehow made  it past their first‑line defenses. These little models can be deployed right inside the email gateway. They’re cheap, fast, and—most importantly—they don’t send your sensitive email traffic to some cloud API that might be training on your data. Because let’s be honest, the last thing you want is your own SIEM accidentally feeding the enemy. Anomaly Detection That Actually Understands Humans User and Entity Behavior Analytics (UEBA) has been around for a while, but  AI is making it less terrible. The old approach was to look for statistical outliers—someone logging in from a new location, downloading an unusual number of files. Attackers learned to blend in. Now, with AI‑driven anomaly detection, you can model the context of behavior. Did the CFO suddenly start writing emails with a slightly different rhythm and vocabulary? That might be a compromised account being used by an LLM to issue fraudulent wire transfers. Did a developer clone a repository at 3 AM using a weird Git client? Maybe it’s fine; maybe it’s an AI‑powered backdoor being deployed. context The key is that the defensive models are getting better at understanding what “normal” looks like—not just in terms of data points, but in terms of intent. It’s still early days, and I’ve seen plenty of false positives that sent the whole SOC into a  frenzy over what turned out to be a tired sysadmin doing their job. But  the direction is promising. intent Using AI to Reverse‑Engineer AI‑Generated Malware Here’s where it gets almost poetic. Attackers use AI to write malware. Defenders can use AI to reverse‑engineer that malware. I’ve seen teams take a suspicious binary, feed its decompiled code into a well‑prompted LLM, and get back a plain‑English explanation of what it does, complete with potential IOCs and even suggested YARA rules. In one case, a model identified that a piece of ransomware was using a custom encryption routine that was essentially a slight mutation of a known open‑source library. The analyst went from “what is this mess?” to “aha,  here’s how we decrypt it” in about fifteen minutes. Now,  you have to be careful—if you’re uploading malware to a public LLM, you might be training the model that’s about to be used against you. So smart teams are using local models (like CodeLlama or a fine‑tuned variant) to do this analysis in‑house. Air‑gapped, no funny business. It’s the equivalent of having a junior malware analyst who never sleeps, never complains about the coffee, and occasionally hallucinates a variable name, but you learn to fact‑check it. The Asymmetric Reality: Speed, Scale, and the Human Element Let’s step back for a second. The thing that makes AI so dangerous for defenders isn’t that it’s magic. It’s that it changes the economics of attacks. Before AI, launching a sophisticated, targeted attack required time, skill, and money. You had to hire people who knew what they were doing. Now, one determined individual with a few hundred dollars in API credits can run a campaign that would’ve taken a nation‑state a year to build a decade ago. Defenders are stuck with the same budgets, the same tired tools, and the same number of analysts who are already overworked. We can’t just throw bodies at the problem. We have to be smarter. That’s why the “AI on AI” approach isn’t just a buzzword—it’s survival. We need defensive AI that operates at the same speed and scale as the  offensive AI. We need models that can sift through terabytes of logs, correlate events across disparate systems, and surface the two or three things that actually matter before the attacker has already moved laterally and sold our secrets to the highest bidder. And we need to stop pretending that our human analysts can out‑think an LLM that’s been fine‑tuned on every breach report from the last ten years. We’re not going to win by being smarter. We’re going to win by being faster and by using AI to augment our own judgment, not replace it. faster Where We Go From Here: A Few Unsolicited Opinions If you’ve made it this far, you probably want something actionable. I’ve got three thoughts, and they’re not the kind you’ll see in a glossy  vendor brochure. 1. Stop banning AI, start governing it. I  know, I know—your CISO sent out that stern email about not using ChatGPT for work. But let’s be real. People are using it anyway. They’re pasting logs into it, asking it to write queries, maybe even uploading sensitive configuration files. Instead of pretending it’s not happening, give them a safe way to do it. Deploy a local model. Use an enterprise‑sanctioned instance with data controls. Because if your team is using shadow AI, you’ve already lost control of your data, and you probably don’t even know it. 2. Train for the AI‑powered attack. Your phishing simulations are cute, but if you’re still using the same old “click here for your bonus” template, you’re wasting everyone’s time. Start using AI to generate your phishing tests. Make them personalized, contextual, and genuinely convincing. See who clicks. Then, when a real attacker does it, you’ll have a fighting chance. And your users will hate you for a week, but they’ll thank you later. Probably. 3. Build your own small models. Don’t rely on the big cloud providers for everything. The technology to fine‑tune a capable 7‑billion‑parameter model is available, open‑source,  and can run on a single decent GPU. Build models for your own  environment: for detecting phishing, for analyzing scripts, for spotting anomalies in your specific business logic. You’ll have more control, less data leakage, and you’ll learn a ton in the process. Plus, it’s a fantastic way to justify that GPU purchase to your manager. The Parting Shot We’re in a strange moment. AI is simultaneously the sharpest tool in our defensive toolbox and the biggest threat we’ve faced since the early days of the internet. It’s like we handed every hacker a lightsaber and then told the security team, “Here, you get a slightly bigger lightsaber. Go figure it out.” But here’s the thing: we’ve been here before. Every major shift—the rise of  the cloud, the explosion of mobile, the dawn of ransomware—felt like the end of the world. And it wasn’t. We adapted, we built new tools, and we got smarter. This time is no different. It’s just moving a hell of a  lot faster. So, grab your coffee, fire up that local LLM, and start experimenting. Because the attacks are coming—they’re already here, in fact—and the only way we’re going to stay ahead is to embrace the same technology that’s being used against us. Just maybe with a little less of the “hacking for profit” part. And if you see my junior red‑teamer with the bagel again, tell him I’m still looking for the source code for that dropper. I need to use it to train my detection model. — A tired SOC manager who has seen things. A tired SOC manager who has seen things.

This story contains AI-generated text. The author has used AI either for research, to generate outlines, or write the text itself. 

Google

Catch

How to Create Your Own Dark Website (.onion) on Linux

Hacker's AI: The Messy Reality of Weaponized AI

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

A Guide To Web Security Testing: Part 1 - Mapping Contents

The Noonification: How Often Do NFTs Pass The Howey Test? (1/13/2023)

Darwin's Hybrid Intelligence to Align AI & Human Goals for Startups & VCs

The Noonification: White Man (11/26/2022)

The Noonification: The Metaverse is a Sh*tshow (11/2/2022)

100 Days of AI Day 1: From Newsletter to Podcast, Leveraging AI for Audio Transformation

A Guide To Web Security Testing: Part 1 - Mapping Contents

The Noonification: How Often Do NFTs Pass The Howey Test? (1/13/2023)

Darwin's Hybrid Intelligence to Align AI & Human Goals for Startups & VCs

The Noonification: White Man (11/26/2022)

The Noonification: The Metaverse is a Sh*tshow (11/2/2022)

100 Days of AI Day 1: From Newsletter to Podcast, Leveraging AI for Audio Transformation

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps