Europol on Monday released a report highlighting the different ways cybercriminals can exploit large language models (LLMs) like Open AI’s ChatGPT and how they can aid law enforcement. The report — the first of its kind — is the result of a series of workshops organized by the Europol Innovation Lab to explore the subject.
“This report is a first exploration of this emerging field,” Europol said. ChatGPT, now in its fourth iteration, has left many stunned by its capabilities.
In the report, Europol described LLMs like ChatGPT as a double-edged sword. While they have several positive use cases, like aiding law enforcement in investigations and stopping sophisticated cyberattacks, the technology can also be used to create harmful content, facilitate terrorism, and aid cybercrime.
Earlier this month, threat intelligence firm HYAS developed a proof of concept, AI-generated “polymorphic malware,” which it said represents a “new breed of cyber threats.” Europol also highlighted a Check Point report from January that demonstrated “how ChatGPT can be used to create a full infection flow, from spear-phishing to running a reverse shell that accepts commands in English.”
Europol noted that its report only explores a fraction of the potential positive and negative applications of LLMs.
ChatGPT Can Be Exploited for Various Crimes
Cybercriminals can use LLMs to “learn about a vast number of potential crime areas with no prior knowledge,” Europol said. For example, they can use ChatGPT to learn how to break into homes or conduct terrorist activities online.
ChatGPT’s ability to craft “highly authentic texts” also makes it the perfect tool for crafting phishing emails. With its language prowess, cybercriminals with low-level command of the English language can create convincing emails for business email compromise and CEO fraud schemes.
“To date, these types of deceptive communications have been something criminals would have to produce on their own,” Europol noted. But, with LLMs like ChatGPT, “these types of phishing and online fraud can be created faster, much more authentically, and at significantly increased scale.”
ChatGP “can be used to generally gather more information that may facilitate terrorist activities, such as, for instance, terrorism financing or anonymous file sharing,” Europol said.
“Not only would this type of application facilitate the perpetration of disinformation, hate speech, and terrorist content online – it would also allow users to give it misplaced credibility, having been generated by a machine and, thus, possibly appearing more objective to some than if it was produced by a human,” the report added.
Circumventing ChatGPT’s Safeguards to Write Malicious Code
Users can also modify ChatGPT to do what it is not programmed to do.
Europol said several of ChatGPT’s safety systems have since been bypassed or jailbroken via “prompt engineering,” which is “a relatively new concept in the field of language processing,” whereby users can “bypass content moderation limitations to produce potentially harmful content.”
Europol said “some of the most advanced and powerful workarounds are sets of specific instructions aimed at jailbreaking the model.” This includes a workaround called “DAN” (Do Anything Now), which makes ChatGPT “respond to any input, regardless of its potentially harmful nature.”
OpenAI has fixed the DAN loophole, but more complex versions have again emerged, the report said — although no DAN workarounds are circulating at this time.
Once safeguards are lifted, cybercriminals can leverage ChatGPT’s ability to write code in several programming languages “with little to no knowledge of coding and development,” the report warned.
“The newer model [ChatGPT 4] is better at understanding the context of code, as well as correcting error messages and fixing programming mistakes,” Europol said, adding advanced users can go further by refining or automating the model.
Controlling AI-Generated Content Safely and Responsibly
To limit the potentially harmful uses of LLMs like ChatGPT, Europol said the Partnership on AI (PAI) and the European Union are working hard to regulate these systems.
Partnership on AI is a research NGO that set guidelines signed by a group of ten companies, including OpenAI, “pledging to adhere to a number of best practices.” These include “informing users that they are interacting with AI-generated content (i.e., through watermarks, disclaimers, or traceable elements),” Europol said.
In the future, we may see an emergence of “dark LLMs,” hosted on the dark web, possibly trained with harmful data, Europol added.
As cybercriminals increasingly adopt ChatGPT, you may begin to receive more sophisticated scam emails, including links to AI-generated misinformation and deep fakes. To learn how to protect yourself from these “convincing” scams, check out our guide to social engineering.
