I attended as many Black Hat briefings as possible this year. As a data scientist, I paid particular attention to the data science, machine learning, and artificial intelligence talks. Before we get into details, let’s address the elephant in the room: ChatGPT and LLMs.
Yes, it did seem like every third talk was about trying to apply a large language model (LLM) to either hack or secure a computer. I don’t doubt that the hacker and security communities will continue to extract value from the compressed knowledge stored in large language models, and made available in user-friendly interfaces such as ChatGPT. But the organizations that extract the most value from LLMs will be the ones that are best able to take advantage of these models’ power for interpreting language. One talk in particular did an excellent job of this.
In my opinion, the talk that made the best use of a large language model was “IRonMAN: InterpRetable Incident Inspector Based ON Large-Scale Language Model and Association miNing” by Sian-Yao Huang, Cheng-Lin Yang, and Chung-Kuan Chen at CyCraft Technology. The basic idea is to borrow the strength of LLMs in interpreting natural language, and use that interpretive power to create vector representations of Windows command lines.
Natural languages are very flexible, with many ways to express the same information. Likewise, command lines have a certain amount of flexibility. The first kind is obfuscation: There are several equivalent, distinct ways to write the same command. This is important in a security context because these obfuscations can evade detection with tools like regular expressions.
The second source of nuance is that the same string can have different meanings in different contexts. As an example, a user could invoke a command foo and also pass an argument named foo to the command; these usages could have the same or distinct meanings. A regular expression would hit on both of those usages, even if our intention is to only capture one of them.
LLMs take (encoded) strings as inputs and yield numerical representations as outputs. The speakers in this talk leverage these numerical outputs, by observing that equivalent command lines tend to be closer to each other than they are to dissimilar command lines, even when obfuscation is used. This gives the model a certain amount of robustness to variations in the input, and allows one to group together command lines from different incidents, facilitating attribution.
I want to emphasize that the main reason this talk is so intriguing to me is that it really leaned on the LLM for the thing that it is best at (interpreting text inputs) and incorporated that utility into a security workflow. Interpreting texts has enormous value for security researchers; using LLMs to do at machine speed what was previously a human-speed task is a big deal.
This talk does not rely on the chat interface at all! Instead, it peeks “under the hood” to work directly with the numerical representations that the model uses to interpret text.
In my humble opinion, the weaker LLM talks focused on the cat-and-mouse aspects of the chatbot interfaces—using ChatGPT to create “black hat” stuff. While the chat interface is impressive, and it can generate some amusing outputs, I don’t see them as a big value-add for security researchers, especially in light of the untrustworthiness of the results. For instance, a recent study found that ChatGPT can generate plausible-but-incorrect answers to Stack Overflow questions 52% of the time. I would expect even worse results if we asked ChatGPT to respond to a security incident.
LLMs are a powerful tool, but to truly leverage that power, security researchers will need to think carefully about how to wield that tool. Where are security researchers bottlenecked? Are those bottlenecks related to interpreting large amounts of text? In the security space, I anticipate that LLM methods will find homes in assessing security risks of source code, the code on web pages, and even decompiled binary executable. I hope that we see some of these applications at next year’s Black Hat.