From Threat Hunter to Red Teamer and How AI Changed Both

Cole French:

Penetration testing, threat hunting, red teaming. Few topics generate more excitement or confusion in cybersecurity, but what often gets overlooked is how these disciplines connect, evolve, and build on each other. If you’ve ever wondered what threat hunting actually looks like day to day, why attackers still rely on basic techniques or how defenders make sense of massive amounts of data, you won’t want to miss today’s conversation.

Welcome to the Cyber Compliance & Beyond Podcast, a Kratos podcast that brings clarity to compliance, helping you leverage compliance as a tool to drive your business’s ability to compete in any market. I’m your host, Cole French. Kratos is a leading cybersecurity compliance advisory and assessment organization, providing services to both government and commercial clients across varying sectors, including defense, space, satellite, financial services, and healthcare.

Now, let’s get to today’s episode and help you move cybersecurity forward. Modern cybersecurity teams operate in environments overflowing with logs, alerts, and telemetry. The challenge, making sense of all of it. In today’s episode, we dive into threat hunting, which isn’t about chasing exotic never-before-seen attacker tricks. It’s about recognizing that adversaries, including nation-state actors, frequently rely on fundamental techniques like LSASS dumping, Kerberos ticket theft, and DCSyncs. The technology needed to detect these behaviors already exists in most environments. But what’s difficult is building the right mindset, knowing what to look for, establishing baselines, and cutting through the noise, especially as AI begins accelerating both the volume of data and the speed at which attackers can operate. Tools like Splunk and Security Onion help threat hunters turn raw data into useful insight. Dashboards, process listings, drilldowns, and correlated network telemetry aren’t just nice to have, they’re essential for finding meaningful signals in environments generating terabytes of information.

We also talk about how AI is starting to augment this workflow, helping analysts sort, correlate, and prioritize signals faster while still requiring human judgment to validate what’s actually malicious. From there, we shift to the tension between Red Teams and Blue Teams. Red Teamers love creative novel attacks, but defenders benefit most when Red Teams test realistic behaviors adversaries actually use. Simple lateral movement methods such as WinRM can provide far more defensive value than overly complex one-off techniques. And with AI making it easier to automate both the simple and the sophisticated, aligning Red Team exercises with real-world attacker behavior is becoming more important than ever. Joining us for today’s conversation is Ben Lieberman. Ben is a former Marine Corps threat hunter turned penetration tester here at Kratos, where he tests web applications, conducts internal red team engagements, and designs custom tools that simulate real world adversaries. We hope you enjoy this episode.

Ben, I just want to thank you for joining us this afternoon on the Cyber Compliance and Beyond Podcast to talk about a really interesting topic. And that is not just penetration testing, but the evolution of, how do I become a pen tester, and the different roles and things like that.

And so why don’t you get us started with where did you start? How did you come to become a pen tester? Obviously, we’ll get to that as we go through this conversation. But if you just want to get us started on where you started and then we’ll go from there on where you are today.

Ben Lieberman:

Yeah, absolutely, Cole. Thank you for having me. I’m excited. So I was a Marine. So I joined the Marine Corps in 2020. Right at the start of COVID, so February of 2020. I was just so lucky enough to be assigned 1721, which is cyber defense. And I spent three years on what’s called a cyber protection team in the Marine Corps. So we would travel around and do threat hunting activity, search for malicious activity on all different types of Department of War infrastructure, the Marine Corps infrastructure, Army infrastructure. I’ve worked on pipelines and other things along those lines.

Cole French:

All right. So you mentioned that you were a threat hunter. So why don’t you walk us through what that actually looks like on a day-to-day basis? What are you staring at? What are you looking for? What’s catching your eye? Things like that.

Ben Lieberman:

Okay. So yeah, I was on what was considered a traveling team. So we would work with, I’ll call this customer environments just to simplify it a little bit. But basically, when we were back in the rear, so I was stationed at Fort Meade, so we were at Fort Meade. We’d spent a lot of time setting up what we had, this was called a collection kit. So this consisted of censors, taps in order to collect network traffic, firewalls and then other virtualization servers. And we would then set these up with ESXI, Security Onion and other basic tools like Splunk. And once these were set up, deployed, we’d bring these out to the customer environment and we would tap network collection points on switches or we would actually intercept links with physical taps. And then we would also put host agents on the machine, start collecting logs. We would input all of this into Splunk and Security Onion and we would use custom query languages on Splunk in order to find malicious cyber activity.

Cole French:

So was there a point at which you realized you were seeing the same techniques over and over and defenders were missing them? Were there consistent blind spots? Just talk about what you were seeing when you were looking at some of that data and querying that data.

Ben Lieberman:

Well, what we started to see was that these nation states, that they were overcomplicating and saying we were creating all these super complicated novel new techniques. That is true, yes. But we also found that water goes through the paths of least resistance and so do they. So they would use very simple techniques, dumping LSASS, performing DCSyncs, stealing Kerberos tickets. All the basic techniques that you see on LiDAR the advanced APTs were also doing. And we would spend a lot of time hunting for these novel techniques instead of just staying at the basics, which would have gotten us very far.

Cole French:

It’s interesting that you mention that because we’ve talked about that a lot. There was a series we did on cyber crime, a four-part series we did on cyber crime. That was one of the things we talked about over and over across that four-part series was if we just secured the most basic things, the simpler things, the things we often overlook, that actually would take care of a lot of the issues that we then have to face, and clean up and all of that fun stuff.

So okay. So jumping off from there. So what’s the hardest thing about threat hunting that people outside of a security operations environment don’t understand? I know there’s things like alert fatigue, telemetry gaps, analyst burnout. But what actually grinds somebody down from a threat hunting perspective?

Ben Lieberman:

Those things you’ve mentioned are absolutely real, but the biggest issue that we would run into is just the sheer volume of information, right? If we weren’t tapped into the right place, we had too many host agents. Even if we were tapped to the right place, sometimes it was terabytes, upon terabytes, upon terabytes of information. So figuring out how to whittle that down. And then when you actually do find something that you want to investigate, how do you further investigate with that amount of information in mind?

Let’s say if you detect malicious activity, I have to be able to move quickly to determine what the activity was because the attacker can move faster than I can, right? So I need to be able to take that information, find step one, go to step two, go to step three and do that quickly so we can isolate the activity.

Cole French:

Just curious, what was your, I guess, standard approach? Maybe that’s not the right phrasing. But how did you approach trying to whittle that data down? You can just talk to it at a high level. I mean, that makes perfect sense and I think we face a lot of that. And even the examples, alert fatigue is just another type of information overload and we very much live in a world today where the most difficult thing is figuring out what is actually important because there’s just an endless amount. So just curious, what were your techniques for whittling that data down to a place where you could actually learn something from it and take action?

Ben Lieberman:

I personally really like the capabilities of Splunk’s dashboards and Splunk’s query language because I can use a dashboard that can show me a bunch of high level information, like for example a process listing. And then I can use drill down or click activities to drill down on a specific process. And then have it update six or seven more dashboards, or analytics basically, and have that analytics all put into my face. And then I can quickly review that information. And then I can go to the next process, click on it, update six or more or so visualization, and I could continue to move on and go very quickly using Splunk’s drill-downs. I can even pivot between dashboards. Those type of things were super helpful to us.

Cole French:

So speaking of the dashboards, and I’ve had some experience with Splunk myself working in a SOC in the past. And so I’m just curious because there’s kind of two part of dashboards, right? It’s knowing the language, knowing how to build the dashboards on one hand. And then on the other hand is knowing what I want to build or what I want to incorporate into a dashboard. So when you looked at a massive set of data, was there something you were looking for in particular to build out these dashboards? So did you know, this is what I’m really curious about in this particular set of information? Or talk a little bit about how you actually built those dashboards, what you were looking for when you were building out those dashboards.

Ben Lieberman:

Yeah. So we took two approaches. We took a much more generalized approach with, for example like we would take SYSMOD and we would just use SYSMOD data to create process tress over a certain host, for example. We think a host, based on what threat intelligence is telling us could be targeted, so we’re going to just take a very general view. And we will take a process listing, we’ll make a tree. Then we might pre-correlate that data with network data. And that is simply for us to look at it and be like, “That stands out or that doesn’t stand out.” We might also take that same data and compare it across a many-host. So we might say, “Okay, this process stands out because it’s the only process running on 5,000 machines,” for example. Those are some different ways that you could do that.

Another thing we might look for is a specific technique. So we may say, “Hey, we want to look for DCSync today.” So we would go into a lab environment, run a DCSync, take the telemetry data out of that activity that we just ran and then create a detection from that activity, and then reapply that back to the environment we’re working in.

So there’s a few ways you could do it. Every analyst has their own preference. But generally, we’d like to start with general information, us that general information to whittle it down to more specific events.

Cole French:

And it goes back to what you were talking about at the beginning, where kind of like the attacks, it’s kind of the same stuff or the easier stuff we’ve seen. And I think with data, you have to set aside the fact that it’s this massive amount of data and still keep your mind focused on, “These are the particular tactics, techniques, et cetera, so I’m going to look for those.” And really, the size of the data doesn’t matter, which speaks back to why you would like a tool like Splunk, and dash-boarding and things like that because it can bring all that. It can comb through all that data and illuminate what it is that you’re looking for.

Ben Lieberman:

Yeah. And I should mention that base-lining is also extremely important. I can take all the data from the 5,000 hosts that I have ingested and I can say, “All right, what’s an outlier here? What’s all the same and what’s different?” And that helps a lot because like I said, one process running on one machine when there’s 5,000 others running different processes, that stands out.

Cole French:

And I’m sure we’ll talk more about that when we get to the AI stuff. But just want to go back to the roles, so threat hunter, red team. So curious, before we get to red teaming, as a threat hunter, what did you wish the red team would test versus what they kept testing, at least in your experience?

Ben Lieberman:

Yeah. So not to sound like a broken record here, but red teamers love their novel approaches just as much as threat hunters think the adversaries do. So you often find red teams that try to create these extremely novel approaches or do things in a crazy, weird way, something brand new and novel. But again, just testing the basic things is super helpful for us. We want to make sure that we’re able to catch the basics before we try to catch the brand new, novel technique that you’ve come up with that you wrote an article about.

Cole French:

So basically what you’re saying is the red teamers tend to focus, or at least in your experience on the threat hunting side, the red teamers tended to focus on trying to come up with their own novel approaches and maybe sometimes they missed testing the most basic things. Is that what you’re getting at?

Ben Lieberman:

They could use a novel protocol for lateral movement that may exist on the environment when they could have just used WinRM and we could have attempted to catch that instead. And we may have caught them and that may have been something that they wouldn’t have been happy about, but that’s what the nation state actor likely would have done.

Cole French:

And so as we continue on in this progression from threat hunting, red teaming, and now we’re going to get into from a defensive approach to an offensive approach, I just wanted to pause real quick and make sure. So define for us real quick, and I think it’s probably clear from our conversation on it so far, but just as we get in, we’re going to get into some more definitions. So real quick, threat hunter, red teamer, just define those real quick for us?

Ben Lieberman:

Okay.

Cole French:

And then we’ll jump into some more defense to offense and what that looks like.

Ben Lieberman:

Yeah. So threat hunting is one of those that has many definitions depending on who you ask. But in my opinion, I would say it’s the human-led initiative to find malicious cyber activity. And the reason I specifically say human-led is because you’re using automated tools, however a lot of the threat hunting aspect is manual review of logs and information. For red teaming specifically, it’s just the emulation of threat actors and replicating their techniques that they would see malicious people do and doing it in an ethical way.

Cole French:

Got it. So the difference is, one is looking for those things, looking for threat. And then the other is attempting to perpetrate or illuminate those threats within an organization. Is that accurate?

Ben Lieberman:

Exactly. Two sides of the same coin.

Cole French:

Yeah, okay. That is helpful. So as we move the conversation, like I mentioned, from defense to offense, go into what made you specifically decide to jump from defense to offense. Was it one engagement? Did you just slowly go in that direction? Just talk to us about what that looked like for you.

Ben Lieberman:

Yeah. So like I was saying, it’s two sides of the same coin. People will often separate these two, however they’re extremely closely related. When I was a threat hunter, I would spend a lot of time doing work on Hack The Box or TryHackMe, or I got my OSCP while I was a threat hunter. What I found was, like I was saying, it’s two sides of the same coin. Learning offensive work or learning red teaming helped me significantly as a threat hunter. So I figured that in terms of my career trajectory, going into red teaming would help me become a threat hunter if I wanted to stay that route, but I got to explore both options.

Cole French:

So what’s the biggest thing a former threat hunter brings to a red team that a career offensive person doesn’t? And before you answer that, I just want to highlight that, yeah, I can see how they are, like you said, two sides of the same coin, but I think this question gets into here’s some of the nuance and the difference between the two. So what’s the biggest thing that a threat hunter brings that a red team maybe doesn’t bring?

Ben Lieberman:

We talked about it earlier, but I have pretty strong understanding of what defenders miss, but also logging configurations and how defenders find my activity. So I know that if I run technique A, it’s going to generate log A, B, C and D. And then if I do another technique, it might only generate B and C, right? So there might be less telemetry. But I could also understand how they might write detections or what detections exist, right? I’ve used all the basic configurations that you can download online, all of the pre-made rules of logging. I understand what gets generated. I spent a lot of time writing detections and so I understand how people who write detections think. It could be very helpful and it’s a lot of the reason I was able to transition so smoothly into red teaming.

Cole French:

So on the flip side of that, what did you have to unlearn? To transition over to the offensive side, what did you have to unlearn? And what does the offensive side require that your threat hunting trained out of you?

Ben Lieberman:

So when I first got into red teaming, I thought I could craft the perfect technique to get around all of the logging, but sometimes you have to take risks and that’s something that I had to accept. It’s like, “Okay, maybe they’ll catch me here, but the reward is greater than the risk of getting caught.” And so you have to make that trade off a lot in red teaming. Not necessarily that I’m going to dump LSASS, but maybe I decide one day that I want to perform a DCSync, which has a very high chance of getting caught, But I think that I can do that activity and perform my follow-up activity before their able to remove me, right? So I need to drop that idea that I could just do everything super silently and take risks.

Cole French:

So you mentioned DCSync. So why don’t you give us a war story just to make this real? So a technique you ran on a red team that, like you mentioned, you took a risk, you knew it would land because you’d watched offender ... Well, I guess you knew defenders would probably miss this, but you also knew there was a possibility, there’s a risk that it might get detected. So yeah, give us a war story, give us a real world example.

Ben Lieberman:

This is probably we use this on every engagement, but token theft. So if you’re not familiar, on Windows, when you log onto let’s say RDP on an activity directory environment to a client machine and there’s an attacker on that client machine, I can then steal the token from that RDP logon and reuse that token to gather Kerberos tickets, certificates. I can use it to authenticate to the machine. It basically becomes usable credentials. The logging for Windows for this mechanism definitely exists, but it can be noisy. And so I knew that defenders constantly missed this and I have yet to be caught stealing a token from a Windows machine and reusing that certificate or that token, or anything that I’ve gathered with it.

Cole French:

Now, curious. Do you have to be on the inside already? I assume so. You have to have some level of access already?

Ben Lieberman:

Yes.

Cole French:

Yeah.

Ben Lieberman:

Yeah. You have to have not necessarily system-level access and we’ll bullet down that simple. But there’s a few permissions that you need on the machine in order to actually take a token from a user that’s logged on.

Cole French:

Well, that is definitely a good war story. And given how prevalent the use of RDP is in organizations, we ... In the world that I live in, we do a lot of assessments on the compliance side, but I mean, I can tell you almost every organization we work with, when we get to security controls that deal with remote access and things like that, we get into RDP and configurations and stuff like that. So while you do have to have access on the inside to perpetrate an attack like this, it is a very prevalent use case for sure.

So, talk to us about living off the land techniques. And why do these consistently surprise defenders? What should hunters be doing differently? This is jumping off from your war story example you gave, but this is kind of a real life thing and I’m just curious to get your thoughts on it.

Ben Lieberman:

Yeah. So if you’re not familiar with living off the land binaries, living of the land binaries or living off the land techniques are native to the operating system which you’re operating on. So if I’m on Windows, a great example of this would be BITS. So it’s Microsoft’s binary for updating Windows. You can actually use this binary to perform similar actions, so like a custom cyber tool or cURL or downloading something, downloading a binary. These are not flagged by traditional security measures because they’re Microsoft-owned tools, right? These are Microsoft SignTools, these were built by Microsoft so a lot of times, these would stand out in a process listing as something that’s not happening all the time, but it’s not going to flag automated detection systems that are looking for unsigned binary, we’ve never seen this hash before, or this is coming from a protocol that doesn’t make sense. Those things, they’re all going to use legitimate utilities that already exist on that operating system.

Cole French:

So the question begs then from that, how do you detect those? How do hunters see that that’s what’s happening? Because like you just mentioned, the attack is being perpetrated specifically to evade detection. It’s using what’s native to the system in a nefarious way, so how do you typically detect something like that?

Ben Lieberman:

So you have to baseline the behavior of the environment, like I was talking about before, but also base-lining the utility. There’s a pretty standard way these utilities are used on Windows. For example, like BITS admin is going to have a pretty standard subset of flags, and arguments and ways that it’s being run. If you could compare that with the way that you’re seeing in your logs it’s being run with malicious flags, going out to a weird IP address, those things should tip you off, right? Now you’re base-lining, instead of an entire environment, you’re base-lining processes, process command line arguments. You’re basically, you’re digging down even further.

Also, having a good understanding of which binaries or native functionalities can be abused for what. So a project with LOLBAS, so if you go to LOLBAS, just Google that online, it has a list of binaries and their functions and the way that they can be abused. Just having a good understanding of those so when you do see them in your logs, you see something like Certutil being ran. Okay, I know Certutil can be abused to download alternate data streams or download a binary, or do some other malicious activity. Okay, is Certutil being used like that in my environment? How do I search for that or how do I write a detection for it?

Cole French:

So again, like you mentioned earlier, understanding the techniques and proceeding accordingly. There’s different ways to perpetrate these, but the techniques are similar. So understanding, knowing the techniques is how you get to the detection part.

Ben Lieberman:

Exactly.

Cole French:

Going on from there, now let’s get into what we want to talk about next, which is purple teaming. And if you go ahead and define that for us. What it is and why it’s a natural home for someone who sits on both the offensive and defensive side of the penetration testing equation.

Ben Lieberman:

Right. And going back to that same coin analogy that I used earlier but, it’s just red and blue team working on the same engagement, same time in order to build up the defensive posture of the blue team. So what happens in a purple team is a red teamer will conduct a technique or engagement and work with the blue team in real time to identify that activity, write detections for it, re-test the detections, and essentially build up the posture on a live engagement working with each other. Instead of it being, you know, “Hey I run a red team engagement and I send you a report in two weeks”, we work with each other live on the engagement to group the security.

Cole French:

And so then you mentioned the blue teaming. So, is the blue team internal?

Ben Lieberman:

Yes.

Cole French:

Ok. So then the red team, purple team, you guys are all working together and, the results of that get fed back into the blue team, which is the internal team working to keep the organization secure. Is that a fair way of describing kind of how these pieces work together?

Ben Lieberman:

It is. And the red team does gain a lot of insights from this as well, right. I have a better understanding of telemetry that my attacks create, maybe I can modify my attack to maybe create a little bit less telemetry and we can do it again and it’s an iterative approach.

Cole French:

I’m curious though, does it impact, or have you seen an impact on the way people approach it, right? Because I would imagine, and I’m not a pen tester so I could be off base on this, but I would imagine you know like we’ve talked about, there’s different thought processes involved in each of these different roles and so I’m curious if actually working together sometimes with some people can make it more challenging, right. There’s like this, the fact that I don’t know as much information, or that I am more on the outside, and things like that, makes it easier to think about whereas, can working together or have you seen… Yeah just talk about, is working together, I can see how it makes it better and we have more information and we share things and stuff like that. But I could see how also, maybe we might miss things cause we’re all kind of on the inside. Does that make sense?

Ben Lieberman:

Typically, the way we run purple teams is we will designate a certain subset of techniques and, kind of what you were saying before, there always is the human aspect to this, right. We’re having to work together with unfamiliar people, people could get their egos hurt, “Oh I thought this would work”, it didn’t work, you know, so those types of things always come into play. But that’s the reality of the world we live in and after working with each other. But this has a much more collaborative effort for a better result than treating them as separate entities, and like I said before, a report 4 weeks later when things are not fresh on people’s minds, when they can read this report and they can dismiss it, right. When we collaborate with each other, in almost every scenario we have better results.

Cole French:

All right. So yeah, you just talked about how it’s a collaborative environment having all these teams working together. So how does it not turn into a blame game? What does the culture need to look like for this purple teaming approach to actually work?

Ben Lieberman:

Yeah. Like I was saying, it’s a human game so egos take over. And when egos take over and the red team decides, “Oh, I want to tear this environment apart, I want all the access to everything, I to do this,” and then the blue teams think they have the world’s best detection, they’re going to catch everything and it becomes almost like a competition in an unhealthy way. Competition can be healthy, but in an unhealthy way where everyone’s egos taking over. They’re upset or they’re angry that the red team found something or the red team is upset that they got caught, that’s not productive. So making sure that egos are controlled and that we’re working together in a collaborative, but competitive and fun way.

Cole French:

We always say in compliance that the objective is to move security forward. That’s the mantra or the place that we come from. It’s not to ... Because we deal with similar things where we point things out or we find things that are not compliant and we have folks that, it goes in two different directions. People become evasive or defensive, or fill-in-the-blank. That similar ... It feels ego driven because it’s like they don’t want anybody telling them what’s wrong with their system. But the reality is, hey, if you want your system to get better, if you want to have better security, then you should want people to point these things out.

So I think it is a subtle, it’s a very subtle thing, but it really, the difference tips pretty noticeably. I think any of us who’ve worked on either side of that have seen what it looks like when folks come at it from an egotistical, defensive point of view and when folks come at it from a I want to move security forward and I want to make improvements to my system. I want to make sure I don’t have any blind spots. They’re very different, but yeah.

Ben Lieberman:

That’s a great example. Compliance is one that you see that in a lot, where people get really hurt that you’re saying something is wrong.

Cole French:

Absolutely, yes. And it’s refreshing when we have folks that we work with that are like, “Oh, I’m really glad you pointed that out and I’m going to make that improvement.” And it’s like, okay, this is when you know you’re working with folks that have an interest in moving things in the right direction because of the right mindset.

Ben Lieberman:

Exactly.

Cole French:

Moving into where we’re at today, everybody’s favorite topic, which is AI. So talk to us about what’s AIs impact been on the offensive side of pen testing, the hunting side of pen testing? Is there a feedback loop between AI and all of these different components? Yeah, just talk a little bit about that.

Ben Lieberman:

Okay. So I’ll talk about some of the ways that we’ve been using AI in order to help our red team engagement. I unfortunately was not a threat hunter when AI is as good as it is now, however I do know that the way we are able to write exploits, create malware, perform our activities, learn how to perform different activities. Using it as a question-and-answer as what capabilities exist for my current spot in the environment. But we’re able to create payloads and techniques, and throw away payloads and techniques that would otherwise have taken weeks in 15 minutes. This includes I want a technique to steal tokens. I want a technique to dump LSASS. I want a technique to do this, I need custom code for this. We’re able to do that in 15 to 20 minutes and that code just works. And if it gets caught or detected or signatured, we just throw it away. We have no personal investment in that tool.

And blue team, they can use AI too, to correlate logs, basically point them in a place to search for investigations faster. Okay, AI flagged this specific section or this host because something happened on this host, you should investigate this. And that can be very helpful, too. But the traditional signature and the traditional antivirus system, we can defeat that in, like I said, 15 minutes.

Cole French:

So then in terms of what you’re seeing today, so you just mentioned, recon, payload generation, phishing, social engineering, all that stuff, would you say it’s the same? You know, where you can create all that kind of stuff, build it all in 15 minutes and tear it all down if you wanted to. Is it the same? Or is there a component of hype out there, I guess, when it comes to AI in penetration testing? Or is it all as good as it ... What is it they say? If it’s too good to be true, it probably is. Is there some of that, or is it too good to be true?

Ben Lieberman:

I would say somewhere in the middle, like most things are. And what I see is you’ll see a lot of people freaking out on Twitter or other places in the security community about how AI is going to be this ultimate hacker and every system is screwed. And in my experience, we’re not even remotely close to that. It still requires somebody with some pretty significant security knowledge in order to perform these activities, even if the AI is telling you. And to do it appropriately where it’s actually going to get through, or not cause any issues or break systems, that still requires a human operator.

Cole French:

Yeah, it’s what I say all the time with folks when I’m having conversations about AI. I’m not using AI necessarily for anything super technical, so I can only imagine that this is even more true on the technical side. But it’s true, AI is as good as the prompts that you give it, right? So you have to have some level of knowledge. You have to know questions to ask or how to answer questions in the right way, all that. So there’s definitely a human art form to it, it isn’t just machine driven. So I would definitely agree with that.

Now, in terms of what you’re seeing out there, offense versus defense. Is AI pushing one versus the other in terms of advantage? Is AI giving the offensive side more of an advantage, the defensive side, is it a mix?

Ben Lieberman:

I would say that in the current way that AI stands, defenders are absolutely losing this battle. And it’s not as if I think they’re the loser forever. However, when a new model drops for example or a new capability, something, I can integrate that into my workflow extremely fast. I’m not bound by rules, regulations, token usage, other things that would cost defenders to have to talk to an office. “Okay, we need to get a budget for this. All right, now we need to get this new tool.” So then they have to do a lot more work in order to keep up.

Some of the ways that we’re using AI, for example, is we created a custom harness that will connect to a machine with a world-class EDR, whether it’s SentinelOne, CrowdStrike. And we’ll say, “Keep running,” for example LSASS dump, “until you find an LSASS dump or a method that does not get detected by the EDR.” And it will continue to do that up until it a payload’s been created that bypasses that specific EDR. So now, we’ve completely blinded defenders with almost no work on our end. And so they don’t have the luxury of those really strong tools that they pay a whole lot of money for because we’ve been able to blindside them so quickly.

Cole French:

That is fascinating and that isn’t even something that I thought of. But man, that’s a great real world example of, yeah, just how being on the defensive side, there’s a lot of disadvantages. Not just the ones that you mention, which I think are pretty obvious, which is you’re subjected to whatever the organizational security parameters and all of that stuff are. There’s a lot of limits. It’s a very defensive posture, which I feel like most of the time when it’s a defensive posture, you’re already at a disadvantage because something’s coming at you. But that’s a great example, that EDR example is a great example of, yeah, it’s like you put up your defense and it sits there, and then that becomes the blanket of security that you’re relying on to the point that you don’t even see. And now, you have tools that can build exploits so quickly and you can deploy them in such a way that they aren’t detected. But then, once they do bypass, they’ll never be detected. So yeah.

Ben Lieberman:

Unless the defenders are actively monitoring logs. And what we’re seeing on environments is that the over-reliance on these EDRs or other security tools is hurting them significantly.

Cole French:

Yeah. And I would say, like I mentioned earlier, I used to work in a SOC. I used to lead a SOC team and that was one of the things, one of the biggest challenges I had in that role was the tendency to think that tools provided an answer for everything. And I mean, this was 15 years ago. Well, not 15 years ago, like 10 years ago before AI and all of that stuff. But even then, it was already an entrenched way of thinking. It’s like, “Well, I got a tool for that.” And then I remember coming onboard and taking inventory of all these different tools, and then finding out that, oh, we have this vulnerability scanning tool that’s great and we’ve deployed it, but literally nobody has sat and actually configured it so it’s just turned on. Nobody’s scanning anything and nobody’s doing anything with the data that’s in this particular tool.

So it’s exactly what you’re saying. Tools are great, but if you’re not doing anything with the data that they’re giving you, if you’re not constantly learning what’s going on in your environment, what good are they? At the end of the day, all it takes is somebody doing something and you’re going to miss it because you’re not looking at it.

Ben Lieberman:

And it could be shocking how much these cost for them to sit unused or not properly managed.

Cole French:

Oh, absolutely. Yeah, definitely. Plenty of tools over the years that I’ve seen sit on the shelf. That’s another thing too, is knowing that there’s folks who, that’s what they love to do, is research and deploy tools because they have an engineering mindset. And then there’s other people who have a data mindset, or they’re interested in how systems work and what the data means, and all that kind of stuff. So we’re going off a little bit on the side here, but it is the importance of, we talked about culture earlier, this is a little bit different view of it. But an important part of the culture of your team is who you have and what are the skillsets that are on your team, right? Everybody needs ... We need a little bit of something put all together. It makes us all stronger.

Cole French:

Cool. So, Ben, I’ve really enjoyed this conversation and appreciate you coming on to share your evolution from the threat hunting side all the way down to purple teaming, and talking through AI and how AI is impacting penetration testing as a whole. But going back to the threat hunting thing, what’s one piece of advice you’d give a threat hunter today whose considering potentially making the jump to the other side?

Ben Lieberman:

Don’t overthink it. If you think that you may want to be a red teamer, there’s plenty of platforms out there, like Hack The Box or TryHackMe, or even getting a certification like OSCP that will help you understand that do I enjoy this thought process of breaking down a systems to its lowest components, and breaking it, and finding different holes in it, or different pathways. If you enjoy that type of thinking and you find you enjoy that thought process and the technical demands of red teaming, then it absolutely could be a treat. It’s very rewarding.

Cole French:

I think that’s great advice. And again, Ben, I appreciate you coming on and I really enjoyed this conversation. I think our listeners will as well.

Ben Lieberman:

Absolutely. Thank you for having me.