WEBVTT 00:00:00.000 --> 00:00:05.840 JACK: Who’s the person with the most power in the workplace? You might think it’s the CEO 00:00:05.840 --> 00:00:10.880 or owner since they can call all the shots and make policy changes that everyone has to adhere 00:00:10.880 --> 00:00:16.920 to. But I think the most powerful person in the workplace might be the sysadmin, the person who 00:00:16.920 --> 00:00:22.640 has administrative access to the core machines that are required for the business to operate. 00:00:22.640 --> 00:00:26.880 They can see what’s in the database and they can read anyone’s e-mail in the whole company, 00:00:26.880 --> 00:00:31.520 and they can see what files are on your computer, and they can sniff all the network traffic from 00:00:31.520 --> 00:00:37.560 your computer to see where you go and what you downloaded. Now, not every network is set up 00:00:37.560 --> 00:00:42.560 like this, where someone can see everything about everyone, and not all networks have one person who 00:00:42.560 --> 00:00:48.520 has all this access. But some networks are set up like this, where one person has control of 00:00:48.520 --> 00:00:53.240 everything. With the press of a button, they can bring business to a halt or potentially 00:00:53.240 --> 00:00:59.680 reroute customer payments or pay checks to them. It’s crazy how much power they have. 00:00:59.680 --> 00:01:06.360 So, it goes without saying; you never, ever want some unauthorized person to have admin 00:01:06.360 --> 00:01:12.800 access to your network, because using this power maliciously can be incredibly destructive to your 00:01:12.800 --> 00:01:19.720 business. But there’s another person who also has a lot of power that we sometimes forget about; 00:01:19.720 --> 00:01:25.760 that’s the overnight janitor, the person who has a key to the building and every room in the office, 00:01:25.760 --> 00:01:31.200 including the CEO’s office. On top of that, they’re always there when nobody else is, 00:01:31.200 --> 00:01:37.080 which gives them the opportunity and capability for some serious spying. The only thing they’d 00:01:37.080 --> 00:01:42.400 need is the motivation, and what’s even crazier is that some of these janitorial services have 00:01:42.400 --> 00:01:49.840 many businesses that they service each night, so that’s quite the key ring to have access to, 00:01:49.840 --> 00:01:57.360 especially in the right parts of town. Imagine if the janitor’s key ring got into the wrong hands, 00:01:57.360 --> 00:02:03.400 into the hands of someone with a lot of motivation and malicious intent. What if 00:02:03.400 --> 00:02:14.738 that someone was extremely skilled at computers and hacking? That would surely be trouble. 00:02:14.738 --> 00:02:16.880 (INTRO): [INRO MUSIC] These are true stories from the dark side 00:02:16.880 --> 00:02:40.720 of the internet. I’m Jack Rhysider. This is Darknet Diaries. (INTRO MUSIC ENDS) 00:02:40.720 --> 00:02:43.780 JACK: Start with your name; what’s your name and what do you do? 00:02:43.780 --> 00:02:47.680 FABIO: Yeah, so my name is Fabio Viggiani. I’m an incident responder, 00:02:47.680 --> 00:02:50.100 threat analyst, and red teamer at Truesec. 00:02:50.100 --> 00:02:54.160 JACK: Red teaming means simulating an attack to make sure a network’s 00:02:54.160 --> 00:03:00.000 defense system actually works, which is fun to come and attack a network. Fabio does that, 00:03:00.000 --> 00:03:02.820 but that’s not what this story is about. 00:03:02.820 --> 00:03:06.440 FABIO: Right, so I work a lot with incident response. 00:03:06.440 --> 00:03:09.740 That’s one of my primary areas that I work with. 00:03:09.740 --> 00:03:14.000 JACK: Fabio works for a company called Truesec which is based in Sweden. When 00:03:14.000 --> 00:03:18.960 businesses or organizations get attacked, they can call Truesec up to come investigate and 00:03:18.960 --> 00:03:23.440 remediate the issue. That’s when Fabio will go onsite to a customer’s location 00:03:23.440 --> 00:03:27.120 to help them out. He’s done a lot of this type of incident response work. 00:03:27.120 --> 00:03:34.160 FABIO: So, basically anything from ransomware to espionage. I’m a technical lead for the 00:03:34.160 --> 00:03:38.560 forensics team. When we do incident response, we have a forensics team. That’s basically 00:03:38.560 --> 00:03:42.600 the investigation to understand what has happened, what the threat actor has done, 00:03:42.600 --> 00:03:47.200 how they’ve came – how they come into the environment, and if there is any persistence, 00:03:47.200 --> 00:03:51.520 anything to remove, and how to clean that up. Another team is the recovery team, 00:03:51.520 --> 00:03:56.760 where they do all the infrastructure work. So, in a ransomware case for example, 00:03:56.760 --> 00:04:03.120 there’s a lot of recovery and rebuilding to be done, so we work very close with them to tell them 00:04:03.120 --> 00:04:06.720 these are the things that need to be cleaned out, these are the things that are safe to restore, 00:04:06.720 --> 00:04:11.060 this is the date that the system can be restored, because we have verified there is nothing there. 00:04:11.060 --> 00:04:14.280 JACK: Fabio has been with this company for eight years now and 00:04:14.280 --> 00:04:18.320 in that time he’s seen lots of network intrusions and handled 00:04:18.320 --> 00:04:23.700 many incidents. But there’s one incident in particular that he’ll always remember. 00:04:23.700 --> 00:04:29.240 FABIO: Yeah, so, it was summer of 2016 and we got a call from this customer, 00:04:29.240 --> 00:04:31.826 from this company here in Sweden. 00:04:31.826 --> 00:04:34.880 JACK: [MUSIC] Now, Fabio had actually done work for this company before, 00:04:34.880 --> 00:04:37.680 advising them on how to secure their network better. 00:04:37.680 --> 00:04:41.320 FABIO: Reviews of their security and penetration testing of some 00:04:41.320 --> 00:04:43.320 of their applications, things like that. 00:04:43.320 --> 00:04:47.480 JACK: He didn’t want to say what company this was, because companies really don’t like talking about 00:04:47.480 --> 00:04:52.680 that time when they were attacked. It’s also not cool for him to go and handle an incident for a 00:04:52.680 --> 00:04:57.720 customer and then blab about it to the world on my show. So, he can’t say who this company was, 00:04:57.720 --> 00:05:01.840 but I do know enough about this company that I want you to picture a large, 00:05:01.840 --> 00:05:07.520 typical office-type business. There’s many offices, they’re all pretty big, and what’s 00:05:07.520 --> 00:05:12.440 notable here is that they have thousands and thousands of computers in the network. 00:05:12.440 --> 00:05:15.720 FABIO: What we knew from the beginning is that they had been contacted by the 00:05:15.720 --> 00:05:21.120 Swedish Security Service because one of their system – one of the systems had 00:05:21.120 --> 00:05:25.480 been talking to a command and control server somewhere on the internet. It was located at 00:05:25.480 --> 00:05:30.860 a foreign state. That’s basically all we knew when we got that call. 00:05:30.860 --> 00:05:36.880 JACK: Now, this is actually a big deal. When the Swedish Security Service calls you up to tip you 00:05:36.880 --> 00:05:42.680 off of a potential problem, you should definitely sit up straight in your chair and ask for help, 00:05:42.680 --> 00:05:47.640 because the Swedish Security Service is the government agency in Sweden that investigates 00:05:47.640 --> 00:05:52.400 espionage and counter-terrorism and any threats against national security. It’s 00:05:52.400 --> 00:05:58.600 sort of like the FBI in the US. So, you can imagine if the FBI calls you to say hey, 00:05:58.600 --> 00:06:03.400 one of the computers in your network is reaching out to a really bad computer on the internet, 00:06:03.400 --> 00:06:08.000 you’re gonna want to spring into action. What the Swedish Security Service was saying was 00:06:08.000 --> 00:06:13.720 a computer at this company was talking to a known bad actor, a command and control server, 00:06:13.720 --> 00:06:17.260 and they gave the IPs that were involved with this, but that was about it. 00:06:17.260 --> 00:06:23.560 FABIO: Just the fact that this is information coming from the Security Service tells you that 00:06:23.560 --> 00:06:30.840 it’s not just random command and control server by some – whatever criminal group 00:06:30.840 --> 00:06:36.360 that is doing ransomware. They don’t tell that. They focus on nation state, 00:06:36.360 --> 00:06:40.400 so when you get a call from them, it’s probably related to something bigger. 00:06:40.400 --> 00:06:45.600 JACK: But even though this isn’t much information, it’s coming from such a reliable source that you 00:06:45.600 --> 00:06:50.120 can assume a few things. [MUSIC] First, they said a computer was reaching out to a 00:06:50.120 --> 00:06:54.960 command and control server. When malware infects a computer, it needs instructions on what to do once 00:06:54.960 --> 00:06:59.840 it’s there. Sometimes it’s built into the malware. Other times it calls out to another computer and 00:06:59.840 --> 00:07:04.640 says what should I do now, or here’s what I have. This is what a command and control server is, 00:07:04.640 --> 00:07:08.800 something that can interact with an infected computer. The fact that a computer is reaching 00:07:08.800 --> 00:07:13.600 out to a command and control server at all means it’s probably infected with something like malware 00:07:13.600 --> 00:07:18.760 that you definitely want to remove. But this is the Swedish Security Service notifying them, 00:07:18.760 --> 00:07:23.040 which might mean that this is either a very serious threat actor or it could mean that 00:07:23.040 --> 00:07:28.520 other companies in Sweden have been hit by this too, and notified the Swedish Security Service 00:07:28.520 --> 00:07:33.520 who then looked into it and found this company may also be infected. Anyway, all that is to 00:07:33.520 --> 00:07:39.480 say is that this was such a reliable tip that it really did warrant calling up Fabio and telling 00:07:39.480 --> 00:07:44.920 him to come immediately, and that’s just what he did. He quickly started shoving gear into his bag. 00:07:44.920 --> 00:07:52.040 FABIO: Well, a couple of laptops with all the tooling and everything needed, and then kind of 00:07:52.040 --> 00:08:01.120 equipment like external discs, different type of USB devices for transfers, and usually a lot of 00:08:01.120 --> 00:08:08.240 storage is needed. But it’s also – in this case, it was actually physically close by, so we didn’t 00:08:08.240 --> 00:08:14.640 have to plan so far ahead. So, just the initial things; a lot of storage and a lot of tooling. 00:08:14.640 --> 00:08:20.145 JACK: He packed it all up, jumped in the car, and drove to this customer’s location. 00:08:20.145 --> 00:08:23.400 FABIO: [MUSIC] Initially, they told us that they got this call from the Swedish Security Service 00:08:23.400 --> 00:08:27.800 because they had this connection to a malicious command and control server on the internet, 00:08:27.800 --> 00:08:35.480 and the rest of the details we would get there. So, when we got there, we got in a room with them, 00:08:35.480 --> 00:08:42.160 and they gave us the information that they had received, which was three things; 00:08:42.160 --> 00:08:47.960 two IP addresses of command and control servers that their infrastructure has been 00:08:47.960 --> 00:08:58.400 seen communicating with, time windows, so when those connections started and ended, and internal 00:08:58.400 --> 00:09:03.920 hosting of one of their servers that apparently has been communicating with those IP addresses. 00:09:03.920 --> 00:09:07.720 JACK: This is a good set of clues to start with, especially having that internal host 00:09:07.720 --> 00:09:12.380 name that’s suspected to be infected. Fabio honed in on that server first. 00:09:12.380 --> 00:09:17.280 FABIO: So, obviously the first thing is asking about that server and understand, first of all, 00:09:17.280 --> 00:09:24.480 what type of server it is and then get access to it in whatever state it is and 00:09:24.480 --> 00:09:30.680 take it from there. Turns out it was not just some random server. It was a pretty important 00:09:30.680 --> 00:09:41.340 server because it was one of the jump servers that the MSP used to manage this customer. 00:09:41.340 --> 00:09:48.000 JACK: Oh, well, that’s interesting. So, MSP stands for Managed Service Provider. 00:09:48.000 --> 00:09:53.520 This company outsourced the monitoring and management of most of their servers to another 00:09:53.520 --> 00:09:59.600 company to take care of, which is this MSP. The MSP is who will keep servers patched up 00:09:59.600 --> 00:10:05.680 and make configuration changes for this company. They’ll also monitor for faults and incidents. So, 00:10:05.680 --> 00:10:10.240 if one server in this network had a high CPU, it would alert the MSP, 00:10:10.240 --> 00:10:16.080 and then someone from the MSP would log into that server and fix the issue. But this MSP 00:10:16.080 --> 00:10:21.840 managed all these servers remotely, from another country even, so they needed a good, 00:10:21.840 --> 00:10:27.360 reliable way to access all these computers in this network. To do that, they set up a VPN 00:10:27.360 --> 00:10:31.800 to a server that they could use to jump off of when they needed to get in this network, 00:10:31.800 --> 00:10:37.440 which is a jump server. So, when someone in the MSP needed to check out a server in this network, 00:10:37.440 --> 00:10:43.200 first they connected into this jump server which then had access to all the servers in the network, 00:10:43.200 --> 00:10:48.680 and it was this jump server that was reaching out to a known command and control server. Since this 00:10:48.680 --> 00:10:54.760 jump server is used by an MSP, it meant it had access to pretty much every important server in 00:10:54.760 --> 00:11:00.840 the network. Of all the servers to be infected, this was probably one of the worst possible ones. 00:11:00.840 --> 00:11:08.000 FABIO: That is a really good place to be for a threat actor. First of all, we started asking for 00:11:08.000 --> 00:11:12.060 access ‘cause the system was up and running, so we just asked can we get access to it? 00:11:12.060 --> 00:11:16.360 JACK: But the problem was that server was fully controlled by the MSP, 00:11:16.360 --> 00:11:19.160 so Fabio had to call them to get access to it. 00:11:19.160 --> 00:11:23.680 FABIO: They were very reluctant from the beginning to give us access, saying things 00:11:23.680 --> 00:11:29.800 like we have SLAs with the customer. We can’t just give access to anyone. If things go down, 00:11:29.800 --> 00:11:35.000 then it’s our responsibility and all that stuff, which is stuff we hear, but it never works out 00:11:35.000 --> 00:11:40.160 ‘cause ultimately it’s the customer system. So, we said okay, fine, we’ll get back to the customer 00:11:40.160 --> 00:11:46.680 and figure that out. But in the meantime, if you don’t want us to access the live system, 00:11:46.680 --> 00:11:52.060 which is fine for now, if you can take a disc image and a memory dump so we can start from that. 00:11:52.060 --> 00:11:57.440 JACK: Now, you might think this is suspicious for the MSP to not help Fabio get access to it, 00:11:57.440 --> 00:12:01.120 but that’s typical in situations like this. Managed Service Providers provide 00:12:01.120 --> 00:12:03.720 service to handle computer problems for the customer, 00:12:03.720 --> 00:12:07.040 and Fabio was not the customer. He worked for a different company, Truesec. 00:12:07.040 --> 00:12:13.320 FABIO: Normally, any type of service providers to an organization that is breached, they may 00:12:13.320 --> 00:12:20.320 tend to be defensive because they don’t want to – ‘cause they might be the reason or they 00:12:20.320 --> 00:12:24.720 might be indirectly the reason. Maybe they haven’t really done what they were supposed to be doing, 00:12:24.720 --> 00:12:30.160 they haven’t been respecting the agreements with their customers, so they feel like – they feel a 00:12:30.160 --> 00:12:34.560 little bit threatened. If we find something that shows they’ve done something badly, 00:12:34.560 --> 00:12:39.680 then this could be bad for them. So, that’s why some people may be defensive. 00:12:39.680 --> 00:12:45.080 JACK: That’s another reason for their hesitation. Once you start throwing around the b word, 00:12:45.080 --> 00:12:50.880 breach, the MSP is going to perk up and be extra-careful about what they’re doing, 00:12:50.880 --> 00:12:55.040 because for them, it’s a bit alarming to hear that their customer may have been 00:12:55.040 --> 00:13:00.200 breached. But ultimately, this was the company’s server, not the MSP’s, 00:13:00.200 --> 00:13:06.960 so the company simply demanded that the MSP give Fabio access, and they let him in. With that, 00:13:06.960 --> 00:13:10.640 he was in the system and was capturing the data he needed to do his investigation. 00:13:10.640 --> 00:13:15.600 FABIO: We’ve got the disc image and a memory dump which is really all you need, 00:13:15.600 --> 00:13:18.860 right, for doing a forensic investigation. 00:13:18.860 --> 00:13:24.760 JACK: A disc image is an exact copy of the entire hard drive, and a memory dump is a copy of what’s 00:13:24.760 --> 00:13:30.240 currently stored in the system’s RAM memory, which will tell you what programs are running, including 00:13:30.240 --> 00:13:35.200 any malware. Okay, so, Fabio has been doing this type of work for a while and gets right to work 00:13:35.200 --> 00:13:40.200 analyzing the disc image. The disc image was put onto an external hard drive and so, he just mounts 00:13:40.200 --> 00:13:46.680 it to his computer as an external drive. [MUSIC] But what do you do with this? Where do you even 00:13:46.680 --> 00:13:52.480 look on this hard drive to try to find malware? Yeah, sure, you could run an antivirus scan on it, 00:13:52.480 --> 00:13:58.320 but this jump server already had antivirus running on it, and all was quiet. Nothing had triggered. 00:13:58.320 --> 00:14:03.720 So, now what? Well, this is why you need someone who’s trained in digital forensics, 00:14:03.720 --> 00:14:09.520 and what makes a good digital forensics analyst is the ability to spot things that aren’t normal. But 00:14:09.520 --> 00:14:14.840 in order to know what’s not normal, you really have to know what is normal. So, 00:14:14.840 --> 00:14:18.600 it’s incredibly important for someone who wants to be good at digital forensics to 00:14:18.600 --> 00:14:24.360 know how computers normally work inside and out. What processes are normally supposed 00:14:24.360 --> 00:14:28.240 to be running? Where do those programs typically live? What stuff belongs in the 00:14:28.240 --> 00:14:33.520 Windows directory and what stuff doesn’t? This jump server was a Windows computer, 00:14:33.520 --> 00:14:38.660 and Fabio is pretty familiar with Windows, so he got right to work looking through files manually. 00:14:38.660 --> 00:14:43.400 FABIO: You just mount it, you do a read-all amount on your computer, and you just have 00:14:43.400 --> 00:14:47.520 a quick look at it ‘cause just from experience, you know where these type of things tend to be. 00:14:47.520 --> 00:14:52.440 JACK: The first place he checked was if anything was in the temp folder. He likes checking here 00:14:52.440 --> 00:14:57.400 often because this is where intruders like to stage files and put things. The temp folder 00:14:57.400 --> 00:15:03.000 is a nice spot to stash stuff temporarily, and that is what I mean. Fabio doesn’t start with 00:15:03.000 --> 00:15:08.840 some elaborate scan that might take hours. He manually checks a few places first just to see 00:15:08.840 --> 00:15:14.000 if he can spot anything himself right away. So, he checks the temp folder straight away. 00:15:14.000 --> 00:15:22.360 FABIO: C:\temp, and there was a bunch of files there. [MUSIC] We found a file with 00:15:22.360 --> 00:15:29.740 a pretty obvious name. It was called thehostingoftheserver.mimikatz.hash. 00:15:29.740 --> 00:15:34.560 JACK: Uh-oh. Within the first minute of having access to this server, he already 00:15:34.560 --> 00:15:42.440 found terrible news. Mimikatz was executed on this computer, and that’s bad. See, Windows has 00:15:42.440 --> 00:15:47.800 a major flaw in the way it handles passwords. When you log into a Windows computer itself, 00:15:47.800 --> 00:15:51.960 you have to enter a username and password, and that username and password that you just typed 00:15:51.960 --> 00:15:58.720 in gets stored in memory, often in clear text. Mimikatz is a tool that goes to the 00:15:58.720 --> 00:16:05.240 exact spot in memory and grabs the password so that anyone can see it in clear text. So, 00:16:05.240 --> 00:16:09.680 if you can successfully run Mimikatz, it means that you can see the username and password of 00:16:09.680 --> 00:16:16.400 every single login to this computer since it was rebooted last. There is no reason 00:16:16.400 --> 00:16:22.560 for this MSP to have run Mimikatz, which means this was a smoking gun that yes, 00:16:22.560 --> 00:16:28.520 a threat actor was here and tried to get usernames and passwords of the users of this machine. 00:16:28.520 --> 00:16:36.600 FABIO: That was the output file of the Mimikatz execution. So, we open it up and it contained 00:16:36.600 --> 00:16:43.760 nearly a hundred credentials of users that had been logged on to that system. The credentials 00:16:43.760 --> 00:16:50.600 were in clear text. They can be hashed, but if the system is a little bit older and is 00:16:50.600 --> 00:16:56.200 unpatched and there is no protection for caching passwords in clear text in memory, 00:16:56.200 --> 00:17:00.840 then Mimikatz would be able to extract any clear text, and that’s what we saw in that output file. 00:17:00.840 --> 00:17:08.160 So, we had about a hundred users with their clear text passwords in that text output file, 00:17:08.160 --> 00:17:15.960 including several Active Directory domain administrators, [MUSIC] which immediately 00:17:15.960 --> 00:17:22.240 kind of escalated the whole incident, right, because then you have evidence 00:17:22.240 --> 00:17:31.060 that someone had access to all the highest privilege credentials in Active Directory. 00:17:31.060 --> 00:17:37.640 JACK: Mimikatz found a lot of passwords, and that is not good. You can assume that 00:17:37.640 --> 00:17:42.320 all these usernames and passwords are now in the hands of whatever attacker 00:17:42.320 --> 00:17:48.800 that got into this computer. But again, this was a MSP jump server which had connectivity 00:17:48.800 --> 00:17:54.560 to pretty much every important server in the network. Now, a finding like this is 00:17:54.560 --> 00:17:59.920 scary. It means this has gotten very serious, and it’s like finding a bomb in the building. 00:17:59.920 --> 00:18:04.920 When Fabio and his two colleagues found this, the temperature in the room went up. 00:18:04.920 --> 00:18:09.000 FABIO: It definitely did, and that also made it so we moved to a much bigger room, like 00:18:09.000 --> 00:18:14.560 an actual War Room with screens and everything. Initially, you never know what you come across, 00:18:14.560 --> 00:18:22.200 so we were in this small room, three, four people just looking at this. When these type 00:18:22.200 --> 00:18:28.600 of things happen, then we make sure this gets escalated and we establish a proper working 00:18:28.600 --> 00:18:33.000 environment for a big incident, because this was obviously going to be a big incident. Again, 00:18:33.000 --> 00:18:38.160 we’re talking about an organization with thousands and thousands of systems, 00:18:38.160 --> 00:18:44.120 and you just have just identified that they are very likely fully compromised. So, 00:18:44.120 --> 00:18:48.480 you know that you’re gonna directly or indirectly have to go through everything. 00:18:48.480 --> 00:18:52.000 It’s gonna take a while. It’s gonna take a lot of people, it’s gonna take some time. 00:18:52.000 --> 00:18:56.480 JACK: Stay with us because we’re gonna take a quick break, but after we get back, 00:18:56.480 --> 00:19:02.960 Fabio gets some answers. Finding evidence that an unauthorized person logged into a server, 00:19:02.960 --> 00:19:07.120 ran Mimikatz, collected lots of usernames and passwords, and then pulled them out 00:19:07.120 --> 00:19:12.400 of the network is really bad. It’s like finding a smoking gun in the network, 00:19:12.400 --> 00:19:16.560 but how did it get there and what did it shoot at? The business leaders needed to be 00:19:16.560 --> 00:19:20.440 called in at this point to be made aware of this, because this could potentially 00:19:20.440 --> 00:19:25.080 have big consequences that can disrupt business. Fabio stuck his head, nose, 00:19:25.080 --> 00:19:29.240 and hands back down into his laptop like a dog trying to dig a hole in the ground. 00:19:29.240 --> 00:19:32.280 FABIO: Then of course, you still need to go through the thorough process, 00:19:32.280 --> 00:19:36.880 then you use tooling; like, you build a timeline of all the files that have been created, modified, 00:19:36.880 --> 00:19:40.640 accessed on the disc, and you correlate that with the time of connection to the 00:19:40.640 --> 00:19:45.120 command and control server and say okay, what files were created or touched around the time 00:19:45.120 --> 00:19:49.240 that the connection started? Then you narrow it down to the – all the new files or newly 00:19:49.240 --> 00:19:53.540 modified files on disc around that time. Then you just go through that. 00:19:53.540 --> 00:19:58.240 JACK: Well, he makes it sound easy, but that’s actually a long, arduous task. A thorough scan 00:19:58.240 --> 00:20:03.200 using a tool can take hours or more just to go through all the files and check each one. 00:20:03.200 --> 00:20:07.240 Then when you have it narrowed down to a few directories or files that were changed during 00:20:07.240 --> 00:20:12.600 that specific time, you need to analyze those files more carefully either by hand or using other 00:20:12.600 --> 00:20:18.440 tools. Now, it’s one thing for a digital forensics analyst to be able to find problems on a computer, 00:20:18.440 --> 00:20:23.040 but what makes a really good analyst is the ability to communicate the issues 00:20:23.040 --> 00:20:28.905 to people effectively. Fabio had to give instructions to people on what to do next. 00:20:28.905 --> 00:20:34.440 FABIO: [MUSIC] First of all, from our side, we need more people to get ready to start looking 00:20:34.440 --> 00:20:41.480 into a lot of other systems. So, this has to scale somehow, because you know – already looking 00:20:41.480 --> 00:20:49.720 at that, you know there is gonna be more systems affected. It’s not just gonna be the server. So, 00:20:49.720 --> 00:20:55.080 and then at a higher level – and that’s not something that I would do directly in that case, 00:20:55.080 --> 00:21:01.240 for example. Then, I would get back to our incident manager and say hey, this is what we 00:21:01.240 --> 00:21:08.160 found, so this needs to be communicated to the management level at the customer to make sure 00:21:08.160 --> 00:21:18.400 that they understand what we are finding here and that there will be consequences of a certain – of 00:21:18.400 --> 00:21:25.880 different type when you know – even if you don’t know exactly what happened yet, you know that they 00:21:25.880 --> 00:21:31.640 had that level of access inside your organization, so management needs to know that right away 00:21:31.640 --> 00:21:38.840 because they need to start working on controlling the situation from all different perspective; 00:21:38.840 --> 00:21:42.720 from business perspective, from a marketing, communication perspective and all that stuff. 00:21:42.720 --> 00:21:47.560 So, they need to know as soon as possible. So, we had a couple of parallel activities going 00:21:47.560 --> 00:21:54.560 on. One was looking at the disc and one was looking at the memory. We had the IP address 00:21:54.560 --> 00:22:03.200 of the command and control server. Something you can get out from memory is network connections, 00:22:03.200 --> 00:22:12.440 current or historical, if they’re still left in memory somewhere. So, we did look for that 00:22:12.440 --> 00:22:22.040 IP address and we found a process that had been connected with that IP address. The name of the 00:22:22.040 --> 00:22:35.080 process was vba32arkit.exe, which is not something I recognized immediately. But we took the hash, 00:22:35.080 --> 00:22:42.080 that binary, and checked it, and turned out to be a legitimate software called 00:22:42.080 --> 00:22:54.000 Vba32 AntiRootkit Scanner, which is ironic in a way. It was legitimate software looking for 00:22:54.000 --> 00:23:02.160 rootkits and malware on systems. It was scanning the system looking for malware. That process, 00:23:02.160 --> 00:23:05.320 it was a signed binary by this company for use in this software. 00:23:05.320 --> 00:23:11.440 JACK: Signed binary is a way to show authenticity of a file. That file really was actual software 00:23:11.440 --> 00:23:17.080 that detects malware from a legitimate company. It specifically looked for rootkits, which is 00:23:17.080 --> 00:23:22.800 malware trying to get access to something it’s not supposed to. But it was this anti-rootkit software 00:23:22.800 --> 00:23:27.960 that was connecting to the bad IP, the command and control server that the Swedish Security Service 00:23:27.960 --> 00:23:36.200 told them about. That is very strange, but it’s also a clue as to what might be going on here. 00:23:36.200 --> 00:23:40.760 FABIO: Right, so, we looked at this file a little bit more and there were a couple things sticking 00:23:40.760 --> 00:23:48.760 out immediately. First thing is that it was in a very unusual location. It was under C:\Windows\Web 00:23:48.760 --> 00:23:55.320 which is a folder that exists, but it doesn’t have that type of software in there. So, just 00:23:55.320 --> 00:24:05.320 having that binary located in that directory was strange. Next to that file, there were a few file; 00:24:05.320 --> 00:24:12.440 a couple of DLLs and another couple of files. So, that immediately smelled like DLL side-loading. 00:24:12.440 --> 00:24:16.600 JACK: DLL side-loading is an interesting attack technique. Here’s 00:24:16.600 --> 00:24:21.200 how it works; [MUSIC] in Windows, programs often require more than one file to run, 00:24:21.200 --> 00:24:26.200 like a driver or a .config file or a DLL file. A DLL file is just some extra 00:24:26.200 --> 00:24:31.000 data that the program needs in order for it to load properly. When the program tries to load, 00:24:31.000 --> 00:24:37.280 it’ll try to find the required files. This process can be manipulated by placing a malicious DLL file 00:24:37.280 --> 00:24:42.840 in a certain place so the program will load it into memory. Programs have sort of an order of 00:24:42.840 --> 00:24:49.040 operation of how they look for their needed DLLs, and this can be exploited. So, this particular 00:24:49.040 --> 00:24:55.080 DLL had instructions to communicate with an outside server. This type of attack is much 00:24:55.080 --> 00:25:00.800 harder for antivirus scans to pick up because the programs that are running are all fine and good, 00:25:00.800 --> 00:25:06.500 but it was the files that those programs called to start running is where the problem was. 00:25:06.500 --> 00:25:11.160 FABIO: It’s a very well-known technique and very effective. Also very easy to do 00:25:11.160 --> 00:25:14.840 because there’s all kinds of software vulnerable to get on side-loading, 00:25:14.840 --> 00:25:20.760 because it’s practically a very hard issue to fix. Not technically; 00:25:20.760 --> 00:25:24.960 technically you just need to verify you’re loading the right DLLs, but in practice when 00:25:24.960 --> 00:25:29.320 you have software with all kinds of DLLs, all kinds of updates, just maintaining that 00:25:29.320 --> 00:25:34.840 is really challenging and expensive. So, there are a lot of products that don’t do it right. 00:25:34.840 --> 00:25:39.400 JACK: So, Fabio examined this DLL and yes, sure enough, 00:25:39.400 --> 00:25:44.200 this normal and benign program was loading this malicious DLL file. 00:25:44.200 --> 00:25:49.440 FABIO: It was very simple. It only had one job; when it was loaded, 00:25:49.440 --> 00:25:55.880 it would read another file from disc which was just a binary BLOB. It was actually encrypted 00:25:55.880 --> 00:26:04.240 data. It would decrypt it with a key that was stored into it. It was stored inside the DLL. 00:26:04.240 --> 00:26:11.200 It would decompress it and then it would just load it and execute it in memory. So, again, 00:26:11.200 --> 00:26:16.640 this would be done within the context of the legitimate binary. So, if you look at who is 00:26:16.640 --> 00:26:21.720 doing what on the system, you would see that it’s this process that now is executing this code. 00:26:21.720 --> 00:26:25.560 JACK: Now that they know this threat actor likes to inject itself into known, 00:26:25.560 --> 00:26:30.840 good processes, they start looking for more instances of DLL side-loading. 00:26:30.840 --> 00:26:38.160 FABIO: We found three more instances of DLL side-loading implants, and they would 00:26:38.160 --> 00:26:46.440 start the same type of malware, but connecting to different command and control servers. You could 00:26:46.440 --> 00:26:52.520 also see that they had been started at different points in time, and there was actually weeks in 00:26:52.520 --> 00:27:00.840 between these executions. So, we suddenly got our timeline a few weeks back, which means the first 00:27:00.840 --> 00:27:06.460 instance of this RAT had already been running for – I think it was more than a month at that time. 00:27:06.460 --> 00:27:12.120 JACK: So, actually, this investigation is going very well so far. Yes, it’s always bad to find 00:27:12.120 --> 00:27:16.400 that someone came and ate your lunch when you weren’t looking, but now they have lots of 00:27:16.400 --> 00:27:21.720 pieces of evidence to go with. So, they take what they’ve learned from this and start spreading out 00:27:21.720 --> 00:27:27.520 their search to find out what other computers might have these same indicators of compromise. 00:27:27.520 --> 00:27:33.280 FABIO: Yeah, so there are two directions you move from here, [MUSIC] and we normally have different 00:27:33.280 --> 00:27:38.240 people working parallel on different tracks. One is to figure out what happened after that, 00:27:38.240 --> 00:27:43.400 what other systems had been affected after this server had been affected, 00:27:43.400 --> 00:27:45.720 and the other track is how they got in in the first place. 00:27:45.720 --> 00:27:50.400 JACK: So, they start searching everywhere to see if these DLL files were on any other computer 00:27:50.400 --> 00:27:54.680 in the network and see what connections were to and from this computer during that 00:27:54.680 --> 00:28:00.080 time. They basically were just following the path of evidence. But wait a minute; 00:28:00.080 --> 00:28:05.760 hold on. Discovering these malicious DLL files means for certain there was 00:28:05.760 --> 00:28:12.360 an unwanted intruder in this server doing things, pushing buttons, executing programs, 00:28:12.360 --> 00:28:18.640 so wouldn’t you want to immediately kick out all users and lock this system down 00:28:18.640 --> 00:28:22.760 so that whatever malicious person has access to this can’t do anything else? 00:28:22.760 --> 00:28:29.200 FABIO: That is always a call that needs to be made and there is no default answer. It depends, 00:28:29.200 --> 00:28:37.240 but looking at the situation here, that had been running for weeks, right? So, if he would be up 00:28:37.240 --> 00:28:43.560 for a couple more hours while we investigate, the chances that something specifically happen 00:28:43.560 --> 00:28:50.960 within those couple hours is low, provided that you haven’t given away that you’re onto 00:28:50.960 --> 00:28:55.160 them. That’s why it’s so important that the right actions are taken from the beginning, 00:28:55.160 --> 00:28:59.760 and especially that the wrong action are not taken from the beginning when you communicate this, 00:28:59.760 --> 00:29:06.320 because then you’re gonna have to take these tough decisions; say okay, do we think that they know? 00:29:06.320 --> 00:29:15.520 If they do, then we may prefer to shut this down right now so they can’t hide better now. Or if you 00:29:15.520 --> 00:29:21.040 have a feeling like we’ve been very stealthy in our investigation, they probably don’t know, then 00:29:21.040 --> 00:29:28.240 it’s actually easier for us to work with something that is ongoing given the time window here. It’s 00:29:28.240 --> 00:29:32.260 been going on for weeks. What are the odds that it’s gonna happen within the next couple hours? 00:29:32.260 --> 00:29:37.040 JACK: In this case, the call was to allow the investigation to continue a little while longer 00:29:37.040 --> 00:29:41.800 without wiping this compromised server down and disinfecting it, because the main thing 00:29:41.800 --> 00:29:47.200 they still needed to figure out was how did this malware get on here? It didn’t show up by itself. 00:29:47.200 --> 00:29:52.400 Someone put it there, so they looked through the logs and pretty easily discovered that someone 00:29:52.400 --> 00:29:58.240 simply logged into this computer normally, through Remote Desktop, and put it there, which is not 00:29:58.240 --> 00:30:03.280 an exploit or a hack at all. It means someone had the username and password to get into this server. 00:30:03.280 --> 00:30:09.080 They logged into it and put the malware on. Okay, so, they know how it got on and they know what 00:30:09.080 --> 00:30:14.800 files it left on the system, but they’re curious to see if there’s anything currently running now. 00:30:14.800 --> 00:30:20.300 That’s where memory analysis comes in, because whatever’s in RAM is what’s actively running. 00:30:20.300 --> 00:30:23.960 FABIO: So, we were looking at the memory analysis. We found a few interesting things in there. 00:30:23.960 --> 00:30:30.240 JACK: They found that yeah, there was malware in the memory, but as they looked at it closer, 00:30:30.240 --> 00:30:36.160 they found a note in the malware, [MUSIC] which was odd. This appeared to be a note 00:30:36.160 --> 00:30:41.640 for the forensic investigators that were looking for this malware, like Fabio. 00:30:41.640 --> 00:30:47.280 FABIO: Which said like this; I have it written here. It said, ‘Have your bosses given you 00:30:47.280 --> 00:30:53.220 the space to try to be a hacker? Come on man, don’t kill me.’ That’s what it said. 00:30:53.220 --> 00:31:01.120 JACK: ‘Have your bosses given you the space to be a hacker? Come on man, don’t kill me.’ What 00:31:01.120 --> 00:31:09.480 in the world does that mean? It’s not clear. Does it mean to tell your boss you want to be a hacker 00:31:09.480 --> 00:31:15.600 or to leave the malware here and just ignore it? This message is confusing. Whoever wrote it 00:31:15.600 --> 00:31:21.840 missed the chance at saying something effective. But if it does mean to leave the malware there, 00:31:21.840 --> 00:31:25.680 then it totally reminds me of this scene from the TV show Mr. Robot, 00:31:25.680 --> 00:31:29.440 where Elliot is investigating an infected computer after an attack. 00:31:29.440 --> 00:31:31.480 ELLIOT: I’m gonna take a look at the infected server, 00:31:31.480 --> 00:31:35.400 okay? Give me a minute. [MUSIC] They must have left a mark or something. Every hacker 00:31:35.400 --> 00:31:41.360 loves attention. They don’t just do DDoS attacks for no reason. This is it. Is that 00:31:41.360 --> 00:31:45.760 supposed to be a joke? This was way too easy. They didn’t hide it well at all. 00:31:45.760 --> 00:31:48.200 JACK: Elliot looks at the message and it says, 00:31:48.200 --> 00:31:53.620 ‘LEAVE ME HERE’ in all upper-case. See, that’s a clear message. 00:31:53.620 --> 00:32:02.080 ELLIOT: This notice for me. They’re telling me to leave it here. But why? 00:32:02.080 --> 00:32:05.180 FABIO: It’s funny; that’s my wallpaper on the desktop right now. 00:32:05.180 --> 00:32:09.460 JACK: As he analyzed things closer, he saw some more interesting evidence. 00:32:09.460 --> 00:32:12.840 FABIO: On that critical server, in the same directory where we had the 00:32:12.840 --> 00:32:18.880 DLL side-loading malware, there were three more files that kind of changed the perspective of this 00:32:18.880 --> 00:32:29.480 whole thing. [MUSIC] There were three files. One was an executable. It was called nbt.exe 00:32:29.480 --> 00:32:34.260 which is a tool. It’s a legitimate tool called NBTscan. It’s a NetBIOS main network scanner. 00:32:34.260 --> 00:32:39.760 JACK: Basically, NetBIOS is how Windows computers connect to shared network drives. So, 00:32:39.760 --> 00:32:43.560 this NetBIOS scanner can scan a whole network and find what servers have 00:32:43.560 --> 00:32:48.600 shared network drives on them. Then if a computer has a shared network drive, 00:32:48.600 --> 00:32:51.800 you may be able to connect to it to see what files are on that server. 00:32:51.800 --> 00:32:58.200 FABIO: Then it was a text file called p.txt which was empty. Then you had a batch file 00:32:58.200 --> 00:33:07.720 called pp.cmd. Pp.cmd had something like thirty-three lines, and each line 00:33:07.720 --> 00:33:18.480 was a command that was executing nbt.exe, so the NetBIOS scanner, followed by a public IP range, 00:33:18.480 --> 00:33:28.760 and then putting that output into the p.txt file. So, you had those thirty-three or something public 00:33:28.760 --> 00:33:36.320 IP ranges in that batch file that were scanned. Obviously the first thing you do is you start 00:33:36.320 --> 00:33:45.720 looking into what are those public IP ranges? Who owns them and what were they scanning? Well, 00:33:45.720 --> 00:33:56.080 I think nineteen of those public IP ranges belonged to the US Department of Defense. 00:33:56.080 --> 00:34:02.800 JACK: Whoa, that’s interesting. This threat actor was using this server and this company’s network 00:34:02.800 --> 00:34:10.760 to scan the US Department of Defense’s servers to check if any of them have open file-sharing 00:34:10.760 --> 00:34:18.480 connections with this company. Now, the Department of Defense is huge; it’s the military, so Navy, 00:34:18.480 --> 00:34:25.240 Air Force, Army, but also the NSA is part of DoD. The output of this scan was blank, 00:34:25.240 --> 00:34:31.880 so it did not find any shared drives on the DoD’s network. But what’s interesting is when 00:34:31.880 --> 00:34:37.200 they follow the timeline, it looks like there was an initial infection, then the user logged out, 00:34:37.200 --> 00:34:41.560 and then a new user logged in quite quickly after that, ran this script, saw that the 00:34:41.560 --> 00:34:47.740 output was blank, and then logged out. Then there was no activity on this server for quite a while. 00:34:47.740 --> 00:34:54.480 FABIO: Right, so this kind of tells you that our customer was most likely not the primary 00:34:54.480 --> 00:35:00.160 target for this. They were trying to see from the network they were in, 00:35:00.160 --> 00:35:06.360 can they go into what was maybe their final target. Once they realized that’s not the case, 00:35:06.360 --> 00:35:08.760 they just dropped it, at least for a while. 00:35:08.760 --> 00:35:14.440 JACK: Which suggests this is likely a nation state actor they’re dealing with, 00:35:14.440 --> 00:35:19.640 and not some criminal group or hacktivist, because look at what they immediately went for; 00:35:19.640 --> 00:35:25.640 it wasn’t the customer’s data or money or ransomware. They immediately went to scan the 00:35:25.640 --> 00:35:31.560 DoD. Fabio checked with this company to make sure there aren’t any connections with them in the US 00:35:31.560 --> 00:35:37.440 Department of Defense. No, this company was not connected with them in any way. After the scan, 00:35:37.440 --> 00:35:43.520 there was no malicious activity on the server for two whole weeks. Then this threat actor 00:35:43.520 --> 00:35:49.400 logged back in, but installed all-new malware and all-new tools which talk to 00:35:49.400 --> 00:35:53.880 a totally different command and control server. They didn’t use any of the tools 00:35:53.880 --> 00:35:57.640 that were already there. In fact, they brought in a known malware called PlugX. 00:35:57.640 --> 00:36:00.780 FABIO: PlugX is a known RAT… 00:36:00.780 --> 00:36:03.920 JACK: RAT stands for Remote Access Trojan, 00:36:03.920 --> 00:36:06.480 and it’s a type of malware that can control your computer. 00:36:06.480 --> 00:36:11.920 FABIO: …that had been used over many years. It’s still used by 00:36:11.920 --> 00:36:17.386 many different threat actors groups that are all based out of China. 00:36:17.386 --> 00:36:20.760 JACK: [MUSIC] Looking at the forensic data, it’s as if there were two or three different 00:36:20.760 --> 00:36:25.520 teams that were part of this attack, one team to just establish initial connection to the system; 00:36:25.520 --> 00:36:30.600 once that happened, there was an immediate scan of the DoD’s network to look for shared connections, 00:36:30.600 --> 00:36:34.640 then two weeks later, another connection into this server where they brought all-new malware 00:36:34.640 --> 00:36:39.000 and tools, and it was then when Mimikatz was run where they grabbed all the credentials 00:36:39.000 --> 00:36:43.000 and started pivoting and traversing to other systems in the network. In fact, 00:36:43.000 --> 00:36:48.840 they got into the domain controller of this network and had full admin access there, 00:36:48.840 --> 00:36:54.360 which pretty much gave them control over the whole business. This is consistent with how 00:36:54.360 --> 00:36:59.160 nation state attackers work. There’s sometimes one team that’s just there to get initial access, and 00:36:59.160 --> 00:37:05.240 then another team takes it from there and carries out objectives. Now, Fabio is suspecting that this 00:37:05.240 --> 00:37:11.640 threat actor might be from China. But he can’t tell for sure, and there’s still more research to 00:37:11.640 --> 00:37:17.400 be done. Fabio wanted to know more about how they initially got into this jump server to begin with. 00:37:17.400 --> 00:37:22.840 They looked at what user logged in and placed the initial malware on the system. It was a username 00:37:22.840 --> 00:37:28.825 of someone who worked at the MSP, the company that managed the computers in this network. 00:37:28.825 --> 00:37:35.400 FABIO: [MUSIC] The source of that log-on was an IP address at the MSP side. So, there was 00:37:35.400 --> 00:37:40.200 a log-on from the MSP infrastructure into the customer infrastructure. So, 00:37:40.200 --> 00:37:49.720 we checked with the MSP and we asked about that user and if that person was working that day, 00:37:49.720 --> 00:37:58.620 if that person had logged on at that time. Turns out that that person wasn’t even working that day. 00:37:58.620 --> 00:38:03.280 JACK: With some more questions to the MSP, Fabio concluded that this was a 00:38:03.280 --> 00:38:08.080 malicious log-on. That employee at the MSP did not log into this server and place the 00:38:08.080 --> 00:38:13.400 malware on there. Someone had stolen their credentials and did it. But wait a minute, 00:38:13.400 --> 00:38:18.360 in order for this malicious actor to get into this jump server, they connected into it from 00:38:18.360 --> 00:38:29.400 the MSP’s network. This means the attacker had control of a computer inside the MSP. Oh man, 00:38:29.400 --> 00:38:35.040 this just made the incident so much bigger, because the MSP has more than just one customer. 00:38:35.040 --> 00:38:40.440 In fact, this is one of the biggest MSPs in the world. They have hundreds if not thousands 00:38:40.440 --> 00:38:45.380 of customers where they’re able to get into networks and manage all those computers, too. 00:38:45.380 --> 00:38:50.760 FABIO: So, we just had to ask questions at that point. I mean, we said look, 00:38:50.760 --> 00:38:55.640 we see this malicious log-on of this user that dropped malware, 00:38:55.640 --> 00:39:05.520 and it comes from that particular IP address located within your infrastructure. So, 00:39:05.520 --> 00:39:10.000 they took in that information and they were doing their own investigation. It 00:39:10.000 --> 00:39:21.960 took I think three weeks after we have given that evidence for them to get back to us and say yeah, 00:39:21.960 --> 00:39:29.060 we see malware on our jump station as well. So, within the MSP infrastructure, the same malware. 00:39:29.060 --> 00:39:35.720 JACK: The MSP had been hacked into and didn’t know it until Fabio showed them the evidence. For them, 00:39:35.720 --> 00:39:40.600 this was much worse than one of their customers being breached. They were breached now too, 00:39:40.600 --> 00:39:44.640 and they may have facilitated a breach on many of their customers. 00:39:44.640 --> 00:39:49.080 This must have been a really bad day for the MSP to discover this. 00:39:49.080 --> 00:39:55.720 FABIO: After another few weeks, [MUSIC] then we also got to know that more of their customers 00:39:55.720 --> 00:40:02.880 had been compromised with the same malware. So, our customer was not the only one. We 00:40:02.880 --> 00:40:12.640 also know that they found key loggers on the jump servers at the MSP’s site. Those were – 00:40:12.640 --> 00:40:18.840 just to give the picture of the infrastructure here; the MSP has a lot of customer to manage. 00:40:18.840 --> 00:40:25.760 We’re talking about a global one, so a name that everyone knows about. They manage a lot 00:40:25.760 --> 00:40:31.760 of customers and they have an infrastructure as a jump layer between their internal infrastructure 00:40:31.760 --> 00:40:37.800 and the different customers’ infrastructure. Jump servers are used to access more than one customer, 00:40:37.800 --> 00:40:42.920 so the one that was used to jump into the customer we were handling was also used to 00:40:42.920 --> 00:40:48.840 jump into a lot of other customers, and that system had key loggers on it. So, 00:40:48.840 --> 00:40:55.480 the threat actor was able to see the credentials for the different MSP customers and were able to 00:40:55.480 --> 00:41:01.880 jump into multiple customer environments from there. I mean, if you put this together with 00:41:01.880 --> 00:41:07.480 what we found in our investigation, I could only imagine. Like, they did the scan on the server 00:41:07.480 --> 00:41:14.800 that we were investigating, they scanned the DoD ranges. I would expect they’ve done a similar scan 00:41:14.800 --> 00:41:24.160 from other customer environments as well, and then just prioritized the ones that had trust with DoD. 00:41:24.160 --> 00:41:29.280 JACK: Oh, interesting. This is now starting to come together for me. If a Chinese threat 00:41:29.280 --> 00:41:34.120 actor wants to get into the US Department of Defense’s network, how could they do it? Well, 00:41:34.120 --> 00:41:38.320 they might have intelligence that says well, some companies do have a shared connection with the 00:41:38.320 --> 00:41:43.000 DoD, maybe because they’re outsourcing something or connected with them in some way, and so, 00:41:43.000 --> 00:41:48.400 the threat actor might know that the DoD allows some companies to connect to it through NetBIOS, 00:41:48.400 --> 00:41:52.880 only specific companies or countries. So, their thought was maybe they could 00:41:52.880 --> 00:41:57.520 find the network or company that does have access to DoD’s network, and to do that, 00:41:57.520 --> 00:42:02.000 they could just hack into an MSP who has access to lots of networks, and then spider into each 00:42:02.000 --> 00:42:06.960 of the customers’ networks and run scans on the DoD’s IPs to see if there’s any shared 00:42:06.960 --> 00:42:14.640 folders open to that company or network. Wow. This is what an advanced persistent threat is, 00:42:14.640 --> 00:42:21.800 an APT. Whoever was behind this had quite the resources to try to penetrate DoD’s network and 00:42:21.800 --> 00:42:30.280 had no problem hacking into potentially hundreds of networks around the world to try. Unbelievable. 00:42:30.280 --> 00:42:37.520 As this incident winded down, Fabio still had no idea who did this, and that mystery held up for 00:42:37.520 --> 00:42:45.920 years. But a few years after that, news hit that told him exactly who did it. Here’s a clip from a 00:42:45.920 --> 00:42:51.440 press conference where the US Deputy Attorney General Rod Rosenstein addressed the public. 00:42:51.440 --> 00:42:55.920 ROD: Good morning. Today, the Department of Justice is announcing a criminal indictment 00:42:55.920 --> 00:43:02.880 of two hackers associated with the Chinese government. The charges include conspiracy 00:43:02.880 --> 00:43:08.960 to commit computer intrusions against dozens of companies in the United States and around the 00:43:08.960 --> 00:43:14.000 world. This case is significant because the defendants are accused of targeting 00:43:14.000 --> 00:43:22.640 and compromising Managed Service Providers or MSPs. MSPs are firms that are trusted to store, 00:43:22.640 --> 00:43:27.360 process, and protect commercial data including intellectual property and other 00:43:27.360 --> 00:43:32.880 confidential business information. When hackers gain access to MSPs, 00:43:32.880 --> 00:43:39.480 they can steal sensitive business information that gives competitors an unfair advantage. The 00:43:39.480 --> 00:43:47.520 indictment alleges that the defendants work for a group known to cyber-security experts as APT10. 00:43:47.520 --> 00:43:53.400 These groups are designated as APTs or Advanced Persistent Threats because they use malware to 00:43:53.400 --> 00:44:01.160 gain access to computer networks and to exfiltrate or steal data over an extended period of time. 00:44:01.160 --> 00:44:07.040 These defendants allegedly compromised MSP clients in at least a dozen countries; 00:44:07.040 --> 00:44:13.280 the United States and eleven other countries. The victims included companies in banking and finance, 00:44:13.280 --> 00:44:18.840 telecommunications and computer – consumer electronics, medical equipment, packaging, 00:44:18.840 --> 00:44:26.080 manufacturing, consulting, health care, bio technology, automotive, oil and gas, exploration, 00:44:26.080 --> 00:44:33.480 and mining. The defendants allegedly committed these crimes in association with a Chinese 00:44:33.480 --> 00:44:39.440 intelligence agency known as the Ministry of State Security. There is no free pass to 00:44:39.440 --> 00:44:45.040 violate American laws merely because they do so under the protection of a foreign state. 00:44:45.040 --> 00:44:47.840 JACK: Later on in this press conference, Jeffrey Burman, 00:44:47.840 --> 00:44:51.000 the US attorney for the Southern District of New York, had some remarks. 00:44:51.000 --> 00:44:56.720 JEFF: The defendant’s hacking campaigns also targeted US government agencies 00:44:56.720 --> 00:45:01.480 including the laboratories of NASA, the United States Department of Energy, 00:45:01.480 --> 00:45:08.480 and the US Navy. Members of APT10 stole personal, confidential information 00:45:08.480 --> 00:45:15.466 including social security numbers and dates of birth from over 100,000 Navy personnel. 00:45:15.466 --> 00:45:22.920 JACK: [MUSIC] Whoa, they did it. Those crazy hackers did it. They found a way into the 00:45:22.920 --> 00:45:29.160 Department of Defense, specifically the US Navy’s network. If there’s one thing the history of 00:45:29.160 --> 00:45:35.200 hacking has taught us, it’s that data will not be contained. People will break in and expand 00:45:35.200 --> 00:45:40.000 to new territories, and they’ll crash through barriers painfully, maybe even dangerously, 00:45:40.000 --> 00:45:47.480 but well, there it is. The hackers found a way. They got into the US Navy and stole 00:45:47.480 --> 00:45:55.240 100,000 records of Navy personnel, including social security numbers. Incredible. Well, 00:45:55.240 --> 00:45:59.560 once this indictment came out, more details started to emerge. Reuters journalists Jack 00:45:59.560 --> 00:46:03.600 Stubbs, Joseph Menn, and Christopher Bing did an investigation and found that seven 00:46:03.600 --> 00:46:08.800 different service providers were compromised, and they listed Hewlett Packard Enterprise, IBM, 00:46:08.800 --> 00:46:15.240 Fujitsu, TADA Consultancy, NTT Data, Dimension Data, and Computer Sciences Corporation. Yes, 00:46:15.240 --> 00:46:22.200 all these provide IT services to other companies. So, if someone hacked into any of these, 00:46:22.200 --> 00:46:28.240 they would probably be able to get into their customers. The Reuters article goes on to list 00:46:28.240 --> 00:46:32.920 some of the customers that were hit by this, which includes the telecom giant Ericsson, 00:46:32.920 --> 00:46:37.040 a Navy ship builder, and the travel reservation service Sabre. 00:46:37.040 --> 00:46:41.560 Now, some of these companies listed do have contracts with the US Navy, especially that 00:46:41.560 --> 00:46:48.000 Navy ship builder, so it’s quite possible that one of these companies did have privileged access 00:46:48.000 --> 00:46:53.920 into the US Navy’s network, which is a fascinating attack, right? Don’t come in through the fortified 00:46:53.920 --> 00:46:59.600 front door when you can just disguise yourself as a caterer and just get welcomed in through the 00:46:59.600 --> 00:47:05.280 side door. This was obviously a massive campaign which seemed to have a primary objective of 00:47:05.280 --> 00:47:10.480 getting into US government networks, and that’s kind of what we expect espionage to be, right? 00:47:10.480 --> 00:47:15.600 When one government wants information on another government, they’ll use electronics or computers 00:47:15.600 --> 00:47:22.200 to carry out their data collection and spy on the enemy. But the concerning thing here is that the 00:47:22.200 --> 00:47:28.240 Chinese government hacked into US companies in order to complete their mission. On top of that, 00:47:28.240 --> 00:47:32.920 when they got into these companies, they sucked up any intellectual property they found along the 00:47:32.920 --> 00:47:38.480 way. That’s straight-up theft. That a foreign government stole proprietary information from 00:47:38.480 --> 00:47:44.320 a corporation is astounding because that kind of thing just doesn’t sound right to me. But it 00:47:44.320 --> 00:47:49.800 doesn’t sound right to the US feds, either. Here’s the Deputy Attorney General Rod Rosenstein again. 00:47:49.800 --> 00:47:55.480 ROD: In 2015, China promised to stop stealing trade secrets and other confidential business 00:47:55.480 --> 00:48:00.760 information through computer hacking with the intent of providing a competitive advantage to 00:48:00.760 --> 00:48:05.960 companies in the commercial sector. But the activity alleged in this indictment 00:48:05.960 --> 00:48:09.640 violates the commitment that China made. That was a commitment they made to members 00:48:09.640 --> 00:48:14.640 of the international community, to the United States, to the G20, and to APAC. 00:48:14.640 --> 00:48:19.680 JACK: It’s one thing for governments to spy on each other, but it’s a totally different thing 00:48:19.680 --> 00:48:24.680 when a government hacks into a private company to steal data from them so that they can benefit 00:48:24.680 --> 00:48:31.600 from it economically. But really, the rules of cyberspace have yet to be fully formed. The way 00:48:31.600 --> 00:48:37.200 this space is innovating and changing every day makes it extremely difficult to lay a set of 00:48:37.200 --> 00:48:43.040 international laws down and actually enforce them. The people who were here in this space early were 00:48:43.040 --> 00:48:48.760 able to sneak by because there weren’t any rules, and the advanced players today surely would only 00:48:48.760 --> 00:48:54.240 make rules which allow them to continue to have power and control in this domain. But regardless 00:48:54.240 --> 00:49:02.920 of what rules are made in cyberspace, it’ll only work with nations who agree to abide by the rules. 00:49:02.920 --> 00:49:07.600 Well, Fabio and his team at Truesec were able to clean up this client’s infection, 00:49:07.600 --> 00:49:12.760 which was not easy. I mean, one thing they had to do was change every single Active Directory 00:49:12.760 --> 00:49:18.160 password in the entire company. There were thousands of passwords. But not only that; 00:49:18.160 --> 00:49:21.800 there are lots of computers that have accounts that talk to other computers, 00:49:21.800 --> 00:49:26.360 so these services all had to have their password changed too, and that took a long 00:49:26.360 --> 00:49:30.600 time because there were so many things that would break along the way. Just think about 00:49:30.600 --> 00:49:34.960 all the old servers in a network that nobody has touched for ten years, and the person who 00:49:34.960 --> 00:49:39.680 set them up is long gone from the company. Yeah, well, suddenly it’s not working now, 00:49:39.680 --> 00:49:44.760 and the current admins have no idea where their credentials are stored in this custom application 00:49:44.760 --> 00:49:49.680 that was made. It’s a mess which causes businesses to be impacted for quite a while. 00:49:49.680 --> 00:49:54.120 FABIO: There’s a lot of consequence when you need to do a full, proper 00:49:54.120 --> 00:50:04.400 Active Directory reset. Then on top of that, we introduced active monitoring and EDR tooling, 00:50:04.400 --> 00:50:07.800 because you can never be 100% sure that the investigation has found everything, 00:50:07.800 --> 00:50:12.000 so you still want to have your eyes on everything that is happening at least for 00:50:12.000 --> 00:50:17.680 a while after this. Ideally forever, right? You always want to have your eyes on things. 00:50:17.680 --> 00:50:22.480 JACK: In addition to that, this company cut ties with the MSP that got them infected. 00:50:22.480 --> 00:50:26.040 They were already in the process of renegotiating a contract with them, 00:50:26.040 --> 00:50:30.680 and this just made the decision easier to not go forward with them. They got a different group to 00:50:30.680 --> 00:50:35.840 come manage their servers after that. This is an interesting story since the threat actor 00:50:35.840 --> 00:50:41.920 targeted MSPs to go after their customers and then carry out their objectives from there. Then, 00:50:41.920 --> 00:50:47.200 MSPs are pretty common. More and more companies are outsourcing their IT infrastructure, so to 00:50:47.200 --> 00:50:53.400 target them makes a lot of sense if your goal is to steal intellectual property. It’s sort of like 00:50:53.400 --> 00:51:00.800 going after the janitor’s key ring which can get you access into many buildings in town. So far, 00:51:00.800 --> 00:51:05.280 the people indicted have not been arrested or brought to court. They’re still hiding out 00:51:05.280 --> 00:51:11.000 somewhere, but they have been named and identified and are considered fugitives in the eyes of the 00:51:11.000 --> 00:51:21.255 US. If they’re ever caught, they’re gonna have to go to New York to face their charges. 00:51:21.255 --> 00:51:24.360 (OUTRO): [OUTRO MUSIC] A big thank you to Fabio Viggiani for sharing this story with us. It’s 00:51:24.360 --> 00:51:29.200 crazy to think that as an incident responder, you might wake up some day and go face off against a 00:51:29.200 --> 00:51:34.720 Chinese advanced persistent threat. Yeah, that happens sometimes. You know there are bonus 00:51:34.720 --> 00:51:39.360 episodes of Darknet Diaries, right? There’s also an ad-free version of the show too, and there’s 00:51:39.360 --> 00:51:44.160 two ways to get this. If you’re an Apple Podcast user, you can sign up to Darknet Diaries Plus 00:51:44.160 --> 00:51:50.280 right there in Apple Podcasts, or you can visit patreon.com/darknetdiaries. By joining either of 00:51:50.280 --> 00:51:54.880 these, you will directly be supporting the show and it’ll give you a better listening experience. 00:51:54.880 --> 00:51:58.720 I really have to say thank you to all the people who joined because they really do make this show 00:51:58.720 --> 00:52:04.240 much better. So, thanks. This show is made by me, the cloudy dragon, Jack Rhysider. Sound design and 00:52:04.240 --> 00:52:09.560 editing by the hidden tiger, Andrew Meriwether. Our theme music is by the fiery crane, Breakmaster 00:52:09.560 --> 00:52:25.280 Cylinder. Doing math and binary is slow. You have to go bit by bit. This is Darknet Diaries.