WEBVTT 00:00:00.000 --> 00:00:05.400 JACK: Whenever we have a computer problem that we need to troubleshoot, we often want to know why 00:00:05.400 --> 00:00:10.560 that was a problem. How did it break? You know what? Sometimes you never get a good answer. 00:00:10.560 --> 00:00:15.840 One time when I was at work, a router suddenly crashed. The internet was down for that office and 00:00:15.840 --> 00:00:19.920 my teammate jumped on the problem to try to figure out what was going on. A few minutes later, the 00:00:19.920 --> 00:00:27.120 router was back up and online and was working fine all on its own. This router crashed and rebooted, 00:00:27.120 --> 00:00:34.440 but why? My teammate wanted to know, so he began a forensic analysis. [MUSIC] He looked at the 00:00:34.440 --> 00:00:39.540 environmental data before the crash. It was not showing high CPU or out of memory. It did not have 00:00:39.540 --> 00:00:45.420 a heavy amount of traffic going over it either, so this wasn’t an over-utilization issue. Next, 00:00:45.420 --> 00:00:49.200 he grabbed core dumps, memory snapshots of what was present at the time of the crash, 00:00:49.200 --> 00:00:52.920 and he sent that to the manufacturer of the router to see if they could figure it 00:00:52.920 --> 00:00:58.260 out. A few days later, the manufacturer told us they analyzed the core dumps and said the 00:00:58.260 --> 00:01:06.720 reason for the crash was spurious emissions from space. Spurious emissions from space. 00:01:06.720 --> 00:01:13.140 That’s what caused this router to crash. What the heck is that? Are they saying an asteroid 00:01:13.140 --> 00:01:17.820 hit this thing? We looked into this further and apparently there are cosmic rays that 00:01:17.820 --> 00:01:22.200 are constantly bombarding Earth, and sometimes they can come down, pass right through the roof, 00:01:22.200 --> 00:01:25.980 right on through the outer chassis of the router, and go right through the circuit 00:01:25.980 --> 00:01:31.560 board of the router which can cause a slight electromagnetic change in the circuitry, 00:01:31.560 --> 00:01:38.640 just enough to make a bit flip from a zero to a one or a one to a zero. If the wrong bit flips, 00:01:38.640 --> 00:01:45.660 it could cause the device to malfunction and crash. Cosmic rays can cause this, which is 00:01:45.660 --> 00:01:51.060 incredible that that’s even possible. But really, I thought this manufacturer was just using this as 00:01:51.060 --> 00:01:56.100 some kind of excuse, because they can’t prove that cosmic rays did this. So, in my opinion, 00:01:56.100 --> 00:02:02.400 it meant that we’ll never know what caused this router to crash. It’ll always be a mystery, 00:02:02.400 --> 00:02:09.060 and I wonder how many mysterious things happen to computers that are caused by cosmic rays. 00:02:09.060 --> 00:02:20.340 (INTRO): [INTRO MUSIC] These are true stories from the dark side of the internet. 00:02:20.340 --> 00:02:31.080 I’m Jack Rhysider. This is Darknet Diaries. [INTRO MUSIC ENDS] 00:02:31.080 --> 00:02:43.140 JACK: Nicole Beckwith started out with a strong interest in computers 00:02:43.140 --> 00:02:48.120 and IT. She studied and learned how to be a programmer, among other things. But somehow, 00:02:48.120 --> 00:02:51.960 at some point of her career, she decided she wanted to be a cop. 00:02:51.960 --> 00:02:59.280 NICOLE: I am a former state police officer and federally sworn US marshal. I worked as 00:02:59.280 --> 00:03:04.200 a financial firm investigator and a digital forensic examiner for the state of Ohio. 00:03:04.200 --> 00:03:09.600 JACK: Now, while she was serving as a police officer, she would see cases where hacking 00:03:09.600 --> 00:03:14.940 or digital harassment was involved. These were cases that interested her the most. 00:03:14.940 --> 00:03:19.680 NICOLE: My background is in computers and computer programming. So, 00:03:19.680 --> 00:03:22.200 because of my background, I started taking all those cases. 00:03:22.200 --> 00:03:26.760 JACK: Now, because the internet connects us all together, she’d often be investigating a case 00:03:26.760 --> 00:03:31.080 and find out that the suspect is in another state, so this would often mean that the case 00:03:31.080 --> 00:03:35.760 would turn into a federal investigation, where it landed in the hands of the FBI or 00:03:35.760 --> 00:03:40.860 Department of Homeland Security, or even the Secret Service. So, these cases that started 00:03:40.860 --> 00:03:45.780 out at her police department would sometimes get handed over to one of these other federal units. 00:03:45.780 --> 00:03:50.820 NICOLE: So, the Secret Service kept seeing my name in all these reports. 00:03:50.820 --> 00:03:55.680 One day I got a call, sitting at my desk, from the Secret Service which I can tell you even 00:03:55.680 --> 00:04:00.720 as an officer is kind of daunting, right? To get a phone call and the agent on the 00:04:00.720 --> 00:04:08.040 other line’s like, hi from the Secret Service. You’re like oh gosh, what did I do, you know? 00:04:08.040 --> 00:04:12.360 They were like yeah, we keep seeing your name pop up on these cases and we’d really like to 00:04:12.360 --> 00:04:19.980 talk to you. So, I went in. It actually was just across the street from my office at the state. 00:04:19.980 --> 00:04:25.980 I went and met with them and told them my background and explained that I love 00:04:25.980 --> 00:04:33.960 computers and it’s a hobby of mine, and I like to work on all kinds of projects. 00:04:33.960 --> 00:04:39.240 So, they said that’s awesome. How would you like to work for us 00:04:39.240 --> 00:04:45.540 as a task force officer? We will send you to training, we’ll pay for everything; we just want 00:04:45.540 --> 00:04:53.280 you to help with any of the cases that we get. So, of course I jumped at the opportunity and 00:04:53.280 --> 00:04:59.100 they swore me in as a task force officer for their Financial and Electronic Crimes Division. 00:04:59.100 --> 00:05:06.060 JACK: [MUSIC] [00:05:00] A task force officer for the Secret Service? That sounds pretty badass. 00:05:06.060 --> 00:05:08.700 NICOLE: Thank you. Yeah, it was a lot of fun. 00:05:08.700 --> 00:05:12.420 JACK: So, Secret Service; that’s who protects the president, right? 00:05:12.420 --> 00:05:16.500 NICOLE: Correct, yeah. Yeah, so, most people don’t know 00:05:16.500 --> 00:05:22.740 in addition to their everyday duties in protecting the president and foreign dignitaries and 00:05:22.740 --> 00:05:30.600 other public servants and politicians, they actually are staffed with 00:05:30.600 --> 00:05:37.320 – or assigned to investigate financial and electronic crimes, including cyber-crime. 00:05:37.320 --> 00:05:42.120 JACK: That’s where they wanted her to focus; investigating cyber-crime cases 00:05:42.120 --> 00:05:46.560 for the Secret Service. But before she could start investigating cases, they had to give her 00:05:46.560 --> 00:05:51.164 some training and teach her how to do digital forensics like the Secret Service knows how. 00:05:51.164 --> 00:05:56.820 NICOLE: [MUSIC] I got, oh gosh, a whole host of different training. I started out with the basics, 00:05:56.820 --> 00:06:02.040 so you go through basic digital forensics, dead-box forensics, and then they work 00:06:02.040 --> 00:06:10.860 up to network investigations and then network intrusions and virtual currency investigations. 00:06:10.860 --> 00:06:14.640 So, all-in-all, I think I did seven different trainings, 00:06:14.640 --> 00:06:22.380 roughly eighteen months worth off and on, going back and forth from home to Hoover, Alabama, 00:06:22.380 --> 00:06:31.260 and then was able to investigate all these cases. It was like drinking from a fire hose. 00:06:31.260 --> 00:06:39.780 These training courses are – could vary from one week to five weeks in length. 00:06:39.780 --> 00:06:47.100 It was very intensive sunup to sundown. You’re doing extra work at night in your hotel room, 00:06:47.100 --> 00:06:53.700 and you still have to keep learning when you go back. Obviously, that’s not enough 00:06:53.700 --> 00:06:56.760 as we all know in this field, so you have to keep learning. 00:06:56.760 --> 00:07:00.840 JACK: She worked a lot with the Secret Service investigating different cyber-crimes. Her 00:07:00.840 --> 00:07:05.520 training took her to another level, but then the experience of doing digital forensics gave 00:07:05.520 --> 00:07:11.280 her more insight and wisdom. Then one day, about seven years into doing digital forensics work, 00:07:11.280 --> 00:07:20.220 she saw some news that a police station in her jurisdiction was hit with ransomware. [MUSIC] 00:07:20.220 --> 00:07:23.880 Like, all the computers in the police department were no longer functioning. 00:07:23.880 --> 00:07:30.660 NICOLE: It was ransomware across the entire network. It took down the patrol vehicles, 00:07:30.660 --> 00:07:34.500 it took down the entire police department, 00:07:34.500 --> 00:07:41.760 and I’m told also some of the city laptops because they ended up being connected in a few 00:07:41.760 --> 00:07:46.080 different places. It didn’t take the entire city down, but at least the entire police department. 00:07:46.080 --> 00:07:49.980 JACK: She called them up as a courtesy to see if they needed any help. 00:07:49.980 --> 00:07:55.500 NICOLE: So, for this story I’m gonna tell, I was in my role as a task force officer for the Secret 00:07:55.500 --> 00:08:03.000 Service. As such, like I said, I was called out to respond to cyber incidents. I do want 00:08:03.000 --> 00:08:07.620 to do a quick disclaimer of what I discuss in this episode is either publicly available information 00:08:07.620 --> 00:08:13.740 or I received prior approval to discuss this, so, I do want to get that out there. As a little bit 00:08:13.740 --> 00:08:20.220 of backstory and to set the stage a bit, this is a small-sized city, so approximately 28,000 00:08:20.220 --> 00:08:29.100 residents, ten square miles. Not a huge city, but big enough that you – a ransomware incident would 00:08:29.100 --> 00:08:37.440 take them down. So, I’m already aware of this agency because it’s in my jurisdiction, so we 00:08:37.440 --> 00:08:42.720 had reached out when they were hit to offer any assistance. We were told that they had it handled. 00:08:42.720 --> 00:08:44.640 JACK: [MUSIC] 00:08:44.640 --> 00:08:49.740 The IT team at this police department was doing daily backups of all their systems in the network, 00:08:49.740 --> 00:08:54.660 so they never even considered paying the ransom. They knew they could just restore from backup and 00:08:54.660 --> 00:08:58.740 everything would be fine again, because that’s a great way to mitigate the threat of ransomware. 00:08:58.740 --> 00:09:03.960 NICOLE: As a lot of us know, you always have to make sure that your backups are 00:09:03.960 --> 00:09:09.180 good, and they did not test their backups prior to deploying them, so 00:09:09.180 --> 00:09:14.760 they simply restored the system from backup, checked the box, and said we’re good. 00:09:14.760 --> 00:09:20.640 JACK: Something happened months earlier which meant their backups weren’t actually working. 00:09:20.640 --> 00:09:27.540 The latest backup they had was from ten months ago. That’s a really frustrating thing to realize, 00:09:27.540 --> 00:09:32.340 but by the time they had figured that out, they had already restored a bunch of their 00:09:32.340 --> 00:09:38.100 systems already, and the network was back up and online. So, they just went with it like that. It 00:09:38.100 --> 00:09:42.960 wasn’t the best restore, but it allowed people to get up and working fairly quickly. They just 00:09:42.960 --> 00:09:47.400 had to re-enter in all that stuff from the last ten months back into the systems again. 00:09:47.400 --> 00:09:52.560 NICOLE: So, during the conversation when I’m asking if they need assistance, 00:09:52.560 --> 00:09:59.040 they’re explaining to me that IT has it. When I’m probing them for a little bit more details like 00:09:59.040 --> 00:10:02.760 hey, do you know what happened? [00:10:00] Did somebody click on a phishing e-mail? 00:10:02.760 --> 00:10:11.940 Do you understand the attack vector on this? They’re saying no; all we know is that morning 00:10:11.940 --> 00:10:19.860 our printers went down and then the next thing we know, all of our computers were down. So, that was 00:10:19.860 --> 00:10:21.780 pretty much all that they could tell me. 00:10:21.780 --> 00:10:27.780 JACK: [MUSIC] So, time passes. How much time passes? 00:10:27.780 --> 00:10:29.100 NICOLE: A week. 00:10:29.100 --> 00:10:33.720 JACK: Okay. So… 00:10:33.720 --> 00:10:34.260 NICOLE: Right? 00:10:34.260 --> 00:10:36.840 JACK: Yeah, okay. So, a week later, what happens? 00:10:36.840 --> 00:10:42.900 NICOLE: So, a week later, I’m actually – I just happened to be on the phone with the lieutenant 00:10:42.900 --> 00:10:53.100 on an unrelated matter. He paused and he said oh, crap, our printers are down again. [MUSIC] I said 00:10:53.100 --> 00:10:58.080 wait, isn’t that what happened the first time you guys were hit? It happened to be the same exact 00:10:58.080 --> 00:11:05.160 day, so Friday to Friday. He said yeah, actually, this is exactly what happened that morning. I’m 00:11:05.160 --> 00:11:11.820 like okay, stop everything. Don’t touch a thing. Let’s triage this. Let’s grab some evidence if 00:11:11.820 --> 00:11:19.740 we can. Can I please come help you? So, he’s like yes, please. We would love the assistance. 00:11:19.740 --> 00:11:29.040 When can you be here? I always have a go-bag in my car. I did happen to be at my office that 00:11:29.040 --> 00:11:34.260 morning but I always have a go-bag in my car, so I know that any given time if I need to jump in 00:11:34.260 --> 00:11:39.600 my car and respond, if at home or wherever, that I have all of my essentials in my car. 00:11:39.600 --> 00:11:41.940 JACK: Well, hang on, now; when I hear go-bag, 00:11:41.940 --> 00:11:47.864 I think seventy-two hours of food and water and some Band-Aids. What’s in your go-bag, though? 00:11:47.864 --> 00:11:54.780 NICOLE: [MUSIC] Yeah, so, in my go-bag I have a whole bunch of other – of things, including 00:11:54.780 --> 00:12:01.080 food and clothes and all of that that you just mentioned, but I have what we call a toaster. So, 00:12:01.080 --> 00:12:08.160 a toaster is a hard drive or a SATA dock that you can plug a hard drive into and do imaging 00:12:08.160 --> 00:12:15.540 or whatever. I have hoards of USB drives and CDs with all sorts of mobile triage and analysis 00:12:15.540 --> 00:12:23.400 software such as Paladin, Volatility, password cracking, mobile apps. I have several hard drives 00:12:23.400 --> 00:12:29.580 for evidence collection, both SATA and external. Then I always had a box of cables and adapters, 00:12:29.580 --> 00:12:36.660 tools just in case I needed to take the computer apart, so, you know, screwdrivers and stuff. A 00:12:36.660 --> 00:12:41.940 mouse and a keyboard obviously, because you never know what kind of system you’re gonna encounter. 00:12:41.940 --> 00:12:48.180 Sometimes, like you mentioned, most folks forget that you might be at an incident for quite some 00:12:48.180 --> 00:12:54.780 time, so I always had non-perishable food items ready. I always had bottles of water and granola 00:12:54.780 --> 00:13:01.440 bars or energy bars, change of clothes, bath wipes, deodorant, other hygiene items, 00:13:01.440 --> 00:13:07.800 all of those things, of course. Then on top of that, for forensics, I would also include 00:13:07.800 --> 00:13:13.920 my WiebeTech Ditto machine for imaging. I also had two triage laptops, so, both a Mac and a 00:13:13.920 --> 00:13:21.660 PC. Then of course gloves after a really bad scare once where I thought I had gotten into 00:13:21.660 --> 00:13:26.580 something nasty on a computer. I learned to wear gloves no matter what type of case I was working. 00:13:26.580 --> 00:13:31.920 JACK: Dang, that’s a pretty awesome-sounding go-bag, packed full of tools and items to 00:13:31.920 --> 00:13:36.360 help go onsite and quickly get to work. So, she grabs this thing and jumps in her car, 00:13:36.360 --> 00:13:41.340 and starts driving to the police department. But on the way, she starts making tons of phone calls. 00:13:41.340 --> 00:13:46.860 NICOLE: Oh, yeah. So, the drive over, I’m immediately on the phone getting 00:13:46.860 --> 00:13:52.800 permission from all sorts of people to even be at this police department. So, 00:13:52.800 --> 00:13:58.140 I’m making sure the police department is okay with it, getting permission from the police chief, 00:13:58.140 --> 00:14:04.740 from the city manager, the mayor, my director and my chief at the state, as well as the resident 00:14:04.740 --> 00:14:09.780 agent in charge or my boss at the Secret Service, because there is a lot of red tape that you have 00:14:09.780 --> 00:14:16.440 to work through in order to even lay hands on a system to start an investigation. So, you have to 00:14:16.440 --> 00:14:22.440 have all those bases covered, so, I’m making a lot of phone calls. I’m also working to make sure that 00:14:22.440 --> 00:14:29.100 there is a systems administrator there to give me access to the servers, log-in details, making 00:14:29.100 --> 00:14:35.400 sure I have access to the room to even get to the server. It’s a police department, so, a badge to 00:14:35.400 --> 00:14:41.400 get in and out of rooms, or at least an escort to allow me to get in and out of places that I need 00:14:41.400 --> 00:14:49.620 to get to. I’m also calling a secondary agent and backup for me. I don’t ever want to be the only 00:14:49.620 --> 00:14:56.100 person there. You always want to have a second person with you for a number of reasons, but… 00:14:56.100 --> 00:14:59.700 JACK: It’s funny though because you’re calling for backup to go to the police 00:14:59.700 --> 00:15:03.180 department. [00:15:00] Like, there’s enough officers ready to back you up, aren’t there? 00:15:03.180 --> 00:15:09.300 NICOLE: Right, yeah. Not necessarily backup for physical security, although 00:15:09.300 --> 00:15:14.340 in this case maybe I wasn’t worried about it, but in other cases maybe I am, right? 00:15:14.340 --> 00:15:20.520 Maybe I’m responding to some place where the hostile actor is actually an internal person, and 00:15:20.520 --> 00:15:28.260 you don’t ever want to be with your back against a door or somewhere where you can be ambushed. Even 00:15:28.260 --> 00:15:33.960 in incident response you have to worry about your physical security. In this case, backup 00:15:33.960 --> 00:15:40.680 just for the forensics, but in some cases I am asking for backup for physical security as well. 00:15:40.680 --> 00:15:46.740 JACK: She also keeps questioning herself; is all this even worth the fuss? So far the only 00:15:46.740 --> 00:15:50.880 problem reported were that printers were not working. You don’t deploy the Secret Service 00:15:50.880 --> 00:15:56.340 to go onsite just to fix printers. Maybe she’s just way overthinking this whole thing and she’ll 00:15:56.340 --> 00:16:01.140 get there and it’s just a false alarm. But it didn’t matter; she’s already invested and wants 00:16:01.140 --> 00:16:06.600 to check on it just in case. Okay, so, this is how I picture it; you’re arriving in your car, 00:16:06.600 --> 00:16:11.160 you’ve got your go-bag in your hand, you’ve got the curly earpiece that all the Secret 00:16:11.160 --> 00:16:15.420 Service agents use, your aviator sunglasses, and you’re just busting in the front door. 00:16:15.420 --> 00:16:21.047 NICOLE: Exactly. Picture Lara Croft with cyber stuff, yeah. No. 00:16:21.047 --> 00:16:23.250 JACK: Is it really, though? 00:16:23.250 --> 00:16:27.000 NICOLE: Yeah, no, probably not. 00:16:27.000 --> 00:16:31.380 Yeah, I like to think that, but I’m sure that’s not how I actually looked. So, yeah, no, 00:16:31.380 --> 00:16:40.020 I’m arriving, I’m grabbing all this stuff out of my – the trunk of my car, meeting the lieutenant 00:16:40.020 --> 00:16:47.040 and the chief and kinda doing a data dump on hey, what’s happened since I talked to you last, 00:16:47.040 --> 00:16:51.420 letting all my other bosses know I have arrived on-scene and I’m going to start. 00:16:51.420 --> 00:16:55.620 JACK: She knows she needs access to the computers in the building, and the best way to get into the 00:16:55.620 --> 00:17:00.900 computers is to have someone from IT help you with that. Well, since this was a small agency, 00:17:00.900 --> 00:17:06.660 the IT team was just one person. One guy was running all the computers in this place. 00:17:06.660 --> 00:17:15.120 NICOLE: So, I’m on the phone with him when I first get there. I’m also trying to figure out where 00:17:15.120 --> 00:17:21.120 is the server actually located, which in this case was way back in the back of the building. 00:17:21.120 --> 00:17:27.720 When you walk in, it looks kinda like a garage or a storage place, I guess; dark, bicycles 00:17:27.720 --> 00:17:34.560 and boxes, and just everything that they didn’t want in the police department back in this room, 00:17:34.560 --> 00:17:40.140 cables, and just all sorts of things all over the place. The server’s kinda sitting not 00:17:40.140 --> 00:17:45.900 in the middle of the room but kinda away from the wall, so just picture wires and 00:17:45.900 --> 00:17:50.640 stuff all over the place. It’s a little bit messy, so a little bit concerned there. 00:17:50.640 --> 00:17:54.780 JACK: She finds the server but then starts asking more questions. Is there anyone else 00:17:54.780 --> 00:17:59.400 who manages these computers? Yes, they outsource some of the computer management 00:17:59.400 --> 00:18:04.200 to another company. They had another company do updates to the computers and do security 00:18:04.200 --> 00:18:10.860 monitoring. But they were more reactive, not very proactive at handling security incidents. So, 00:18:10.860 --> 00:18:14.400 yeah, so you go into the back, you’re on the phone with the local IT admin, 00:18:14.400 --> 00:18:20.100 you’re trying to figure out what’s going on. What system do you try to get into first? 00:18:20.100 --> 00:18:28.560 NICOLE: So, they had their main server which had multiple BMs on it. But I’m just getting into the 00:18:28.560 --> 00:18:36.840 main production server, what I thought was just a server for the police department. Turns out, 00:18:36.840 --> 00:18:45.120 it actually housed a couple other applications for the city, but at least everything for the 00:18:45.120 --> 00:18:53.520 police department. When I’m initially responding, I’m looking at the server, getting the log-in 00:18:53.520 --> 00:19:03.180 information from the lieutenant. You successfully log-in. Now, you – in this case, normally when 00:19:03.180 --> 00:19:10.140 you’re responding to a case like this, you’re trying as hard as possible not to leave a digital 00:19:10.140 --> 00:19:15.780 footprint. You’re being really careful about what you touch ‘cause you don’t want to alter the data. 00:19:15.780 --> 00:19:22.620 This case was a little different because of the ransomware in the past and knowing that as soon 00:19:22.620 --> 00:19:30.480 as they lost their printers, it was within an hour that the ransomware was deployed. So, I didn’t 00:19:30.480 --> 00:19:37.680 know how much time I had before what I assumed was going to be ransomware was likely deployed again. 00:19:37.680 --> 00:19:48.060 So, I was trying to hurry and capture whatever I could for forensics right away, before something 00:19:48.060 --> 00:19:58.920 went down. I log into the server. I immediately start dumping the memory, so Volatility is one 00:19:58.920 --> 00:20:05.760 of my hands-down favorite tools to use. [00:20:00] I’m doing dumps of data on Volatility. I’m pulling 00:20:05.760 --> 00:20:13.020 reports, dumping that to a USB drive. I also – once that is running, I wanted to grab network 00:20:13.020 --> 00:20:19.800 traffic and so, I started Wireshark up and I’m dumping network traffic to a USB also. 00:20:19.800 --> 00:20:23.460 JACK: Okay, so, Volatility and Wireshark; let’s jump into these tools for a second, 00:20:23.460 --> 00:20:27.540 because I think they’re really cool. [MUSIC] Volatility is an open-source free tool which is 00:20:27.540 --> 00:20:32.400 used in digital forensics. So, Step One is she’s gotta get into that domain controller which is 00:20:32.400 --> 00:20:37.440 like the central brain of the network, and take a snapshot of the memory which is what’s in RAM, 00:20:37.440 --> 00:20:43.620 because whatever data is in memory is what’s being ran right now, and it changes moment to moment. 00:20:43.620 --> 00:20:48.360 Now, this can take a while to complete. You’ve got to sit there waiting for all the memory to 00:20:48.360 --> 00:20:52.620 be copied over to the USB drive, but it’s more than just whatever memory is active in 00:20:52.620 --> 00:20:56.520 RAM. It’s also going to show what processes are running, what apps are open, the names of all 00:20:56.520 --> 00:21:01.020 the files on the systems, the registry, network connections, users logged in, and system logs. 00:21:01.020 --> 00:21:07.380 Once she has this raw dump of everything on her USB drive, she’ll switch the USB drive over to 00:21:07.380 --> 00:21:11.700 her computer to begin analyzing everything. Are there any suspicious programs running? 00:21:11.700 --> 00:21:16.860 What connections are active, and what activity are the users doing right now? But depending 00:21:16.860 --> 00:21:22.680 on how big these snapshots are, each of these questions can take a while to get answers to. So, 00:21:22.680 --> 00:21:28.800 it’s a slow process to do all this. In the meantime, she fires up Wireshark which is 00:21:28.800 --> 00:21:33.780 a packet-capture tool. Any traffic coming in and out of this domain server is captured to 00:21:33.780 --> 00:21:37.980 be analyzed later. Basically, by capturing all traffic to and from this computer, 00:21:37.980 --> 00:21:43.020 she’ll be able to capture any malware that’s been sent to it, or malicious commands, or suspicious 00:21:43.020 --> 00:21:49.200 activity. As you can imagine though, capturing all network traffic is a lot of stuff to process. 00:21:49.200 --> 00:21:53.280 You’re basically looking at a beach full of sand and trying to figure out that one grain 00:21:53.280 --> 00:21:57.300 of sand that shouldn’t be there. It’s hard to narrow down all the packets to 00:21:57.300 --> 00:22:01.680 find just what you need. It takes a long time, but it’s better to capture it now, 00:22:01.680 --> 00:22:07.140 because nothing else will, and it’s good to have something to go back to and look at just in case. 00:22:07.140 --> 00:22:12.120 Now, what really was fortunate for her was that she got there early enough and set up quickly 00:22:12.120 --> 00:22:19.200 enough that no ransomware had been activated yet. But she had all her listeners open and ready in 00:22:19.200 --> 00:22:24.840 case something did happen. While all that’s going on, she’s poking around in the server, 00:22:24.840 --> 00:22:28.740 looking for anything out of the ordinary, and she finds something. 00:22:28.740 --> 00:22:33.720 NICOLE: After I run all of the quick stuff with Volatility, 00:22:33.720 --> 00:22:40.200 I’m analyzing that really quickly to see what accounts are active, who’s logged in, 00:22:40.200 --> 00:22:46.680 are there any accounts that are rogue? I immediately see another active logged-in account. 00:22:46.680 --> 00:22:54.960 JACK: [MUSIC] Another system admin was logged into this server at the same time she was. She asked 00:22:54.960 --> 00:23:02.160 the IT guy, are you also logged into this server? He said no. She asks, do you think that company 00:23:02.160 --> 00:23:07.320 that manages the network is logged into this server? He checks with them and says nope, nobody 00:23:07.320 --> 00:23:12.540 is logged into our servers right now, either. She gets up and starts asking around the station. 00:23:12.540 --> 00:23:18.540 NICOLE: So, I’m asking the police chief, I’m asking the police lieutenant, 00:23:18.540 --> 00:23:23.640 who else has access to this? They’re like, nobody should be logged in except 00:23:23.640 --> 00:23:30.800 for you. There’s only one access. So, my heart sinks at that point. 00:23:30.800 --> 00:23:36.600 JACK: There wasn’t just one other active user, either; there were a few other people logged 00:23:36.600 --> 00:23:43.260 into this domain controller as admin right now. She’s baffled as to why, and starts to think 00:23:43.260 --> 00:23:47.490 maybe she’s just got there fast enough to actually catch this hacker mid-hack. 00:23:47.490 --> 00:23:55.380 NICOLE: Yeah, so, for somebody that has complete admin access as a couple of these folks did, 00:23:55.380 --> 00:24:01.560 they potentially have access to everything that’s on this server. So, because this is a police 00:24:01.560 --> 00:24:10.620 department, you have case files and reports, you have access to public information or – and PII. 00:24:10.620 --> 00:24:17.640 So, social security numbers and birthdates, and drivers license, and sensitive information about 00:24:17.640 --> 00:24:24.600 cases as well as a whole host of other things that a police department has overseen, right? So, 00:24:24.600 --> 00:24:30.420 you’re looking at officers and officer security and their names and information, 00:24:30.420 --> 00:24:35.820 and e-mail addresses. There’s a whole lot of things that they have 00:24:35.820 --> 00:24:40.500 access to when you’re an admin on a police department server. 00:24:40.500 --> 00:24:45.360 JACK: At this point, she knows for sure whoever is logged into this server should 00:24:45.360 --> 00:24:50.520 not be there. It’s crazy because even as a seasoned incident responder like Nicole, 00:24:50.520 --> 00:24:52.560 it can still affect you emotionally. 00:24:52.560 --> 00:24:56.400 NICOLE: Because your heart sinks when you see that. 00:24:56.400 --> 00:25:04.080 You kinda get that adrenaline pumping and you [00:25:00] see that this isn’t a false positive, 00:25:04.080 --> 00:25:08.220 ‘cause going over there I’m wondering, right, like, okay, so their printers went down; 00:25:08.220 --> 00:25:15.000 is this another ransomware, potential ransomware incident? So, that was the moment when your heart 00:25:15.000 --> 00:25:19.220 starts beating a little bit faster and you know that there actually is something to this. 00:25:19.220 --> 00:25:24.240 JACK: It’s clear to her that she needs to kick the admins out immediately, 00:25:24.240 --> 00:25:27.120 but another thought comes into her head. 00:25:27.120 --> 00:25:35.700 NICOLE: So, right now, as I’m seeing the log-ins, I have to weigh in my head, do we leave 00:25:35.700 --> 00:25:45.420 them logged in and potentially allow them to do additional harm or do I immediately revoke them? 00:25:45.420 --> 00:25:49.560 JACK: Because her tools are still trying to finish their snapshots. [MUSIC] If she 00:25:49.560 --> 00:25:53.820 kicked out the hacker, that might cause her tools to miss the information she 00:25:53.820 --> 00:25:57.480 needs to prove what’s going on. As a digital forensics investigator, 00:25:57.480 --> 00:26:01.740 it’s not often you’re in this situation. Usually you’re called in months after the fact to figure 00:26:01.740 --> 00:26:06.060 out what happened. Trying to both figure out what happened and fight off an active intruder 00:26:06.060 --> 00:26:11.520 is just on another level. She checks the status of her Volatility tool, and it’s almost done 00:26:11.520 --> 00:26:16.980 collecting what she needs. So, she just waits for it to finish, but the wait is killing her. 00:26:16.980 --> 00:26:24.900 NICOLE: Right, yeah, so, of course I’m just letting Wireshark run, but then Volatility – yeah, 00:26:24.900 --> 00:26:31.680 there’s a whole host of scripts and data points that I want dumped. As soon as that finishes, 00:26:31.680 --> 00:26:39.000 then I’m immediately like alright, you’re done; out. Click, revoking access. So, 00:26:39.000 --> 00:26:43.560 as soon as you kick that person out of the system, you breathe a very faint sigh of relief, 00:26:43.560 --> 00:26:46.320 right, ‘cause you still don’t – you have a lot of unknowns, 00:26:46.320 --> 00:26:54.900 but at least you know that one big threat is eliminated for the moment. Because of the fact 00:26:54.900 --> 00:27:02.520 that we weren’t sure what the intrusion vector was at that point, like how they initially got in, 00:27:02.520 --> 00:27:09.360 I’m also changing the password of the supposed admin, the person who’s supposed to have access. 00:27:09.360 --> 00:27:15.000 So, I’m changing his password as well because I don’t know if that’s how they initially got in. 00:27:15.000 --> 00:27:22.080 So, I’m resetting that. Then I’m gonna go back in and grab all the other stuff that I need to grab, 00:27:22.080 --> 00:27:23.220 doing images and whatnot. 00:27:23.220 --> 00:27:27.600 JACK: She swivels around in her chair, moving the USB stick from the domain controller to 00:27:27.600 --> 00:27:31.860 her laptop to start analyzing it, then swivels back to the domain controller to look for more 00:27:31.860 --> 00:27:37.020 stuff. She’s collecting data and analyzing it, but she knows she needs more data. That’s when 00:27:37.020 --> 00:27:40.500 she calls up the company that’s supposed to be monitoring the security for this network. 00:27:40.500 --> 00:27:45.960 NICOLE: I wanted to make contact at that point. Now that I had what I needed, I didn’t want the 00:27:45.960 --> 00:27:53.640 IT contractor to immediately start restoring from backup or doing something that would just ruin my 00:27:53.640 --> 00:28:00.960 evidence. So, now I’m on the phone with them and I’m wanting to make sure that they had backups, 00:28:00.960 --> 00:28:07.740 that they’re currently running a backup just in case, asking them what data they had, like could 00:28:07.740 --> 00:28:16.500 they give me logs? Could they see the initial access point? Basically asking me to – asking them 00:28:16.500 --> 00:28:22.260 to send me anything that they could in the logs that could potentially help me with this case. 00:28:22.260 --> 00:28:25.860 JACK: How did they respond to you? Were they friendly and nice? 00:28:25.860 --> 00:28:31.200 NICOLE: No, they were a little upset that I was there and had not called them. 00:28:31.200 --> 00:28:36.420 They were upset with the police department. But then we had to explain like, look, we got 00:28:36.420 --> 00:28:42.420 permission from the mayor. We got permission from the police department, so they wanted us to come 00:28:42.420 --> 00:28:48.180 in. This is a law enforcement investigation at this point. So, I need your cooperation. 00:28:48.180 --> 00:28:50.820 JACK: [MUSIC] 00:28:50.820 --> 00:28:57.600 They were upset because they were supposed to be the first contact if something happened. They were 00:28:57.600 --> 00:29:01.740 just learning now that all this happened, that the printers went down, that there were unauthorized 00:29:01.740 --> 00:29:07.680 admins accessing the network, and that the Secret Service is there onsite doing an investigation. 00:29:07.680 --> 00:29:14.520 I can see why they’re upset but professionally, there’s no time for that. If your job is to help 00:29:14.520 --> 00:29:19.500 your client be safe, oh well if you want the first to be called. Your help is needed now, 00:29:19.500 --> 00:29:25.260 so let’s get to work now. She kindly asked them, please send me the logs you’ve captured. 00:29:25.260 --> 00:29:31.920 NICOLE: In addition to logs, I had asked them if – from the prior incident – they had saved 00:29:31.920 --> 00:29:39.240 a variant or a file of malware, if they were able to find a ransom letter, if what they had, 00:29:39.240 --> 00:29:46.260 that they could potentially hand over to me in addition to that so that we could kinda see 00:29:46.260 --> 00:29:53.100 what strain of malware it was, if we could do soft attribution on it based on that, 00:29:53.100 --> 00:29:57.540 if there were any other details that we could glean from prior evidence. 00:29:57.540 --> 00:30:01.680 JACK: But they’re still upset on how this [00:30:00] incident is being handled. 00:30:01.680 --> 00:30:06.060 NICOLE: Right, yeah, so, they didn’t want to hand over the logs and the data. 00:30:06.060 --> 00:30:10.980 JACK: This is kind of infuriating to me. The police department is paying this company to 00:30:10.980 --> 00:30:15.000 monitor their network for security incidents and they didn’t want to cooperate with the Secret 00:30:15.000 --> 00:30:19.440 Service on this because they felt the incident wasn’t being handled the way they wanted it to 00:30:19.440 --> 00:30:25.440 be handled? I guess maybe they felt threatened or pressured, or maybe embarrassed that they didn’t 00:30:25.440 --> 00:30:29.940 catch this themselves or solve it themselves. By this point, they had internal investigators 00:30:29.940 --> 00:30:34.440 working on this, and I imagine they felt like their work was being undermined. But 00:30:34.440 --> 00:30:39.300 from my point of view, they completely failed the police department on that first incident. 00:30:39.300 --> 00:30:44.340 That was their chance to shine, and they missed it. I guess they didn’t want to fail again though, 00:30:44.340 --> 00:30:49.200 and wanted to show how they can fix it fast this time, and Nicole was just screwing up 00:30:49.200 --> 00:30:53.640 their plans. But she kept asking them to send her data on the previous incident. 00:30:53.640 --> 00:31:02.400 NICOLE: They did end up saying that they had saved a file that was a paint.exe file for 00:31:02.400 --> 00:31:09.660 the original malware and had saved a text file for the ransomware that was the ransom note. 00:31:09.660 --> 00:31:13.860 JACK: Well, that’s something for her at least to look at. Every little bit helps to build a 00:31:13.860 --> 00:31:17.820 complete picture of what happened and what could happen in this incident. 00:31:17.820 --> 00:31:22.920 NICOLE: As I’m analyzing all of the data that I collected and the evidence, 00:31:22.920 --> 00:31:29.040 I ended up seeing that there was an external IP address that had been logged in at that time. 00:31:29.040 --> 00:31:35.820 JACK: What she realized was this police station’s domain controller was accessible from the internet 00:31:35.820 --> 00:31:41.640 over Remote Desktop. The brains of the network was accessible from anywhere in the world without 00:31:41.640 --> 00:31:47.460 a VPN. You just needed the username and password to get into this thing or if you had an exploit 00:31:47.460 --> 00:31:53.100 for this version of Windows. But this, this is a bad design. This system should not be accessible 00:31:53.100 --> 00:31:57.480 from the internet. Ideally, you should be onsite at the police department to get into this system. 00:31:57.480 --> 00:32:01.200 But if you really need someone to get into this remotely, you should probably set up 00:32:01.200 --> 00:32:06.660 a VPN for admins to connect to first and then get into this. Having a system running Remote 00:32:06.660 --> 00:32:13.440 Desktop right on the internet just attracts a ton of people to try to abuse the system. So, 00:32:13.440 --> 00:32:18.120 she’s seeing all these external public IPs that just keep logging into this system, 00:32:18.120 --> 00:32:22.740 and she’s kicking them out one by one, but she’s realizing this has to stop. 00:32:22.740 --> 00:32:30.780 NICOLE: So, with this, I politely asked them, I need you to turn off all external access, like who 00:32:30.780 --> 00:32:38.280 – how are these people getting in? Take down remote access from this server. There’s no 00:32:38.280 --> 00:32:47.820 reason for it. They refused to do it. [MUSIC] So, I made the request; they just basically said sure, 00:32:47.820 --> 00:32:56.400 whatever. I think it was a day later that I checked and it still was not taken care of. So, 00:32:56.400 --> 00:33:01.140 at that point I went right to their office, showed up to the office, knocked on the door, 00:33:01.140 --> 00:33:09.120 asked for the person that I was working with, and stood in front of his desk and just told him, 00:33:09.120 --> 00:33:15.060 you’re gonna lock this down right now. It wasn’t nice and I don’t have to do that 00:33:15.060 --> 00:33:20.600 very often, but I stood in front of his computer until he locked it down. 00:33:20.600 --> 00:33:28.080 JACK: Whoa, it’s crazy to think that this IT company had to have the Secret Service 00:33:28.080 --> 00:33:33.540 explain the dangers of why this is a problem. Nicole is right; this should not be allowed. 00:33:33.540 --> 00:33:38.400 But I’ve personally tried to convince people to turn this off before myself, and what I’ve been 00:33:38.400 --> 00:33:43.020 told is it’s required because certain tools and systems need it to be open for things to work, 00:33:43.020 --> 00:33:47.760 and you’ll break things if you turn it off. Something about legacy equipment, too. Yeah, 00:33:47.760 --> 00:33:52.320 well, that might have been true even in this case. Certain vendors or apps might have no 00:33:52.320 --> 00:33:56.580 longer worked if you turned that off. I just think vendors that require this are 00:33:56.580 --> 00:34:01.740 dumb because the consequences of having your domain controller hacked is far greater than 00:34:01.740 --> 00:34:06.600 your app going down. In this case, the police department was hit with ransomware because this 00:34:06.600 --> 00:34:12.300 system was accessible from the internet which caused ten months of lost work. 00:34:12.300 --> 00:34:16.020 It would have been hit again if it wasn’t for Nicole’s quick reactions. 00:34:16.020 --> 00:34:20.880 So, she was happy that they finally turned off public access to this computer, and left. 00:34:20.880 --> 00:34:27.000 NICOLE: So, after this conversation with the security contractor, I go back and do an analysis. 00:34:27.000 --> 00:34:35.520 JACK: [MUSIC] She tries to figure out more about who was logged in as an admin at the same time 00:34:35.520 --> 00:34:40.260 as her. Looking through the logs and data she collected, she looks at the IP address of the 00:34:40.260 --> 00:34:45.900 user, which is sort of a digital address. Obviously they connected from a public IP, 00:34:45.900 --> 00:34:51.000 and she had that, but then from there she did a geo-IP lookup to see where this IP address 00:34:51.000 --> 00:34:57.360 may be located physically in the world. When she looked at that, the IP was in the exact same town 00:34:57.360 --> 00:35:03.780 as where this police department was. [00:35:00] That’s interesting. A local person did this? 00:35:03.780 --> 00:35:11.340 NICOLE: So, I write a search warrant to that ISP asking for who this IP address comes back to. 00:35:11.340 --> 00:35:16.440 JACK: So, what law enforcement can do is issue a search warrant to the ISP to figure 00:35:16.440 --> 00:35:21.000 out what user was assigned that public IP at the time. But this takes a while; 00:35:21.000 --> 00:35:26.100 a few days, maybe weeks. In that time, she starts thinking about why someone 00:35:26.100 --> 00:35:30.420 locally in this town might want to hack into the police department’s computers. 00:35:30.420 --> 00:35:36.900 NICOLE: For me, I’m thinking that it’s somebody local that has a beef 00:35:36.900 --> 00:35:42.720 with the police department. Maybe a suspect or there’s a case or they got 00:35:42.720 --> 00:35:47.100 pulled over. A whole host of things are running through my head at this point. 00:35:47.100 --> 00:35:53.940 JACK: Stay with us because after the break, things don’t go as planned. Okay, so at this point, 00:35:53.940 --> 00:35:58.920 she’s analyzed the system pretty well and found that this user did upload some malware and looks 00:35:58.920 --> 00:36:04.440 like they were staging it to infect the network with ransomware again, which means this was an 00:36:04.440 --> 00:36:09.480 actual and serious attack that she was able to intercept and neutralize before it had a 00:36:09.480 --> 00:36:15.180 chance to detonate. On top of that, she’s traced this hacker to come from a person who’s local to 00:36:15.180 --> 00:36:19.800 the city where this police department was, and issued a search warrant with the ISP to figure 00:36:19.800 --> 00:36:26.280 out exactly who was assigned that IP. She gets the documents back from the ISP and opens it to see. 00:36:26.280 --> 00:36:35.820 NICOLE: [MUSIC] So, when I see the address and the person that is connected to this search warrant, 00:36:35.820 --> 00:36:45.780 I’m a little bit baffled. I’m shocked, I’m concerned, 00:36:45.780 --> 00:36:53.460 not really fully understanding what I’m looking at. Confusion comes into play there. A roller 00:36:53.460 --> 00:36:58.440 coaster of emotions are going through my head when I’m seeing who it’s tied back to. 00:36:58.440 --> 00:36:59.600 JACK: Why? 00:36:59.600 --> 00:37:03.920 NICOLE: Because it came back to the mayor of the city. 00:37:03.920 --> 00:37:10.740 JACK: Whoa. The mayor of the city is who hacked into the computer and planted malware 00:37:10.740 --> 00:37:15.780 on it and was about to detonate it to take the police department’s network down again? 00:37:15.780 --> 00:37:16.800 NICOLE: Correct. 00:37:16.800 --> 00:37:18.060 JACK: What? 00:37:18.060 --> 00:37:27.720 NICOLE: So, at this point, I’m running scenarios in my head as to why in the world a mayor would be 00:37:27.720 --> 00:37:36.720 connected to this server. Doing reconnaissance on this case and looking at some of the past 00:37:36.720 --> 00:37:45.300 cases and just knowing the city and wondering who could potentially have an issue with the police 00:37:45.300 --> 00:37:54.480 department, I did run across some information that suggested that the mayor of the city may have 00:37:54.480 --> 00:37:58.920 taken an issue with the police department because he was actually previously, 00:37:58.920 --> 00:38:07.680 prior to becoming mayor, arrested by this police department. So, having that in the back of my 00:38:07.680 --> 00:38:16.320 head, of course you’re wondering why is this person logged in and then, he does have motive 00:38:16.320 --> 00:38:22.680 to be upset with the police department. You’re running through a lot of things. 00:38:22.680 --> 00:38:27.180 You’re told you shouldn’t make snap judgments. Obviously in police work, 00:38:27.180 --> 00:38:30.960 you never want to do that, right? But you’re still gonna think through the theories and the thought 00:38:30.960 --> 00:38:37.500 – you’re gonna have these thoughts and things are gonna pop into your head. So, you have to look at 00:38:37.500 --> 00:38:44.640 every possible scenario because you don’t want to be blindsided or put yourself into a potentially – 00:38:44.640 --> 00:38:54.060 a bad situation. So, armed with this information, obviously I have to make my leadership aware. 00:38:54.060 --> 00:39:00.120 I’m talking to the agent in charge, I’m talking to my bosses and just letting them know hey, 00:39:00.120 --> 00:39:06.120 this is what I’m seeing. We really need to go have a conversation with the mayor so it gets out, 00:39:06.120 --> 00:39:12.840 figure out why he’s logged into this computer at this time. So, 00:39:12.840 --> 00:39:14.880 we end up setting up a meeting with the mayor. 00:39:14.880 --> 00:39:20.580 JACK: [MUSIC] So, on your way to meet with the mayor, how are you going – I mean, 00:39:20.580 --> 00:39:25.320 you’ve got a different couple ways of doing this. Are you going to 00:39:25.320 --> 00:39:29.880 get your backup to distract him while you grab his computer off his desk or are you 00:39:29.880 --> 00:39:34.260 going to do bad cop, good cop and sit him down and say we know what you’ve been up to, 00:39:34.260 --> 00:39:38.640 and we can make this easy or hard – like, what’s your strategy of confronting the mayor here? 00:39:38.640 --> 00:39:45.960 NICOLE: Right, so, I am not the beat-around-the-bush type of person. 00:39:45.960 --> 00:39:53.880 I’m very direct typically, especially when I’m doing an interview or an interrogation. 00:39:53.880 --> 00:39:58.980 I tried good cop, bad cop; I’m not a very scary person, so that doesn’t work very well unless 00:39:58.980 --> 00:40:06.120 I’m the good cop. [00:40:00] We go meet with the mayor, and I start the conversation. He’s like oh, 00:40:06.120 --> 00:40:11.700 can you give me an update? I’m just walking through and I’m like yeah, so, you know, 00:40:11.700 --> 00:40:20.580 we did the search warrant. We see there’s a local IP address that’s on the network at this time. 00:40:20.580 --> 00:40:25.500 We really need to talk to you about this because it’s coming back to you. 00:40:25.500 --> 00:40:29.760 JACK: She shows him the date and times when someone logged into the police department. 00:40:29.760 --> 00:40:36.240 He says no way; it couldn’t have been me because I was at work in the mayor’s office at the time. 00:40:36.240 --> 00:40:39.720 This alibi checks out, because people did see him in the office then. 00:40:39.720 --> 00:40:47.460 NICOLE: Obviously we’re asking do you have kids, do you have somebody else staying at your house, 00:40:47.460 --> 00:40:53.700 is there additional people that have access to your computer or these credentials that would 00:40:53.700 --> 00:41:02.400 be able to access this server? He’s saying no, he should be the only one with access to this server. 00:41:02.400 --> 00:41:08.700 JACK: Now, at this point, Nicole is doing more mental gymnastics to try to figure out how and 00:41:08.700 --> 00:41:14.640 why. How did the mayor’s home computer connect to the police department’s server at that time? It’s 00:41:14.640 --> 00:41:20.100 possible he’s lying and was either home that day or had some kind of remote access connection to 00:41:20.100 --> 00:41:25.080 his home computer and then connected in, but if he’s going to do something bad against the police 00:41:25.080 --> 00:41:29.760 department, he’d probably want to hide his tracks and not do it from his home computer. I mean, 00:41:29.760 --> 00:41:35.220 if he’s savvy enough to do remote connections and hack into things, then he would know he needed to 00:41:35.220 --> 00:41:42.060 hide his tracks better, right? She believes him but is hesitant. She looks at her boss 00:41:42.060 --> 00:41:47.640 who’s also in the room and then back to the mayor, and asks him another question. Well, 00:41:47.640 --> 00:41:52.920 have you ever used your home computer to log into the police department’s server before? He says… 00:41:52.920 --> 00:42:00.600 NICOLE: Yeah, I was probably logging in to check my mail, my e-mail. 00:42:00.600 --> 00:42:12.360 I’m, again, completely floored at this point, not quite understanding what just came out of his 00:42:12.360 --> 00:42:22.080 mouth, right? I reiterate; okay, you’re logging in from your house to the police department’s 00:42:22.080 --> 00:42:31.020 domain server to check your e-mail? He’s like oh yeah, we all do it, every one of us. I’m like, 00:42:31.020 --> 00:42:37.380 what do you mean, we all? Who is we all? ‘Cause then I’m really starting to get concerned, right? 00:42:37.380 --> 00:42:43.440 He says well, I do, the city council does. Yeah, whenever we’re working from home or we’re remote, 00:42:43.440 --> 00:42:53.160 we just – and we’re not in front of our computer, we just log into the server and check our e-mail. 00:42:53.160 --> 00:43:03.000 I’m thinking, okay. I said, do you – what are your credentials to log in? Do you have 00:43:03.000 --> 00:43:08.820 separate e-mail address, password? Like, it’s set up for every person? Am I gonna 00:43:08.820 --> 00:43:17.820 see multiple accounts logging in? [MUSIC] He’s like oh no, we all have the admin credentials; 00:43:17.820 --> 00:43:24.000 they’re all the same. All of us log in. We just check whatever e-mail we want. 00:43:24.000 --> 00:43:32.880 JACK: Apparently what him and others were doing were logging into this server through 00:43:32.880 --> 00:43:39.240 Remote Desktop and then using this computer to log into their webmail to check e-mail? 00:43:39.240 --> 00:43:45.120 NICOLE: Correct, yeah. Yeah, so, admin credentials to this 00:43:45.120 --> 00:43:56.760 server, to RDP in, and then they’re checking their e-mail. I have seen a lot of stuff in my life, 00:43:56.760 --> 00:44:06.600 but that’s the – takes – that takes the cake. So, I just look at my boss and shake my head ‘cause 00:44:06.600 --> 00:44:15.980 at that point, I don’t really know what to say. We’re just like alright, thank you for your time. 00:44:15.980 --> 00:44:22.800 JACK: This threw a monkey wrench in all of her hunches and theories. The network was not set up 00:44:22.800 --> 00:44:27.180 right. For whatever reason, someone decided that it was too much of a risk to have the webmail 00:44:27.180 --> 00:44:31.680 server exposed to the internet for people to log into, but thought it was perfectly fine to have 00:44:31.680 --> 00:44:36.540 the domain controller exposed to the internet for people to log into instead? Not only that, 00:44:36.540 --> 00:44:40.500 but to have them log in as admins, which means they have full permission to change 00:44:40.500 --> 00:44:45.300 anything they want or do whatever they want in the network? For instance, with domain admin access, 00:44:45.300 --> 00:44:50.340 the mayor could easily read anyone’s e-mail, not just his. He could sabotage users like 00:44:50.340 --> 00:44:56.520 change their passwords or delete records. Admins have full control of everything. The thing is, 00:44:56.520 --> 00:45:01.260 the domain server is not something the users should ever log into. [00:45:00] There’s just 00:45:01.260 --> 00:45:06.480 nothing there to help them be productive. It’s not where files are stored or even e-mails. This 00:45:06.480 --> 00:45:11.940 server does behind-the-scenes work, authorizing and authenticating connections among other stuff. 00:45:11.940 --> 00:45:16.260 Again, in this case, the mayor wasn’t accessing e-mails that were on this server. He was getting 00:45:16.260 --> 00:45:22.260 on this server and then using a browser to access e-mails on another server. It’s just silly. 00:45:22.260 --> 00:45:27.600 So, Nicole packs up and leaves the mayor’s office with more questions now than before she 00:45:27.600 --> 00:45:32.340 arrived. She calls up the security monitoring company to ask them for more information. 00:45:32.340 --> 00:45:39.180 NICOLE: I have a conversation with the security vendor and say look, can you give me a list of all 00:45:39.180 --> 00:45:47.160 of the admins that have access to this computer? A) They’re with you or with the city, or anybody 00:45:47.160 --> 00:45:56.820 you know. Pull up on your computer who has access to this computer, this server. So, they give me a 00:45:56.820 --> 00:46:05.520 list and there are actually several people on this list, the mayor being one of them, and all of the 00:46:05.520 --> 00:46:11.160 city council, a secretary. So, there’s a whole host of people that have access to this server. 00:46:11.160 --> 00:46:16.920 JACK: What’s more is that some of these people are sharing their admin log-ins with others. So like, 00:46:16.920 --> 00:46:21.300 if the city council member has a secretary, sure, go ahead, give the secretary this admin 00:46:21.300 --> 00:46:26.280 log-in so they can check their e-mail, too. This is a personal pet peeve of mine; I hate 00:46:26.280 --> 00:46:31.020 it when admin log-ins are shared, because when you have multiple people logged into one account, 00:46:31.020 --> 00:46:37.020 you have no idea which person is doing stuff. Is it the secretary that just logged in? The city 00:46:37.020 --> 00:46:42.780 council member? The mayor? Nobody knows, which is horrible when you’re trying to account for 00:46:42.780 --> 00:46:47.220 what’s going on in your network. She then told the IT company what to do. 00:46:47.220 --> 00:46:53.340 NICOLE: Again, immediately it’s obviously you shut that down. I want you to delete 00:46:53.340 --> 00:46:58.320 those credentials and reset all the credentials for this server. 00:46:58.320 --> 00:47:03.540 JACK: Of course, the IT company did not like this idea since it meant that city council members and 00:47:03.540 --> 00:47:08.280 everyone couldn’t check their e-mail remotely anymore. But the network obviously needed to 00:47:08.280 --> 00:47:15.360 be redesigned badly. But Nicole still had this mystery; who the hell logged into the police 00:47:15.360 --> 00:47:19.920 station from the mayor’s home? Well, they asked the mayor if they could investigate 00:47:19.920 --> 00:47:26.520 his home PC and he said yes. But it was around this time when Nicole moved on to another case 00:47:26.520 --> 00:47:31.260 and someone else took over that investigation. But she did follow up to see what happened. 00:47:31.260 --> 00:47:38.160 NICOLE: Yeah, I did hear after the fact that they were able to find a phishing e-mail. 00:47:38.160 --> 00:47:44.280 There was credentials stolen. There was somebody in the mayor’s computer 00:47:44.280 --> 00:47:49.320 that ended up gaining access to the server through the mayor’s home computer. 00:47:49.320 --> 00:47:54.900 JACK: Someone sent the mayor a phishing e-mail. He clicked it; this gave the attacker remote access 00:47:54.900 --> 00:47:59.220 to his computer. The attacker put a keystroke logger on the computer and watched what the 00:47:59.220 --> 00:48:03.300 mayor did. The mayor went and logged into the police department’s computer to check his e-mail, 00:48:03.300 --> 00:48:09.060 and the attacker saw all this, including his password he typed. From there, the attacker 00:48:09.060 --> 00:48:13.320 logged into the police station, and that’s how the police station got infected with ransomware 00:48:13.320 --> 00:48:19.020 the first time and almost a second time. The investigators were able to see whoever hacked into 00:48:19.020 --> 00:48:24.240 the mayor’s computer was coming from somewhere in Europe. But they didn’t track this down 00:48:24.240 --> 00:48:30.180 any further. That would just cost more time and money and probably wouldn’t result in anything. 00:48:30.180 --> 00:48:35.400 So, there’s this practice in IT security of giving your users least privilege. Just give 00:48:35.400 --> 00:48:39.000 them the minimum necessary rights to do what they need to do, and maybe only give them the 00:48:39.000 --> 00:48:44.340 rights for a short duration, because this severely limits what a potential attacker can do. When you 00:48:44.340 --> 00:48:49.920 give someone full admin rights, it really opens up the attack surface. People can make mistakes, 00:48:49.920 --> 00:48:56.460 too. Maybe they accidentally shut down the domain server because they can as admin. Another thing 00:48:56.460 --> 00:49:01.560 to watch out for is when actual admins use their admin log-ins for non-admin things. 00:49:01.560 --> 00:49:06.600 Admins should only use their admin accounts to do admin-type things. They shouldn’t be 00:49:06.600 --> 00:49:11.520 logging in from home as admin just to check their e-mail. What did the police department 00:49:11.520 --> 00:49:16.740 do after this as far as changing their posture on the network or anything at all? 00:49:16.740 --> 00:49:23.580 NICOLE: Yeah, so, they did a lot. They ended up firing the security vendor that 00:49:23.580 --> 00:49:31.860 they were using. They hired a new security vendor which has been fabulous. They ended 00:49:31.860 --> 00:49:41.400 up choosing a new virus protection software. They completely wiped all of the computers one by one, 00:49:41.400 --> 00:49:52.920 especially those in the patrol vehicles, upgraded those to new operating systems, they started being 00:49:52.920 --> 00:49:59.040 more vigilant about restricting the permissions that were given to staff for certain things, 00:49:59.040 --> 00:50:06.540 [00:50:00] reinstalled their VPN, thankfully, and had no network lag there. They changed and 00:50:06.540 --> 00:50:12.960 updated all the passwords. So, there was a lot that they did after the fact. My understanding 00:50:12.960 --> 00:50:20.880 is they’re – that’s a process because it costs so much money and obviously it’s a government 00:50:20.880 --> 00:50:26.040 agency – budgets only allow for certain things at certain times. But this was a process over 00:50:26.040 --> 00:50:31.860 time. I’m sure that they’re continuing to work on that, but they did quite a bit right away. 00:50:31.860 --> 00:50:37.980 JACK: Yeah, a redesign like this does cost a lot, but they had their hand forced because 00:50:37.980 --> 00:50:42.960 the attorney general found out about these security incidents and was not 00:50:42.960 --> 00:50:48.840 happy. The attorney general revoked the police department’s access to the gateway network. 00:50:48.840 --> 00:50:54.780 NICOLE: The gateway network is how this police department gets access to new suspect information, 00:50:54.780 --> 00:51:03.900 how we run suspects, how we run for doing traffic stuff, how we run plates. There’s a 00:51:03.900 --> 00:51:08.820 lot of information that’s coming back from this system. For a police department to be shut off 00:51:08.820 --> 00:51:14.100 from that system, which they were denied access to that, they had to use another agency to pull 00:51:14.100 --> 00:51:20.940 data. Obviously it’s both good and bad, right? It’s good because the attorney general is taking 00:51:20.940 --> 00:51:27.600 a very hard and fast stance with that in saying if you can’t control your networks and your systems, 00:51:27.600 --> 00:51:33.120 then we’re not allowing you access to ours because you’re a security risk. But in – at the same time, 00:51:33.120 --> 00:51:41.640 this is then also hindering the operations of the police department and could potentially put 00:51:41.640 --> 00:51:49.980 officers’ lives in risk for not being able to run a suspect for warrants or if they’re on a call. 00:51:49.980 --> 00:51:54.540 So, it – I see both sides of that coin. But they did eventually get 00:51:54.540 --> 00:51:59.520 granted access back after they could prove that they had done all of these upgrades. 00:51:59.520 --> 00:52:05.160 JACK: With their network secure and redesigned and their access to the gateway network reinstated, 00:52:05.160 --> 00:52:08.940 things returned to normal. But it was certainly 00:52:08.940 --> 00:52:13.620 disruptive and costly for the police department to handle this incident. 00:52:13.620 --> 00:52:18.840 Nicole has since moved on from working with the Secret Service and is currently a security 00:52:18.840 --> 00:52:24.037 engineer where she plans, designs, and builds network security architectures. 00:52:24.037 --> 00:52:33.300 (OUTRO): [OUTRO MUSIC] A big thank you to Nicole Beckwith for sharing this story with 00:52:33.300 --> 00:52:36.900 us. I have a link to her Twitter account in the show notes and you should totally 00:52:36.900 --> 00:52:41.640 follow her. Hey, I just released the ninth bonus episode of Darknet Diaries. Currently, 00:52:41.640 --> 00:52:46.680 it’s only available for Patreon users, but I am in the process of getting bonus content over to 00:52:46.680 --> 00:52:51.300 Apple Podcasts for paying subscribers there, too. The latest bonus episode is about a lady 00:52:51.300 --> 00:52:57.180 named Mary who got a job as a web developer, but things went crazy there which resulted in 00:52:57.180 --> 00:53:01.980 her getting interrogated by the FBI and facing prison time. To hear her story, 00:53:01.980 --> 00:53:09.360 head on over to patron.com/darknetdiaries. Thank you. This show is made by me, running at 7200 RPM, 00:53:09.360 --> 00:53:14.340 Jack Rhysider. Editing help this episode by the decompiled Damienne. Our theme music is 00:53:14.340 --> 00:53:18.360 by the beat-weaver Breakmaster Cylinder. You know what? I don’t like calling it a War Room. 00:53:18.360 --> 00:53:24.600 I’d rather call it a Peace Room since peace is our actual goal. This is Darknet Diaries.