WEBVTT 00:00:00.000 --> 00:00:04.620 JACK: Hey, it’s Jack, host of the show. Let me ask you a question; whose job is it to 00:00:04.620 --> 00:00:09.780 keep the roads you drive on safe? [MUSIC] Is it the driver’s sole responsibility? 00:00:09.780 --> 00:00:15.000 What about the car-makers? Are they responsible for keeping the roads safe for other drivers? 00:00:15.000 --> 00:00:19.080 What about the cops? Maybe they need to come by and watch everyone and make sure they’re obeying 00:00:19.080 --> 00:00:23.460 the law and keeping everyone safe. Or wait; maybe it’s the job of the civil engineers, 00:00:23.460 --> 00:00:29.040 the people who design the roads. I mean, a crazy, curvy, bumpy road with a speed limit of 100 miles 00:00:29.040 --> 00:00:36.660 per hour is obviously not safe. It must be their job to design it to be as safe as possible, right? 00:00:36.660 --> 00:00:40.800 Whose job is it to keep the roads safe? All these people. We need drivers to drive safe, 00:00:40.800 --> 00:00:43.980 we need cars to be built with safety in mind, and we need cops to catch 00:00:43.980 --> 00:00:48.780 people who aren’t being safe, and we need civil engineers to design us safe roads. 00:00:48.780 --> 00:00:52.800 I think this analogy applies to keeping our networks and computers safe, 00:00:52.800 --> 00:00:58.200 too. We need users to be smart at what they click on and do, and we need software makers 00:00:58.200 --> 00:01:02.640 to design the software to be secure. We need the cops to arrest people when they break the law, 00:01:02.640 --> 00:01:08.460 and we need groups who set up industry standards that guide us to safety. We cannot rely on one 00:01:08.460 --> 00:01:13.680 person to keep our networks safe. It takes all these people to always be vigilant to 00:01:13.680 --> 00:01:19.440 keep our computers safe. This is a story about what happens when a software maker finds a bug 00:01:19.440 --> 00:01:25.080 in their own software and what those effects were. Specifically, this is a story about when 00:01:25.080 --> 00:01:33.840 Microsoft found a massive bug in Windows which paved the way for the largest worm in history. 00:01:33.840 --> 00:01:43.860 JACK (INTRO): [INTRO MUSIC] These are true stories from the dark side of the internet. I’m 00:01:43.860 --> 00:01:58.860 Jack Rhysider. This is Darknet Diaries. [INTRO MUSIC ENDS] 00:01:58.860 --> 00:02:02.460 JACK: 00:02:02.460 --> 00:02:05.580 I’m very excited about this episode because I think this is a rare story 00:02:05.580 --> 00:02:09.300 to hear. We aren’t going to hear it from the hacker who found the exploit, 00:02:09.300 --> 00:02:13.080 and we aren’t going to hear from the company who got hacked by this exploit. Instead, 00:02:13.080 --> 00:02:17.880 we’re gonna go right to the source to hear the story of one of the most notorious bugs ever, 00:02:17.880 --> 00:02:23.460 right from the horse’s mouth; Microsoft. [SKYPE] Hello! 00:02:23.460 --> 00:02:27.480 JOHN: Hey Jack, can you hear me? 00:02:27.480 --> 00:02:29.820 JACK: Yeah, I hear you pretty well. What kind of mic are you using? 00:02:29.820 --> 00:02:35.520 JOHN: This is a headset. It’s actually made by Microsoft. 00:02:35.520 --> 00:02:36.600 JACK: This is John Lambert. 00:02:36.600 --> 00:02:38.730 JOHN: My name is John Lambert and I work at Microsoft. 00:02:38.730 --> 00:02:41.940 JACK: He’s been with Microsoft a long time. While he was there, 00:02:41.940 --> 00:02:45.840 he spent ten years working on a team called the Trustworthy Computing Group. 00:02:45.840 --> 00:02:55.020 JOHN: That was a group that was created after the early worms of Code Red, Nimda, Blaster, Slammer, 00:02:55.020 --> 00:03:03.600 and it was designed to help improve customer trust in Microsoft products by fortifying them from a 00:03:03.600 --> 00:03:09.780 security privacy and reliability perspective. The domain I worked in was security and so a 00:03:09.780 --> 00:03:15.720 lot of that focus was on finding and eliminating vulnerabilities and designing and coding from the 00:03:15.720 --> 00:03:21.180 products, and working across the Microsoft product suite; not just Windows or Office but all of the 00:03:21.180 --> 00:03:27.120 products to sort of build the techniques that security researchers had familiarity with and 00:03:27.120 --> 00:03:32.520 then inculcating that ethos and know-how inside of the Microsoft development cycle. 00:03:32.520 --> 00:03:36.420 JACK: Whoa, what an interesting role to have, right? His mission is to improve 00:03:36.420 --> 00:03:42.660 the trust people have in Microsoft products by making them more secure. I just looked this up; 00:03:42.660 --> 00:03:48.300 in 2008, Microsoft had 91,000 employees. With that many employees, yeah, I guess it 00:03:48.300 --> 00:03:52.620 makes sense to make [00:05:00] a team to improve the trust of their products. Oh, 00:03:52.620 --> 00:03:58.260 and in case you’re wondering, in 2019, they had 144,000 employees. Now, of course, 00:03:58.260 --> 00:04:02.100 the biggest Microsoft product that John would work with is Windows, 00:04:02.100 --> 00:04:07.920 the operating system itself. John got the ability to examine Windows in unique ways, by looking at 00:04:07.920 --> 00:04:12.360 the source code and building relationships with developers and conducting security tests against 00:04:12.360 --> 00:04:16.980 it. It’s not really his job to make sure all the bugs are squashed or to find the bugs, 00:04:16.980 --> 00:04:21.960 but he was certainly involved in getting all the teams together to help find the bugs and get them 00:04:21.960 --> 00:04:27.240 fixed. Okay, now when a team at Microsoft finds a bug in Windows, they give it to the developers who 00:04:27.240 --> 00:04:32.280 then work on a fix and create a patch for it. They issue these patches on Tuesdays. 00:04:32.280 --> 00:04:37.320 JOHN: [MUSIC] Every Patch Tuesday which is every second Tuesday of the month, 00:04:37.320 --> 00:04:42.360 we will create a number of security bulletins. The bulletins essentially are the advisory 00:04:42.360 --> 00:04:48.120 that describes here are the vulnerabilities that we are fixing in say, Internet Explorer, 00:04:48.120 --> 00:04:55.740 or Windows, or Office. Every bulletin may have one or more vulnerabilities that is 00:04:55.740 --> 00:05:01.740 being fixed. Those have individual CVE identifiers that security professionals 00:05:01.740 --> 00:05:05.340 are familiar with. But the bulletin is essentially a grouping of them for 00:05:05.340 --> 00:05:11.460 a product. The bulletin is rated critical important, moderate; the level of severity. 00:05:11.460 --> 00:05:15.420 JACK: Now, these advisory bulletins put out on Patch Tuesday might have 00:05:15.420 --> 00:05:25.320 a name like MS07-029 or MS08-067 and if you see something like this, MS07-029, 00:05:25.320 --> 00:05:32.940 it means the advisory was published in 2007 and it was the 29th advisory of the year. 00:05:32.940 --> 00:05:39.300 For this story, we’re going to be talking about the security bulletin MS08-067 which means we’re 00:05:39.300 --> 00:05:45.180 talking about this bug which was found in 2008. Vista had just come out the year before but its 00:05:45.180 --> 00:05:50.280 adoption rate wasn’t that strong, so the majority of Microsoft Windows customers were using Windows 00:05:50.280 --> 00:05:57.180 XP still. Take a guess at how many lines of code it took to create Windows XP. You ready? 00:05:57.180 --> 00:06:02.880 Forty-five million lines of code. [MUSIC] Of course, that was spread out among many 00:06:02.880 --> 00:06:09.000 teams. Trying to keep a program that big bug-free is, well, in my opinion impossible. 00:06:09.000 --> 00:06:13.260 It’s just way too big to keep it bug-free. There has to be some function or routine 00:06:13.260 --> 00:06:18.900 in the code that didn’t get tested properly or coded properly and has a glaring bug. Or 00:06:18.900 --> 00:06:25.140 sometimes it gets tested and coded properly but there’s a bug that’s not so glaring in there. It 00:06:25.140 --> 00:06:31.680 only appears under weird circumstances, like only when the memory is filled and it’s only 00:06:31.680 --> 00:06:36.480 a certain millisecond on a certain time of day. Weird stuff happens when you have a codebase 00:06:36.480 --> 00:06:42.000 that big. All this is to say because Windows XP was forty-five million lines of code long, 00:06:42.000 --> 00:06:50.280 just by that size alone it must have had a lot of bugs. But before we get into MS08-067, 00:06:50.280 --> 00:06:55.140 maybe we should talk about a bug found the year before called MS07-029. 00:06:55.140 --> 00:07:02.040 JOHN: Okay, so that MS07-029 was a very important – it was sort of the 00:07:02.040 --> 00:07:05.700 moment of insight, if you will, for all the things that came after it. 00:07:05.700 --> 00:07:16.620 What MS07-029 was, that was a bulletin that corrected a vulnerability with Windows DNS. 00:07:16.620 --> 00:07:22.620 When Microsoft became aware of it, a customer that was being exploited in the wild contacted 00:07:22.620 --> 00:07:27.840 us and said somehow, we just got attacked. Here’s an attack tool that we were able to 00:07:27.840 --> 00:07:33.106 find associated with it. We think it’s exploiting a vulnerability somehow. That goes into MSRC. 00:07:33.106 --> 00:07:39.600 JACK: [MUSIC] MSRC is the Microsoft Security Response Center. This is the team of security 00:07:39.600 --> 00:07:44.100 professionals within Microsoft that is in charge of making Windows secure and 00:07:44.100 --> 00:07:48.660 looking for bugs and getting them fixed. They took this report they got and tested it to see 00:07:48.660 --> 00:07:54.300 if you could hack into a fully-updated Windows computer with it. It worked. 00:07:54.300 --> 00:07:58.320 JOHN: Then, they investigated and found there’s a zero-day in Windows DNS that 00:07:58.320 --> 00:08:06.780 is being used. They put out an advisory, they put out a bulletin. What I was doing, is I was 00:08:06.780 --> 00:08:11.640 looking at this crash dump system that Microsoft has, this Windows error reporting, 00:08:11.640 --> 00:08:17.760 and I was working with some engineers that were looking at the crash reports coming in 00:08:17.760 --> 00:08:24.720 associated with Windows DNS. The DNS product was otherwise very reliable and hardly ever crashed. 00:08:24.720 --> 00:08:31.620 The only crashes coming in at that point, when we looked at them, were actually attacks in the wild. 00:08:31.620 --> 00:08:37.920 When we started to examine them, we could tell that actually, we could have known about this 00:08:37.920 --> 00:08:43.140 attack in the wild much earlier than the customer contacting us if only we’d been able to pull, 00:08:43.140 --> 00:08:46.950 you know, these needles out of the massive haystack of crashes coming into Microsoft. 00:08:46.950 --> 00:08:49.380 JACK: Do you ever use Windows or Internet Explorer 00:08:49.380 --> 00:08:52.740 or [00:10:00] Microsoft Word and the app suddenly crashes and it says, 00:08:52.740 --> 00:08:57.780 ‘Do you want to send the crashed data to Microsoft?’ This is what’s known as W-E-R; WER. 00:08:57.780 --> 00:09:04.140 JOHN: Yeah, so Windows Error Reporting, WER. It’s also known internally as Dr. Watson. It’s 00:09:04.140 --> 00:09:11.520 really a technology that goes back to the late 90s. Both Office and Windows were independently 00:09:11.520 --> 00:09:16.680 working on how do we get better data on how the products are crashing before we ship them? Then 00:09:16.680 --> 00:09:21.420 after we ship them, other than customers calling customer support, how do we get data about how 00:09:21.420 --> 00:09:28.380 they’re faring in the wild? Both Office and Windows had built features to collect crash 00:09:28.380 --> 00:09:33.600 reports at scale that could be automatically submitted by customers and then run analysis 00:09:33.600 --> 00:09:39.120 tools against them to try to root cause and bucket them to say okay, is this a new bug we don’t know 00:09:39.120 --> 00:09:44.700 about or is this an existing bug that we do know about happening again? Those two efforts came 00:09:44.700 --> 00:09:49.620 together and that feature was built into Windows and that was called Windows Error Reporting. 00:09:49.620 --> 00:09:52.920 JACK: Yeah, that little box that pops up when the app crashes will ask you 00:09:52.920 --> 00:09:55.200 if you want to send the crash report to Microsoft. 00:09:55.200 --> 00:10:00.600 JOHN: If they say yes, then that communicates with Microsoft and the back-end system decides 00:10:00.600 --> 00:10:03.780 is this something we already know about or is this something new? If it’s new, 00:10:03.780 --> 00:10:10.440 it starts to prompt the user to potentially upload more data so we can root cause the issue further. 00:10:10.440 --> 00:10:15.480 JACK: Now, remember I said that XP had forty-five million lines of code. A program that big is 00:10:15.480 --> 00:10:21.000 impossible to be bug-free. Yeah, I don’t think it’s just a dozen bugs in a program that big. 00:10:21.000 --> 00:10:26.640 I don’t even think it’s just hundreds of bugs, or thousands of bugs. I think there’s more like tens 00:10:26.640 --> 00:10:32.160 of thousands of bugs in a program with forty-five million lines of code in it. At the same time, 00:10:32.160 --> 00:10:38.220 Windows was installed on a billion computers in 2008, so Microsoft was seeing millions of crash 00:10:38.220 --> 00:10:43.500 dumps a day from these computers. Let’s try to be a project manager for a second here and come 00:10:43.500 --> 00:10:48.180 up with a strategy to tackle these millions of crash dumps a day. We have a few options; 00:10:48.180 --> 00:10:52.980 first, let’s just filter out all the known bugs we’ve already fixed. People’s 00:10:52.980 --> 00:10:56.580 computers are crashing but they just need to patch it and it won’t crash anymore. Okay, 00:10:56.580 --> 00:11:02.400 that’s out of the way. But now when we try to prioritize what to look at next, it’s not so easy. 00:11:02.400 --> 00:11:07.860 It might make sense to tackle the bugs that show up the most but these might be very low-severity, 00:11:07.860 --> 00:11:13.500 maybe not as important. Maybe there’s something with a more high severity but not as many crash 00:11:13.500 --> 00:11:17.820 reports. Then you might decide to look at the highest severity crash reports instead. 00:11:17.820 --> 00:11:22.200 Or maybe some apps are more important than others. Like, if MS Paint crashes, 00:11:22.200 --> 00:11:26.700 it might not be as big of a deal as if the whole computer was crashing. Or maybe you can 00:11:26.700 --> 00:11:30.960 look at the easiest-to-fix problems first; get those out of the way. It’s really hard to know 00:11:30.960 --> 00:11:35.460 what to prioritize here so this might give you a better understanding of how hard it would be 00:11:35.460 --> 00:11:41.820 to look through millions of crash reports a day to figure out what to fix first. But it was still 00:11:41.820 --> 00:11:47.280 very important for Microsoft to collect these dumps and analyze them. At this time in 2007, 00:11:47.280 --> 00:11:51.240 John was starting to look at these crash reports to try to make sense of them. 00:11:51.240 --> 00:11:58.800 JOHN: [MUSIC] They go up to a massive automated system. These tools run against them at scale 00:11:58.800 --> 00:12:03.300 in a completely headless way. They’re sort of bucketed and binned in a way that makes 00:12:03.300 --> 00:12:08.220 it easy for engineers to know is there something new coming along or how active, 00:12:08.220 --> 00:12:14.460 how prevalent is this issue? Then a wide variety of engineers across the company, 00:12:14.460 --> 00:12:18.900 if you work on Office, you look in the Office crash buckets and see what is hot for you there. 00:12:18.900 --> 00:12:24.060 If you work in Windows or any other product, you can kind of see how is my product crashing? 00:12:24.060 --> 00:12:30.240 Engineers across the company were using this to fix bugs. For example, in Windows Vista, 00:12:30.240 --> 00:12:36.600 from the Trustworthy Computing side, they did all kinds of static analysis tools to find and remove 00:12:36.600 --> 00:12:43.560 reliability issues. I think they fixed 100,000 bugs through those efforts. Through Windows Error 00:12:43.560 --> 00:12:48.660 Reporting they found another 5,000 bugs that had escaped all of those processes that they 00:12:48.660 --> 00:12:54.540 went and fixed. Every service pack, every new product from Microsoft is more reliable because 00:12:54.540 --> 00:13:00.840 they’re finding and fixing all these bugs that are manifesting in the wild. I was there on the side 00:13:00.840 --> 00:13:07.080 saying this is a fascinating telemetry system. How do we look at this from a security point of view? 00:13:07.080 --> 00:13:11.160 JACK: You said 100,000 bugs? 00:13:11.160 --> 00:13:15.960 JOHN: 100,000 changes to the product that were done because of – not all of those are 00:13:15.960 --> 00:13:23.940 bugs. One way to think about it is there’s coding practices that are outdated. Developers will know 00:13:23.940 --> 00:13:29.460 about these things as calling these unsafe string functions like strcpy and so forth. 00:13:29.460 --> 00:13:32.580 Instead of trying to figure out which ones are vulnerabilities and which ones are not, they 00:13:32.580 --> 00:13:37.200 said let’s just go remove all of them from the product. That’s a massive amount of code changes. 00:13:37.200 --> 00:13:42.720 Many of those are not bugs. Many of those are not vulnerabilities but it’s an easy way to just go 00:13:42.720 --> 00:13:45.480 say look, we’re going to have a new standard of engineering. We’re going to get rid of that 00:13:45.480 --> 00:13:49.500 whole class of thing by tackling that. That’s an example of some of those changes, [00:15:00] yeah. 00:13:49.500 --> 00:13:52.980 JACK: Yeah, but I mean, are you exaggerating when you say 100,000, or…? 00:13:52.980 --> 00:13:54.720 JOHN: No, no. 00:13:54.720 --> 00:14:02.040 JACK: Jeez, can you imagine trying to keep something like this secure? To fix 100,000 00:14:02.040 --> 00:14:08.220 things in the code? It just sounds like madness to try to complete. I guess this is why they needed 00:14:08.220 --> 00:14:13.380 91,000 Microsoft employees to tackle huge issues like this. These crash reports were really helping 00:14:13.380 --> 00:14:18.840 them identify the problems that needed fixing. But even though these might be bugs, they might 00:14:18.840 --> 00:14:23.700 not all be security problems. Like, if you click Print and nothing happens, that might be a bug 00:14:23.700 --> 00:14:29.760 in the code, but is it really a security issue? A hacker probably can’t use that to take control of 00:14:29.760 --> 00:14:34.740 your computer or hack you, but John was looking at this and wondering if there was anything in these 00:14:34.740 --> 00:14:40.020 crash reports that is something a hacker could use, or maybe signs that a hacker caused a crash. 00:14:40.020 --> 00:14:44.940 JOHN: Some vulnerabilities are discovered by attackers and exploited in the wild before 00:14:44.940 --> 00:14:53.880 Microsoft is aware of them. How could we go find that before customers contact us? 00:14:53.880 --> 00:15:01.440 There are a lot of reasons that exploits actually fail in practice, and that was the idea which is 00:15:01.440 --> 00:15:06.000 instead of trying to find exploits when they’re working, sometimes they don’t. There’s a bunch of 00:15:06.000 --> 00:15:13.200 interesting reasons why they don’t. By studying the causes of failure of those exploits in the 00:15:13.200 --> 00:15:18.060 wild, that would lead us to potentially discover them. Some of the reasons they 00:15:18.060 --> 00:15:22.380 fail are because the exploit was written for, say, an English language version of Windows 00:15:22.380 --> 00:15:27.300 and it’s run on a Chinese language version and they’re slightly different internally. 00:15:27.300 --> 00:15:35.760 A variety of reasons that exploits would fail in practice. I studied extensively all the different 00:15:35.760 --> 00:15:40.860 patterns of how they fail ‘cause that was what I needed to be able to understand to find them. 00:15:40.860 --> 00:15:48.420 JACK: [MUSIC] The hunt was on for John. He wanted to find traces of hackers in the WER logs. But 00:15:48.420 --> 00:15:53.280 like I was saying, there are millions of error logs a day, so this wasn’t going to be easy. 00:15:53.280 --> 00:16:00.660 JOHN: I think back then it was over a billion a month. Recognize that it’s like, how could 00:16:00.660 --> 00:16:05.700 you look through a billion a month? Because of the way Windows Error Report buckets the issues, 00:16:05.700 --> 00:16:10.800 you don’t have to look at every single one one-by-one. You’re really trying to find new ones, 00:16:10.800 --> 00:16:16.920 new ones that are interesting in a different way. Then another way to think about it is there’s only 00:16:16.920 --> 00:16:24.720 a certain number of ways that zero days are going to show up and affect an application. 00:16:24.720 --> 00:16:28.620 People can send malicious documents that’ll crash Word, so Word is interesting to look 00:16:28.620 --> 00:16:34.560 at. People can send exploits through the browser, so Internet Explorer would be interesting to look 00:16:34.560 --> 00:16:39.960 at. Then, if you think about even inside of Microsoft Word, if an attack is gonna happen, 00:16:39.960 --> 00:16:44.340 likely it’s gonna happen when the user is first opening the document. 00:16:44.340 --> 00:16:48.840 It’s rare that some exploit is designed to work when the user’s been in Word for an hour and they 00:16:48.840 --> 00:16:53.040 finally click Tools > Options > Spellcheck > Add-In, and then it goes boom. Those are 00:16:53.040 --> 00:16:57.180 not the paths where an exploit’s gonna work. It’s gonna be in the File > Open code path. 00:16:57.180 --> 00:17:03.360 That further narrowed down the places that we needed to look. Exploits fail in certain 00:17:03.360 --> 00:17:10.260 specific patterns that are the kind of patterns that we knew to go sifting over. We were able to 00:17:10.260 --> 00:17:16.980 work that funnel down from a billion reports a month to millions that seemed interesting 00:17:16.980 --> 00:17:24.140 to hundreds of thousands that had other clues in them and so forth, and whittled the funnel down. 00:17:24.140 --> 00:17:30.900 JACK: John dedicated a lot of time to try to find this unknown vulnerability that some hacker was 00:17:30.900 --> 00:17:36.840 exploiting. When we come back from the break, John finds something that he’ll never forget. 00:17:36.840 --> 00:17:42.180 JOHN: [00:20:00] 00:17:42.180 --> 00:17:50.280 On September 25th, I remember opening a crash report for the Service Host process 00:17:50.280 --> 00:17:54.660 that we have seen many millions of crash reports for already 00:17:54.660 --> 00:18:00.900 because a couple years earlier there was this other vulnerability, MS06-040, 00:18:00.900 --> 00:18:07.440 that had been adopted by bots and worms to spread. It was causing millions of crashes 00:18:07.440 --> 00:18:14.580 against machines that had not put on that patch. But this one was a little different. 00:18:14.580 --> 00:18:20.160 [MUSIC] First of all, it had an exploit. It had exploit code in it, okay? That didn’t mean it 00:18:20.160 --> 00:18:26.640 could be new; it could be an old one. It was an exploit on the stack which is a critical part of 00:18:26.640 --> 00:18:33.840 memory that tells you that exploit was trying to be activated right now. It had a difference in it, 00:18:33.840 --> 00:18:39.420 what I call an egg hunt, which was an exploit technique that I had never seen before for any 00:18:39.420 --> 00:18:48.480 exploit in MS06-040. This was either a new strain of that or something new for that service host. 00:18:48.480 --> 00:18:53.220 This egg hunt is an exploit-writing technique where, as the bad guy, imagine you’re gonna go 00:18:53.220 --> 00:18:58.020 scale a wall and you put all your attack tools in a – your thieving tools in a bundle and you 00:18:58.020 --> 00:19:03.240 throw it over the wall and then later you go scale the wall and go get that. An egg hunt breaks the 00:19:03.240 --> 00:19:11.160 exploit into two pieces to serve a purpose. It had that technique in it so that alone drew my 00:19:11.160 --> 00:19:19.320 eye to look at it. Then, the odds of there being a buffer over on an exploit in the same area as 00:19:19.320 --> 00:19:27.420 this MS06-040 just seemed unlikely and yet I felt like if it was new, this was really important. So, 00:19:27.420 --> 00:19:33.060 I just tried to stick with it and do what I could to rule in or rule out whether this was 00:19:33.060 --> 00:19:40.800 new. One of the most stubborn clues was in a crash report, it has information about every DLL that 00:19:40.800 --> 00:19:50.700 is loaded into it. The DLL that had MS06-040 was loaded in it. That is this netapi32. The version 00:19:50.700 --> 00:19:56.280 information told me it was fully-patched so it clearly could not have been exploiting 00:19:56.280 --> 00:20:01.220 that vulnerability because that vulnerability did not exist in that version of the product. 00:20:01.220 --> 00:20:07.680 JACK: John looked at the logs here and it looked like a hacker was exploiting two processes. One 00:20:07.680 --> 00:20:12.420 was basically injecting some hacker tools into the system, hiding an egg if you will, 00:20:12.420 --> 00:20:18.720 and the other was going in and using those tools. A strange combo but like John said, 00:20:18.720 --> 00:20:22.560 it was sort of like throwing your tools over the fence and then jumping over the fence to 00:20:22.560 --> 00:20:29.220 get them. The two processes being exploited here were SVC Host, or Service Host, and netapi32. 00:20:29.220 --> 00:20:37.440 JOHN: Yeah, so sometimes when you are writing an exploit, you have constraints that you have 00:20:37.440 --> 00:20:44.220 to work within. Once your exploit starts to run, you have a number of steps as the 00:20:44.220 --> 00:20:49.920 attacker you’re gonna want to do. Typically, that involves downloading some external piece 00:20:49.920 --> 00:20:54.720 of malware to that system and then launching that, and then the rest of your attack, then 00:20:54.720 --> 00:20:58.680 you’re a resident on your computer and you can do whatever attacks that you want to do further from 00:20:58.680 --> 00:21:06.720 there. But you have to get it going and sometimes that shell code, as it’s called, to get going, 00:21:06.720 --> 00:21:11.700 can’t fit within the constraints of the vulnerability. 00:21:11.700 --> 00:21:16.740 The buffer, for example, needs to fit and is just too small a box to package all those instructions 00:21:16.740 --> 00:21:24.840 with it. So, an egg hunt is a technique that is designed to solve that by first sending over some 00:21:24.840 --> 00:21:29.220 data and interacting with the vulnerable program in a way that doesn’t use the vulnerability at 00:21:29.220 --> 00:21:35.760 all. It just tries to get data and memory that is that bag of tools, that shell code that they are 00:21:35.760 --> 00:21:41.100 going to use later. The only thing that they need to get when they actually run the attack is a very 00:21:41.100 --> 00:21:45.720 small piece of shell code that basically goes and searches memory to find that bag of tools. 00:21:45.720 --> 00:21:51.480 JACK: What a tricky exploit. Because there’s two parts to this bug, it makes it really hard 00:21:51.480 --> 00:21:57.240 to find. The more John looked at this particular crash report, the more he started to realize this 00:21:57.240 --> 00:22:03.180 was actively exploiting an unknown vulnerability in Windows which makes it a zero-day bug. 00:22:03.180 --> 00:22:08.700 JOHN: [MUSIC] This, in a way, was the hardest moment of this entire 00:22:08.700 --> 00:22:14.100 thing for me because I clearly had enough evidence to say this was a new attack, 00:22:14.100 --> 00:22:22.320 a new vulnerability that we did not know about, and to get the company to act, you need to 00:22:22.320 --> 00:22:26.820 actually pinpoint the vulnerability. We have to know what’s the code that’s wrong? Otherwise, 00:22:26.820 --> 00:22:34.980 nothing can happen. I pored over the code and pored over the code and I could not figure it out. 00:22:34.980 --> 00:22:40.380 At one point I decide the clock is ticking; [00:25:00] this is a potentially really bad 00:22:40.380 --> 00:22:47.040 situation. I need to go ask for help if I’m gonna figure this out. I walked over to the office of 00:22:47.040 --> 00:22:52.980 Andrew. Andrew was a colleague of mine. We had worked together previously, actually. 00:22:52.980 --> 00:23:01.140 Andrew had looked at many crash reports for security objectives for a long time. 00:23:01.140 --> 00:23:06.000 He knew what this hunt was like, which was in a lot of ways, it can be a very frustrating hunt 00:23:06.000 --> 00:23:13.020 and you can spend hours looking at something you think is real and then it’s not. What I wanted is, 00:23:13.020 --> 00:23:21.360 hand this to one of the MSRC engineers and they will go figure this out. He was reluctant to 00:23:21.360 --> 00:23:27.480 take an engineer off of an existing confirmed vulnerability that somebody else had reported 00:23:27.480 --> 00:23:34.260 and was in the middle of to go potentially chase a ghost. He took it on to go look at it, 00:23:34.260 --> 00:23:43.800 to, I think in his mind say look, this is a false positive. This is not a real issue. Case closed. 00:23:43.800 --> 00:23:49.440 There was some tension in the air with me really pushing a busy team to go look at something that 00:23:49.440 --> 00:23:57.180 likely was not going to pan out but I felt like it could be important enough; it’s worth doing that, 00:23:57.180 --> 00:24:02.580 Andrew wanting to protect his team but do the due diligence to make sure we got the answer right. 00:24:02.580 --> 00:24:09.600 JACK: Andrew got a copy of this crash report and the code for these processes. He started 00:24:09.600 --> 00:24:15.060 analyzing all this. He spent a couple days looking it over and working on it, trying to 00:24:15.060 --> 00:24:20.340 find what the crash report means. There’s an egg somewhere in the code but neither of them could 00:24:20.340 --> 00:24:26.520 find it. But this error report did say that there was something causing a crash. Now, keep in mind, 00:24:26.520 --> 00:24:32.820 John had only seen one crash report from this. This wasn’t a widespread issue so both John and 00:24:32.820 --> 00:24:36.660 Andrew weren’t entirely sure how much effort they should spend into looking for something 00:24:36.660 --> 00:24:42.600 that only happened once on one computer. But if there was an unknown vulnerability, 00:24:42.600 --> 00:24:51.180 both John and Andrew wanted to find it and fix it, so Andrew kept looking. 00:24:51.180 --> 00:24:56.580 JOHN: Then at one point he stops by my office and the look on his face told me everything. The look 00:24:56.580 --> 00:25:07.260 was the look that a security researcher has when they found something. There’s a goofy, happy smile 00:25:07.260 --> 00:25:14.460 that is also full of holy cow, I can’t believe how serious this thing is that I just found. [MUSIC] 00:25:14.460 --> 00:25:20.400 He said I found a vulnerability. When I heard that, I knew everything was 00:25:20.400 --> 00:25:27.060 gonna – the next two weeks of my life were gonna be completely different because this 00:25:27.060 --> 00:25:31.680 vulnerability was in an area that would allow an internet worm to be written against it. 00:25:31.680 --> 00:25:36.300 JACK: The vulnerability they found allowed an attacker to take remote control of a Windows 00:25:36.300 --> 00:25:41.580 computer. No need for a username and password, no need for RDP to be enabled or anything like that. 00:25:41.580 --> 00:25:47.160 Just give me full control of that computer and now I can issue my own commands on it, see any files, 00:25:47.160 --> 00:25:51.840 do whatever I want on that computer as if I’m right in front of it. Of course, to top it off, 00:25:51.840 --> 00:25:56.880 it’s a vulnerability in a fully-updated version of Windows. In the security world we call this 00:25:56.880 --> 00:26:02.940 kind of vulnerability an RCE which means you can do Remote Code Execution. Like, 00:26:02.940 --> 00:26:06.540 a hacker can execute whatever commands they want on the victim’s machine, 00:26:06.540 --> 00:26:12.420 and this is the worst kind of vulnerability. It’s the most critical, the most severe because 00:26:12.420 --> 00:26:17.940 you absolutely never want just any Joe Shmoe executing commands on your computer which 00:26:17.940 --> 00:26:22.500 has full defenses up. If you can run whatever commands you want on someone else’s computer, 00:26:22.500 --> 00:26:28.680 it pretty much means it’s your computer. But to top it off as if this highly critical 00:26:28.680 --> 00:26:33.300 and severe vulnerability wasn’t enough, this exploit was wormable. 00:26:33.300 --> 00:26:40.740 JOHN: Wormable means that you have a vulnerability that an attacker can write an exploit for and it 00:26:40.740 --> 00:26:46.440 can propagate across the internet, exploit that vulnerability, and then turn around and continue 00:26:46.440 --> 00:26:55.440 to repeat the process and propagate further. It becomes a viral outbreak and it is the most 00:26:55.440 --> 00:27:04.680 damaging, devastating, disruptive kind of attack that can take place. The world had seen many worms 00:27:04.680 --> 00:27:14.160 in the form of Blaster, Code Red, Nimda, Slammer, Zotob. We knew how devastating these attacks can 00:27:14.160 --> 00:27:22.440 be; entire businesses are disrupted, systems are taken offline, network traffic gets clogged with 00:27:22.440 --> 00:27:29.700 worms replicating out of control, using up all available bandwidth. We knew what the potential 00:27:29.700 --> 00:27:36.180 was. We didn’t think that that state had been happening yet but we knew that was possible. 00:27:36.180 --> 00:27:39.780 JACK: When Andrew came to John’s office and [00:30:00] told him this 00:27:39.780 --> 00:27:45.960 vulnerability is real and wormable, the two immediately jumped up and sprang into action. 00:27:45.960 --> 00:27:51.540 JOHN: Andrew and I both were on the engineering side and we knew we needed to go activate the 00:27:51.540 --> 00:27:57.120 crisis response part of Microsoft. He and I immediately leave my office. 00:27:57.120 --> 00:28:03.840 We walk down the hallway to the office of the crisis manager. His name is Phillip. 00:28:03.840 --> 00:28:11.700 Phillip has worked on many of Microsoft’s most important security crises and he’s chatting in 00:28:11.700 --> 00:28:19.800 his office with a colleague. We show up. He knew that wasn’t a good sign. We look at him and say 00:28:19.800 --> 00:28:27.180 we need to talk. He looks at his colleague and says I’ll talk to you later. Then I said we have a 00:28:27.180 --> 00:28:35.040 zero-day. He just knew by the two of us showing up in that fashion that something bad was happening. 00:28:35.040 --> 00:28:41.580 I mostly had two emotions going on; one is we need to get all 00:28:41.580 --> 00:28:47.580 of Microsoft engaged on this that can do something about this. That is an impetus 00:28:47.580 --> 00:28:51.660 to go make sure you’re informing people so they understand the severity right 00:28:51.660 --> 00:28:58.200 away and they can begin the mobilization process. Then the other side of me was 00:28:58.200 --> 00:29:04.620 what is really going on out there? I have one crash report. I don’t know if there are a hundred 00:29:04.620 --> 00:29:10.500 thousand out there just like this and this is a worm that is raging across different parts of the 00:29:10.500 --> 00:29:19.560 world, or this is something that’s just getting started. I immediately wanted to go and get some 00:29:19.560 --> 00:29:23.880 better situational awareness about scope and scale to know what we were really dealing with. 00:29:23.880 --> 00:29:29.460 JACK: John and Andrew briefed the Crisis Manager on what’s going on, that a very serious wormable 00:29:29.460 --> 00:29:34.320 and extremely critical vulnerability is present in Windows and needs to be fixed immediately. 00:29:34.320 --> 00:29:40.140 [MUSIC] While they found this in Windows XP, the vulnerability existed in more than just XP. 00:29:40.140 --> 00:29:46.020 JOHN: Yes, and it affected every version of Windows up until we had at that point. 00:29:46.020 --> 00:29:52.680 Microsoft has a crisis response process that they invoke when one of these things occur. 00:29:52.680 --> 00:30:03.360 It’s called a CRP, internally. Then a call bridge is stood up and then all of the crisis 00:30:03.360 --> 00:30:07.920 partners across the company join that call and then Phillip would take them through; 00:30:07.920 --> 00:30:14.820 here’s a summary of the situation. Generally speaking, if there’s a vulnerability in Windows, 00:30:14.820 --> 00:30:21.600 teams know what to do. This would involve a lot more teams because they had to be ready 00:30:21.600 --> 00:30:28.020 for customer support calls. If there’s an attack, what is the malware? What does the threat side 00:30:28.020 --> 00:30:32.400 of it look like? The anti-malware team will start building signatures for that. We knew 00:30:32.400 --> 00:30:38.460 we had to prepare data for all of the security partner companies across Symantec, MacAfee, 00:30:38.460 --> 00:30:46.500 or whatnot for them to help go protect their customers and the ecosystem. The engineering 00:30:46.500 --> 00:30:50.880 team needs to go okay, what do we have to do to fix this vulnerability? Are there any 00:30:50.880 --> 00:30:57.380 others just like it waiting that we need to fix at the same time so we get the patch dead right? 00:30:57.380 --> 00:31:02.760 JACK: A huge conference call was set up with pretty much someone representing all the 00:31:02.760 --> 00:31:07.680 different departments of Microsoft. The goal was to get everyone engaged in helping solve this as 00:31:07.680 --> 00:31:13.020 quickly as possible. This vulnerability was much more severe than any of the others they found. 00:31:13.020 --> 00:31:17.580 JOHN: Once I knew that the right people were engaged and working to get the right patch 00:31:17.580 --> 00:31:22.440 out the door, the thing I could go help on was how often is this happening and where is 00:31:22.440 --> 00:31:29.160 it happening? I call that dialing into the crash buckets to find any information I could about how 00:31:29.160 --> 00:31:37.140 often this was occurring. I was able to bring back the situational awareness to the crisis response 00:31:37.140 --> 00:31:44.880 that said okay, I saw it five more times today. It just spread from Malaysia to Japan to Singapore to 00:31:44.880 --> 00:31:51.720 whatnot and they can kind of get a sense of okay, it’s the same attack payload communicating with 00:31:51.720 --> 00:31:56.940 the same IP address and then you’re gonna have to pull it down. But it’s spreading from geography 00:31:56.940 --> 00:32:04.020 to geography to geography. It didn’t look like a worm but it looked like some set of hackers that 00:32:04.020 --> 00:32:09.540 were going around using this tool across a variety of targets, expanding its scope every single day. 00:32:09.540 --> 00:32:16.020 JACK: See, that’s fascinating. Somebody knew this vulnerability. They found this before 00:32:16.020 --> 00:32:21.060 Microsoft did and was using it. This wasn’t just some accidental bit flip or something 00:32:21.060 --> 00:32:28.340 that caused the crash report. This was, actually, somebody had coded this and was using it in Asia. 00:32:28.340 --> 00:32:34.740 JOHN: That’s right, so somebody found this vulnerability, probably got the bonus of the 00:32:34.740 --> 00:32:41.880 year for finding it. Then it was [00:35:00] weaponized and then some group unknown to 00:32:41.880 --> 00:32:49.800 this day was using it in those geographies where we saw these crash reports coming. 00:32:49.800 --> 00:32:56.700 The tool was not completely reliable and so the crashes that I were seeing, that’s when it 00:32:56.700 --> 00:33:04.260 failed. But obviously, it worked and it succeeded a bunch of the time, and I didn’t know how often. 00:33:04.260 --> 00:33:08.880 For every crash is that one in a hundred times that it’s crashing or is it one in a thousand? 00:33:08.880 --> 00:33:16.680 What shadow of the real activity am I seeing? That was an unknown that we were working against 00:33:16.680 --> 00:33:22.020 but I could use this to get a sense of its spread and its reach and its pace. 00:33:22.020 --> 00:33:26.340 JACK: This became a top priority for the development team to fix. They dove into 00:33:26.340 --> 00:33:30.300 the code and fiercely started to rewrite the code to fix this problem [MUSIC] and they did; 00:33:30.300 --> 00:33:35.040 they fixed the vulnerability. But this is not the end of our story, no, no, 00:33:35.040 --> 00:33:41.040 no. This is just the genesis. Stay with us because after the break, we’ll hear how this 00:33:41.040 --> 00:33:56.460 went on to become one of the most notorious bugs ever to be found in Windows. Now that a fix was 00:33:56.460 --> 00:34:01.500 written and a patch was ready, a decision had to be made about how to release this to the world. 00:34:01.500 --> 00:34:08.160 JOHN: Right, so there’s a decision point as part of this response process which is okay, 00:34:08.160 --> 00:34:14.640 when is the patch gonna be ready? The ideal time to release a patch is on Patch Tuesday 00:34:14.640 --> 00:34:21.240 ‘cause every IT team in every company knows on the second Tuesday of every month there will be 00:34:21.240 --> 00:34:27.360 a set of security patches from Microsoft and they know to organize their patching of their 00:34:27.360 --> 00:34:33.300 fleet of systems and take their down-time and their maintenance time to go put them on. They 00:34:33.300 --> 00:34:38.700 know some of them will be severe and they should do that right away. IT teams are best ready to 00:34:38.700 --> 00:34:44.520 consume those updates and put them out at that time. The other option is you just ship it when 00:34:44.520 --> 00:34:50.640 you’re ready. We call that going out of band. That gets the solution out there sooner but 00:34:50.640 --> 00:34:57.780 at the same time, none of those IT teams are necessarily ready and poised to go grab it. 00:34:57.780 --> 00:35:03.120 We had a situation where there obviously was an attack in the wild that attackers were doing and 00:35:03.120 --> 00:35:09.000 we wanted to go stop that from happening but we knew as soon as we put this out, copycat attacks 00:35:09.000 --> 00:35:13.740 would happen by people reverse-engineering the patch and understanding. That would spawn 00:35:13.740 --> 00:35:18.780 a whole other wave of attacks and so there was this weighing of if you go out of band, 00:35:18.780 --> 00:35:24.060 you could spawn a bunch of copycats of a very damaging vulnerability when IT teams are not 00:35:24.060 --> 00:35:28.020 ready to go put it on and there could be more victimization, theoretically. 00:35:28.020 --> 00:35:35.280 The bottom line in terms of our calculus on it was we had a very severe vulnerability. It was 00:35:35.280 --> 00:35:42.000 wormable. It affected every version of Windows. We had a solution and we went out of band. 00:35:42.000 --> 00:35:46.800 JACK: This vulnerability was so severe that they decided to not wait until Patch 00:35:46.800 --> 00:35:51.120 Tuesday and just push this out immediately, as soon as they got it. This patch went out 00:35:51.120 --> 00:35:58.440 in 2008 and it was the 67th patch of the year which famously made this MS08-067. 00:35:58.440 --> 00:36:03.900 JOHN: There were two most interesting things that happened after we went out of band. The 00:36:03.900 --> 00:36:09.840 first one was that attacker where I was seeing crash reports still coming in every day until 00:36:09.840 --> 00:36:19.500 we went public. As soon as we went public that attack disappeared and I never saw him again. 00:36:19.500 --> 00:36:26.640 JACK: How can you trace that? You have a signature for certain machines that kept getting attacked? 00:36:26.640 --> 00:36:31.080 JOHN: Well, part of the attack details [00:40:00] that were hidden – so one, 00:36:31.080 --> 00:36:35.100 the shell code and egg hunt were kind of a fingerprint that let me see, 00:36:35.100 --> 00:36:40.500 that were consistent across all the crashes I had gotten to date for this issue. 00:36:40.500 --> 00:36:47.280 All of them were contacting the same IP address of a server in Japan and downloading a payload 00:36:47.280 --> 00:36:53.460 from it. All of that was consistent. Then the day the patch goes live and we announce it, 00:36:53.460 --> 00:37:01.380 I see no more crash reports from this anymore. Then there’s this period where, 00:37:01.380 --> 00:37:09.660 for a number of hours or a day, nothing’s coming in anymore. Then we start to see new 00:37:09.660 --> 00:37:17.040 crash reports for the same issue that were clearly security companies reproducing this vulnerability. 00:37:17.040 --> 00:37:21.660 They had reverse-engineered what we had fixed in the patch. They were writing their own proof of 00:37:21.660 --> 00:37:28.560 concepts to crash it; not write exploits for it but just to probe the vulnerability to make sure 00:37:28.560 --> 00:37:36.420 they understood it. We started seeing those crash reports come in from the security researcher side. 00:37:36.420 --> 00:37:44.340 Soon enough, a few days later after that, we started to see new attacks from the first wave 00:37:44.340 --> 00:37:50.220 attackers that was clearly different and new than the original ones, that look like bots 00:37:50.220 --> 00:37:57.060 or botnet programs adopting this as a spreading mechanism. We started to see that wave as well. 00:37:57.060 --> 00:38:01.860 JACK: See, this is what I mean by this being only the genesis. Now that the patch is out there, 00:38:01.860 --> 00:38:07.140 both white hat and black hat hackers analyzed the patch to figure out what this exploit was 00:38:07.140 --> 00:38:13.260 and how to run it which means now this tool went from being just used by a single hacking group 00:38:13.260 --> 00:38:19.740 somewhere in the world to now being known by the general hacker community. Within a short time, 00:38:19.740 --> 00:38:24.660 it would become available for anyone in the world to just download and use. [MUSIC] Isn’t that 00:38:24.660 --> 00:38:30.120 a strange dilemma or decision to have to make, though? Knowing that if you put a patch out, this 00:38:30.120 --> 00:38:35.160 reveals the vulnerability to the world for any hacker to use? But Microsoft has a duty to make 00:38:35.160 --> 00:38:39.780 secure products so they absolutely have to release a patch whenever they do find a vulnerability 00:38:39.780 --> 00:38:44.940 like this because this has far-reaching effects on helping people stay secure. 00:38:44.940 --> 00:38:54.060 JOHN: In the first week it was on Windows Update and within seven days I think we had 00:38:54.060 --> 00:39:00.420 patched four hundred million machines. This is sort of the awesome part about Windows Update; 00:39:00.420 --> 00:39:06.900 it’s a system that had been built to patch the Windows ecosystem at scale and this is 00:39:06.900 --> 00:39:11.280 one of those times when you really needed it. It came through in terms of our ability 00:39:11.280 --> 00:39:19.620 to essentially inoculate a huge swath of the world against this vulnerability 00:39:19.620 --> 00:39:25.340 in a very short period of time. It was very effective about that. 00:39:25.340 --> 00:39:29.100 JACK: It’s hard to know for sure but my best research tells me there was 00:39:29.100 --> 00:39:33.720 about one billion Windows computers in the world at that time and this vulnerability 00:39:33.720 --> 00:39:38.520 affected all of them. By having four hundred million of them patched in the first week, 00:39:38.520 --> 00:39:44.940 that was a huge win for helping the world become more secure. 40% of the Windows computers in the 00:39:44.940 --> 00:39:49.680 world were no longer vulnerable to this right away. That’s awesome and amazing. 00:39:49.680 --> 00:39:56.400 JOHN: All of this was in October and sometime in early December, 00:39:56.400 --> 00:40:03.480 a Conficker worm that had been this sort of small-scale thing for a time, 00:40:03.480 --> 00:40:09.060 adopted this vulnerability as a spreading mechanism and then it began to use that 00:40:09.060 --> 00:40:15.600 against systems around the world that had not put this patch on yet. If you think about well, what 00:40:15.600 --> 00:40:22.680 if only 1% of the Windows PCs in the world don’t patch? That’s still millions of systems. [MUSIC] A 00:40:22.680 --> 00:40:30.300 lot of the damage that was done by Conficker which was significant, imagine what would have happened 00:40:30.300 --> 00:40:37.040 if it had a half a billion more PCs that were vulnerable and not patched against this issue. 00:40:37.040 --> 00:40:42.960 JACK: Conficker was the first big attack to use this exploit. Just months after John discovered 00:40:42.960 --> 00:40:48.240 this vulnerability, Conficker figured it out and used it to arm itself to infect Windows machines. 00:40:48.240 --> 00:40:55.860 Because the vulnerability in MS08-067 was so effective, Conficker spread rapidly worldwide, 00:40:55.860 --> 00:41:01.380 ultimately infecting computers in over 190 countries worldwide. It would eventually 00:41:01.380 --> 00:41:07.740 infect millions of computers. Conficker was spreading in a terrible way. Think about how 00:41:07.740 --> 00:41:12.900 horrible this is, for some hacker group to have full control of millions of Windows 00:41:12.900 --> 00:41:17.640 computers worldwide. I’m talking about government agencies, businesses, home PCs. 00:41:17.640 --> 00:41:22.560 The hackers could see all the files on them, run any clients they wanted, install key loggers, 00:41:22.560 --> 00:41:27.000 take screenshots, install rootkits, or do whatever they want on these computers. 00:41:27.000 --> 00:41:31.620 It’s so frightening to think about that. [00:45:00] Conficker continued to spread, 00:41:31.620 --> 00:41:37.800 seemingly unstoppable. By January, 30% of Windows computers still did not apply the patch to protect 00:41:37.800 --> 00:41:43.740 themself from MS08-067 or Conficker, which means hundreds of millions of computers were 00:41:43.740 --> 00:41:49.080 still vulnerable to this. Conficker had a field day with everyone who didn’t bother to patch and 00:41:49.080 --> 00:41:55.320 eventually was able to infect ten million Windows computers worldwide. Ten million. 00:41:55.320 --> 00:42:02.520 As far as I know, this makes Conficker the largest worm ever, all thanks to MS08-067. 00:42:02.520 --> 00:42:10.380 JOHN: It was disappointing because – at the same time a strong lesson because we had put the patch 00:42:10.380 --> 00:42:15.060 out, right? The job is done. It’s time for the world to put the patch on their systems. That’s 00:42:15.060 --> 00:42:21.300 the step they have to do. There isn’t anything further that we thought we could do and yet 00:42:21.300 --> 00:42:26.760 look what happened and what came after that; all of this damage from Conficker. Now, 00:42:26.760 --> 00:42:32.220 Conficker had other ways to spread, through USB drive infections and scanning file shares 00:42:32.220 --> 00:42:37.680 and so forth, but still there was a large part of the world that had not patched against this 00:42:37.680 --> 00:42:46.020 vulnerability. It was a bit of a lesson for just how damaging things can become and how much of 00:42:46.020 --> 00:42:50.880 the world can be exposed to these attacks even once we think we’ve done our part of the job. 00:42:50.880 --> 00:42:55.500 JACK: It’s fascinating to me to see that John here was the one who decided to take 00:42:55.500 --> 00:43:00.780 it upon himself to look at that crash report and to discover this. If it wasn’t for him, 00:43:00.780 --> 00:43:04.290 who knows how long this would have stayed out in the wild before being discovered? 00:43:04.290 --> 00:43:13.320 JOHN: Yeah, I think these operators using this tool thought they had a really great new attack 00:43:13.320 --> 00:43:20.760 and probably would have used it if it was more reliable and we had not seen this for potentially 00:43:20.760 --> 00:43:30.120 years if they were stealthy and choosey about how they used it. Sometimes these exploits we do learn 00:43:30.120 --> 00:43:35.820 about zero-day have been in use for years in very disciplined ways. The noisier operators are with 00:43:35.820 --> 00:43:41.160 it, the more likely that some victim is gonna find out about it and somehow, they’ll get a whiff of 00:43:41.160 --> 00:43:48.780 how the attack is working and the thing can get patched. But I think if we had not have seen 00:43:48.780 --> 00:43:53.700 this one in this way, very likely this operator could have continued for a very long time with it. 00:43:53.700 --> 00:44:00.420 JACK: [MUSIC] Hm, this makes me think about how stealthy some hackers could be. I mean, 00:44:00.420 --> 00:44:06.420 imagine if the hackers disabled WER or blocked all connections to Microsoft. This would have 00:44:06.420 --> 00:44:10.920 been an effective technique to keep Microsoft from seeing these crash reports and discovering 00:44:10.920 --> 00:44:15.720 this. But maybe that’s going too crazy with it. But see, sophisticated hackers 00:44:15.720 --> 00:44:21.240 take extreme measures to hide their tracks. I mean, that’s almost one of your adversaries, 00:44:21.240 --> 00:44:27.360 right, are these APTs like the NSA. You’re battling with them like it’s a arms race 00:44:27.360 --> 00:44:33.420 between you as the vendor and them as the aggressor. Do you feel that way sometimes? 00:44:33.420 --> 00:44:38.940 JOHN: It does feel like that way sometimes in the sense of there are people that have weaponized 00:44:38.940 --> 00:44:46.080 vulnerabilities and are using in the wild. In a way, the defenders that are at Microsoft, 00:44:46.080 --> 00:44:52.440 Google, and other companies don’t care who these people are or why they’re doing it. We just want 00:44:52.440 --> 00:44:58.260 to find what they’re doing and take those tools away from them. Every time a zero-day is found in 00:44:58.260 --> 00:45:04.800 the wild by some defender or organization and it gets patched, that is a happy day for me. 00:45:04.800 --> 00:45:12.120 JACK: What a strange thing to think about. Does that put you in deep thought too? The 00:45:12.120 --> 00:45:17.280 truth of the matter is that the NSA is actively looking for vulnerabilities in Windows so that 00:45:17.280 --> 00:45:22.860 they can use that against their adversaries. Then here’s John actively trying to figure out 00:45:22.860 --> 00:45:28.620 what the NSA is up to so he can basically expose their secret weapons. I don’t know what to think 00:45:28.620 --> 00:45:32.940 about that. I don’t know who the bad guy or the good guy is in this. NSA is supposed to 00:45:32.940 --> 00:45:37.500 be working towards keeping our country safe but at the same time, they have to develop 00:45:37.500 --> 00:45:43.500 cyber-weapons to attack other nations, so it almost seems like the NSA would see John and 00:45:43.500 --> 00:45:49.500 Microsoft as the enemy. John might see the NSA as the enemy. I just never thought about how 00:45:49.500 --> 00:45:55.940 these two would be battling each other like this. It’s just wild to think about this relationship. 00:45:55.940 --> 00:46:02.760 JOHN: Yeah, and then in many ways I feel like I relived this whole moment a couple of years 00:46:02.760 --> 00:46:11.280 ago when the EternalBlue exploit was discovered. This is one of these NSA tools that the Shadow 00:46:11.280 --> 00:46:21.420 Brokers group leaked onto the internet. A patch was produced for it; that was MS17-010. A couple 00:46:21.420 --> 00:46:28.200 months later, the WannaCry worm was unleashed and spread in a very similar way. It had a more 00:46:28.200 --> 00:46:32.600 destructive payload and [00:50:00] ran across the globe against systems that had not patched. 00:46:32.600 --> 00:46:39.420 JACK: Now, if you recall in earlier episodes, NSA actually did tell Microsoft about EternalBlue just 00:46:39.420 --> 00:46:44.460 before the Shadow Brokers published it to the world, giving Microsoft an early warning who 00:46:44.460 --> 00:46:49.200 were able to move quickly and patch this before it was released. I’m guessing the NSA knew it was 00:46:49.200 --> 00:46:54.660 going to be published and wanted to help the world just stay slightly ahead of the game. 00:46:54.660 --> 00:46:59.940 But that must have been an awkward phone call, or whatever; for Microsoft to get the memo that 00:46:59.940 --> 00:47:05.940 NSA has found a devastating exploit in Windows and this exploit got leaked to Shadow Brokers 00:47:05.940 --> 00:47:12.120 and they’re about to leak it to the world. Phew, I don’t know. I don’t know, it’s a weird, tangly 00:47:12.120 --> 00:47:17.160 mess when you get into the relationships between NSA and Microsoft. Just to be clear, there’s no 00:47:17.160 --> 00:47:23.760 link between MS08-067 being found by the NSA. Oh, something happened last week that’s kind 00:47:23.760 --> 00:47:30.060 of interesting, too. Last week was Patch Tuesday and boy, was it a doozy. There was a bug patched. 00:47:30.060 --> 00:47:35.400 It was a cryptographic API bug in Windows 10 which basically allows an attacker to pose as 00:47:35.400 --> 00:47:40.080 someone they aren’t and your computer could send trusted information to it. But here’s the thing; 00:47:40.080 --> 00:47:47.280 this bug was reported to Microsoft by the NSA. In fact, it was so important, the NSA even held 00:47:47.280 --> 00:47:52.800 a press conference to urge people to patch. This is very rare; I mean, we don’t know how 00:47:52.800 --> 00:47:57.480 many times the NSA has reported bugs to Microsoft. They could be doing this all the time. But we do 00:47:57.480 --> 00:48:01.680 know for sure that there were two times that they did do it; once when Shadow Brokers got 00:48:01.680 --> 00:48:08.100 the NSA’s exploits, and now again last week. The NSA says they told Microsoft because they want 00:48:08.100 --> 00:48:15.000 to build more trust with people and help keep computers secure. Hm, really? It could be that. 00:48:15.000 --> 00:48:20.100 Our world is changing and new things are happening and the NSA might be doing this from now on to try 00:48:20.100 --> 00:48:24.000 to keep the country safer by working with vendors to get things patched. In fact, 00:48:24.000 --> 00:48:29.580 they have done stuff like that for a while but it’s all been small potatoes versus the god-mode 00:48:29.580 --> 00:48:36.420 bugs that the NSA keeps to themselves. My theory is this; the NSA gave this bug to Microsoft. Why? 00:48:36.420 --> 00:48:42.240 Maybe it was just to build better PR. Okay. But then maybe the NSA knows something we don’t, 00:48:42.240 --> 00:48:46.740 like maybe they uncovered a huge man-in-the-middle campaign that some 00:48:46.740 --> 00:48:51.120 foreign government was doing to many Americans and thought it could have devastating results, 00:48:51.120 --> 00:48:57.060 so this was their way to stop it. Or maybe the NSA lost another exploit and didn’t want their 00:48:57.060 --> 00:49:02.640 enemy to have it. I don’t know but I have a feeling there’s more to this story, though. 00:49:02.640 --> 00:49:09.840 [MUSIC] Back to Conficker. You might be wondering what happened there. Like I was saying, 00:49:09.840 --> 00:49:14.940 Conficker infected ten million computers and was growing. However, it was a mystery as to what 00:49:14.940 --> 00:49:20.280 Conficker actually did or who made it or what it was supposed to do. While it was infecting systems 00:49:20.280 --> 00:49:26.100 worldwide, it was apparently not doing anything once it was infected. On a periodic basis, 00:49:26.100 --> 00:49:30.660 the computer that had Conficker running on it would reach out to certain domains 00:49:30.660 --> 00:49:35.400 to receive instructions on what to do next, but it just didn’t get any instructions to 00:49:35.400 --> 00:49:40.800 do. Security teams all over the world feared that maybe there were instructions to do on 00:49:40.800 --> 00:49:46.620 a particular day and in fact, for some weird reason, we thought that on April 1st, 2009, 00:49:46.620 --> 00:49:50.640 there was gonna be some big surprise that Conficker was going to give us. I remember 00:49:50.640 --> 00:49:54.780 being in the office that day and setting up a conference bridge in a war room with everyone 00:49:54.780 --> 00:50:00.240 from IT and looking for signs of Conficker kicking up or something, but nothing happened. 00:50:00.240 --> 00:50:05.820 A few companies got really fed up with Conficker spreading everywhere so they decided to do 00:50:05.820 --> 00:50:11.640 something about it. In February 2009, something called the Microsoft Cabal was formed. This 00:50:11.640 --> 00:50:17.880 included people from Microsoft, Verizon, NEWSTAR, America Online, Symantec, F-Secure, 00:50:17.880 --> 00:50:22.860 researchers from Georgia Tech in the Shadowserver Foundation, and so many more organizations. It was 00:50:22.860 --> 00:50:27.360 a huge list of companies that came together to figure out a way to stop Conficker. They 00:50:27.360 --> 00:50:31.560 would do things like reverse-engineer Conficker to see how it worked and then write fixes to 00:50:31.560 --> 00:50:35.940 block Conficker from spreading more. But then, whoever made Conficker would change how it was 00:50:35.940 --> 00:50:40.320 infecting machines so it could keep infecting machines, creating a new variant of the worm. 00:50:40.320 --> 00:50:44.460 It became a game of cat and mouse, with the security professionals blocking it and the 00:50:44.460 --> 00:50:50.100 worm creator getting around that. At some point, Microsoft said they’re willing to pay 00:50:50.100 --> 00:50:54.240 $250,000 for information that leads up to the capture of whoever created the 00:50:54.240 --> 00:50:58.980 Conficker worm. They were taking this pretty serious. The FBI started getting involved 00:50:58.980 --> 00:51:04.200 and the hunt began to look for whoever was running this worm. The author Mark Bowden 00:51:04.200 --> 00:51:09.960 did some extraordinary research into this worm in his book which is just titled Worm. 00:51:09.960 --> 00:51:14.400 He writes that there are a few theories in what Conficker was. One is that is was just a 00:51:14.400 --> 00:51:19.140 security researcher playing around, making a crazy worm, but never intended to do any harm with it; 00:51:19.140 --> 00:51:23.820 just trying to see how big it could get. Another theory was that a government created this worm and 00:51:23.820 --> 00:51:29.160 was waiting for instructions to maybe attack on command or spy on people or something. 00:51:29.160 --> 00:51:34.320 [00:55:00] But as more organizations joined the Microsoft Cabal, their more effort put 00:51:34.320 --> 00:51:39.060 into looking into this, and they may have found the answer. They reverse-engineered 00:51:39.060 --> 00:51:42.480 the code and did everything they could to trace it back to their creators. 00:51:42.480 --> 00:51:48.600 They handed this information over to the FBI who then arrested three Ukrainians; Sergey, 00:51:48.600 --> 00:51:54.420 Yevgen, and Dmytro. These Ukrainians all were millionaires. They drove black Porsches and 00:51:54.420 --> 00:51:59.040 they lived in penthouse apartments. Their story was that they ran a website with employees and 00:51:59.040 --> 00:52:04.500 everything and they paid themselves. But they weren’t paying themselves very much. They were 00:52:04.500 --> 00:52:10.500 arrested on tax evasion charges and the feds seemed to have found some evidence of Conficker 00:52:10.500 --> 00:52:15.960 code on their work computers but I don’t think we have any idea what happened to them next. 00:52:15.960 --> 00:52:20.520 It doesn’t seem like the FBI was able to extradite them to the US and they 00:52:20.520 --> 00:52:27.600 just disappeared into the Ukrainian courts. [MUSIC] But the FBI also arrested one other guy; 00:52:27.600 --> 00:52:33.300 a Swede named Mikael. Mikael was arrested in Denmark in connection with this and 00:52:33.300 --> 00:52:38.880 extradited to the US. The court records don’t say anything about Conficker, though. Instead, 00:52:38.880 --> 00:52:44.400 the FBI found evidence that Mikael was infecting computers and putting scareware on them. This is 00:52:44.400 --> 00:52:48.600 where he would infect a computer and say there’s a virus on it and you need to buy this antivirus. 00:52:48.600 --> 00:52:53.400 But when the victim buys the antivirus, nothing actually gets fixed. The FBI claims 00:52:53.400 --> 00:52:59.580 Mikael made seventy-one million dollars from his scareware campaigns which is a really big haul. 00:52:59.580 --> 00:53:06.000 To get that much, you must have had a lot of infected machines and there was evidence in 00:53:06.000 --> 00:53:11.580 some of the variants of Conficker that it was capable of running scareware, so it might have 00:53:11.580 --> 00:53:18.300 been getting ready to launch a big campaign to do just that, but it never did. Mikael got two 00:53:18.300 --> 00:53:23.700 years in prison for his scareware scams that he was running and it’s alleged that he had ties 00:53:23.700 --> 00:53:30.360 to those Ukrainians who also got arrested over Conficker. It seems like the best theory now was 00:53:30.360 --> 00:53:35.880 that Conficker was made by a group of Ukrainian cyber-criminals who may have been planning on 00:53:35.880 --> 00:53:43.320 using it to send spam e-mails or running scareware to scams its victims, but they never got to it. 00:53:43.320 --> 00:53:47.700 What’s truly fascinating about Conficker is that it’s still out there, infected on 00:53:47.700 --> 00:53:54.360 a ton of systems. Even though MS08-067 was patched in 2008, there are still computers 00:53:54.360 --> 00:53:58.200 out there that are running systems older than that that haven’t been patched still, 00:53:58.200 --> 00:54:07.200 and the latest estimate is that Conficker is still present on 400,000 computers today. 00:54:07.200 --> 00:54:12.480 MS08-067 will go down in history as one of the most notorious vulnerabilities in Windows, 00:54:12.480 --> 00:54:17.400 ever. The reason for it is because of how effective it is. I personally love playing 00:54:17.400 --> 00:54:21.540 around with this vulnerability and exploiting Windows computers with it because it’s so easy 00:54:21.540 --> 00:54:26.760 to do. I want to walk you through how I’ve done it. First, you need a version of Windows from 00:54:26.760 --> 00:54:31.560 before 2008 which is actually quite easy; you just install Windows XP on a computer and don’t 00:54:31.560 --> 00:54:36.300 patch it. This will be vulnerable to it. Then you need to run this exploit against it. Now, 00:54:36.300 --> 00:54:39.840 instead of knowing what shell code to send to the computer and work all that out, 00:54:39.840 --> 00:54:45.180 there’s a crazy shortcut. It’s called Metasploit. Metasploit is an incredible hacking framework. It 00:54:45.180 --> 00:54:51.840 has over a thousand exploits all pre-programmed and ready to run. You pick the MS08-067 exploit, 00:54:51.840 --> 00:54:56.820 then point at your Windows XP machine, type run, and boom. You’re in. 00:54:56.820 --> 00:55:02.460 Now, when I mean you’re in, you’re really in. Metasploit has tools to allow you to use that 00:55:02.460 --> 00:55:06.900 computer you just infected as if you’re right in front of it. You can run any command you want 00:55:06.900 --> 00:55:11.280 on that computer, all through the command line, take screenshots of the desktop, enable the mic, 00:55:11.280 --> 00:55:16.080 enable the camera, run a key logger to watch what someone types. You can do all that and 00:55:16.080 --> 00:55:21.180 more. Metasploit is an amazing hacker tool which is the standard for any hacker to know how to use 00:55:21.180 --> 00:55:26.280 today. The best part about Metasploit is that it’s free and open-source. Anyone can grab it, 00:55:26.280 --> 00:55:30.840 study a few commands, and have over a thousand exploits ready at their fingertips. It’s really 00:55:30.840 --> 00:55:36.180 powerful and fun to play with, and if you attend any ethical hacking training, chances are you’ll 00:55:36.180 --> 00:55:42.060 be given Metasploit and a system vulnerable with MS08-067 as one of your first hacks you’ll do 00:55:42.060 --> 00:55:46.800 using it, because it’s pretty easy to use and you can see how effective Metasploit can be. 00:55:46.800 --> 00:55:54.420 Because of that, penetration testers all over the world are very familiar with MS08-067 and all have 00:55:54.420 --> 00:56:01.200 that number memorized. However, it’s now 2020 so MS08-067 is a dozen years old now which means 00:56:01.200 --> 00:56:06.540 there are far fewer computers running Windows XP or that are vulnerable to this attack. This bug 00:56:06.540 --> 00:56:12.360 is losing its notoriety. It’s much more rare to find a system vulnerable to this today but they do 00:56:12.360 --> 00:56:18.000 exist. This story is another example on why it’s so important to update your software as soon as 00:56:18.000 --> 00:56:23.640 you can. However, it’s not always that easy. Some networks have very strict controls; like, they 00:56:23.640 --> 00:56:28.920 can’t patch. A patch might break everything. The applications and software running on the computers 00:56:28.920 --> 00:56:34.260 doesn’t always work with the [01:00:00] latest OS updates applied. This quickly becomes a nightmare. 00:56:34.260 --> 00:56:39.660 I recently updated my home computer and a lot of the software that I was running stopped working 00:56:39.660 --> 00:56:44.700 so I had to wait for each application maker to put out an update for me to be back up 00:56:44.700 --> 00:56:49.920 and working again. Something like this is totally unacceptable in critical networks like hospitals 00:56:49.920 --> 00:56:55.020 or power plants. It’s not always as simple as just patching it. Like I was saying at the beginning, 00:56:55.020 --> 00:56:59.460 it’s a lot of different people’s jobs to keep our network secure. In this case, 00:56:59.460 --> 00:57:04.260 Microsoft did their job at finding a bug and issuing a patch for it, and now that fix needs to 00:57:04.260 --> 00:57:09.480 trickle all the way down to everyone’s computers. Because by being on the most up-to-date version, 00:57:09.480 --> 00:57:13.500 it is like giving yourself a vaccination to render yourself safe from the known 00:57:13.500 --> 00:57:18.600 attacks in the world. Updating your apps, operating systems and programs, in my opinion, 00:57:18.600 --> 00:57:22.920 is the single most effective thing you can do to protect yourself on the internet today. 00:57:22.920 --> 00:57:31.140 JACK (OUTRO): [OUTRO MUSIC] 00:57:31.140 --> 00:57:35.280 A big thank you to John Lambert from Microsoft for coming on the show and sharing this story 00:57:35.280 --> 00:57:39.960 with us. I found it to be really cool. Also, if you want to know more about Conficker, 00:57:39.960 --> 00:57:44.760 check out the book Worm by Mark Bowden. It goes into a lot of details about it. It’s pretty good. 00:57:44.760 --> 00:57:49.440 Hey, if you’re all caught up with this podcast and want more episodes, check out the Darknet 00:57:49.440 --> 00:57:55.140 Diaries Patreon page. You can find bonus episodes there. This show is made by me, the cyber-ghost, 00:57:55.140 --> 00:58:01.620 Jack Rhysider. Music in this episode was special. Typically, I grab songs from all over but in this 00:58:01.620 --> 00:58:07.500 one, every single song was created by the top talented Breakmaster Cylinder. Even though 00:58:07.500 --> 00:58:13.560 hoodies go up and drawstrings get pulled tight every time I say it, this is Darknet Diaries.