WEBVTT 00:00:00.579 --> 00:00:05.400 JACK: Hey, it’s Jack, host of the show. Now, a lot of you write to me and tell me your favorite 00:00:05.400 --> 00:00:10.200 episodes are the ones with social engineers or penetration testers. Yeah, sure, being on the red 00:00:10.200 --> 00:00:15.660 team is fun, to break into things, but my heart is with the blue team, the defenders of the network, 00:00:15.660 --> 00:00:20.520 because that’s what I did for ten years professionally. I was configuring firewalls, 00:00:20.520 --> 00:00:24.900 intrusion detection systems, and reviewing logs to find threats in the network. 00:00:24.900 --> 00:00:30.300 I felt like it was my job to stop or restrict bad things from happening in my clients’ networks. It 00:00:30.300 --> 00:00:35.880 was a game of Cat and Mouse. I had to learn what the bad guys knew so I could stop them and I’ll 00:00:35.880 --> 00:00:41.880 tell you, it was exciting. At times it felt like the battle at Helm’s Deep, with a never-ending 00:00:41.880 --> 00:00:48.000 onslaught of attackers and I had to embody Legolas to defend them off one at a time. Now, this story 00:00:48.000 --> 00:00:54.600 is about a defender and how she uncovered a serious breach in the network of a major bank. 00:00:54.600 --> 00:00:59.520 (INTRO): [INTRO MUSIC] These are 00:00:59.520 --> 00:01:05.760 true stories from the dark side of the internet. 00:01:05.760 --> 00:01:15.660 I’m Jack Rhysider. This is Darknet Diaries. [INTRO MUSIC ENDS] 00:01:15.660 --> 00:01:26.220 JACK: Alright, so you ready to get into it? 00:01:26.220 --> 00:01:27.300 AMÉLIE: Yeah, sure. 00:01:27.300 --> 00:01:31.220 JACK: Alright, so let’s start with what’s your name? 00:01:31.220 --> 00:01:32.220 AMÉLIE: 00:01:32.220 --> 00:01:40.080 My name is Amélie Koran. That wasn’t always my name but that’s a story for another time. 00:01:40.080 --> 00:01:41.960 JACK: What are you known as online? 00:01:41.960 --> 00:01:46.320 AMÉLIE: My handle is webjedi although back when I was in the BBS world, 00:01:46.320 --> 00:01:51.360 it was Thunderball ‘cause I was a big fan of James Bond, but the webjedi moniker has 00:01:51.360 --> 00:01:55.260 been mine for probably a quarter century, so I’ll stick with that. 00:01:55.260 --> 00:02:00.540 JACK: That is a really cool name. How did it come along, webjedi? 00:02:00.540 --> 00:02:06.720 AMÉLIE: So, I entered college back in the early nineties, so ‘93, and being the nerd that I am, 00:02:06.720 --> 00:02:11.580 I’m a big fan of science fiction and particularly Star Wars. About that same 00:02:11.580 --> 00:02:17.040 time is the birth of the web. So, ‘93, ‘94, you saw the first couple websites show up online. 00:02:17.040 --> 00:02:19.980 JACK: This was back in her college days. She was going to school at 00:02:19.980 --> 00:02:23.580 Carnegie Mellon at the time where she was studying electrical engineering 00:02:23.580 --> 00:02:27.150 and computer science. She was building this website on the school’s computers. 00:02:27.150 --> 00:02:29.250 AMÉLIE: It was a Star Wars multimedia archive. 00:02:29.250 --> 00:02:33.300 JACK: As she was building this site, she was looking around at what other websites did, 00:02:33.300 --> 00:02:35.940 and there weren’t actually many on the internet at that time. 00:02:35.940 --> 00:02:40.440 AMÉLIE: So I busted my chops to create a fan website, you know, 00:02:40.440 --> 00:02:41.940 one of the first thousand websites on the internet. 00:02:41.940 --> 00:02:45.840 JACK: One of the things she noticed is people who created websites were referring 00:02:45.840 --> 00:02:51.390 to themselves as web masters. [MUSIC] She thought about this term, web master. 00:02:51.390 --> 00:02:57.420 AMÉLIE: What’s better than a web master? I figured a Jedi master, so I took webjedi. 00:02:57.420 --> 00:03:02.040 JACK: So, webjedi became her screen name and that still carries over to 00:03:02.040 --> 00:03:04.680 today. I mean, webjedi is her Twitter name now. 00:03:04.680 --> 00:03:08.340 AMÉLIE: For the longest time, my e-mail address was jedi@cmu.edu, 00:03:08.340 --> 00:03:11.400 so I kinda stuck with that for the time I was there. 00:03:11.400 --> 00:03:13.500 JACK: She was really fascinated with the internet, 00:03:13.500 --> 00:03:18.300 learning all kinds of stuff while at CMU, even doing things that weren’t taught in class. 00:03:18.300 --> 00:03:20.400 AMÉLIE: But yeah, no, it was a lot of the cases 00:03:20.400 --> 00:03:24.900 of learning a lot of technology stuff that wasn’t getting taught in classes, 00:03:24.900 --> 00:03:30.240 so I used this as my test bed to learn how to set up a web server, a mail server, 00:03:30.240 --> 00:03:37.560 security permissions for files, setting up network ports on shared services. It was a great learning 00:03:37.560 --> 00:03:42.240 tool that definitely got me interested in stuff well beyond what I was being taught in school. 00:03:42.240 --> 00:03:44.280 JACK: She was getting really geeky with it, 00:03:44.280 --> 00:03:48.840 practicing doing things on computers on her free time and picking up all kinds of new skills. 00:03:48.840 --> 00:03:58.080 AMÉLIE: But since I was such a poorly-skilled programmer, I failed out of my CS classes 00:03:58.080 --> 00:04:05.220 rather hard, and switched into a Social Sciences degree, and that’s what I graduated as. But I 00:04:05.220 --> 00:04:09.060 actually spent more time as a computer engineer than I actually did as a social scientist, 00:04:09.060 --> 00:04:16.320 so I had a good mix of the policy theory and information system stuff from the 00:04:16.320 --> 00:04:23.040 social sciences team and then knew all the technical stuff of hardware and analog 00:04:23.040 --> 00:04:29.280 circuits from the EC courses. It was a really weird thing to kind of graduate as, skill-wise. 00:04:29.280 --> 00:04:35.820 JACK: But her heart was in tech and computers, so she pursued jobs as a computer engineer. At first, 00:04:35.820 --> 00:04:41.040 she got a job as a user interface designer at Xerox, then she got a job at a different 00:04:41.040 --> 00:04:44.940 company being a system administrator where she was taking care of the servers in the network, 00:04:44.940 --> 00:04:49.080 updating them, configuring them, keeping them going. Then she moved on from there. 00:04:49.080 --> 00:04:52.740 AMÉLIE: [MUSIC] Just kinda worked up from there; was securing web servers 00:04:52.740 --> 00:04:57.300 for The American Chemical Society and running those. Eventually I ended up to go work for 00:04:57.300 --> 00:05:03.000 Stan Lee of Marvel fame, [00:05:00] running his IT shop when he left Marvel. 00:05:03.000 --> 00:05:05.760 JACK: She then moved out of California and 00:05:05.760 --> 00:05:10.080 got a job at a utility company. They provided gas and electric to people. 00:05:10.080 --> 00:05:14.100 AMÉLIE: It was during a very interesting time because I think within the first year I was there, 00:05:14.100 --> 00:05:19.860 we had a major hurricane roll through and knock out a good swath of power up in the 00:05:19.860 --> 00:05:26.280 Chesapeake Bay. Shortly after that, we then had the northeast blackout. 00:05:26.280 --> 00:05:33.360 It taught a lot about designing for resiliency but also just generally how 00:05:33.360 --> 00:05:37.140 do you work at scale when you don’t have all the resources that you typically would have? 00:05:37.140 --> 00:05:41.160 JACK: She said she was impressed at how this company learned from their mistakes. Yeah, 00:05:41.160 --> 00:05:47.160 sure, a hurricane is not a normal event, but they knew that another one might happen again someday, 00:05:47.160 --> 00:05:51.660 so it’s best to build a more resilient network in case it does happen. So, 00:05:51.660 --> 00:05:57.000 they redesigned the network and built out a pretty robust data-recovery center, a secondary 00:05:57.000 --> 00:06:01.920 place that can handle the full load in the event that their main data center would go down. 00:06:01.920 --> 00:06:07.920 AMÉLIE: I think it came into full effect – is that they had a bathroom explode one day and 00:06:07.920 --> 00:06:13.620 actually flooded parts of the training floor, so that practice kept them running without having 00:06:13.620 --> 00:06:19.500 to worry about if those procedures had actually worked again. But you know, working for a critical 00:06:19.500 --> 00:06:25.320 infrastructure company such as a power company, the mantra was just be a less appetizing target 00:06:25.320 --> 00:06:31.320 than the next network down the road, so a lot of what we built was on detection, not 00:06:31.320 --> 00:06:37.740 necessarily response. I learned a lot there and I carried that forward to where my career is now. 00:06:37.740 --> 00:06:43.860 JACK: It was here where she really got into DFIR. So, DFIR stands for Digital Forensics 00:06:43.860 --> 00:06:48.960 and Incident Response. This is the team to be called in when there’s an incident and they’ll 00:06:48.960 --> 00:06:54.600 handle the situation. I sometimes like to think of the DR team like Winston Wolf in Pulp Fiction. 00:06:54.600 --> 00:06:57.240 WINSTON: You’re Jimmy, right? This is your house? 00:06:57.240 --> 00:06:58.200 JIMMY: Sure is. 00:06:58.200 --> 00:07:01.080 WINSTON: I’m Winston Wolf. I solve problems. 00:07:01.080 --> 00:07:02.340 JIMMY: Good. We got one. 00:07:02.340 --> 00:07:06.840 JACK: See, in big companies, incidents are never handled by one person. First, you have the people 00:07:06.840 --> 00:07:12.360 in the operations room who first saw the alert and notified somebody. Then you have network engineers 00:07:12.360 --> 00:07:16.920 and system administrators who are engaged in investigating it further. You might get someone 00:07:16.920 --> 00:07:21.720 from leadership entering the room, asking tons of questions and wanting updates and is there to make 00:07:21.720 --> 00:07:26.460 decisions, and then you might also have a bunch of angry customers who want to know why their power 00:07:26.460 --> 00:07:31.740 is out, and so customer support might be looking for updates, too. The operation center quickly 00:07:31.740 --> 00:07:37.260 becomes a mess during large incidents, and so the DFIR team steps in to get things under control. 00:07:37.260 --> 00:07:41.520 They get the latest updates and then disseminate that information to everyone who needs to know, 00:07:41.520 --> 00:07:45.360 and they’ll get the right teams engaged to get things under control and work with leadership 00:07:45.360 --> 00:07:50.700 to present the details and work out any big decisions that need to be made. While Amélie 00:07:50.700 --> 00:07:55.020 was doing this type of work at this utility company, she was also a security engineer and 00:07:55.020 --> 00:07:59.640 an architect there, too. She learned a lot from there but then left that place to get 00:07:59.640 --> 00:08:03.990 a job somewhere else for a bit. She got bored there and then went to go work for Mandiant. 00:08:03.990 --> 00:08:07.560 AMÉLIE: As their first full-time IT manager. 00:08:07.560 --> 00:08:11.280 JACK: Mandiant is a security company focusing on incident response and 00:08:11.280 --> 00:08:14.700 threat intelligence. Recently, they were acquired by FireEye. 00:08:14.700 --> 00:08:18.480 AMÉLIE: They were not the Mandiant that most people know about today. They were small, 00:08:18.480 --> 00:08:24.900 if you call sixty people small. But we had three or four people sharing a cube desk, 00:08:24.900 --> 00:08:31.740 we had a lab that was literally a closet. We’re working a bunch of different major cases 00:08:31.740 --> 00:08:39.420 that people look back at and those are the things that – you remember Home Depot, you remember TJX; 00:08:39.420 --> 00:08:42.870 those were cases that they worked but this was back when they were much smaller. 00:08:42.870 --> 00:08:46.860 JACK: After Mandiant, she went to work for the World Bank. 00:08:46.860 --> 00:08:51.000 HOST1: With thousands of employees and 189 member countries, 00:08:51.000 --> 00:08:54.900 the World Band is one of the most powerful institutions in the world, 00:08:54.900 --> 00:08:58.980 funneling billions of dollars every year into ending global poverty. 00:08:58.980 --> 00:09:02.700 JACK: If you’re not familiar with what the World Bank is, let me give you a quick 00:09:02.700 --> 00:09:08.100 catch-up. [MUSIC] It was created after World War II to loan countries money to help rebuild 00:09:08.100 --> 00:09:13.200 after the war. After that, they continued to loan countries money to build bridges 00:09:13.200 --> 00:09:18.480 and other infrastructure for the nation. So, countries around the world owe money to the 00:09:18.480 --> 00:09:23.220 World Bank. The World Bank is actually an NGO, a non-government organization and because of that, 00:09:23.220 --> 00:09:27.840 it falls under different regulations and laws. Now, their mission is to help people in extreme 00:09:27.840 --> 00:09:32.220 poverty. They help fund projects around the world to try to combat extreme poverty, 00:09:32.220 --> 00:09:37.920 but it’s also still a bank which issues bonds and loans and stuff to people with interest, 00:09:37.920 --> 00:09:42.300 and that’s how they make their money off this, is because all these loans have interest. Now, 00:09:42.300 --> 00:09:46.680 the headquarters of the World Bank is like, five blocks away from the White House in Washington, 00:09:46.680 --> 00:09:52.260 DC. While the World Bank is a non-government organization, they also have connections with 00:09:52.260 --> 00:09:57.540 the government in some ways. For instance, the president of the United States of America can 00:09:57.540 --> 00:10:02.040 nominate who the president of the World Bank should be. [00:10:00] If the board agrees, 00:10:02.040 --> 00:10:06.360 then that person becomes the president. So, on March 16, 2005… 00:10:06.360 --> 00:10:09.420 BUSH: Thank you for giving me a chance to come by and say hello. 00:10:09.420 --> 00:10:12.540 JACK: President Bush holds a press conference… 00:10:12.540 --> 00:10:17.040 BUSH: Preparing for my trip out of town for Easter. 00:10:17.040 --> 00:10:21.240 JACK: …and nominates Paul Wolfowitz as the president of the World Bank. 00:10:21.240 --> 00:10:26.340 BUSH: Paul is committed to development. He’s a compassionate, decent man 00:10:26.340 --> 00:10:28.740 who will do a fine job in the World Bank. 00:10:28.740 --> 00:10:33.900 JACK: Now, let’s back up a second. During 9/11, Paul Wolfowitz was the deputy secretary of 00:10:33.900 --> 00:10:39.360 defense. That’s the second-highest ranked person in the entire Department of Defense. He was an 00:10:39.360 --> 00:10:45.420 early advocate for the US to invade Iraq under the belief that Iraq had weapons of mass destruction 00:10:45.420 --> 00:10:50.940 and so, it was a little odd to hear about this nomination. I mean, on one hand you have Wolfowitz 00:10:50.940 --> 00:10:56.280 attacking Iraq and then what, on the other hand he might loan them money to rebuild the country? 00:10:56.280 --> 00:11:02.940 It just seemed really odd. But here’s how one reporter asked a question to President Bush. 00:11:02.940 --> 00:11:05.580 REPORTER: Paul Wolfowitz was the chief architect 00:11:05.580 --> 00:11:08.454 of one of the most unpopular wars in our history. Is your choice to be… 00:11:08.454 --> 00:11:10.680 BUSH: That’s an interesting start. 00:11:10.680 --> 00:11:14.460 REPORTER: …is your choice to be the president of the World Bank. 00:11:14.460 --> 00:11:16.620 What kind of signal does that send to the rest of the world? 00:11:16.620 --> 00:11:21.960 BUSH: First of all, I think people appreciate the world leaders taking my phone calls as 00:11:21.960 --> 00:11:27.646 I explain to them why I think Paul will be a strong president of the World Bank. 00:11:27.646 --> 00:11:31.500 JACK: [MUSIC] Well, Paul Wolfowitz didn’t last long as the president 00:11:31.500 --> 00:11:35.220 of the World Bank. Two years into this role as president… 00:11:35.220 --> 00:11:39.600 HOST2: President Paul Wolfowitz is at the center of a scandal over alleged favoritism. 00:11:39.600 --> 00:11:45.540 HOST3: By arranging a promotion and pay raise for his female companion, Shaha Riza, his credibility 00:11:45.540 --> 00:11:51.360 on this and other issues was undermined. An issue for many member of states was Wolfowitz’s 00:11:51.360 --> 00:11:56.220 effectiveness in the wake of the scandal. Former World Bank official Gene Rotberg… 00:11:56.220 --> 00:12:00.600 GENE: If the countries say your effectiveness is damaged and if the 00:12:00.600 --> 00:12:05.460 staff say your effective – is damaged, then by definition, you are damaged. 00:12:05.460 --> 00:12:09.900 HOST3: From the start, Wolfowitz was the center of controversy as bank chief 00:12:09.900 --> 00:12:14.160 because of his role in planning the Iraq war when he was a top Defense Department 00:12:14.160 --> 00:12:19.080 official. Much of the bank’s staff held this against him, according to Rotberg. 00:12:19.080 --> 00:12:25.740 GENE: It is a rack. He is looked upon as one of the 00:12:25.740 --> 00:12:34.260 persons who form both the intellectual and practical support for that war, 00:12:34.260 --> 00:12:41.160 that it had cost hundreds of thousands of lives. That is what the staff thinks. 00:12:41.160 --> 00:12:45.240 JACK: For reals; if the World’s Bank mission is to help people in extreme poverty, 00:12:45.240 --> 00:12:49.980 having a president who architected the Iraq war and now at the center of a scandal where 00:12:49.980 --> 00:12:54.060 he helped use his position as president to get his girlfriend a job in the World Bank, 00:12:54.060 --> 00:12:59.520 yeah, he was fired, or I guess the term is that he was forced to resign. 00:12:59.520 --> 00:13:04.800 BUSH: I regret that it’s come to this. I admire Paul Wolfowitz. 00:13:04.800 --> 00:13:09.120 JACK: Anyway, that’s what the World Bank was like when Amélie got a job 00:13:09.120 --> 00:13:14.220 working there back in 2008. A new president had just come on board and she was hired on, too. 00:13:14.220 --> 00:13:18.180 AMÉLIE: I was just an information security engineer. I was just basic utility player. 00:13:18.180 --> 00:13:22.200 I was a contractor so I was just kinda like, plug you in here, plug you in there. 00:13:22.200 --> 00:13:25.200 JACK: Now, at the time, Amélie was a meticulous note-taker. 00:13:25.200 --> 00:13:33.480 AMÉLIE: Well, this being back in pre-iPhone, pre-iPad or anything like that, 00:13:33.480 --> 00:13:39.180 no one had tablet computers, so it was just normal for an incident handler to 00:13:39.180 --> 00:13:45.780 walk around – every time they had a case, it was their paper notebook, a steno pad, 00:13:45.780 --> 00:13:52.680 that you were frantically taking notes on. There was no real easy way to do this and secure it. 00:13:52.680 --> 00:13:58.020 I think that’s one of the other challenges too, is a handler’s notebook is their Bible. 00:13:58.020 --> 00:14:01.800 JACK: I just wanted to mention her notepad here because for her to retell this story 00:14:01.800 --> 00:14:06.540 that happened back in 2008, she brought these old notepads out to confirm the story. 00:14:06.540 --> 00:14:09.720 AMÉLIE: Flipping back through some of them and 00:14:09.720 --> 00:14:13.140 I don’t know why I ended up having these, but it was just given to me as my box when 00:14:13.140 --> 00:14:17.640 I left out and they didn’t ask for any of the stuff back, so it is what it is. 00:14:17.640 --> 00:14:22.800 JACK: For this story, she finds the date in her notebook where all this started to happen. 00:14:22.800 --> 00:14:29.100 AMÉLIE: I think when I look through my notebooks, stuff I was looking at before was some full disc 00:14:29.100 --> 00:14:34.320 encryption stuff, but my notebook ends looking at BitLocker and then immediately 00:14:34.320 --> 00:14:40.080 the next page is here are all the notes that just got handed to you about this incident. 00:14:40.080 --> 00:14:42.840 It was like, okay, I’m drinking from the fire hose, here. 00:14:42.840 --> 00:14:49.123 JACK: What was handed to her was a very serious security incident going on in the World Bank. 00:14:49.123 --> 00:14:56.520 AMÉLIE: [MUSIC] So, what had happened from my best memory, and this is mainly because I was 00:14:56.520 --> 00:15:02.640 on the outside for the first two weeks of this, was [00:15:00] something got triggered on a log. 00:15:02.640 --> 00:15:10.320 Someone saw some weird traffic happening and that spun up some folks who investigated, and this was 00:15:10.320 --> 00:15:15.300 a team of probably about five or seven people that were in a conference room down the hall from me. 00:15:15.300 --> 00:15:18.000 JACK: You were in Washington, DC at the time? 00:15:18.000 --> 00:15:18.930 AMÉLIE: Yeah, this was Washington, DC. 00:15:18.930 --> 00:15:20.400 JACK: Yeah, but that’s where you were, right? 00:15:20.400 --> 00:15:21.060 AMÉLIE: Yeah, yeah. 00:15:21.060 --> 00:15:21.480 JACK: Okay. 00:15:21.480 --> 00:15:29.160 AMÉLIE: Yeah, they just noticed there was some weird traffic and they saw some stuff 00:15:29.160 --> 00:15:33.060 through I believe some basic integrity checking that some stuff had changed 00:15:33.060 --> 00:15:37.560 on a server that was not supposed to generally be touched by people. 00:15:37.560 --> 00:15:41.460 JACK: Oh yeah, file integrity monitoring. This is a helpful tool that companies use 00:15:41.460 --> 00:15:46.560 to monitor security problems. This is where a system checks all the important servers in a 00:15:46.560 --> 00:15:51.000 network to make sure nothing has changed on it that shouldn’t have changed. So like, if there 00:15:51.000 --> 00:15:55.620 was a configuration change, the file integrity checker would notice that and create an alert, 00:15:55.620 --> 00:16:00.240 and the monitoring team would now have to go to the system administrator and ask hey, 00:16:00.240 --> 00:16:04.980 did you make a change on this server? If that system administrator did, then everything would be 00:16:04.980 --> 00:16:10.140 okay. But in this case, the file integrity monitor triggered an alert and it showed that someone 00:16:10.140 --> 00:16:16.380 had made changes to a server which wasn’t made by the system administrator or anyone else in the IT 00:16:16.380 --> 00:16:22.800 team. This meant that some unauthorized person had been inside one of their servers in the World Bank 00:16:22.800 --> 00:16:28.980 and was doing stuff to the servers. This is where Amélie started getting pulled into the incident to 00:16:28.980 --> 00:16:34.080 see if there was a way she could help. She took a look at the systems that had unauthorized changes. 00:16:34.080 --> 00:16:36.780 AMÉLIE: Some of the triggers were which machine was hit, 00:16:36.780 --> 00:16:40.320 and one of the machines that got touched was our HSM. 00:16:40.320 --> 00:16:43.500 JACK: An HSM is a hardware security module. It’s 00:16:43.500 --> 00:16:47.100 a device that specifically does cryptographic computations which 00:16:47.100 --> 00:16:51.120 means this is the device that does their encryption, decryption, and authentication. 00:16:51.120 --> 00:16:55.440 AMÉLIE: Essentially, our locker for all our cryptologic material. 00:16:55.440 --> 00:17:01.920 That server was shown to be touched and that was like okay, they were going after the crown jewels. 00:17:01.920 --> 00:17:05.460 That, I think, was probably the triggers – when that machine name came up in that list, 00:17:05.460 --> 00:17:10.260 that was immediately like, this has gotten bigger than just a couple database servers being touched. 00:17:10.260 --> 00:17:16.080 JACK: The company quickly put everyone in IT to work in this incident. The amount of work 00:17:16.080 --> 00:17:20.760 that everyone needed to do was staggering. There were tons of logs to look through, 00:17:20.760 --> 00:17:25.560 systems to analyze, connections to review. This is where Amélie started getting involved. 00:17:25.560 --> 00:17:30.360 AMÉLIE: They had one full-time forensics guy and the folks who were running the 00:17:30.360 --> 00:17:37.020 incident handling group before I was called in were basically saying image everything. 00:17:37.020 --> 00:17:41.040 JACK: Which is a good first thing to do in this situation. It’s really just like making 00:17:41.040 --> 00:17:46.380 a copy of everything on that server to analyze offline, because an attacker might erase their 00:17:46.380 --> 00:17:52.440 tracks or systems might change, so having a copy of an infected machine is great. But there’s a 00:17:52.440 --> 00:17:56.760 sort of fog of war when you’re dealing with an incident like this. You can’t really see 00:17:56.760 --> 00:18:02.280 all of what the attacker did, so it’s hard to know how big of an issue this really is, 00:18:02.280 --> 00:18:07.680 and it’s hard to know where to even look for clues. After looking through the logs and alerts, 00:18:07.680 --> 00:18:13.200 they had evidence that this attacker had accessed and changed configurations on thirty different 00:18:13.200 --> 00:18:18.840 servers in the bank. [MUSIC] So, they were taking images and snapshots of these thirty computers, 00:18:18.840 --> 00:18:24.480 but they only had one guy with one computer who was capable of analyzing these images to look for 00:18:24.480 --> 00:18:29.220 clues of malicious activity. To analyze one machine might take hours and hours, 00:18:29.220 --> 00:18:33.720 but that’s just for a computer to analyze it. For a human to analyze it, it takes even longer, 00:18:33.720 --> 00:18:40.740 so the process was going very slow. At the same time, some of the employees were in a panic, 00:18:40.740 --> 00:18:45.780 stressed out to the brim over this incident. People from management were also filled with 00:18:45.780 --> 00:18:49.680 anxiety when things were not moving as fast as they wanted. Emergency meetings 00:18:49.680 --> 00:18:53.400 were spun up to try to get people to move faster, and leadership was freaking out. 00:18:53.400 --> 00:18:57.840 AMÉLIE: I was coming back from lunch one day and got a conference call about this thing. As 00:18:57.840 --> 00:19:02.580 a contractor, you had the CIO and the CTO on the call, and the CISO, and as a contractor, 00:19:02.580 --> 00:19:10.260 I said calm the fuck down. Literally, the entire phone line just went silent. 00:19:10.260 --> 00:19:15.300 That’s what is needed sometimes, a good incident handler, and learning from these experiences is 00:19:15.300 --> 00:19:22.620 to maintain calm. Throughout the story, it – their initial two weeks, hair on fire, 00:19:22.620 --> 00:19:27.600 but when you have somebody who is giving them good information, giving them actual things 00:19:27.600 --> 00:19:33.360 they need to do, helping them solve their problem, that’s the best thing an incident handler can do, 00:19:33.360 --> 00:19:39.300 is to maintain a sense of calm and transparency. If there’s anything that anybody walks away from 00:19:39.300 --> 00:19:45.120 this – with is that’s the thing that makes a good incident handler, not what cool tools you know or 00:19:45.120 --> 00:19:50.820 anything like that; it’s just speaking the truth, sharing evidence, and being – having a cool head. 00:19:50.820 --> 00:19:54.540 JACK: I often find that when people are handling incidents like this, 00:19:54.540 --> 00:19:59.040 they sometimes go through all the stages of grief. You know, first you’re shocked that a 00:19:59.040 --> 00:20:02.640 hacker got into your network, [00:20:00] and then you might deny that it happened. Like, 00:20:02.640 --> 00:20:08.460 wait, no, no, no; no way they got into my server. That’s crazy. Then when there’s no denying it, 00:20:08.460 --> 00:20:13.860 you might feel angry that it happened. Then you might bargain; like well, at least they didn’t get 00:20:13.860 --> 00:20:18.840 into that database over there. But then when the reality hits that all this is really happening, 00:20:18.840 --> 00:20:26.100 you might feel depressed, like this is such a big problem that maybe we’ll never solve. Once 00:20:26.100 --> 00:20:32.100 you process all that, can you really accept the situation and move on? As Amélie puts it… 00:20:32.100 --> 00:20:33.540 AMÉLIE: Your lunch has already been eaten. 00:20:33.540 --> 00:20:37.740 JACK: Because she’s dealt with this kind of thing so many times that she can quickly move 00:20:37.740 --> 00:20:41.820 to this acceptance stage and just start working on solutions while others might 00:20:41.820 --> 00:20:46.543 still be busy dealing with their emotions. Amélie was now fully immersed in this problem. 00:20:46.543 --> 00:20:49.380 AMÉLIE: [MUSIC] If you’re an incident handler and you’re thrown into it, if you’re not the 00:20:49.380 --> 00:20:53.040 one who’s actually on the detection – and a lot of times that could occur with another team, 00:20:53.040 --> 00:20:58.800 so network team or server team or something like that – yeah, you have fire hoses aimed 00:20:58.800 --> 00:21:04.080 at you of information coming from every which direction. It’s a matter of which one you’re 00:21:04.080 --> 00:21:10.320 gonna turn and open your mouth to, and in these cases, coming in two weeks late, it was like, 00:21:10.320 --> 00:21:15.900 give me the dump and give me an afternoon or a day to kind of sort through this and if 00:21:15.900 --> 00:21:22.320 anybody’s got any inferences, try to summarize them. Unfortunately this team didn’t have it, 00:21:22.320 --> 00:21:26.880 so it was literally like give me what you got. I sat down in my office and 00:21:26.880 --> 00:21:31.220 tried to pore over as much of the data I could and try to make sense of it. 00:21:31.220 --> 00:21:35.400 JACK: She knew just what to do in a situation like this and started 00:21:35.400 --> 00:21:37.860 asking for forensic images and logs to review. 00:21:37.860 --> 00:21:44.880 AMÉLIE: Being that I was called in two weeks late, getting a memory image was near impossible, so we 00:21:44.880 --> 00:21:51.420 were working with a lot of imperfect information. By the time I got a chance to call for stuff from 00:21:51.420 --> 00:21:57.720 say ArcSight which was their log repository, and even Mazu which I think was their NetFlow logging 00:21:57.720 --> 00:22:05.520 tool, they had such little online storage that most of that evidence was gone. It was playing 00:22:05.520 --> 00:22:12.420 catch-up with stuff that was disappearing, trying to grab sand as it was flying through your hands. 00:22:12.420 --> 00:22:17.820 Time was very much of the essence and trying to narrow down what we needed to pull was also very 00:22:17.820 --> 00:22:24.180 important because again, time was of the essence because at – when this was handed off, I had no 00:22:24.180 --> 00:22:30.300 idea if the aggressor, the attacker, or whoever was in the network was still on the network. 00:22:30.300 --> 00:22:35.160 JACK: On top of her doing all this incident handling, the bank realized they needed even 00:22:35.160 --> 00:22:39.360 more help, so they called Mandiant up which is where she used to work, but they called 00:22:39.360 --> 00:22:43.680 them up just to help with incident response, too. Mandiant had a whole team of incident 00:22:43.680 --> 00:22:49.080 responders ready to be deployed onsite to help troubleshoot major attacks like this, but it 00:22:49.080 --> 00:22:53.160 would take them a few days to arrive, to Amélie stayed busy working on the issue in the meantime. 00:22:53.160 --> 00:22:57.660 AMÉLIE: I was literally trying to map out a picture of what the entire incident looked 00:22:57.660 --> 00:23:02.880 like. [MUSIC] So, looking at log files, looking at servers that they may have already identified, 00:23:02.880 --> 00:23:10.560 and drawing out a map. Like, where the hops were, who was affected, when did they do this, 00:23:10.560 --> 00:23:16.860 what was the timeline, and I spent most of that weekend – I got approved for extra hours to work 00:23:16.860 --> 00:23:22.140 that entire weekend, and I think I was doing this on a Mac, so it was probably the first version 00:23:22.140 --> 00:23:28.020 of OmniGraffle that was ever released, so I’m doing all this stuff in Visio and OmniGraffle, 00:23:28.020 --> 00:23:34.740 mapping out all the paths, so it was like a board game. You just kinda watched; they got on 00:23:34.740 --> 00:23:40.740 this server and then they went lateral to these two servers and they touched these files, and 00:23:40.740 --> 00:23:45.600 it really does end up occasionally looking like a digital version of that yarn and pushpin thing. 00:23:45.600 --> 00:23:49.500 JACK: By that time, the World Bank was having daily meetings, a War Room, 00:23:49.500 --> 00:23:52.860 if you like, to bring everyone up to speed on the latest with this incident 00:23:52.860 --> 00:23:57.240 and to make decisions on what to do next. While a lot of people were working on this, 00:23:57.240 --> 00:24:01.320 only a small group had access to the details of this incident. 00:24:01.320 --> 00:24:06.060 AMÉLIE: As we were initially handling this response, and it was probably timed about that two 00:24:06.060 --> 00:24:12.360 weeks when I was called in, there was a story that ended up getting leaked to Fox News [MUSIC] that 00:24:12.360 --> 00:24:21.840 had particularly detailed recounting of a lot of what was going on with our incidence response, 00:24:21.840 --> 00:24:29.940 as if someone was – in the room was also leaking stuff to the press. 00:24:29.940 --> 00:24:36.900 JACK: That’s not good. Typically when you have an intrusion like this, you want to be very careful 00:24:36.900 --> 00:24:42.120 how you publicly disclose this. The wording needs to be precise and at the very least, 00:24:42.120 --> 00:24:46.740 you want to be able to control the messaging that the press knows. But on top of that, 00:24:46.740 --> 00:24:50.160 they just notified the hacker that the bank is onto them. 00:24:50.160 --> 00:24:51.660 AMÉLIE: There’s just a lot of pressure of 00:24:51.660 --> 00:24:57.780 trying to stem the bleeding and make sure that the message is controlled. 00:24:57.780 --> 00:25:03.240 JACK: Someone had spoken to a reporter [00:25:00] at Fox News and told them about this incident. 00:25:03.240 --> 00:25:08.580 Here, I’ll read the article for you. The headline says World Bank Under Cyber Siege 00:25:08.580 --> 00:25:14.340 in Unprecedented Crisis. Then the story reads “It is still not known how much information was 00:25:14.340 --> 00:25:19.020 stolen but sources inside the bank confirm that servers in the institution’s highly-restricted 00:25:19.020 --> 00:25:24.180 Treasury unit were deeply penetrated with spy software last April. Invaders had full 00:25:24.180 --> 00:25:28.320 access of the rest of the bank’s network for nearly a month in June and July. In 00:25:28.320 --> 00:25:32.760 a frantic midnight e-mail to colleagues, the bank’s senior technology manager referred to 00:25:32.760 --> 00:25:40.560 the situation as an unprecedented crisis.” Hm, that’s some pretty specific information that 00:25:40.560 --> 00:25:46.500 this inside source had leaked. Like, they saw that e-mail from the senior manager and they 00:25:46.500 --> 00:25:52.980 know information that was only discussed in that War Room. Something strange was going on here. 00:25:52.980 --> 00:25:59.160 AMÉLIE: There’s an internal integrity group at the World Bank which basically is a watcher for 00:25:59.160 --> 00:26:06.000 the watchers, I guess. They audit how the bank does programs. They’re kinda like internal review 00:26:06.000 --> 00:26:12.240 groups and whatnot. Part of something I had to do when I first signed on as a contractor 00:26:12.240 --> 00:26:18.180 there was asked to go and investigate some news stories that were leaked out that seemed 00:26:18.180 --> 00:26:23.640 to be originating from data that was passing through some of the executive branches there. 00:26:23.640 --> 00:26:29.220 JACK: Oh, so Amélie had already been looking for this leaker before this news story even hit. Now, 00:26:29.220 --> 00:26:32.340 even though the bank is in the middle of an unprecedented crisis, 00:26:32.340 --> 00:26:37.920 she’s gotta find out who this person is that’s leaking information to Fox News. 00:26:37.920 --> 00:26:43.020 AMÉLIE: [MUSIC] I think that earlier case involved the Wall Street Journal as well, 00:26:43.020 --> 00:26:46.340 so there was an intersection between Fox News and Wall Street Journal. 00:26:46.340 --> 00:26:52.140 JACK: Trying to figure out who the leaker is is like a game of Among Us where you’re 00:26:52.140 --> 00:26:56.580 trying to figure out who the impostor is. Who would have the motivation to talk to 00:26:56.580 --> 00:27:01.260 Wall Street Journal and Fox News? Who would have access to this kind of information, 00:27:01.260 --> 00:27:04.320 to be in the War Room where this stuff is discussed, 00:27:04.320 --> 00:27:10.620 or to see those e-mails? Amélie started becoming very observant of everyone in the IT department, 00:27:10.620 --> 00:27:21.180 trying to figure out who this leaker was. Stay with us because after the break, she sets a trap. 00:27:21.180 --> 00:27:26.040 Amélie starts making a list of names in her notebook of who this leaker might be, 00:27:26.040 --> 00:27:30.300 and crossing off names that just don’t seem possible for it to be that person. 00:27:30.300 --> 00:27:33.300 She had already started researching this before. 00:27:33.300 --> 00:27:37.080 This was actually the second case of someone leaking stories to the press. 00:27:37.080 --> 00:27:42.600 AMÉLIE: So, that original case – and I had notes for that one, too – were looking at 00:27:42.600 --> 00:27:48.720 some linguistical analysis, of all things, finding out what people – how people wrote 00:27:48.720 --> 00:27:53.280 certain things, looking at quotes from the Wall Street Journal and Fox News, 00:27:53.280 --> 00:27:59.760 and then looking at e-mail as well as documents that may have been accessed, 00:27:59.760 --> 00:28:06.900 and seeing who had accessed them and then with the e-mail, seeing how these peoples’ quotes were. 00:28:06.900 --> 00:28:13.200 So, when you’d see stuff that was called out, we could search the mail system and try to find out 00:28:13.200 --> 00:28:19.200 where those particular quotes came from. With this incident, the data that was getting leaked 00:28:19.200 --> 00:28:26.760 out was making it out to be – we had keystone cops running this response. So, parallel to all this, 00:28:26.760 --> 00:28:30.600 because obviously we knew that the incident response team was potentially compromised, 00:28:30.600 --> 00:28:36.480 I was – it was almost like incident response inception, so I was actually investigating the 00:28:36.480 --> 00:28:42.000 investigators who had tripped upon the fact that we had this data getting leaked out. 00:28:42.000 --> 00:28:47.700 JACK: Imagine being in that incident War Room. Your eyes come up over the laptop and 00:28:47.700 --> 00:28:54.480 you gaze around the room, sizing up everyone, wondering if they’re the leaker. At the same 00:28:54.480 --> 00:28:59.640 time they’re looking back at you wondering if you’re the inside source. Amélie got an idea. 00:28:59.640 --> 00:29:06.660 AMÉLIE: We set a trap because we had kinda narrowed it down to a few suspects. 00:29:06.660 --> 00:29:11.340 Due to the prior investigation, we had I think probably about five or six people 00:29:11.340 --> 00:29:13.320 that we knew possibly were the leakers. 00:29:13.320 --> 00:29:16.860 JACK: Her suspicion was that it was someone inside the IT department. 00:29:16.860 --> 00:29:22.560 AMÉLIE: We decided to set up a honeypot. [MUSIC] So, we had a conference room where a lot of this 00:29:22.560 --> 00:29:28.200 outbriefing was done to the CIO and the CTO and some of the senior IT people in the World Bank. 00:29:28.200 --> 00:29:32.940 JACK: The debrief meetings were with leadership and senior people and it was only a few people. 00:29:32.940 --> 00:29:38.460 Her hunch was that it wasn’t anyone in these meetings but it was someone who might be 00:29:38.460 --> 00:29:43.740 snooping in on one of these meetings, trying to find the latest news to give to a reporter. 00:29:43.740 --> 00:29:48.900 So when the meeting was over and everyone left, Amélie planted some fake information. 00:29:48.900 --> 00:29:57.720 AMÉLIE: Started writing some fake stuff on the walls; put up some papers on the table and 00:29:57.720 --> 00:30:04.400 whatnot, and we kinda waited for [00:30:00] those notes to start showing up somewhere. 00:30:04.400 --> 00:30:10.920 JACK: I love how low-tech of a honeypot this is, just a few notes left in a conference room. Well, 00:30:10.920 --> 00:30:13.680 now that the honey trap was set, they just had to wait. 00:30:13.680 --> 00:30:22.620 AMÉLIE: [MUSIC] We saw some stuff that had popped up with the Fox News story a day later, 00:30:22.620 --> 00:30:28.980 so we’d done the honeypot. We laid it out and we cleaned it up, and – when we knew one of our 00:30:28.980 --> 00:30:33.120 suspects was in the office – and then a day or two later we saw it show up in a Fox News report. So, 00:30:33.120 --> 00:30:37.440 we knew what they were reporting was wrong and we had narrowed it down I think two or three 00:30:37.440 --> 00:30:44.640 people. Eventually it was narrowed down to this one person. We’d honeypotted a second time and 00:30:44.640 --> 00:30:49.560 we actually had somebody walk by the office of the suspect to make sure they were there at that time, 00:30:49.560 --> 00:30:58.320 and we got a message back. I think we we were still using PalmPilots to send e-mail and stuff at 00:30:58.320 --> 00:31:02.940 the time, to tell you how long ago it was, to let them know that yeah, that person was in the office 00:31:02.940 --> 00:31:06.720 and they were on their computer at the time, and they were the only one that was alive in that 00:31:06.720 --> 00:31:14.100 hallway. We kept an eye on them because sometimes as the phrase goes, there’s a useful idiot. 00:31:14.100 --> 00:31:18.420 JACK: But at the same time, they wanted to build this case further to prove it was him. 00:31:18.420 --> 00:31:25.200 AMÉLIE: So, now we had a suspect and a machine. We could push out our remote forensic imaging element 00:31:25.200 --> 00:31:30.540 out to them and grab an image of their hard drive. When we eventually pulled it through using stuff 00:31:30.540 --> 00:31:37.920 like EnCase and some other tools, reconstruct the web caches so you could see the Yahoo e-mails that 00:31:37.920 --> 00:31:42.300 went out, so we had evidence that they were using their work computer to leak the information out 00:31:42.300 --> 00:31:47.400 and it wasn’t – ‘cause we weren’t detecting it in the mail system which was Lotus Notes; 00:31:47.400 --> 00:31:52.320 we knew it was through webmail, so we were then able to dive through the entire history of what he 00:31:52.320 --> 00:31:56.280 had sent out and found out that he was connected not only to the leaking of the information 00:31:56.280 --> 00:32:01.460 regarding this incident but was also tied to some of the connections with the prior leadership. 00:32:01.460 --> 00:32:08.550 JACK: Wait, the prior leadership of the World Bank? As in Paul Wolfowitz? 00:32:08.550 --> 00:32:09.540 AMÉLIE: Yeah. 00:32:09.540 --> 00:32:14.640 JACK: So, when Amélie analyzed this IT person’s computer, the leaker, 00:32:14.640 --> 00:32:21.120 she found that he was also working with another person about these leaks. This 00:32:21.120 --> 00:32:26.400 person who was helping him was an internal investigator also with the World Bank. 00:32:26.400 --> 00:32:33.720 AMÉLIE: They had an investigator that was one of the internal integrity investigators. 00:32:33.720 --> 00:32:37.980 I had done an analysis on their hard drive, so I was the watcher watching the watcher 00:32:37.980 --> 00:32:42.900 watching the watcher-type kinda thing, and saw some of his personal items on there. 00:32:42.900 --> 00:32:46.440 JACK: What she found was evidence that this investigator was being 00:32:46.440 --> 00:32:51.000 blackmailed by the former leadership which was Paul Wolfowitz or his team, 00:32:51.000 --> 00:32:54.180 and they somehow figured out a secret about this guy. 00:32:54.180 --> 00:33:05.100 AMÉLIE: Who was gay but wasn’t openly gay. They were basically going to use that information 00:33:05.100 --> 00:33:12.300 as a leverage to out him. Remember, this was during the Bush administration and it’s a very 00:33:12.300 --> 00:33:19.200 conservative organization overall within the World Bank. That’s what they were using to leverage 00:33:19.200 --> 00:33:24.000 and I let my boss know that this was most likely the reason why they were – for blackmailing him. 00:33:24.000 --> 00:33:27.960 JACK: [MUSIC] So, to tie it all together or what Amélie believes is that Wolfowitz 00:33:27.960 --> 00:33:31.020 seemed to be a little upset that he had to leave the World Bank and we 00:33:31.020 --> 00:33:34.200 believed he was trying to make the current leadership of the World Bank look bad, 00:33:34.200 --> 00:33:40.080 so his people were blackmailing this investigator to somehow find a way to make the bank look bad, 00:33:40.080 --> 00:33:46.140 and the investigator was just using the IT guy to get this information and to leak it to the 00:33:46.140 --> 00:33:52.320 press to make the bank look bad. Places like Fox News were adding in all kinds of extra narratives, 00:33:52.320 --> 00:33:57.060 like talking about how the current president isn’t doing as good of a job and stuff like that. 00:33:57.060 --> 00:34:01.380 AMÉLIE: DC is very much a long smear of campaigns and whatever kind of leverage, 00:34:01.380 --> 00:34:05.280 no matter who it can hurt, this is how this town operates. It’s pretty sad. 00:34:05.280 --> 00:34:09.600 JACK: So, she prepared all this for HR to handle and her boss. 00:34:09.600 --> 00:34:13.500 AMÉLIE: The investigator 00:34:13.500 --> 00:34:17.160 I think still was there, just probably waylaid a little bit. I didn’t see much of him. It was 00:34:17.160 --> 00:34:24.240 more or less that – I’m sure the pestering by the outside folks went down quite significantly 00:34:24.240 --> 00:34:31.800 but I was told that that was handled at a level above me. The IT person was eventually let go. 00:34:31.800 --> 00:34:38.280 JACK: Okay, problem solved. What a relief. Time to take a break. But oh, wait a minute, no, 00:34:38.280 --> 00:34:42.900 we can’t. This bank is under attack, remember? There’s a hacker that got into thirty servers 00:34:42.900 --> 00:34:47.580 in this network, and this was all going on at the same time. What a mess to try to handle 00:34:47.580 --> 00:34:52.680 two different incidents at once. All this was just putting a lot of extra stress on people. 00:34:52.680 --> 00:34:57.060 AMÉLIE: That first weekend was me going – working as much as I can, 00:34:57.060 --> 00:35:00.600 and then going – getting something to eat and maybe going home. [00:35:00] I needed 00:35:00.600 --> 00:35:05.520 to sleep in my own bed. But I think the next day when I came back, I brought a pillow and a 00:35:05.520 --> 00:35:11.580 blanket to sleep under my desk so I could just continuously work through the day. 00:35:11.580 --> 00:35:14.400 Eventually I just left a pillow and a blanket there ‘cause I just figured this 00:35:14.400 --> 00:35:19.920 was what life was gonna be like for a while. Sleeping under your desk is not fun at all. 00:35:19.920 --> 00:35:22.800 JACK: Okay, so, back to the network intrusion. 00:35:22.800 --> 00:35:27.600 AMÉLIE: We eventually got to the point where we’re actually pinpointing machines of interest. So, 00:35:27.600 --> 00:35:33.600 we had thirty machines involved but to kind of only forensically look at a couple, I think 00:35:33.600 --> 00:35:38.520 we narrowed it down to like, seven machines. I had a list of these are the ones that you have 00:35:38.520 --> 00:35:43.980 images for and these are the things we’re looking for. [MUSIC] A lot of this was done in parallel, 00:35:43.980 --> 00:35:48.840 so we had the forensics images of I think those five to seven servers but at the same time, 00:35:48.840 --> 00:35:54.480 knowing that we instrumented relatively well but there was a lot of data that started to disappear, 00:35:54.480 --> 00:36:03.720 like the Mazu gateway data, that was the NetFlow stuff, and that the ArcSight was no longer online. 00:36:03.720 --> 00:36:08.820 It was trying to piece together the story from logs, trying to figure out alright, 00:36:08.820 --> 00:36:17.220 so, here’s the map of what happened and what do the logs tell us about this and how they got in? 00:36:17.220 --> 00:36:23.460 So, it was more or less a lot of the forensics were a confirmation of our inferences, so as we 00:36:23.460 --> 00:36:30.120 started to pinpoint what got accessed and whatnot, we still didn’t know the motivation. If I remember 00:36:30.120 --> 00:36:34.200 correctly, we didn’t actually end up getting to where we found the motivation as to what they were 00:36:34.200 --> 00:36:44.820 interested in. Most of this came down to what tools did they use, how did they gain access, 00:36:44.820 --> 00:36:49.680 and what machines do we need to reimage, what data do we need 00:36:49.680 --> 00:36:55.140 to be on the lookout for if it’s gonna be used by an adversary, and so forth. 00:36:55.140 --> 00:36:57.780 JACK: What was it that they – that was compromised? 00:36:57.780 --> 00:37:03.200 AMÉLIE: As it came down to it, it was some machines associated with the HR system. 00:37:03.200 --> 00:37:10.680 JACK: Hm, okay. A bank has a lot of money so you’d think that a hacker getting – going through all 00:37:10.680 --> 00:37:14.720 the effort of getting into a bank, it’s kind of a surprise that they were going after HR. 00:37:14.720 --> 00:37:22.380 AMÉLIE: Yeah, the thing was with the data that was on there, there was a little bit of the HR 00:37:22.380 --> 00:37:28.140 stuff and when you start asking questions as to figure out what all the connections are, 00:37:28.140 --> 00:37:33.240 they’re shared databases. We considered it was probably HR but there may have been some other 00:37:33.240 --> 00:37:38.280 databases that they were interested on there. At the point and times that were mainly – they 00:37:38.280 --> 00:37:42.720 – the leadership was mainly interested in how they got in and if they could clean up. 00:37:42.720 --> 00:37:46.140 JACK: I’m curious about this, too. How did the hacker get in? 00:37:46.140 --> 00:37:51.060 AMÉLIE: One of the issues was with the bank is that they had abandoned a multi-factor access 00:37:51.060 --> 00:38:02.100 program, so not only were they not using a hard card or a smart card for credentialing for all 00:38:02.100 --> 00:38:11.760 users, the users between enterprise admin and the regular old, every Jane or Joe user on the World 00:38:11.760 --> 00:38:20.220 Bank network, it was just a matter of your user ID and your password. So, our enterprise admin, 00:38:20.220 --> 00:38:24.060 one of our three enterprise admins for AD’s account was compromised. 00:38:24.060 --> 00:38:30.480 JACK: Mm-hm. All it takes is the right username and password to get in sometimes. You might ask 00:38:30.480 --> 00:38:35.640 why was multi-factor authentication turned off? Well, they did try multi-factor authentication 00:38:35.640 --> 00:38:41.340 but for whatever reason, they didn’t like it. It was too slow, too complex. It was impacting how 00:38:41.340 --> 00:38:45.840 business worked, so they removed it and were in the process of switching everyone to use 00:38:45.840 --> 00:38:49.680 smart cards for authentication. This is where you have to insert a little credit card-like 00:38:49.680 --> 00:38:53.460 thing and then type in your password to get into a computer. [MUSIC] So, they turned off 00:38:53.460 --> 00:38:57.840 the token method for authenticating users and were in the process of switching everyone to use these 00:38:57.840 --> 00:39:03.780 physical cards. So, it was just really bad timing that the hackers got in during that transition. 00:39:03.780 --> 00:39:07.200 Once the hacker got in, they tried running a hacker tool, 00:39:07.200 --> 00:39:12.000 but the antivirus on that computer blocked it, so they tried another exploit, a newer 00:39:12.000 --> 00:39:18.780 one, and even though this computer had antivirus running, it didn’t trigger on this newer exploit. 00:39:18.780 --> 00:39:22.560 Once they were in one computer, they were able to traverse the network to get into 00:39:22.560 --> 00:39:27.480 other computers. The hacker eventually got their hands on password hashes. Now, 00:39:27.480 --> 00:39:31.860 you need to run a password-cracking tool to figure out what the password is once you get the hashes, 00:39:31.860 --> 00:39:36.720 but Amélie saw that they had gotten these password hashes and she knew they had the enterprise 00:39:36.720 --> 00:39:42.720 admin’s password, so she decided to get the hashes herself and see if she could crack the password. 00:39:42.720 --> 00:39:47.460 AMÉLIE: And was able to pull out the passwords for the enterprise admin 00:39:47.460 --> 00:39:49.140 rather quickly, I think in about five minutes. 00:39:49.140 --> 00:39:54.060 JACK: A-ha, so, if she can crack the password that quickly, that meant that the password was 00:39:54.060 --> 00:39:59.400 not very strong. This helped her connect a bunch of pieces together to know how the hacker moved 00:39:59.400 --> 00:40:03.600 around and got different things. [00:40:00] So, when she went to the close-out meeting with the 00:40:03.600 --> 00:40:08.520 head of IT to wrap this whole thing up, she decided to do a sort of magic trick for them. 00:40:08.520 --> 00:40:13.740 AMÉLIE: [MUSIC] I had Joyce Lin ask the enterprise admin for his password. 00:40:13.740 --> 00:40:18.780 JACK: Joyce Lin was from Mandiant who was called in to help with this whole incident. So, the admin 00:40:18.780 --> 00:40:24.420 wrote the password on a piece of paper and then folded it so nobody could see it, and put it on 00:40:24.420 --> 00:40:30.060 the table. At the same time, Amélie started John the Ripper, a tool used to crack passwords, and 00:40:30.060 --> 00:40:35.640 everyone in the meeting watched. Quite quickly, she had managed to crack the enterprise password. 00:40:35.640 --> 00:40:41.400 AMÉLIE: Then turned the screen around and I said, is this your card? Sure enough, that was the 00:40:41.400 --> 00:40:47.880 password that was written on that sheet, and we – she, Joyce, showed the screen and that password, 00:40:47.880 --> 00:40:53.460 and it was a really simple password. I think it was his daughter’s name with a year or something. 00:40:53.460 --> 00:41:00.900 It was really bad password policy. So, that instantiated some policy changes. They started 00:41:00.900 --> 00:41:05.760 doing some user account separation. A couple of months later, we had called in Microsoft 00:41:05.760 --> 00:41:13.320 to go through an entire analysis of the Active Directory forest management for the entire bank. 00:41:13.320 --> 00:41:18.120 JACK: Ah, yes, nothing like a good old-fashioned password audit. This is where you take the hash 00:41:18.120 --> 00:41:23.340 dump for everyone’s passwords on the entire network and see how many can be easily cracked. 00:41:23.340 --> 00:41:28.200 So, you see some pretty bad behavior. Like, a lot of people use the company name inside their 00:41:28.200 --> 00:41:31.980 password, and that kind of stuff might show up on the audit which then might allow the security 00:41:31.980 --> 00:41:36.360 team to make new rules, to restrict certain passwords in an attempt to make things harder 00:41:36.360 --> 00:41:40.980 to crack. On top of that, they finished rolling out that smart card authentication for everyone. 00:41:40.980 --> 00:41:44.820 AMÉLIE: This was kind of a come to Jesus incident for them, 00:41:44.820 --> 00:41:48.840 to try to get serious about increasing their security. 00:41:48.840 --> 00:41:52.380 JACK: They also brought in more FTEs, or full-time employees, 00:41:52.380 --> 00:41:57.000 to help do security. They built out a SOC, they developed an incident-handling playbook, 00:41:57.000 --> 00:42:01.860 and they improved their security overall for the whole company to keep this from happening again. 00:42:01.860 --> 00:42:06.780 AMÉLIE: As much as they were doing everything you could potentially do wrong before this, 00:42:06.780 --> 00:42:13.380 that it was enough of a punch that it got them to really kinda try to do something better. 00:42:13.380 --> 00:42:18.840 JACK: Now, the biggest question that always comes up from a hack like this is who did this? They 00:42:18.840 --> 00:42:24.240 weren’t able to figure that out for sure but there were a few clues that suggested that this 00:42:24.240 --> 00:42:30.600 attack came from China. So, they packaged up these findings and sent it to the World Bank leadership. 00:42:30.600 --> 00:42:38.160 AMÉLIE: The initial response was from the bank’s CIO; was oh my god, and I think there 00:42:38.160 --> 00:42:44.220 was an explicative in there ‘cause she was very much more reserved than most, but she 00:42:44.220 --> 00:42:51.780 was literally ready to march up to the Chinese embassy or up there and around UDC and basically 00:42:51.780 --> 00:42:56.700 chew them out and say that they were gonna pull all the bank funding from all of their projects. 00:42:56.700 --> 00:43:02.400 JACK: Yeah, this is interesting because the World Bank was loaning money to China for various 00:43:02.400 --> 00:43:06.960 projects. I don’t know what exactly, but maybe a loan to build a bridge or maybe to help people in 00:43:06.960 --> 00:43:11.760 extreme poverty in China. But it doesn’t matter what project. The thing is is that the World Bank 00:43:11.760 --> 00:43:17.940 was directly helping China and didn’t like that China was behind this attack. Yeah, we don’t know 00:43:17.940 --> 00:43:23.520 if they actually spoke to anyone in China about this but it’s certainly interesting if they did. 00:43:23.520 --> 00:43:31.020 AMÉLIE: [MUSIC] Why that was significant was – is that the challenge with anything like this 00:43:31.020 --> 00:43:37.080 is attribution. There was stuff that wasn’t shared with us once we – as I mentioned, you asked well, 00:43:37.080 --> 00:43:40.980 why HR systems? There was definitely something that triggered in the back of the mind of 00:43:40.980 --> 00:43:48.300 those executives that when we showed them that evidence, they knew that it was China 00:43:48.300 --> 00:43:54.240 or some Chinese asset that was looking to get this information, because there was something higher 00:43:54.240 --> 00:44:02.160 up at the bank that was happening that us lowly contractors and other employees weren’t party to. 00:44:02.160 --> 00:44:11.640 So, it wasn’t necessarily the way that typically you would map your TTPs to a particular actor, 00:44:11.640 --> 00:44:17.340 so China has their threat actors with their names and stuff, and then Russia has theirs, 00:44:17.340 --> 00:44:21.540 and Iran has theirs, and North Korea has theirs. 00:44:21.540 --> 00:44:28.140 But some of that too, the reason we also were able to be somewhat confident other 00:44:28.140 --> 00:44:33.600 than the fact that the bank executives were like yeah, we kinda – this makes sense, 00:44:33.600 --> 00:44:39.300 where the – Mandiant, as part as some of the malware stuff that we had looked at using 00:44:39.300 --> 00:44:43.380 their Mirror tool – it was just the first release of their Mirror tool at the time 00:44:43.380 --> 00:44:47.820 – was that some of the stuff that we had found on the systems started matching some 00:44:47.820 --> 00:44:54.420 of their early work that they had had to map to some of the Chinese threat actors. 00:44:54.420 --> 00:44:58.140 It all kinda came together, but this is very much in the early days, like I said, 00:44:58.140 --> 00:45:03.840 2007, 2008, the very early days of [00:45:00] being able to have some level of confidence 00:45:03.840 --> 00:45:08.040 that these were the teams that were particularly poking and prodding. 00:45:08.040 --> 00:45:12.480 JACK: So, that concluded the investigation. It was time for Amélie to bring her pillow back 00:45:12.480 --> 00:45:18.780 home and sleep in her own bed again. But honestly, Amélie loves handling incidents like this. 00:45:18.780 --> 00:45:26.460 AMÉLIE: It’s the chase, it’s the finding something new, finding something clever, the thrill of the 00:45:26.460 --> 00:45:33.480 chase overall. Yeah, I kinda sometimes thrive on stuff. I’ve done forensics stuff where we had to 00:45:33.480 --> 00:45:40.740 do an entire imaging of a lab while people were out from the time work closed to the next day 00:45:40.740 --> 00:45:47.340 they walked in, trying to do it on the sly. Just the challenge of – we didn’t have the right tools 00:45:47.340 --> 00:45:53.100 and we’re improvising and that kinda stuff. You’re running on adrenaline and endorphins and whatnot, 00:45:53.100 --> 00:45:59.220 but the whole thing; somebody inside, this is an inside job, and then you have the realization 00:45:59.220 --> 00:46:03.660 this is somebody I had been investigating, so this is a much bigger picture. Then you 00:46:03.660 --> 00:46:08.760 start thinking about the political intrigue and you’re just kinda like, I can’t believe I’m here. 00:46:08.760 --> 00:46:14.940 It’s really weird to be at the right place at the wrong time or the wrong place at the right time, 00:46:14.940 --> 00:46:22.080 or whatever it is. It just seems – for me, I kinda run on that, I guess. I get this – that’s my MO. 00:46:22.080 --> 00:46:29.220 [MUSIC] I was at the White House when we had the OPM breach, and Heartbleed, and working at a power 00:46:29.220 --> 00:46:33.780 company during a blackout in a hurricane, and it’s just like, I don’t think anybody would ever want 00:46:33.780 --> 00:46:41.220 to hire me because trouble follows and it’s not my fault. But the idea is you can kind of help these 00:46:41.220 --> 00:46:48.120 organizations fix stuff and the emotion after that is alright, well, we did this clean-up. Your 00:46:48.120 --> 00:46:54.720 lunch was eaten. We told you the truth; this is what happened, but what can we do to get better? 00:46:54.720 --> 00:47:01.740 If you’re not somebody who loves the thrill of the chase – and a lot of IT people are builders; 00:47:01.740 --> 00:47:08.640 building back better, which is the mantra of the current new administration as of today. The build 00:47:08.640 --> 00:47:14.520 back better is the thing that also thrills people. They get to tool up, they get to construct stuff, 00:47:14.520 --> 00:47:18.120 they get to do the things that really are exciting. They get to put the good stuff into 00:47:18.120 --> 00:47:24.660 practice, and that’s another emotional high. If you get to play all of that, man, it’s great as 00:47:24.660 --> 00:47:31.680 a blue team person. Very infrequently I ever got the red team, so maybe capturing flags and stuff 00:47:31.680 --> 00:47:35.520 were the big emotional high for red team people, but for blue team it’s like yeah, I kept them out 00:47:35.520 --> 00:47:41.420 or yeah, we fixed this thing, and go ahead, try me. So yeah, that’s kinda how it feels. 00:47:41.420 --> 00:47:48.660 JACK: So, that’s it. This incident was all wrapped up, all with the help of Amélie and Joyce Lin. 00:47:48.660 --> 00:47:52.020 AMÉLIE: Joyce Lin, she was the project manager 00:47:52.020 --> 00:47:56.580 from Mandiant. She unfortunately passed away in an aircraft crash. 00:47:56.580 --> 00:48:00.000 JACK: She died in May of 2020. 00:48:00.000 --> 00:48:03.240 AMÉLIE: She was delivering medical supplies, medical and food supplies, 00:48:03.240 --> 00:48:10.620 to I think somewhere in Southeast Asia and her plane crashed. She was a Air Force reservist. 00:48:10.620 --> 00:48:15.600 The funny thing was is that after that, I ran into her when I was stationed at the Defense Cyber 00:48:15.600 --> 00:48:22.020 Crime Center and she was doing some reserve duty up there, and it was connecting old times. It’s 00:48:22.020 --> 00:48:26.820 a small world. You never know who you run into but when I saw her walking the halls up there, 00:48:26.820 --> 00:48:32.100 and I was like, I just smiled and she smiled back and we just knew we’d been through hell. It wasn’t 00:48:32.100 --> 00:48:38.460 necessarily in a fox hole anywhere but we knew and trusted one another and it was a shame when 00:48:38.460 --> 00:48:43.860 I had heard from some Mandiant people that she had perished, but she’d led her life pretty well. 00:48:43.860 --> 00:48:46.380 JACK: Amélie kept working at the World Bank for 00:48:46.380 --> 00:48:49.920 a while but something awkward happened which made her leave. 00:48:49.920 --> 00:48:53.880 AMÉLIE: What we had found out because I was working late, late hours for a lot of 00:48:53.880 --> 00:48:59.820 these incidents that we had – and I usually just like working late in general ‘cause it was quiet 00:48:59.820 --> 00:49:05.700 – I ended up starting to get suspicious of my boss. He would show up at weird hours and stuff 00:49:05.700 --> 00:49:13.560 like that. He was having some issues with his Mac and the like, and we ended up kinda finding 00:49:13.560 --> 00:49:25.200 out that he was – after I left, found out that he was cheating on his recently-pregnant wife with a 00:49:25.200 --> 00:49:32.520 co-worker. The fact that I was there late and seeing his coming and goings, I think he felt 00:49:32.520 --> 00:49:41.040 kind of threatened, so I was – my contract wasn’t renewed which is kind of a shame. The fact is is 00:49:41.040 --> 00:49:50.040 that going through all this work and whatnot and then getting let go because you’re getting 00:49:50.040 --> 00:49:56.820 too close to something else that was trying to be swept under the carpet was a little annoying but 00:49:56.820 --> 00:50:04.920 I felt that [00:50:00] they’ll do what they need to do. Then I became the chief enterprise security 00:50:04.920 --> 00:50:08.220 architect for Department of the Interior and I was there for nearly five years. 00:50:08.220 --> 00:50:12.540 JACK: Chief security architect for Department of Interior; that sounds huge. 00:50:12.540 --> 00:50:19.560 AMÉLIE: Yeah, yeah. Yeah, it was a big project to work on. I took over, 00:50:19.560 --> 00:50:26.220 helped develop their mobile security program, some of the remote work policies and stuff like that, 00:50:26.220 --> 00:50:31.620 lots of different things. During my time there I did a leadership rotation as part of the 00:50:31.620 --> 00:50:37.020 president’s management council at the White House, at the Office of Management and Budget and worked 00:50:37.020 --> 00:50:42.240 for the chief information officer, the federal chief information officer. Oddly enough, I think 00:50:42.240 --> 00:50:47.160 within a week of me getting there is when they had Heartbleed, the SSL, the open SSL incident, 00:50:47.160 --> 00:50:55.080 which was an interesting experiment in trying to explain to senior political officials how 00:50:55.080 --> 00:51:04.080 open-source projects were governed. Shortly after, we kind of handled the Heartbleed data, data call. 00:51:04.080 --> 00:51:11.160 There was some news that came out regarding USIS, which is one of the companies that did – that was 00:51:11.160 --> 00:51:17.880 contracted out to do background investigations for the Office of Personal Management, OPM. 00:51:17.880 --> 00:51:23.520 The news that came out was just that their contract was put on hold or terminated. 00:51:23.520 --> 00:51:28.080 They performed about 50% of the background investigations. Then later on in the summer, 00:51:28.080 --> 00:51:35.880 KeyPoint was also terminated, and then a few weeks later DHS released the notice that they 00:51:35.880 --> 00:51:42.420 had found some intrusions on the OPM network. But that wasn’t necessarily when the OPM breach 00:51:42.420 --> 00:51:49.020 was disclosed. As you know, the timeline the OPM breach was disclosed I believe in March of 2015. 00:51:49.020 --> 00:51:54.480 The fact is is that things like a history of incident handling and response, when you start 00:51:54.480 --> 00:52:01.920 to look at what these companies did, how they were connected to the network over VPN and whatnot, 00:52:01.920 --> 00:52:09.240 back into OPM, it started looking like they were using these companies as a way 00:52:09.240 --> 00:52:14.940 into OPM’s soft underbelly on their network, and that’s exactly what ended up happening. 00:52:14.940 --> 00:52:19.020 I had mentioned to the federal CIO at the time, Steve VanRoekel, 00:52:19.020 --> 00:52:23.880 that this looks like you have a breach in progress at some point. 00:52:23.880 --> 00:52:30.720 Unfortunately, my rotation ended but I at least let him know this looked bad, so just be prepared 00:52:30.720 --> 00:52:36.480 that in time, this will probably get much, much worse. At that point in time, I think they started 00:52:36.480 --> 00:52:41.460 – DHS worked with OPM to do the investigation and then that was disclosed in March. 00:52:41.460 --> 00:52:44.940 JACK: This incident where the Office of Personnel Management was breached 00:52:44.940 --> 00:52:48.720 was a major incident that I’ll have to cover in another episode someday, 00:52:48.720 --> 00:52:53.983 but handling all these crazy incidents just made Amélie a pro at incident response. 00:52:53.983 --> 00:52:58.140 AMÉLIE: [MUSIC] Well, you know, then I helped found the US Digital Service when I was there 00:52:58.140 --> 00:53:02.040 because no one else – the person who was originally working on it decided they were 00:53:02.040 --> 00:53:06.720 thinking about leaving government, so it was just random stuff I got assigned to 00:53:06.720 --> 00:53:12.120 go and do and put the feather in the hat. But I think a federal government career; 00:53:12.120 --> 00:53:17.940 you’re at the White House. This is like the Superbowl of federal employment and yeah, I 00:53:17.940 --> 00:53:22.500 was to return to Interior to my old position there and I was really excited about bringing what we 00:53:22.500 --> 00:53:28.980 were doing for the US Digital Service back to the agency. We had just switched the CIO in that time. 00:53:28.980 --> 00:53:34.980 It really was saying hey, you haven’t had this chief technology officer position filled in a 00:53:34.980 --> 00:53:39.780 while. Are you guys gonna fill it? I really would love to have that. If not, I’d really like to push 00:53:39.780 --> 00:53:47.580 the Digital Service stuff at Interior. They just didn’t act and I was just feeling kind of pent up, 00:53:47.580 --> 00:53:56.820 so I decided to go work for Disney. I left federal service for about a year to go work as 00:53:56.820 --> 00:54:03.240 a enterprise architect doing technology strategy for Disney. Spent a year in LA, hated Los Angeles, 00:54:03.240 --> 00:54:13.920 moved back to DC, went to go work for Treasury in the GSOC, the government SOC in Vienna, Virginia, 00:54:13.920 --> 00:54:22.200 and lead the Continuous Diagnostics and Mitigation Program for all of Treasury. Then most recently 00:54:22.200 --> 00:54:27.420 I got offered a position to work as a deputy chief information officer at the HHS, Health and 00:54:27.420 --> 00:54:33.840 Human Services inspector general’s office which I really always kinda wanted my career to go and do, 00:54:33.840 --> 00:54:37.620 and learn stuff like budgeting which most techies don’t learn, 00:54:37.620 --> 00:54:41.940 learn how to do HR which most techies don’t tend to learn. 00:54:41.940 --> 00:54:47.640 I got a chance to lead teams, and when our CTO left, I was dual-hatted as the chief 00:54:47.640 --> 00:54:52.740 technology officer leading of development efforts as well as the deputy CIO until we 00:54:52.740 --> 00:54:58.740 hired a new deputy CIO and I stuck as a CTO for a year doing a lot of cool stuff 00:54:58.740 --> 00:55:03.780 there. I’m currently at [00:55:00] Splunk as a technology advocate ‘cause every manager needs 00:55:03.780 --> 00:55:10.200 a break and I like not having to manage people for the last year, so – but the cool thing is 00:55:10.200 --> 00:55:16.980 is as a technology advocate, I get to go out and speak about ways that you can do things better. 00:55:16.980 --> 00:55:22.980 JACK: She has given a lot of talks. If you go to her website, webjedi.net, right in the front page 00:55:22.980 --> 00:55:27.780 you see a link to fifteen different talks she’s given at places like Defcon and ShmooCon and 00:55:27.780 --> 00:55:33.360 other DevOp conferences, too. If you want to hear more from her, definitely check out her talks. 00:55:33.360 --> 00:55:36.240 AMÉLIE: This is my job now and I really enjoy doing it ‘cause it’s 00:55:36.240 --> 00:55:39.480 a way to maybe spread the knowledge around to people that may not get 00:55:39.480 --> 00:55:43.380 experience to it and hopefully make their lives a little bit easier. 00:55:43.380 --> 00:55:54.300 (OUTRO): [OUTRO MUSIC] A big thank you to Amélie Koran. You can find her on Twitter 00:55:54.300 --> 00:56:00.120 which is @webjedi or visit her blog which is webjedi.net. I bring you this show free 00:56:00.120 --> 00:56:04.260 of charge every two weeks and one reason I can keep it going is because of all the wonderful 00:56:04.260 --> 00:56:08.700 people who give to the show through Patreon. This is the most direct way to show support 00:56:08.700 --> 00:56:15.600 for content you appreciate, so consider donating by going to patreon.com/darknetdiaries. The show 00:56:15.600 --> 00:56:19.200 will continue to be free whether you give or not. I’ll still be here making the show because 00:56:19.200 --> 00:56:24.420 I don’t want to leave you hanging and I hope you don’t leave me hanging on the other end. 00:56:24.420 --> 00:56:30.480 This show is made by me, the ID10T award-winner, Jack Rhysider. Sound design this episode by the 00:56:30.480 --> 00:56:35.280 digitized Andrew Meriwether, editing help this episode by the 3D-printed Damienne, and 00:56:35.280 --> 00:56:41.280 our theme music is by the never-cold, always hot, Breakmaster Cylinder. Even though when a SQL query 00:56:41.280 --> 00:56:47.520 walked into a bar, it went up to two tables and asked, can I join you? This is Darknet Diaries.