WEBVTT

00:00:00.120 --> 00:00:05.220
JACK: At this point, every single one of my listeners has been the victim of some kind of

00:00:05.220 --> 00:00:09.120
data breach; [MUSIC] whether that’s getting your personal data stolen from the Equifax

00:00:09.120 --> 00:00:14.400
breach or some other company that had info on you but that got stolen. But how impacted are

00:00:14.400 --> 00:00:19.140
we when this happens? At the least, you should change your passwords and tighten up your own

00:00:19.140 --> 00:00:23.100
personal security and stuff like that. But there’s not much more you can do after that,

00:00:23.100 --> 00:00:29.160
so we’re kind of stuck waiting for whoever stole our data to see what they do with it. Sometimes

00:00:29.160 --> 00:00:34.020
nothing happens; we’re just not impacted at all, but I’m willing to bet in the future

00:00:34.020 --> 00:00:40.740
we’ll all each be impacted by a different kind of hack, something that will certainly

00:00:40.740 --> 00:00:47.400
impact our daily lives in a major way, like one that might take out our electricity or water,

00:00:47.400 --> 00:00:53.640
or a hack that might cause a major disaster. Like, what if a dam got opened up and let out

00:00:53.640 --> 00:00:59.400
a bunch of water and flooded a whole city? That would have a big impact on our lives.

00:00:59.400 --> 00:01:04.200
JACK (INTRO): [INTRO MUSIC] These are

00:01:04.200 --> 00:01:10.440
true stories from the dark side of the internet.

00:01:10.440 --> 00:01:25.950
I’m Jack Rhysider. This is Darknet Diaries. [INTRO MUSIC ENDS]

00:01:25.950 --> 00:01:29.520
JACK:

00:01:29.520 --> 00:01:34.440
This story takes place in the Kingdom of Saudi Arabia, in the Middle East. Saudi Arabia has a

00:01:34.440 --> 00:01:39.840
massive amount of natural resources, primarily oil which makes it a very rich country. In fact,

00:01:39.840 --> 00:01:44.580
the oil company Saudi Aramco is probably the most valuable company in the world because of

00:01:44.580 --> 00:01:49.440
the oil there. In Episode 30, I actually cover a hack that was done against Saudi

00:01:49.440 --> 00:01:53.880
Aramco called Shamoon. It came through and wiped out almost all the [MUSIC] computers

00:01:53.880 --> 00:01:59.340
in the whole company. It was devastating. But there’s another massive company in Saudi Arabia.

00:01:59.340 --> 00:02:03.930
HOST: On the west coast of Saudi Arabia, something remarkable is happening.

00:02:03.930 --> 00:02:05.640
JACK: It’s a petrochemical company.

00:02:05.640 --> 00:02:10.560
HOST: The world’s largest integrated refinery and petrochemical single-phase project.

00:02:10.560 --> 00:02:14.160
JACK: They produce 140 million barrels of products every year.

00:02:14.160 --> 00:02:17.670
HOST: …produces a wide range of high-quality, high-demand products.

00:02:17.670 --> 00:02:21.330
JACK: They produce components that go into manufacturing things we use like…

00:02:21.330 --> 00:02:27.780
HOST: …clothes to fertilizers to packaging to medical equipment to electronics to automobiles,

00:02:27.780 --> 00:02:33.060
and countless other items that make everyday life easier, safer, and more comfortable.

00:02:33.060 --> 00:02:36.720
JACK: I’m not gonna say the name of the company. You can look that up yourself if you want.

00:02:36.720 --> 00:02:38.520
HOST: Where innovation, investment,

00:02:38.520 --> 00:02:42.720
and human potential are being exploited to the full to enrich life.

00:02:42.720 --> 00:02:47.760
JACK: This chemical plant is huge. From a distance it looks like a downtown skyline of a whole city;

00:02:47.760 --> 00:02:53.700
huge tanks, towers, pipes going everywhere, lots of lights on at night, and each structure is a

00:02:53.700 --> 00:02:59.700
building with no walls. You can see right through it. It’s almost skeleton-like. Very industrial;

00:02:59.700 --> 00:03:04.080
it’s a massive plant with lots of chemicals, oil, and people all working together to make

00:03:04.080 --> 00:03:11.880
petrol-based products that you and I use. But in 2017, something big happened there.

00:03:11.880 --> 00:03:18.660
[ALARMS] In June 2017, a Triconex controller shut down.

00:03:18.660 --> 00:03:21.120
HOST: Redefining process safety.

00:03:21.120 --> 00:03:23.866
JACK: These are the emergency shutdown systems.

00:03:23.866 --> 00:03:26.940
HOST: [MUSIC] Market-leading Triconex Safety Systems have, for example,

00:03:26.940 --> 00:03:32.640
run for more than 600 million hours without failure on-demand and are still going strong.

00:03:32.640 --> 00:03:37.980
JACK: Safety systems like this have to be extremely robust and resilient and never fail.

00:03:37.980 --> 00:03:43.440
HOST: But today, technology is only part of the safety equation. Production complexity,

00:03:43.440 --> 00:03:49.080
aging systems, changing workforce, cyber-crime and complacency are

00:03:49.080 --> 00:03:52.770
just a few of the factors introducing new threats to operational integrity.

00:03:52.770 --> 00:03:54.060
JACK: That’s for sure.

00:03:54.060 --> 00:03:59.820
HOST: Your culture is enriched; your people are safe. Your business is sound. Triconex

00:03:59.820 --> 00:04:01.920
process [00:05:00] safety by Schneider Electric.

00:04:01.920 --> 00:04:05.760
JACK: [MUSIC] Okay, hang on a second. In order to understand what happened at this plant,

00:04:05.760 --> 00:04:11.400
we need to learn a little bit more about what OT is. You probably already know what IT is, right?

00:04:11.400 --> 00:04:16.320
Information technology. It’s where computers store, manipulate, and transfer information.

00:04:16.320 --> 00:04:21.060
OT is operational technology and this is the hardware and software that’s used to control

00:04:21.060 --> 00:04:27.000
physical things in the world like valves and pumps and other machinery. Think about all the

00:04:27.000 --> 00:04:33.480
electronics that control a factory, a plant, or a utility company. A chemical and petrol plant

00:04:33.480 --> 00:04:39.120
like this has a ton of OT systems. There are electrical devices that open valves,

00:04:39.120 --> 00:04:43.860
pour chemicals, release gases, and pump fluids. But an important component of all

00:04:43.860 --> 00:04:50.040
this is the safety instrumented systems, or SIS. So many of the chemicals at the plant

00:04:50.040 --> 00:04:57.000
are toxic and must be handled very carefully. These SIS or safety systems will monitor the

00:04:57.000 --> 00:05:02.100
environment very closely and trigger a shutdown if anything becomes dangerous.

00:05:02.100 --> 00:05:07.860
Those safety systems that are responsible for conducting an emergency shutdown are the Triconex

00:05:07.860 --> 00:05:16.620
controllers. In June 2017, something had gone terribly wrong. One of the emergency shutdown

00:05:16.620 --> 00:05:21.960
systems stopped working; it malfunctioned. When the emergency shutdown device malfunctions,

00:05:21.960 --> 00:05:26.820
then if there was a real emergency at the plant, this could result in a disaster.

00:05:26.820 --> 00:05:32.340
This is a big problem, like when the brakes go out on your car. But when this system malfunctioned,

00:05:32.340 --> 00:05:37.140
it triggered an alert on another system which alerted the engineers to go shut the plant down

00:05:37.140 --> 00:05:42.900
and inspect this controller. The manufacturer of the Triconex system came out and they examined

00:05:42.900 --> 00:05:46.980
it but didn’t find anything wrong with it. The plant was able to get back online pretty

00:05:46.980 --> 00:05:52.920
quick. That’s because they weren’t looking in the right place for the problem. [MUSIC]

00:05:52.920 --> 00:06:01.140
Fast-forward two months. It’s August 4th, 2017. It’s 7:43 p.m. on a Friday night.

00:06:01.140 --> 00:06:07.200
Six of the Triconex Safety Systems had malfunctioned and tripped an alarm. When the

00:06:07.200 --> 00:06:12.060
safety systems fail like this, it automatically causes a shutdown at the plant because if you

00:06:12.060 --> 00:06:17.160
don’t have properly operating safety systems, you have nothing protecting you in case something goes

00:06:17.160 --> 00:06:22.200
wrong. Those systems that had problems were in charge of issuing a shutdown if either the

00:06:22.200 --> 00:06:27.360
sulfur recovery unit or the burner management systems had detected a dangerous condition.

00:06:27.360 --> 00:06:32.760
This is a big chemical plant and there are many technicians and engineers who work there and can

00:06:32.760 --> 00:06:40.320
troubleshoot this kind of issue, but it’s 8:00 p.m. on a Friday night. It’s the weekend so the

00:06:40.320 --> 00:06:45.240
crew was minimal. There’s also a lot of vendors who work there who could also troubleshoot this

00:06:45.240 --> 00:06:51.120
equipment but their staff is also minimal too because it’s the night and on a weekend.

00:06:51.120 --> 00:06:54.240
Troubleshooting began on these Triconex systems.

00:06:54.240 --> 00:06:59.340
Logs showed that some configuration changes had been pushed to the controllers. Now,

00:06:59.340 --> 00:07:04.080
to make a change on the Triconex controller, yeah, you need to use a computer to interact with it.

00:07:04.080 --> 00:07:09.600
But someone had to physically be present at the controller to make the change. Specifically,

00:07:09.600 --> 00:07:14.640
there’s a key that needs to be inserted into the controller and you have to turn that key to the

00:07:14.640 --> 00:07:20.580
mode Program. Once the key is in that setting, someone back in the control room can push a

00:07:20.580 --> 00:07:25.680
configuration change to that controller. Well, it just so happened that someone had left six of

00:07:25.680 --> 00:07:31.560
these controllers in the Program state and that’s not right. It’s 8:00 p.m. on a Friday night;

00:07:31.560 --> 00:07:37.560
no authorized changes were approved for those controllers at that time of night. The key

00:07:37.560 --> 00:07:43.440
should not have been left on that setting, but I guess it was just laziness on the plant operators.

00:07:43.440 --> 00:07:47.940
I mean, it takes ten minutes to go from the control room all the way to the controller

00:07:47.940 --> 00:07:51.480
just to put the key in and switch it to Program. Then you need to go all the way

00:07:51.480 --> 00:07:55.140
back to the control room, make the changes you need to make, and then when you’re done,

00:07:55.140 --> 00:07:59.040
hopefully remember to go all the way back to the controller and turn the key back to the Run

00:07:59.040 --> 00:08:03.960
mode. It looks like a few of these were just accidentally left in the Program state which

00:08:03.960 --> 00:08:09.480
was bad practice. Actually, operators had been seeing alerts on a daily basis that the key was

00:08:09.480 --> 00:08:15.240
in the wrong state but once a day they would just clear those alerts and ignore them. I’m

00:08:15.240 --> 00:08:19.980
not sure if it was just laziness of the people monitoring the alerts or the engineers or both,

00:08:19.980 --> 00:08:25.080
because typically you don’t want anyone to be able to make remote changes to these safety

00:08:25.080 --> 00:08:29.520
controllers. You want to cut these things off from the network entirely for safety reasons.

00:08:29.520 --> 00:08:35.400
But when that key was in the Program state, it meant it was now waiting for a configuration

00:08:35.400 --> 00:08:41.760
change from over the network. But something went wrong when the config changes were pushed

00:08:41.760 --> 00:08:47.700
to these controllers. Whatever configuration was sent, it caused a failure state on the units. It

00:08:47.700 --> 00:08:53.100
didn’t like whatever it was getting and caused a reboot of these systems. This is what triggered

00:08:53.100 --> 00:08:58.680
the alerts and caused the plant shutdown. This was similar to the outage two months ago but that one

00:08:58.680 --> 00:09:03.900
was just one controller; this time [00:10:00] it was six at the same time. But what’s more

00:09:03.900 --> 00:09:09.720
suspicious is that because this was a weekend and at night, there were no planned changes to these

00:09:09.720 --> 00:09:18.800
controllers at that time. Whatever config changes were attempted, they were completely unauthorized.

00:09:18.800 --> 00:09:24.540
[MUSIC] As the onsite crew investigated, they found the computer in the operations room which

00:09:24.540 --> 00:09:29.580
was pushing these configurations. When they investigated further, they found this computer

00:09:29.580 --> 00:09:37.380
had an unauthorized RDP session opened on it. This is really scary. To connect the dots here,

00:09:37.380 --> 00:09:44.880
some unknown person has gained remote access to a computer in the operations room. That computer had

00:09:44.880 --> 00:09:51.720
just pushed a config change to six of these safety systems which caused the plant to shut down.

00:09:51.720 --> 00:09:57.840
Something very fishy was going on here. The onsite crew continued to troubleshoot

00:09:57.840 --> 00:10:03.420
for days and even weeks but weren’t getting anywhere further with this investigation.

00:10:03.420 --> 00:10:07.680
It was just above their skill level so they called for additional help.

00:10:07.680 --> 00:10:11.640
JULIAN: My name is Julian Gutmanis. I’m an industrial incident responder.

00:10:11.640 --> 00:10:16.680
JACK: Julian was working as an OT incident responder in Saudi Arabia at the time. He was

00:10:16.680 --> 00:10:20.580
told to hop on a conference call and listen to their problem to see if he had any input.

00:10:20.580 --> 00:10:25.920
JULIAN: The first I was told was that we needed to get on a phone call to provide some guidance

00:10:25.920 --> 00:10:30.180
to the plant as they were having mechanical problems that had resulted in a shutdown.

00:10:30.180 --> 00:10:34.380
They only mentioned the one shutdown in August. All we’d really heard about it

00:10:34.380 --> 00:10:39.600
was mechanical issues. They just want to have a security analyst on the phone to make sure

00:10:39.600 --> 00:10:45.120
that there’s nothing wrong; don’t worry about it, it’s not a big deal, just join the call.

00:10:45.120 --> 00:10:48.120
JACK: See, at this point the plant didn’t even know if this

00:10:48.120 --> 00:10:50.370
was a security incident or a mechanical failure.

00:10:50.370 --> 00:10:54.960
JULIAN: But when I probed a little bit further to say well, what’s actually happening here, they

00:10:54.960 --> 00:10:59.880
started saying that it looks like the emergency shutdown systems have kicked in and shut the

00:10:59.880 --> 00:11:07.080
plant down and they don’t know why. They’re seeing some potentially weird logins and it’s happening

00:11:07.080 --> 00:11:12.900
on a Friday night. I was like, almost double-take. [MUSIC] Like, what are you talking about? This is

00:11:12.900 --> 00:11:18.440
probably the most serious thing I’ve ever heard about in my career. Can we get on a plane now?

00:11:18.440 --> 00:11:24.840
JACK: Julian added everything up quickly; an unknown remote attacker had attempted to make

00:11:24.840 --> 00:11:30.480
configuration changes to an emergency shutdown system of this plant? Why would someone do that?

00:11:30.480 --> 00:11:35.820
Why would someone want to mess with the last line of defense like that? Without

00:11:35.820 --> 00:11:42.120
a properly functioning emergency shutdown system, catastrophic results could occur.

00:11:42.120 --> 00:11:46.740
Julian immediately wanted to travel to the site so he assembled a team.

00:11:46.740 --> 00:11:51.720
NASER: Yeah, hi, I’m Naser Aldossary. I’m currently an industrial incident responder.

00:11:51.720 --> 00:11:55.560
JACK: Naser is also an OT incident responder based in the Kingdom of

00:11:55.560 --> 00:12:00.420
Saudi Arabia. They called Naser up and said hey, get ready; you’re going on a trip.

00:12:00.420 --> 00:12:05.400
NASER: My bags were ready. We’re used to this kind of traveling. I had just picked up one of my ready

00:12:05.400 --> 00:12:11.880
bags and head to the office and was told that we should book the earliest flight, and we did.

00:12:11.880 --> 00:12:14.040
JACK: This is sometimes the life of an incident responder;

00:12:14.040 --> 00:12:19.740
people have to always have a few bags ready to go at any time, like having a three-day go-bag and a

00:12:19.740 --> 00:12:24.000
seven-day go-bag are suggested because when you’re dealing with big incidents like this,

00:12:24.000 --> 00:12:28.920
it’s best to have someone get onsite as soon as possible and help conduct the incident response

00:12:28.920 --> 00:12:33.780
and forensics. Julian and Naser grabbed their go-bags and jumped on the earliest flight to

00:12:33.780 --> 00:12:38.820
the plant. It was an overnight flight which means they were up all night getting there.

00:12:38.820 --> 00:12:45.780
NASER: We arrived there the next day. It was August, hot. It was still early in the morning

00:12:45.780 --> 00:12:49.560
but it was super hot. We were waiting in line to get in through the security checkpoint and get

00:12:49.560 --> 00:12:58.440
our access granted. By the time we made it, the system just decided to malfunction and shut down.

00:12:58.440 --> 00:13:03.000
JULIAN: [MUSIC] I guess one of the funny things we were joking about at the time was when we

00:13:03.000 --> 00:13:06.720
went through the security checkpoint. It was about the time that we handed over our IDs

00:13:06.720 --> 00:13:11.760
and they started looking at who we were that their system just shut down. We were kind of

00:13:11.760 --> 00:13:16.740
joking at the fact that the IT compromise is so bad that these guys are monitoring the security

00:13:16.740 --> 00:13:21.660
desk and blocking people from getting in. It was quite entertaining at the time.

00:13:21.660 --> 00:13:27.180
NASER: The security guard could not grant us access so we waited there for another hour,

00:13:27.180 --> 00:13:31.860
waiting for them to figure out how to restart the system and grant us access.

00:13:31.860 --> 00:13:35.640
JACK: They finally got in. It’s not just two of them, actually. I think four of

00:13:35.640 --> 00:13:40.080
them showed up onsite to help conduct this incident response. They break into two teams

00:13:40.080 --> 00:13:43.740
of two people each and start interviewing everyone just to get a lay of the land.

00:13:43.740 --> 00:13:48.360
JULIAN: I guess from the investigation standpoint, we really wanted to start at

00:13:48.360 --> 00:13:54.300
the systems room impacted. What caused the actual shutdown was the safety controllers. Obviously,

00:13:54.300 --> 00:13:58.260
the engineers have already done some reliability and some mechanical testing on the devices and

00:13:58.260 --> 00:14:04.980
pulled things like diagnostics logs and [00:15:00] other certain artifacts from these devices. After

00:14:04.980 --> 00:14:11.340
analyzing the actual controllers and identifying this, we wanted to figure out what, if anything,

00:14:11.340 --> 00:14:15.120
had actually changed in the controllers. You’ve got to understand that these controllers aren’t

00:14:15.120 --> 00:14:19.980
like Windows or Linux machines. They’re embedded systems. The functionality that

00:14:19.980 --> 00:14:24.060
you can actually get from these devices is relatively limited, especially depending on

00:14:24.060 --> 00:14:29.040
the configuration. Pulling these logs is really plugging in a serial cable and waiting five,

00:14:29.040 --> 00:14:32.580
ten minutes until it actually completes downloading the logs and things like that.

00:14:32.580 --> 00:14:37.920
It’s not a basic process. The other thing you can’t really do is actually pull the programs back

00:14:37.920 --> 00:14:44.460
off the controllers and say hey, this is what’s on there. What you can do is you can jump onto

00:14:44.460 --> 00:14:51.660
the engineering software, TriStation, and issue a – sort of like an integrity verification command.

00:14:51.660 --> 00:14:57.120
This command basically takes what the program and logic files that the engineers worked on within

00:14:57.120 --> 00:15:04.380
the system and then pushes them obviously to the safety controller and does a comparison of what’s

00:15:04.380 --> 00:15:08.880
running on the controller versus what’s on the system. What came back after that was actually

00:15:08.880 --> 00:15:14.940
a number of IO points. There was discrepancies between the IO points which is basically the

00:15:14.940 --> 00:15:19.659
inputs and the outputs that go to the safety systems that would end up shutting down the plant.

00:15:19.659 --> 00:15:24.240
JACK: [PULSING TONE] Keep in mind while they’re down there working on these controllers, they’re

00:15:24.240 --> 00:15:31.740
in the plant where the products are being created. It’s loud, hot, and they have to wear safety gear.

00:15:31.740 --> 00:15:37.380
JULIAN: Taking a step back from that, we wanted to go okay, well, if this is

00:15:37.380 --> 00:15:42.540
occurring, how is it occurring? Who’s doing this? Again, when we arrived, we really weren’t sure

00:15:42.540 --> 00:15:46.260
whether this was an insider that was doing this. Maybe it could have been one of the operators

00:15:46.260 --> 00:15:50.100
that just gained access to the engineering workstation. Or was this somebody coming in

00:15:50.100 --> 00:15:55.260
from the IT network? Could have been some kind of contractor that was – a number of plants, projects

00:15:55.260 --> 00:15:59.460
that were going on at the moment with different vendors and things going on. You have potentially

00:15:59.460 --> 00:16:04.560
a number of untrusted parties wandering around that could have gained access to these systems.

00:16:04.560 --> 00:16:09.360
Realistically at this point in time, the last thought on my mind was this is a remote attack.

00:16:09.360 --> 00:16:13.380
We were really thinking that it could have just been either somebody messing with the systems,

00:16:13.380 --> 00:16:20.040
somebody doing something they shouldn’t be doing, or a malicious internal party, realistically.

00:16:20.040 --> 00:16:23.640
What we started doing there was really investigating the engineering workstations

00:16:23.640 --> 00:16:32.040
which involved taking triage artifacts from the devices, a number of images, and things like that.

00:16:32.040 --> 00:16:37.620
One of the things we were working with that was pretty handy was obviously a pretty confirmed

00:16:37.620 --> 00:16:42.840
timeline. We knew exactly when the controllers shut down and resulted in the plant shutdown.

00:16:42.840 --> 00:16:48.840
If you’re doing an investigation, this is very handy; you know that I can just focus on the

00:16:48.840 --> 00:16:53.880
lead-up to this event and then really narrow down my search on what’s occurring in that timeframe.

00:16:53.880 --> 00:16:59.700
NASER: I remember the press engineer was sitting next to me and I just looked at him;

00:16:59.700 --> 00:17:04.560
I was like by any chance, do you have any kind of HP printers here? It’s unusual in

00:17:04.560 --> 00:17:09.960
these environments. He was like no; why? I was like, there is a folder called HP in here

00:17:09.960 --> 00:17:15.240
and there is a Python DLL. This is where it kind of clicked in my head that this

00:17:15.240 --> 00:17:21.240
is – something is going on. [MUSIC] To be honest, the first thing I started thinking about is I’m

00:17:21.240 --> 00:17:28.500
in this plant. Initially when we went there, we knew that it’s possibly in an unsafe state

00:17:28.500 --> 00:17:33.840
but you’re just sitting there in a place where you’re not sure what was going on.

00:17:33.840 --> 00:17:38.040
To be honest with you, it was scary.

00:17:38.040 --> 00:17:45.960
It’s not something where an e-mail is gonna go down or –these systems, especially – when you

00:17:45.960 --> 00:17:50.640
work in this field, this is – it gets drilled in your head; you’re not supposed to go there until

00:17:50.640 --> 00:17:56.160
you get all these safety trainings. One of the safety trainings they drill in your head is H2S,

00:17:56.160 --> 00:18:01.800
H2S. H2S this, H2S that, it’s poisonous. It just kind of goes to the back of your

00:18:01.800 --> 00:18:06.180
head that if something happens you need to do this and you need to do that, and they give

00:18:06.180 --> 00:18:12.060
you the real scenarios. It’s not something that – it’s instant death, in some cases.

00:18:12.060 --> 00:18:22.680
JACK: H2S is hydrogen sulfide. It’s very poisonous, corrosive, and flammable. The safety

00:18:22.680 --> 00:18:28.140
controller that they are troubleshooting was part of the sulfur recovery unit.

00:18:28.140 --> 00:18:34.380
This system was in charge of shutting down the plant if there were unsafe levels of H2S detected,

00:18:34.380 --> 00:18:41.640
but this safety system itself had gone down. If there were unsafe levels of H2S, there [MUSIC] was

00:18:41.640 --> 00:18:46.860
no safety system to shut things down to protect the people and the equipment in this plant.

00:18:46.860 --> 00:18:52.380
NASER: Knowing that you’re in this unsafe condition, I remember I just

00:18:52.380 --> 00:18:58.380
walked outside and I was like, maybe I shouldn’t be breathing this air. You’re

00:18:58.380 --> 00:19:03.720
really scared. It’s not an easy [00:20:00] thing. Even I remember, when I discussed it

00:19:03.720 --> 00:19:09.240
with my boss when we came back and I was like yeah, this is a really dangerous situation.

00:19:09.240 --> 00:19:13.440
JULIAN: At this stage when we’re being engaged, as I mentioned, it was a couple weeks after the

00:19:13.440 --> 00:19:19.440
actual outage had occurred. Management’s already done the difficult discussions about do I start

00:19:19.440 --> 00:19:24.360
this plant back up or do we need to do further investigations, or what do we do? Obviously,

00:19:24.360 --> 00:19:28.620
they come to the conclusion that leaving the plant in a down-state is extremely expensive.

00:19:28.620 --> 00:19:33.120
We’ve already had to pay for the outage which is obviously a week or something to get back up and

00:19:33.120 --> 00:19:39.720
running. They want to start the plant back up. Even when you’ve detected these kind of attacks

00:19:39.720 --> 00:19:44.340
with the malware and stuff within the plant and you’re providing a report saying you have an

00:19:44.340 --> 00:19:49.560
advanced adversary in your plant, they’re going to be hesitant to even shut the plant down. You

00:19:49.560 --> 00:19:54.840
know you’re dealing with a hot environment when you’re doing the incident response. You know that

00:19:54.840 --> 00:20:00.240
it could be some pretty hairy situations if the attackers choose to do some kind of

00:20:00.240 --> 00:20:04.020
– if they’re still active within the environment or if they’ve triggered some kind of backdoors

00:20:04.020 --> 00:20:07.020
or timebombs in the environment for when the communications are severed.

00:20:07.020 --> 00:20:12.000
JACK: Whoa, this is a lot to think about while onsite. Without having

00:20:12.000 --> 00:20:16.680
any sleep the night before, Julian and Naser had work to do still.

00:20:16.680 --> 00:20:19.860
JULIAN: We wanted to confirm whether or not this was an insider. Realistically,

00:20:19.860 --> 00:20:27.000
that was our main goal. What we initially did is we identified the malfunctioning systems which

00:20:27.000 --> 00:20:32.100
were the controllers, we traced it back to the engineering workstations which then led to the

00:20:32.100 --> 00:20:39.540
investigation that found the Triconex tools; the trilog.exe and the library Python files.

00:20:39.540 --> 00:20:43.620
JACK: They figured out which of the engineering computers had that remote desktop connection

00:20:43.620 --> 00:20:48.420
to it and examined it. They found that computer and immediately took a snapshot of that system,

00:20:48.420 --> 00:20:53.880
copying everything off it; all files of course but on top of that, all the event logs on that system

00:20:53.880 --> 00:20:58.980
and everything that was in memory and all running processes, and all open connections to that

00:20:58.980 --> 00:21:05.520
computer. Yeah, someone had accessed this computer remotely but what did they do once they got in?

00:21:05.520 --> 00:21:13.560
Julian and Naser discovered two files on this computer that were the smoking gun; trilog.exe

00:21:13.560 --> 00:21:22.620
and library.zip. This was malware, very dangerous malware. These files were used to interact with

00:21:22.620 --> 00:21:26.520
those safety controllers and this was the program that was used to push configuration

00:21:26.520 --> 00:21:31.500
changes to those safety systems, and inside that zip file were the binary files that were sent to

00:21:31.500 --> 00:21:38.640
the controller. This would be extremely useful to analyze more in-depth later, but for now they’re

00:21:38.640 --> 00:21:42.180
still trying to track down who connected to this computer to put these files here.

00:21:42.180 --> 00:21:46.860
JULIAN: From there what we did was, we were luckily able to trace a lot of the activity

00:21:46.860 --> 00:21:53.700
through the DMZ firewalls. Luckily, the plant were capturing both successful and failed connection

00:21:53.700 --> 00:21:59.040
attempts through the plant DMZ. [MUSIC] So, leveraging these communications, we’re able to

00:21:59.040 --> 00:22:06.180
trace a number of sessions that overlapped with artifacts being created on engineering

00:22:06.180 --> 00:22:12.060
workstations within the system journals, the NTFS journal. We could see the sessions coming

00:22:12.060 --> 00:22:18.720
through a DMZ chokepoint, through the jump box, and then the DMZ from the perimeter VPN.

00:22:18.720 --> 00:22:22.860
We did track this to an external party that was logging in from the VPN through

00:22:22.860 --> 00:22:28.020
to the DMZ and then through to the engineering workstation and leveraging these attack tools.

00:22:28.020 --> 00:22:32.100
JACK: The network equipment at the plant had some pretty good logging turned on so it made it easy

00:22:32.100 --> 00:22:36.000
for them to connect the dots. The incident response team determined that someone had

00:22:36.000 --> 00:22:42.420
connected from the outside, the internet, the world, and exploited a computer inside the DMZ,

00:22:42.420 --> 00:22:46.740
a separate part of the network inside this chemical plant. It was supposed to

00:22:46.740 --> 00:22:52.440
be separated from the inside of the network but the attackers found a hole in the DMZ which let

00:22:52.440 --> 00:22:57.720
them slip through into the internal network which is how they got to those engineering workstations,

00:22:57.720 --> 00:23:03.300
and that’s how they got trilog.exe and the library.zip file onto that computer.

00:23:03.300 --> 00:23:07.980
Once the attacker was on that engineering workstation, they got a list of safety controllers

00:23:07.980 --> 00:23:14.940
and did a multi-cast ping on all those controllers to see if any of them were in the Program state.

00:23:14.940 --> 00:23:21.660
That’s how they found these six controllers were ready to receive a new configuration. These two

00:23:21.660 --> 00:23:26.580
files that were on the engineering workstation had some advanced malware, something that Julian and

00:23:26.580 --> 00:23:31.440
Naser were totally blown away by, something that the makers of the Triconex controllers, Schneider

00:23:31.440 --> 00:23:36.120
Electronic, had also never seen before and they were flabbergasted by it. Collecting these hacker

00:23:36.120 --> 00:23:41.340
tools was a fantastic find for the security teams to investigate further but when they looked at the

00:23:41.340 --> 00:23:47.820
engineering workstation again later, these tools were suddenly gone. Somebody had deleted them.

00:23:47.820 --> 00:23:52.440
JULIAN: Yeah, I mean, well, obviously they’re still active.

00:23:52.440 --> 00:23:58.140
We kind of thought that they may have taken a break after the shutdown had occurred. Come

00:23:58.140 --> 00:24:01.200
a couple of weeks later, the tool kit’s still there, it seems [00:25:00] like they probably

00:24:01.200 --> 00:24:09.300
haven’t done much. But seeing it being deleted like that was – I keep saying that we got lucky;

00:24:09.300 --> 00:24:14.040
we’re lucky we imaged that machine when we imaged it and we found what was causing the outage.

00:24:14.040 --> 00:24:19.380
JACK: This incident response team is good at industrial control systems and OT. They came in,

00:24:19.380 --> 00:24:22.440
collected enough information, and they determined the problem.

00:24:22.440 --> 00:24:26.460
JULIAN: At that point, realistically, we had

00:24:26.460 --> 00:24:32.040
kind of achieved our goal. Our goal was to realistically do a initial triage and find out

00:24:32.040 --> 00:24:37.080
is this system malfunctioning a controller or is this something more malicious? Our consideration

00:24:37.080 --> 00:24:41.940
at that point was that we had a pretty advanced actor that was potentially interacting with the

00:24:41.940 --> 00:24:47.280
controllers. This is obviously excluding a lot of the stuff that we did find within the environment

00:24:47.280 --> 00:24:55.260
including other malware strains, and things like that. At this point, our goal kind of shifted.

00:24:55.260 --> 00:25:01.560
Our goal wasn’t now initially the incident. We had escalated to a cleanup crew, basically;

00:25:01.560 --> 00:25:06.840
an external party to come in, do a full scoping exercise, and eradicate the threat from the

00:25:06.840 --> 00:25:13.320
environment. That was how we handled that. Our goal from there really shifted to the kingdom.

00:25:13.320 --> 00:25:19.560
We ourselves had 170+ plants that have – a number of them have Schneider Electric

00:25:19.560 --> 00:25:24.480
controllers that we needed to assess to make sure that we aren’t currently being compromised

00:25:24.480 --> 00:25:31.800
or impacted. We were protected ourselves against that state. We also looked at communicating with

00:25:31.800 --> 00:25:37.260
other potentially impacted organizations so other petrochemical facilities, other

00:25:37.260 --> 00:25:42.540
oil and gas facilities within the kingdom ‘cause obviously it was a wide-scale targeting campaign;

00:25:42.540 --> 00:25:49.740
it wasn’t just the victim that was being impacted. From there we were also – Naser was doing

00:25:49.740 --> 00:25:54.420
a huge amount of communication with the Saudi government to ensure that appropriate information

00:25:54.420 --> 00:25:59.280
was [MUSIC] shared within the intelligence circles to be distributed to appropriate teams

00:25:59.280 --> 00:26:05.160
to make sure that they can track what’s going on and be across everything that was going on. Our

00:26:05.160 --> 00:26:14.600
responsibilities didn’t end at the victim and the initial triage. It, if anything, grew from there.

00:26:14.600 --> 00:26:15.600
JACK:

00:26:15.600 --> 00:26:19.980
Julian and Naser got out of there. A new team came in to take a look at the problems

00:26:19.980 --> 00:26:23.820
in the DMZ and the insecure engineering workstations, and of course went through

00:26:23.820 --> 00:26:27.900
and made sure that none of the controllers were in that Program state anymore. Also,

00:26:27.900 --> 00:26:31.800
it should never be allowed for someone to come into this network from the internet and to be

00:26:31.800 --> 00:26:37.860
able to gain control of a safety system in the plant. This is a design flaw of the network. Those

00:26:37.860 --> 00:26:42.300
engineering workstations that had the ability to push configurations to the controllers should be

00:26:42.300 --> 00:26:47.340
totally disconnected from the network so that a remote attacker could never gain access to them.

00:26:47.340 --> 00:26:51.300
This should make it so that the only people who could make changes are the people who are

00:26:51.300 --> 00:26:57.600
onsite and authorized to do so. Something big had happened here, something extremely

00:26:57.600 --> 00:27:03.120
serious and potentially really dangerous. Why would someone hack into this place and target

00:27:03.120 --> 00:27:12.480
the emergency shutdown systems? After the break we’ll try to unravel that mystery as best we can.

00:27:12.480 --> 00:27:16.620
FireEye is a company that is known for investigating cyber-security

00:27:16.620 --> 00:27:20.580
threats, [MUSIC] and FireEye was called down to clean up and investigate this problem.

00:27:20.580 --> 00:27:25.020
MARINA: My name is Marina Krotofil and I’ve been specializing on the security

00:27:25.020 --> 00:27:29.280
of industrial control systems for almost a decade by now.

00:27:29.280 --> 00:27:32.580
JACK: This is Marina Krotofil. As a member of the FireEye team,

00:27:32.580 --> 00:27:36.900
Marina was investigating the incident. She knows her stuff when it comes to attacks and

00:27:36.900 --> 00:27:42.000
exploitation of embedded systems. She focused on this malware analysis and she’s here to tell us

00:27:42.000 --> 00:27:46.500
what was in the public FireEye reports as well as her independent analysis of this.

00:27:46.500 --> 00:27:52.920
MARINA: The attacks seemed to understand overall the culture and [00:30:00] how

00:27:52.920 --> 00:27:54.480
the plants work. They were trying onFriday and Saturday; this is weekdays off, and so

00:27:54.480 --> 00:28:06.540
they were basically targeting their sensitive operations,

00:28:06.540 --> 00:28:13.440
like injection of the implant to this – weekdays off and on the later hour.

00:28:13.440 --> 00:28:17.220
JACK: Okay, good point; they had to know this plan inside and out because let’s face it,

00:28:17.220 --> 00:28:20.400
IT and OT are very different animals. A typical

00:28:20.400 --> 00:28:24.540
hacker is not gonna know how to work a Triconex safety system to take control

00:28:24.540 --> 00:28:28.560
of it or know how to program it. That takes a whole new level of expertise.

00:28:28.560 --> 00:28:35.940
MARINA: I remember there was – one evening, I – once we were still studying the codes

00:28:35.940 --> 00:28:41.700
and you’re still just trying to understand what the malware’s exactly doing, what is the intent?

00:28:41.700 --> 00:28:47.880
You just at the very beginning. They were like, function names, and one of them had

00:28:47.880 --> 00:28:58.140
write ‘ext’, so at the end it was ‘ext’. ‘Ext’, for me, was – the first thought, what I was – had

00:28:58.140 --> 00:29:03.660
in my head is external, so do you want to write in some external memory? Then I started talking

00:29:03.660 --> 00:29:11.940
to some guys who has access to the controller. I received a photo of their – my PCB board.

00:29:11.940 --> 00:29:18.240
JACK: When she looked at the photos of this device, she saw that the safety device also had

00:29:18.240 --> 00:29:24.720
the ability to control the valves. What this meant was if this malware was writing to the external

00:29:24.720 --> 00:29:31.380
memory, it could instruct the valves to operate in an unsafe state which could cause damage. At

00:29:31.380 --> 00:29:35.700
the same time, the malware could instruct the safety systems not to shut down or even

00:29:35.700 --> 00:29:42.840
create an alert. [MUSIC] This meant the attackers could unleash a catastrophic blow to this plant.

00:29:42.840 --> 00:29:52.020
MARINA: I got so scared. I could not even tell you; I could not breathe. My hands were shaking.

00:29:52.020 --> 00:29:57.540
I felt like I had discovered – so important and then later on when I first analyzed the code,

00:29:57.540 --> 00:30:03.360
I realized that this is not external but extended, so if you want to write more than twenty-two – a

00:30:03.360 --> 00:30:11.160
large chunk of code, then you would evoke specific function which allows you to write more so it’s

00:30:11.160 --> 00:30:18.200
not – was external, but extended. But at that time, I swear that I would have a heart attack.

00:30:18.200 --> 00:30:25.200
JACK: They realized when the plant shut down, it was a mistake. The hackers accidentally

00:30:25.200 --> 00:30:30.420
tripped some kind of emergency shutdown system while fumbling around with these systems which

00:30:30.420 --> 00:30:36.000
makes you wonder, what was their objective? FireEye came up with three potential attack

00:30:36.000 --> 00:30:41.400
scenarios. Attack option one; the attackers could force this plant to shut down by triggering the

00:30:41.400 --> 00:30:45.960
emergency shutdown systems, basically a false positive. But by shutting down the plant,

00:30:45.960 --> 00:30:50.700
it could mean a financial impact to the plant. Then there’s attack option two;

00:30:50.700 --> 00:30:55.860
the attackers could reprogram the safety systems so the plant could continue to operate in an

00:30:55.860 --> 00:31:01.860
unsafe state which could cause destruction to the plant or even a disaster. Then there’s attack

00:31:01.860 --> 00:31:07.200
option three and this one is the most scary; the attackers could make the emergency shutdown system

00:31:07.200 --> 00:31:14.940
ignore unsafe operating levels and then somehow cause the plant to operate in an unsafe state.

00:31:14.940 --> 00:31:20.340
In this scenario, the attackers might be able to control the valve for hydrogen sulfide, H2S,

00:31:20.340 --> 00:31:25.680
and somehow pump out high amounts of this dangerous gas and then tell the emergency

00:31:25.680 --> 00:31:31.800
shutdown system to ignore the dangerous levels of H2S. If you just breathe too

00:31:31.800 --> 00:31:36.960
much of this stuff in, you can lose your sense of smell, fall unconscious, or die.

00:31:36.960 --> 00:31:42.720
To top it off, hydrogen sulfide is extremely combustible so one little spark and this could

00:31:42.720 --> 00:31:48.960
cause a major explosion which would almost certainly result in casualties. [MUSIC] As

00:31:48.960 --> 00:31:53.940
the team at FireEye investigated this, they decided to give it a name. Since the file

00:31:53.940 --> 00:32:00.480
was called trilog.exe and this was targeting the Triconex systems, they called the malware Triton.

00:32:00.480 --> 00:32:07.080
MARINA: The Triton malware, if it could have this damage payload which was not uncovered, it might

00:32:07.080 --> 00:32:11.880
keep them up. It means that the process will not shut down and that could be a safety incident.

00:32:11.880 --> 00:32:15.180
JACK: But this malware wasn’t made by someone who was sloppy

00:32:15.180 --> 00:32:19.560
or unskilled. Marina found it to be a pretty sophisticated program.

00:32:19.560 --> 00:32:26.220
MARINA: Right, so the job was not an easy job. Triton is, as such, it’s a passive implant. Why I

00:32:26.220 --> 00:32:30.840
call it passive? Because it does nothing. It sits in the memory. Once you inject it in the memory,

00:32:30.840 --> 00:32:34.920
it sits in the memory and it expects a certain packet to be activated.

00:32:34.920 --> 00:32:39.720
JACK: This malware was very stealthy. As Marina said, it would implant itself into

00:32:39.720 --> 00:32:44.700
the memory. That is, volatile memory like RAM where the system would reboot and it would be

00:32:44.700 --> 00:32:49.860
gone. But these safety systems would often go over ten years without a reboot so hiding

00:32:49.860 --> 00:32:53.280
out in the memory was fine. [00:35:00] Now, once it was hidden in the memory,

00:32:53.280 --> 00:32:57.960
it was designed to act normal and engineers could interact with it just fine without knowing there

00:32:57.960 --> 00:33:03.000
were any problems with this thing. What’s more is that this malware had to rewrite the firmware in

00:33:03.000 --> 00:33:08.580
order to be successful and this was not possible to do remotely as a user accessing it through the

00:33:08.580 --> 00:33:13.500
engineering workstation. You typically needed to bring a flash drive to the system and then plug a

00:33:13.500 --> 00:33:17.820
console cable into it and upgrade the firmware while physically standing next to the system.

00:33:17.820 --> 00:33:23.460
But this malware found an unknown bug in the controller, a zero-day which allowed

00:33:23.460 --> 00:33:29.460
it to elevate its privileges to write into the firmware of this system. Again, for someone to

00:33:29.460 --> 00:33:34.320
have such an advanced knowledge of this particular safety controller running this particular version

00:33:34.320 --> 00:33:40.680
of software and to be able to craft a zero-day to exploit it, this is just top-level stuff. I mean,

00:33:40.680 --> 00:33:45.120
if you think about who could have made this, first of all, it had to be someone who had a lot of time

00:33:45.120 --> 00:33:50.040
because this attack took years to execute and it had to be someone who has a very high skillset

00:33:50.040 --> 00:33:56.160
who can hack both IT and OT environments. Then for them to develop this malware, which

00:33:56.160 --> 00:34:00.960
they probably had full, unrestricted access to these Triconex controllers in a lab or something

00:34:00.960 --> 00:34:07.260
so they could build this on and practice with. Basically, the attackers had unlimited resources

00:34:07.260 --> 00:34:13.620
to carry this attack out with. Okay, why would the attacker want to get into the safety system?

00:34:13.620 --> 00:34:22.560
MARINA: Exactly, and this is where we’re really getting into the large discussion also with the

00:34:22.560 --> 00:34:32.340
human cost of cyber-operations and ethics and so on. Safety systems are – even if the attacker

00:34:32.340 --> 00:34:40.140
would try to engineer a damage scenario and execute it to use it in the main control system

00:34:40.140 --> 00:34:49.860
like DCS, really bad consequences like explosion and toxic release will be always prevented by the

00:34:49.860 --> 00:34:56.460
safety systems. By targeting safety system and potentially preventing it from executing its

00:34:56.460 --> 00:35:05.760
function, the attacker would allow such terrible incidents like explosions and toxic releases.

00:35:05.760 --> 00:35:12.060
You would really have a cyber-attack with very dramatic physical consequences. But

00:35:12.060 --> 00:35:17.520
because people work in those plants and also even in the night, this may also

00:35:17.520 --> 00:35:23.580
result in casualties. You’re basically denying – because safety systems are meant to save life.

00:35:23.580 --> 00:35:30.900
This is the right of every employee to be in safe working conditions. They specifically

00:35:30.900 --> 00:35:40.320
target systems which prevent – protect civilian people. This is already off-limits. You should

00:35:40.320 --> 00:35:45.900
not be targeting those systems in the – when you do not even have war conditions. I’ve been

00:35:45.900 --> 00:35:53.700
working a lot with International Humanitarian – International Institute for Humanitarian Law

00:35:53.700 --> 00:35:59.940
and International Organization of Red Cross and all of these questions. You see like,

00:35:59.940 --> 00:36:06.540
yeah, targeting civilian protecting system is not permitted. It’s off-limits but currently

00:36:06.540 --> 00:36:15.600
these operations are not really specifically regulated. This is why it’s actually, yeah,

00:36:15.600 --> 00:36:20.700
encouraged more active discussions. How should we regulate site operation on the international

00:36:20.700 --> 00:36:28.640
level? Yes, it’s very upsetting because it’s – such an attack may result in human casualties.

00:36:28.640 --> 00:36:29.970
JACK: Wow.

00:36:29.970 --> 00:36:36.660
MARINA: But that means also really bad damage. The reason why I see why they would do that,

00:36:36.660 --> 00:36:42.660
once you want to take a specific refinery for a very long – like, take it down for a very

00:36:42.660 --> 00:36:49.560
prolonged time, you would go for such an attack. This would be really something very dramatic.

00:36:49.560 --> 00:36:52.980
But again, this is connected also with human casualties.

00:36:52.980 --> 00:36:59.160
JACK: Whoa. I can’t believe somebody would be insane enough to attempt something like

00:36:59.160 --> 00:37:06.480
this. [MUSIC] This is straight-up terrorism, cyber-terrorism. Now, while FireEye was

00:37:06.480 --> 00:37:10.440
investigating this to try to figure out what was the purpose of this attack and how it worked and

00:37:10.440 --> 00:37:15.300
who did it, word started to get out because at this point it’s months after the attack and many

00:37:15.300 --> 00:37:19.500
teams have been involved; there was the internal team and then the team Julian and Naser were on,

00:37:19.500 --> 00:37:23.160
and then the Schneider Electric team, and also there were other vendors onsite troubleshooting

00:37:23.160 --> 00:37:29.520
this, and now FireEye. Someone within all these teams started leaking information about this

00:37:29.520 --> 00:37:35.280
attack. First, somehow the US government became aware of this. The Department of Defense began

00:37:35.280 --> 00:37:39.600
tracking this but what also happened is that someone uploaded this malware to VirusTotal.

00:37:39.600 --> 00:37:44.700
VirusTotal is an amazing website; anyone can upload a file to it and when you do,

00:37:44.700 --> 00:37:49.800
it gets ran through like seventy different virus scans to see if it’s known malware

00:37:49.800 --> 00:37:55.380
and then tell you information [00:40:00] about that. Someone uploaded these files to VirusTotal

00:37:55.380 --> 00:38:00.720
and it just came back as unknown. This was probably a mistake for whoever uploaded it

00:38:00.720 --> 00:38:07.260
because when malware like this gets uploaded to VirusTotal, the premium users of the site get to

00:38:07.260 --> 00:38:13.020
see a copy of this malware. When it was uploaded there, it pretty much landed in the hands of all

00:38:13.020 --> 00:38:19.080
the premium users of the site. At that point, the world was not aware of this attack. But if

00:38:19.080 --> 00:38:26.520
whoever did this attack was a premium member of VirusTotal, now they knew their cover was blown.

00:38:26.520 --> 00:38:32.640
Another company comes into picture here; Dragos. They also investigate security threats related

00:38:32.640 --> 00:38:37.320
to industrial control systems. I sat down with their CEO to try to get to the bottom of this.

00:38:37.320 --> 00:38:40.980
ROBERT: My name is Robert Lee and I’m the CEO and co-founder over at Dragos.

00:38:40.980 --> 00:38:46.230
JACK: Now Rob, Rob used to work with the NSA before starting Dragos.

00:38:46.230 --> 00:38:53.040
ROBERT: That’s correct. I built and led the ICS Threat Discovery Mission access

00:38:53.040 --> 00:38:58.620
period. After that, they moved me into offensive operations. The United States government,

00:38:58.620 --> 00:39:05.340
they didn’t like that. They don’t really have a desire to do offense. I saw a gap in the

00:39:05.340 --> 00:39:09.660
private sector around industrial security and I saw this belief that was forming

00:39:09.660 --> 00:39:14.640
that was essentially taking IT security best practices and copy-and-pasting them into ICS,

00:39:14.640 --> 00:39:18.780
not actually thinking about the difference in mission, difference in threats, and similar.

00:39:18.780 --> 00:39:22.740
To be perfectly blunt with you, I would really like my son to have lights and water when he

00:39:22.740 --> 00:39:28.380
grows up so out of necessity of trying to get this right, I jumped ship and created Dragos.

00:39:28.380 --> 00:39:33.000
JACK: There’s a threat intelligence group within Dragos which is looking at what’s

00:39:33.000 --> 00:39:37.590
going on in the world to see what threats there are out there against industrial control systems.

00:39:37.590 --> 00:39:41.130
ROBERT: We ended up finding this malware.

00:39:41.130 --> 00:39:45.000
JACK: I don’t think they found this malware through VirusTotal but this is a company who

00:39:45.000 --> 00:39:49.740
has their finger on the pulse for threats related to industrial control systems. When

00:39:49.740 --> 00:39:54.240
something new like this shows up in the world, they’re probably going to find it pretty quick.

00:39:54.240 --> 00:39:57.240
ROBERT: When we found it, we had never heard of it before, we had never seen it before,

00:39:57.240 --> 00:40:02.040
we didn’t know about what had happened in Saudi Arabia at the time. We analyzed it and

00:40:02.040 --> 00:40:08.220
we started applying it to the set of intrusions that we were tracking. We made this assessment;

00:40:08.220 --> 00:40:11.820
yep, we have enough now that this is a real set that we would be tracking and

00:40:11.820 --> 00:40:15.450
here’s this SIS or safety system targeted malware and we’re gonna name it TRISIS.

00:40:15.450 --> 00:40:21.060
JACK: Yeah, okay, so they didn’t know FireEye had already named this malware Triton, so Dragos

00:40:21.060 --> 00:40:27.120
called this TRISIS. Just so you know, Triton and TRISIS are referring to the same malware.

00:40:27.120 --> 00:40:33.420
ROBERT: At that point we ended up feeling very uncomfortable about what we were looking at.

00:40:33.420 --> 00:40:38.460
[MUSIC] We knew very clearly, just from what we could assess and did malware analysis,

00:40:38.460 --> 00:40:43.200
that we were looking at an adversary that was either already deploying or

00:40:43.200 --> 00:40:48.900
going to be deploying malware to target safety systems and potentially compromise human life.

00:40:48.900 --> 00:40:53.160
JACK: Now, Rob is extremely experienced on the security of industrial control systems.

00:40:53.160 --> 00:40:57.720
I know this because I actually took a class with Rob at SANS once and he just blew my

00:40:57.720 --> 00:41:01.320
mind with his next level of understanding of things. He’s been involved with some of the

00:41:01.320 --> 00:41:05.520
world’s biggest industrial control system hacks ever. He was there for BlackEnergy,

00:41:05.520 --> 00:41:10.260
the attack on Ukraine’s power grid, and has responded to hundreds of serious incidents

00:41:10.260 --> 00:41:15.060
in industrial plants, utilities, dams, you name it. But as he was understanding what

00:41:15.060 --> 00:41:19.500
he was looking at with Triton, this hit him hard like nothing else has.

00:41:19.500 --> 00:41:25.800
ROBERT: To be extremely candid and transparent, I let out an audible ‘fuck’ and like, sat back in

00:41:25.800 --> 00:41:31.680
my chair, went and poured a glass of whiskey, sat there realizing that I had to draft this e-mail to

00:41:31.680 --> 00:41:35.880
the Department of Home Security understanding what could come after if it had went poorly.

00:41:35.880 --> 00:41:40.860
JACK: Rob has a history of working with the US government and feels like something like

00:41:40.860 --> 00:41:44.400
this is important enough to inform the Department of Homeland Security

00:41:44.400 --> 00:41:48.660
that hackers somewhere in the world have broken into a chemical plant in

00:41:48.660 --> 00:41:53.160
Saudi Arabia and had the capability to cause a major terrorist attack.

00:41:53.160 --> 00:41:57.960
ROBERT: Sitting there reading the report of the first ever SIS-targeted malware,

00:41:57.960 --> 00:42:03.360
the first time in human history that somebody tangibly went after human life from a cyber-attack

00:42:03.360 --> 00:42:09.900
and knowing what was gonna happen next, it’s a lot to take in because I also thought about it

00:42:09.900 --> 00:42:13.560
from the industry perspective; I thought about all of the conversations that were

00:42:13.560 --> 00:42:17.340
going to have to take place, the years of my life that I would then be talking about this

00:42:17.340 --> 00:42:21.480
and trying to educate groups and talking to engineering operations and security. Those

00:42:21.480 --> 00:42:26.040
are not fun – I think everyone thinks these are fun situations. These are not fun situations.

00:42:26.040 --> 00:42:32.940
JACK: Yeah, it does sound exciting to be part of this but Rob is right. This stuff can get dark and

00:42:32.940 --> 00:42:40.485
scary real quick and the burden it brings can really bring you down because it’s so intense.

00:42:40.485 --> 00:42:43.380
ROBERT: We don’t always tell governments about what we do. I think it’s very important for us

00:42:43.380 --> 00:42:47.340
to try to keep our customers out of the media and out of government panels a lot of the times. But

00:42:47.340 --> 00:42:54.300
we thought that this was so concerning that the US government [00:45:00] needed to know. I passed the

00:42:54.300 --> 00:43:02.100
information onto the Department of Home Security and said look, this is very, very significant.

00:43:02.100 --> 00:43:05.880
Little did I know, I think – I don’t think they leaked it. [MUSIC] Don’t get me wrong, I

00:43:05.880 --> 00:43:09.960
don’t think there’s any badness happening here but there’s a lot of contractors and

00:43:09.960 --> 00:43:16.560
people inside of DHS. One way or another, it made its way to FireEye. A FireEye executive

00:43:16.560 --> 00:43:21.600
ended up calling me up going hey, we see that you’re tracking this, we saw your analysis and

00:43:21.600 --> 00:43:27.660
stuff. That’s great and wonderful; FYI, we’re already involved. I was like oh, okay, cool.

00:43:27.660 --> 00:43:30.180
You want to partner together on this or analyze it?

00:43:30.180 --> 00:43:35.400
They said they couldn’t which makes sense, from NDAs and similar. I said okay, well, we’re not

00:43:35.400 --> 00:43:41.040
gonna publish on this. We’re gonna report it to our customers but whenever you guys publish it,

00:43:41.040 --> 00:43:46.860
let us know and we’ll publish our analysis as well. I think a lot of people view cyber-security

00:43:46.860 --> 00:43:49.980
teams to always be competitive but behind the scenes, a lot of your cyber-security

00:43:49.980 --> 00:43:57.360
companies work together for the benefit of the community ‘cause we all hate the adversary.

00:43:57.360 --> 00:44:04.200
Anyways, FireEye ended up going forward and deciding to publish this late December.

00:44:04.200 --> 00:44:09.540
We take a stance at our firm that we never publish about threats and their capabilities unless it’s

00:44:09.540 --> 00:44:13.500
already going to be made public because we want our customers and the community to have

00:44:13.500 --> 00:44:16.860
the information as much as possible ahead of the New York Times articles or similar.

00:44:16.860 --> 00:44:21.600
JACK: Okay, so back to FireEye. After all, FireEye had as close to a full picture as

00:44:21.600 --> 00:44:26.820
possible with all the extra data they collected. After analyzing the code and looking for clues and

00:44:26.820 --> 00:44:32.804
understanding its capabilities, they started to form an idea of who might be behind this.

00:44:32.804 --> 00:44:38.520
MARINA: [MUSIC] I think Iran was initially suspected by everybody because it was the

00:44:38.520 --> 00:44:46.560
logical target but it was quickly ruled out. I think FireEye has never confirmed it was Iran

00:44:46.560 --> 00:44:53.520
but in the mass media it was frequently speculated that it could be Iran because

00:44:53.520 --> 00:44:58.620
it was the logical target, but there was no evidence and FireEye did not confirm that. Yes,

00:44:58.620 --> 00:45:06.840
and then there was another report that – which FireEye has attributed

00:45:06.840 --> 00:45:15.180
activities to this National Research Institute of Mechanics – of Chemistry and Mechanics in Moscow.

00:45:15.180 --> 00:45:21.900
JACK: Oh, what? The Central Scientific Research Institute of Chemistry and Mechanics is suspected

00:45:21.900 --> 00:45:30.540
behind this? Let me look this up. Okay, so they’re based in Moscow, Russia but they literally seem

00:45:30.540 --> 00:45:35.580
to be a regular research institute publishing reports about thermal vision, gas dynamics,

00:45:35.580 --> 00:45:40.860
high-energy substances. In my opinion they don’t sound like a hacker group who would

00:45:40.860 --> 00:45:46.380
be intent on blowing up a chemical plant in Saudi Arabia. It just doesn’t make sense.

00:45:46.380 --> 00:45:56.340
But hm, wait a minute, do you remember Stuxnet, the hack against the nuclear enrichment facility

00:45:56.340 --> 00:46:03.780
in Iran? Do you remember where we think Stuxnet was created? In the Idaho National Lab or the

00:46:03.780 --> 00:46:09.480
Oakridge National Lab which are both ran by the Department of Energy and studies science and

00:46:09.480 --> 00:46:15.120
physics. I mean, the story goes is that somebody from the NSA or CIA went to these labs to find

00:46:15.120 --> 00:46:22.200
people who were skilled enough to develop an exploit for a centrifuge. Maybe someone went to

00:46:22.200 --> 00:46:28.500
this scientific institute in Moscow to get their help in developing the OT part of this attack.

00:46:28.500 --> 00:46:36.720
MARINA: It’s not really unusual that something what is built in the lab also has cyber

00:46:36.720 --> 00:46:44.520
capabilities. It sounds illogical, but it does. It’s just that previously, we have not really

00:46:44.520 --> 00:46:52.800
articulated this or never really looked into the practice of such research institutions in-depth.

00:46:52.800 --> 00:47:00.120
But yeah, it’s not a very unusual combination and they have a couple of departments which

00:47:00.120 --> 00:47:06.600
is related to the advanced informatics and security of critical infrastructure.

00:47:06.600 --> 00:47:14.280
JACK: What evidence is there to point that this research institute in Moscow may have done this?

00:47:14.280 --> 00:47:19.320
MARINA: Right, so, FireEye has laid down the facts pretty well,

00:47:19.320 --> 00:47:28.560
actually. This IP address which is – from which they observed intrusion being conducted or at

00:47:28.560 --> 00:47:33.780
least some operations related to intrusions of the Triton team, like, in a known organization

00:47:33.780 --> 00:47:38.580
were conducted from that IP address. [MUSIC] It would be known that the IP address was used to

00:47:38.580 --> 00:47:44.760
monitor the activity related to publications on Triton.

00:47:44.760 --> 00:47:50.100
JACK: I’ve also read in the FireEye report that the same IP of that research institute was doing

00:47:50.100 --> 00:47:55.920
reconnaissance on some [00:50:00] other plants and was seen engaging in other suspicious activity.

00:47:55.920 --> 00:48:05.880
MARINA: Also, a little bit funny; so, Nick Carr was really very vocal about this,

00:48:05.880 --> 00:48:15.780
Tweeting about this incident. In the library.zip there was – one of the files, like calculation

00:48:15.780 --> 00:48:23.700
of the DRC code, was written by Alexander Kotov, so they directly took that file and just used it.

00:48:23.700 --> 00:48:28.740
Then there was a block for but they – Alexander Kotov, he described how he needed to write this

00:48:28.740 --> 00:48:36.720
file and how developed it. Later on, when they found this Department of Advanced Informatic from

00:48:36.720 --> 00:48:44.760
this research institute, they have a group photo and there is – one of the members of this group

00:48:44.760 --> 00:48:51.360
looked like this Alexander Kotov. If they later hired him to work there

00:48:51.360 --> 00:48:57.000
and he posted these two pictures which was a tweet from October 24th, 2018,

00:48:57.000 --> 00:49:02.280
which if I look at the photos, it could be him. It’s just a fun fact.

00:49:02.280 --> 00:49:06.600
JACK: Of course, Russia has some very skilled hackers who work on behalf of the government,

00:49:06.600 --> 00:49:12.780
hackers within the FSB or GRU which are intelligence agencies in Russia.

00:49:12.780 --> 00:49:16.440
It’s possible that they might have been teaming up with this research

00:49:16.440 --> 00:49:21.540
institute which then makes this a multi-disciplinary attack. I mean,

00:49:21.540 --> 00:49:26.220
it makes sense that if one team got into the plant and got access to the engineering workstation,

00:49:26.220 --> 00:49:30.720
then the engineers from the research institute could take over the keyboard and go from there.

00:49:30.720 --> 00:49:35.700
MARINA: It seems like they didn’t have a proper infrastructure, attack infrastructure in place to

00:49:35.700 --> 00:49:44.100
make sure that the attribution will never be done, including this IP address. Which is, you see on

00:49:44.100 --> 00:49:50.580
one hand, it makes sense to move intrusion team to the engineers. On the other hand,

00:49:50.580 --> 00:49:55.740
you’re still better off to conduct an operation from the established governmental

00:49:55.740 --> 00:49:59.460
institutions because you have better attack infrastructure. Maybe they need to work on that.

00:49:59.460 --> 00:50:04.440
JACK: Good point; if it was this research team, they didn’t hide their tracks very well which

00:50:04.440 --> 00:50:08.880
is something a more seasoned government hacking group would have done better at.

00:50:08.880 --> 00:50:13.260
Now, once FireEye published their report on this, Dragos also published a report

00:50:13.260 --> 00:50:18.660
and in their report, they didn’t identify any specific group that did this. But instead,

00:50:18.660 --> 00:50:22.020
they created a name for the threat actor and called them Xenotime.

00:50:22.020 --> 00:50:28.380
ROBERT: When you look at what Xenotime was capable of doing, what they did, is they compromised this

00:50:28.380 --> 00:50:36.120
company back in 2014 and they beelined straight for the industrial networks. They compromised

00:50:36.120 --> 00:50:41.880
their SMS, two-factor authentication, they went directly into the industrial

00:50:41.880 --> 00:50:46.440
networks after compromising the company. After getting into the industrial networks they went

00:50:46.440 --> 00:50:50.760
and profiled, to the best of our knowledge, that safety system, and then they left. They

00:50:50.760 --> 00:50:56.806
didn’t come back until 2017 with a purpose-made capability on a highly-proprietary safety system.

00:50:56.806 --> 00:51:01.260
JACK: [MUSIC] Oh, wow. Okay yeah, so when the attackers have the capability to spend

00:51:01.260 --> 00:51:06.900
years fine-tuning their attack, this pretty much rules out any hacktivism groups simply because

00:51:06.900 --> 00:51:12.900
the sophistication here is just too high for some teenagers or a ragtag group of hackers to do. See,

00:51:12.900 --> 00:51:17.640
while trying to figure out who did it is impossible, we can take pretty good guesses

00:51:17.640 --> 00:51:23.580
at who didn’t do it and try to eliminate certain groups. Next, we can try to look at

00:51:23.580 --> 00:51:28.470
this attack through the lens of a cyber-criminal, someone who would be motivated by financial gain.

00:51:28.470 --> 00:51:31.740
ROBERT: Yeah, so one of the things we think about with cyber-crime and again,

00:51:31.740 --> 00:51:37.200
I don’t think it’s fair to ever eliminate fully, but one of the reasons chiefly that you would

00:51:37.200 --> 00:51:41.340
start to think it’s not cyber-criminal related regardless of this investigation and operation,

00:51:41.340 --> 00:51:45.720
is the impact and what were they trying to achieve. Usually, you think a lot about

00:51:45.720 --> 00:51:52.440
what’s the criminal aspect of this? There was no financial motivation, there was no

00:51:52.440 --> 00:51:57.000
intellectual property they were stealing that they could then sell off to somebody else,

00:51:57.000 --> 00:52:02.940
there was no return on investment to a criminal enterprise easily sussed out.

00:52:02.940 --> 00:52:07.320
You can always try to connect a million things or oh, they’re just shorting the oil markets or

00:52:07.320 --> 00:52:13.800
something. But straight away kind of analysis, there’s not a reliable assessment around this

00:52:13.800 --> 00:52:17.640
being criminal-related. As you look at this case, there’s not enough to support

00:52:17.640 --> 00:52:21.600
that it was hacktivism. There’s not enough to support that it was criminal-related. There’s not

00:52:21.600 --> 00:52:26.880
enough to support that it was a terrorist action or a non-state actor. The overwhelming support,

00:52:26.880 --> 00:52:31.740
the overwhelming evidence, classification to hypothesis would be a state actor.

00:52:31.740 --> 00:52:36.060
JACK: Okay, a state actor is a group of hackers who work on behalf of a

00:52:36.060 --> 00:52:39.060
government organization. When I think about state actors,

00:52:39.060 --> 00:52:43.860
the first group that comes to my mind is the NSA because they’re totally capable of

00:52:43.860 --> 00:52:48.600
pulling something like this off. That’s what NSA stands for, right, nation-state actor?

00:52:48.600 --> 00:52:54.960
ROBERT: This is a good question; would it be the [00:55:00] NSA? Which I think would

00:52:54.960 --> 00:53:01.980
fail all reason that a strong US ally like the NSA are going after it to cause physical events

00:53:01.980 --> 00:53:05.940
and try to kill people. It’s definitely not in anything that we’ve ever seen them do before.

00:53:05.940 --> 00:53:12.360
But let me talk about the attribution in general and my general thoughts on

00:53:12.360 --> 00:53:19.500
it. A number of folks at FireEye came out – a number of folks came out and have attributed this

00:53:19.500 --> 00:53:25.080
to the Russian government. I am not saying that these are incompetent folks,

00:53:25.080 --> 00:53:29.160
that their analysis is bad, or that they’re not supporting their assessments. [MUSIC] I’m not

00:53:29.160 --> 00:53:36.360
ever trying to dismiss other people’s assessments. My assessment of the situation, my knowledge of

00:53:36.360 --> 00:53:39.660
it and working with my intelligence team and some really wonderful professionals

00:53:39.660 --> 00:53:44.520
is that attribution is significantly more difficult than people make it out to be.

00:53:44.520 --> 00:53:49.620
It’s significantly easier to do than the naysayers would position; oh,

00:53:49.620 --> 00:53:54.360
you can’t get to attribution. Well, that’s not true either, but to get to a high-confidence level

00:53:54.360 --> 00:54:01.140
of attribution is incredibly difficult. My own biases from having worked in the National Security

00:54:01.140 --> 00:54:06.780
Agency with intelligence professionals is that a high-confidence level of attribution isn’t just

00:54:06.780 --> 00:54:11.880
related to the forensics and incident response and intrusion or tracking adversaries or doing OSINT.

00:54:11.880 --> 00:54:17.700
Hell, for us, high-confidence would have been, I’ve got screenshots of the person or I’ve got

00:54:17.700 --> 00:54:22.680
camera feeds and intrusion data and signals intelligence and maybe human intelligence.

00:54:22.680 --> 00:54:27.960
It’s so many components working together to get to a high-confidence level of assessment.

00:54:27.960 --> 00:54:32.700
A lot of the private sector high-confidence assessments I see really would have been

00:54:32.700 --> 00:54:38.220
low or moderate-confidence assessments in the government and I’ve never been able to break that.

00:54:38.220 --> 00:54:42.780
I don’t try to – again, I’m not trying to downplay anybody or similar, but when you’re

00:54:42.780 --> 00:54:48.240
talking about national critical infrastructure and cyber-attacks upon it, which is a really,

00:54:48.240 --> 00:54:54.360
really tense situation between state players, the last thing I want to do is have a – my firm,

00:54:54.360 --> 00:54:59.400
as an example, come out and go oh, we are basically positive that it’s Russia. I’m like wow,

00:54:59.400 --> 00:55:05.880
that’s gonna be used diplomatically, potentially militarily, that’s gonna feed into broader

00:55:05.880 --> 00:55:10.920
assessments. You gotta be real careful when you’re talking national disruption state tension.

00:55:10.920 --> 00:55:16.980
But the other reason we push back, well, there’s two other reasons that we push back; the first

00:55:16.980 --> 00:55:22.920
is that what most people want, not all, but what most intelligence requirements in the private

00:55:22.920 --> 00:55:29.760
sector relate to is how to do better security. How do I prioritize things? How do I look to better

00:55:29.760 --> 00:55:33.300
have security controls? What type of behaviors in the environments should I be detecting?

00:55:33.300 --> 00:55:39.360
What should my response plan be? None of those things require true attribution of ‘it was

00:55:39.360 --> 00:55:47.220
Vladimir in Russia.’ That’s not a valuable return on investment in trying to get the defensive

00:55:47.220 --> 00:55:54.660
recommendations. Our customers and largely our wider IT security community most of the time

00:55:54.660 --> 00:55:59.280
don’t care about attribution outside of a talking point to executives. Even then,

00:55:59.280 --> 00:56:03.660
it’s really just talking points. They’re not actually using that information but it’s a

00:56:03.660 --> 00:56:07.680
high cost to try to even get that information and I would argue you probably really can’t

00:56:07.680 --> 00:56:12.360
get high-confidence as often as you would like. Then the last thing, without being too wordy,

00:56:12.360 --> 00:56:16.980
but the last consideration around this and again, not trying to put anybody down,

00:56:16.980 --> 00:56:24.840
but we in InfoSec generally treat attribution as this binary thing; it was Russia or it

00:56:24.840 --> 00:56:30.540
wasn’t. It was China or it wasn’t. But these state players are not so black and white.

00:56:30.540 --> 00:56:37.800
[MUSIC] Russia has a variety of intelligence agencies and military agencies. When we say

00:56:37.800 --> 00:56:43.260
Russia, do we mean SBR? Do we mean GRU? What elements are we talking about? Inside of that,

00:56:43.260 --> 00:56:47.940
there’s the aspect that they have their own supply chain and non-state actors like our

00:56:47.940 --> 00:56:54.120
defense industrial base that they’re using. They might be having vendors of their own capabilities,

00:56:54.120 --> 00:57:00.900
maybe somebody making exploits for them. They have allies; Russia, China, North Korea or Iran teaming

00:57:00.900 --> 00:57:05.880
up at any given point on different operations just like we would do with the UK, Australia, and

00:57:05.880 --> 00:57:13.200
others. This discussion around attribution is way more nuanced at a geopolitical level than

00:57:13.200 --> 00:57:20.700
I generally see from a cyber-security audience. To just come out and go ‘it’s Russia’ I think is

00:57:20.700 --> 00:57:26.820
not a position that I could comfortably take because of what that means in impact, what little

00:57:26.820 --> 00:57:32.300
value it has to the customer, and how nuanced the real answer around that solution might be.

00:57:32.300 --> 00:57:39.240
JACK: Okay, but at the same time you’ve identified a group called Xenotime. How

00:57:39.240 --> 00:57:43.860
do you identify a group behind this without knowing who the group is?

00:57:43.860 --> 00:57:50.400
ROBERT: Yeah, great question. Clustering on intrusions to form a group;

00:57:50.400 --> 00:57:54.600
[01:00:00] diamond analysis, kill chain analysis, however you’re going to do it,

00:57:54.600 --> 00:58:00.840
is an effective tool to trapping an adversary and the methods and tools and infrastructure they

00:58:00.840 --> 00:58:06.360
used to make those defense recommendations. If you’re going to get to ‘it’s Russia’,

00:58:06.360 --> 00:58:10.680
you actually have to go through individual intrusions. You analyze an intrusion,

00:58:10.680 --> 00:58:15.840
you’re probably analyzing hundreds or thousands of pieces or elements of an intrusion,

00:58:15.840 --> 00:58:22.680
if not tens of thousands, to siphon it down to a set. Then once you have a set of intrusions

00:58:22.680 --> 00:58:27.300
and characteristics and similar, then you can start looking at victimology and infrastructure

00:58:27.300 --> 00:58:31.800
patterns and capability patterns and similar to then get to attribution.

00:58:31.800 --> 00:58:37.020
It’s actually not in the other way where you say ‘it’s Russia’ and you want me to follow them;

00:58:37.020 --> 00:58:43.200
you’re first actually creating sets of intrusions that you then follow. If you go and put the

00:58:43.200 --> 00:58:47.760
additional work into it you can try to make assessments around true attribution. You’re

00:58:47.760 --> 00:58:53.940
still doing attribution; you’re attributing this intrusion or this attack that you saw to

00:58:53.940 --> 00:58:59.640
a set but I’m not making the assessment about who that set is. I’m saying it’s this actor,

00:58:59.640 --> 00:59:03.600
this is Xenotime, we can tell that they’ve targeted the other entities,

00:59:03.600 --> 00:59:07.260
we can follow them, we can track them, we can learn from them. I’m just not going

00:59:07.260 --> 00:59:11.880
that additional step to put in the analysis, time, and resources to try to get to true attribution.

00:59:11.880 --> 00:59:16.200
JACK: One of the first lines in this – one of these reports that I’m reading

00:59:16.200 --> 00:59:21.660
on Dragos’s website is ‘Xenotime is easily the most dangerous threat activity publically known.’

00:59:21.660 --> 00:59:22.920
ROBERT: Yep.

00:59:22.920 --> 00:59:25.260
JACK: Can you kind of back that up?

00:59:25.260 --> 00:59:33.960
ROBERT: They’re the only threat publically that we know of that has shown both the intent and the

00:59:33.960 --> 00:59:41.700
capability to go after human life. I don’t think you can measure anything else other than that.

00:59:41.700 --> 00:59:45.120
I think it’s very fair to say there’s threats that have caused a lot of intellectual property loss,

00:59:45.120 --> 00:59:50.880
economic damage and similar, but there is nothing so sacred as human life and for an

00:59:50.880 --> 00:59:56.940
adversary to specifically intend and be capable of targeting that, that puts them in a special

00:59:56.940 --> 01:00:01.680
league of their own of a particularly dangerous and honestly awful threat.

01:00:01.680 --> 01:00:05.460
JACK: I mean, the next question I logically

01:00:05.460 --> 01:00:09.680
have is why would somebody want to actually kill people at this plant?

01:00:09.680 --> 01:00:15.060
ROBERT: There is a wide variety of motives that could go into it. I don’t want to

01:00:15.060 --> 01:00:18.960
speculate. I’ll give you some examples but it shouldn’t be seen as assessments;

01:00:18.960 --> 01:00:24.180
this is just speculation of what could happen. First and foremost,

01:00:24.180 --> 01:00:30.360
if you are a state actor that is competitive with the oil and gas industry in Saudi Arabia,

01:00:30.360 --> 01:00:37.500
which there are numerous, [MUSIC] the loss of life in those plants could not only have an

01:00:37.500 --> 01:00:42.600
immediate impact on production, it could have an immediate impact on morale of the workers

01:00:42.600 --> 01:00:46.920
and similar going back to those plants. It could have a public perception issue inside

01:00:46.920 --> 01:00:50.940
the kingdom that they have to deal with. But a lot of these companies are stock-owned and publically

01:00:50.940 --> 01:00:59.460
traded so you have impacts on actual wealth and capitalization and future operations and similar.

01:00:59.460 --> 01:01:06.540
What you’re basically doing is, with a single cyber-attack, you have an ability

01:01:06.540 --> 01:01:14.820
to help destabilize a strategic regional or non-regional adversary. If you are a

01:01:14.820 --> 01:01:21.420
state adversary that particularly doesn’t like Saudi Arabia or their wealth and oil and gas,

01:01:21.420 --> 01:01:26.700
this is a very effective attack to achieve especially ‘cause Saudi Aramco, even though

01:01:26.700 --> 01:01:32.040
they weren’t the victim, was getting ready to do their IPO at the time. They ended up delaying it.

01:01:32.040 --> 01:01:35.040
We don’t know if it was related to the attack or not that they delayed it, but they ended up

01:01:35.040 --> 01:01:42.000
delaying their IPO until later on. These types of attacks definitely make investors and others very,

01:01:42.000 --> 01:01:47.280
very concerned. The other aspect about it – I mean, there’s so many different motives.

01:01:47.280 --> 01:01:51.600
You could have a motive of simply using this attack, even though it wasn’t a training exercise,

01:01:51.600 --> 01:01:58.260
but using it as training too, for your own team on cool, can we go achieve these attacks? How

01:01:58.260 --> 01:02:02.340
could we make this scalable? What’s the next level of it? You have to get combat experience,

01:02:02.340 --> 01:02:07.500
if you will, not to overplay it, but you have to get experience as the adversary if you want

01:02:07.500 --> 01:02:14.220
to do these types of things. All reasonable analysis points to a state actor targeting

01:02:14.220 --> 01:02:21.120
Saudi Arabia to disrupt a portion of their oil and gas infrastructure. Why they did that is

01:02:21.120 --> 01:02:26.340
a very difficult intelligence requirement to have that really is inside the realm of

01:02:26.340 --> 01:02:30.480
state intelligence agencies, not something that a private sector intelligence agency

01:02:30.480 --> 01:02:35.640
could really reasonably get to. It’s like, a step beyond attribution is understanding why.

01:02:35.640 --> 01:02:43.320
JACK: Is this a story that we should be freaking out about? ‘Cause this could potentially target

01:02:43.320 --> 01:02:47.880
people in the US or places like that, and the whole infrastructure is like aah!

01:02:47.880 --> 01:02:56.040
ROBERT: Yeah, I [01:05:00] share people’s concern and I completely find it reasonable when people

01:02:56.040 --> 01:03:00.240
are concerned but I always try to downplay the hype of it. What’s the hype of it and what’s

01:03:00.240 --> 01:03:05.640
the reality? The hype of it would be to assume that this is some highly-scalable attack that

01:03:05.640 --> 01:03:10.320
immediately could target oil and gas companies or electric companies around the world like,

01:03:10.320 --> 01:03:15.720
all at the same time or similar. [MUSIC] The same way that attacks on an electric system aren’t

01:03:15.720 --> 01:03:21.300
hype, but thinking that there’s one grid that you could take down all at once is hype. On this one,

01:03:21.300 --> 01:03:26.040
how seriously do I take this? I take this so seriously that when I talk to the board

01:03:26.040 --> 01:03:30.720
of directors or talk to security teams in the oil and gas industry, this is one of the first things

01:03:30.720 --> 01:03:38.880
I highlight and I tell them very clearly, if you do not have detective, prevention, and responsive

01:03:38.880 --> 01:03:44.220
capabilities around the style of attack we’ve seen, not taking indicators of prices ‘cause the

01:03:44.220 --> 01:03:49.980
indicators will change, but the style, the TTPs, the behavior of the attack, if you’re not prepared

01:03:49.980 --> 01:03:55.800
to try to prevent that and respond to this, you are doing a disservice to your community.

01:03:55.800 --> 01:04:04.140
What I mean by that is this is the absolute best document case we’ve ever had of what really could

01:04:04.140 --> 01:04:10.920
happen from a cyber-attack to lose life in the community. If people aren’t taking that seriously

01:04:10.920 --> 01:04:16.560
in these industrial operations and industrial environments, I think they’re being negligent.

01:04:16.560 --> 01:04:22.800
Do I think the public should be freaking out about it? No. The work that I see out of these

01:04:22.800 --> 01:04:28.320
infrastructure companies is that so much work is happening that’s not public that they never

01:04:28.320 --> 01:04:32.340
get credit for. We commonly see oh, Electric Utility or whatever is not taking security

01:04:32.340 --> 01:04:36.480
seriously. That’s not true. There are some that aren’t and they need to do better for sure,

01:04:36.480 --> 01:04:42.660
but there is so much good work happening and you just don’t come out and publicise it,

01:04:42.660 --> 01:04:49.380
so we have to find a balance there. But does this attack and this adversary concern me? Absolutely.

01:04:49.380 --> 01:04:54.540
What really concerns me is these attacks and industrial control systems aren’t about the

01:04:54.540 --> 01:05:00.480
malware. It’s not about the vulnerability. It’s about a blueprint of how to go – achieve future

01:05:00.480 --> 01:05:05.820
attacks. You’re revealing knowledge and insight that other adversaries could pick up and use.

01:05:05.820 --> 01:05:12.480
This is how the realm of only state-adversary activity gets into non-state actor’s hands, is

01:05:12.480 --> 01:05:17.700
once a state actor figures out how to do it and publicise it, you get other people trying to do

01:05:17.700 --> 01:05:23.700
those things in the future. The butterfly effect here is that when people start doing

01:05:23.700 --> 01:05:29.100
these types of attacks, they start to become more common. They start to become easier and

01:05:29.100 --> 01:05:33.660
we want to prevent that because these are a particularly damaging style of attacks.

01:05:33.660 --> 01:05:40.080
JACK: Hm, for me at least, this whole attack puts me in deep thought. There are hundreds

01:05:40.080 --> 01:05:45.960
of industrial plants around Saudi Arabia and the world that have these same Triconex safety

01:05:45.960 --> 01:05:51.000
controllers. [MUSIC] It sounds like these hackers were in the network for years before accidentally

01:05:51.000 --> 01:05:57.240
tripping an alarm. It just makes me wonder how many other industrial networks might these

01:05:57.240 --> 01:06:04.980
attackers be in right now, lying in wait, waiting for the need to pull the trigger. It also makes

01:06:04.980 --> 01:06:10.620
me wonder how many other plants might have had a mysterious shutdown and didn’t have the capability

01:06:10.620 --> 01:06:17.760
or care to look deeper for this malware and instead they just started the plant back up.

01:06:17.760 --> 01:06:22.920
Spooky stuff. On one hand I want to know more but on the other hand, I’m kind of afraid to look.

01:06:22.920 --> 01:06:28.380
MARINA: Sometimes we have to let this go because it consumes you so much that

01:06:28.380 --> 01:06:32.640
yeah, sometimes you have to let it go and that’s exactly what I did with Triton. I don’t think

01:06:32.640 --> 01:06:38.760
about this anymore so I’m more concentrated right now working with, for example, Red Cross and with

01:06:38.760 --> 01:06:43.260
people who are involved in humanitarian law so that I’m there helping them with my

01:06:43.260 --> 01:06:47.460
technical knowledge, with my technical inputs to explain them the possible consequences of

01:06:47.460 --> 01:06:52.200
such attacks and cyber-operations in the critical infrastructures so that they could create better

01:06:52.200 --> 01:06:57.540
laws and regulations. How do you regulate such operations? Well, this is my main focus right now.

01:06:57.540 --> 01:07:02.040
ROBERT: Yeah, I think when we look at the attribution side of it where I will say the

01:07:02.040 --> 01:07:05.520
private sector may not need to go the distance and try to come up with a high-confidence

01:07:05.520 --> 01:07:11.880
assessment. I do think governments should. Is it important for clients of Dragos’s technology

01:07:11.880 --> 01:07:18.900
to know that Russia did this? No. But if Russia did do it, then the US and others

01:07:18.900 --> 01:07:24.600
do actually need to know that and it does need a way into discussion between states.

01:07:24.600 --> 01:07:30.660
It could lead the way into economic sanctions or others. This attack was a

01:07:30.660 --> 01:07:37.320
very purposeful and blatant attack against civilians and civilian infrastructure.

01:07:37.320 --> 01:07:43.320
State leaders around the world need to take this attack, attacks like Ukraine, the attack

01:07:43.320 --> 01:07:51.060
like NotPetya, and actually take these style of attacks off the table and penalize the states

01:07:51.060 --> 01:07:55.500
that [01:10:00] do these types of attacks. They should be inexcusable. Whereas on the

01:07:55.500 --> 01:07:59.100
attribution subject, I don’t want to go the distance because I don’t see the value in

01:07:59.100 --> 01:08:03.300
trying to pin it to any given state, the various intelligence agencies around the

01:08:03.300 --> 01:08:07.320
world need to and they need to get it right, and there needs to be action follow-through.

01:08:07.320 --> 01:08:10.680
JACK: I’ve seen the way our nation’s leadership interviews people like Mark

01:08:10.680 --> 01:08:14.220
Zuckerberg. Our leaders simply don’t understand technology enough to know

01:08:14.220 --> 01:08:19.200
what to do about this and it’s embarrassing. Technology defines our current time. There’s

01:08:19.200 --> 01:08:26.280
no excuse for our leaders to not understand technology more in-depth at this point.

01:08:26.280 --> 01:08:31.680
Maybe this was all just a test or practice since the attackers didn’t actually cause damage to the

01:08:31.680 --> 01:08:37.020
plant other than an accidental shutdown. Because I wonder about the people who were behind this;

01:08:37.020 --> 01:08:43.560
did they know this was a mission to kill people? Or were they told this is just a test and that no

01:08:43.560 --> 01:08:49.680
human lives would be lost during this test? When you look at the code long enough, the malware, you

01:08:49.680 --> 01:08:57.000
start to really think about that person who wrote it because it was a human who typed out that code.

01:08:57.000 --> 01:09:00.840
Marina thinks a lot about whatever person wrote this malware.

01:09:00.840 --> 01:09:09.480
MARINA: I spent so much time with these activities and because, you know,

01:09:09.480 --> 01:09:19.320
it’s very typical research – intensive research work that to which I can relate.

01:09:19.320 --> 01:09:24.600
I actually talked to many guys about that, everybody who was investigating the incident

01:09:24.600 --> 01:09:30.540
and spent a lot of time. [MUSIC] You start to really see the incident and can feel more the

01:09:30.540 --> 01:09:38.700
person; the pain, the frustration, that they sometimes also kind of want to see the person.

01:09:38.700 --> 01:09:43.500
Yeah, and I think it’s probably my personal opinion but probably even did not

01:09:43.500 --> 01:09:50.100
really – clearly understood the consequences of what exactly they are doing. Or maybe as you say,

01:09:50.100 --> 01:09:53.940
if it was just a test and they knew they’d never go in to disrupt anything,

01:09:53.940 --> 01:09:58.680
though they did not feel like they were doing really something dangerous because I would not

01:09:58.680 --> 01:10:04.620
be ever comfortable to conduct an operation which may impact human life of civilians.

01:10:04.620 --> 01:10:09.960
JACK: Yeah, there is a lot to think about regarding this incident. These kind of attacks

01:10:09.960 --> 01:10:15.720
on operational technology are slowly becoming more common. We’ve seen Stuxnet try to disable

01:10:15.720 --> 01:10:20.760
a nuclear enrichment facility and we’ve seen attacks on the Ukraine’s energy grid, and now

01:10:20.760 --> 01:10:29.040
we see Triton going after the emergency shutdown systems of a chemical plant. It’s chilling for

01:10:29.040 --> 01:11:02.340
sure. I just hope that whoever created this is not crazy enough to intentionally cause a disaster.

01:11:02.340 --> 01:12:27.000
JACK (OUTRO): [OUTRO MUSIC]

01:12:27.000 --> 01:12:31.620
A big thank you to our guests for coming on the show and sharing this story with us. Julian and

01:12:31.620 --> 01:12:36.240
Naser’s initial investigation was pivotal to everything that followed and both of them

01:12:36.240 --> 01:12:41.760
now work for Dragos with Rob. Marina Krotofil’s research and the team at FireEye was eye-opening

01:12:41.760 --> 01:12:46.440
to the world and Rob Lee’s report really does have an impact and hopefully saves lives in the

01:12:46.440 --> 01:12:51.960
future. Keep up the great work on helping us stay safe from major catastrophic events like this.

01:12:51.960 --> 01:12:57.300
This show was created by me, the crimson bear, Jack Rhysider. Original music created by the

01:12:57.300 --> 01:13:02.280
salty jackal Garrett Tiedemann, editing help this episode by the stardust kitten Damienne,

01:13:02.280 --> 01:13:06.600
and our theme music is by the sonic panda Breakmaster Cylinder.

01:13:06.600 --> 01:13:10.320
Even though when my dad has a computer problem and he calls me up to help him,

01:13:10.320 --> 01:13:15.000
I remind him about he used to nag on me to get off the computer when I was in high

01:13:15.000 --> 01:13:20.340
school and if I did, it wouldn’t be able to help him now, this is Darknet Diaries.