Strangers on a Train

“Mr. Trainman did (almost) all the things right, but …  something got terribly wrong”

This is a short story about how Industrial Control System Cybersecurity goes and the importance of real ICS Cybersecurity companies and professionals. A use case on the railway sector.

Landscape and main characters

Once upon a time, a National Railway company was building a new connection line between two main cities. Public tender for train provisioning on that new line containing Safety and Cybersecurity requirements for new equipment was published.

There was a train vendor manufacturing last generation trains all around the world. with leading edge technology on board equipment to add differential value, always complaining with different Safety and Cybersecurity regulations. Mr. Trainman as corporate CISO in front of these issues.

Nowadays, different networks and systems can be found on board in a train. one of those is CCTV (Inside and Outside):

For this project, they were looking for a unique vendor to provide both solutions (Inside and Outside) on CCTV system, to simplify project management and reduce complexity.

Then Moxa came in as they could provide both solutions, especially the outside CCTV needed for safety on persons going up and down on stops.

Having in mind all the Cybersecurity requirements, Mr trainman had to review that vendor selection. 

Mr. Trainman validation checklist

Being a regulated sector, all onboard equipment has to be complain with some railway safety standards:

But as a cybersecurity professional, Mr. Trainman also had some vendor requirements on the Cybersecurity posture, and he was looking for a true secure product development lifecycle certified organization:

With this on mind, Mr, Trainman began to check all these requirements on the selected vendor and product:

Good !!!!

Safety certifications and Ethernet protocols documented to place the equipment on the right IEC 62433 zone inside train network. Good !!!!

Some security controls available to protect the device:

• Password access protection
• IP Filtering
• HTTPS (Encryption) supported

Good !!!!!

Vendor and equipment approved and inside the new train to be delivered to customer.

Mr. Trainman met his stranger on a train

On May 2021, everything changed:

Four critical vulnerabilities on Outside Moxa IP Camera were published by NIST, and:

Vulnerabilities on product leads to a Denial of Service 

(CRITICAL for customer and safety) 

Firmware update MANDATORY on any already deployed camera

Vulnerabilities come from a Level 2 protocol (LLDP)

LLDP not identified by vendor on Spec Sheets

Level 2 filtering was not on Zoning Firewall rules

A complete revision on train IEC 62433 security Zoning needed ASAP

Now they were facing a huge economic impact on the project due to execution of all the tasks describes above.

Some final advices for Mr. Trainman

• Certification is good, but Not Enough
• Cybersecurity Certification Framework still to come on some sectors. Safety regulations are not security regulations. 

• Don’t trust, TEST because:
• Vendor spec sheets, and even technical manuals, don’t describe all the network ports and protocols products use.
• 0-Day vulnerabilities exists even if none had already tested that product, so do it yourself.
• Testing before deploying is cost saving and investment protection
• Always test your new products with an experienced Industrial Control System Cybersecurity Company

Dragonfly 2.0: Testing the Watering Hole risk

Three years after Dragonfly 1.0, now we are facing Dragonfly 2.0. In both campaigns, Watering Hole is one of the attack vectors described in reports, so protecting the website accessed by ICS engineers to get important resources from their daily work should be very important. It seems like ICS vendors and asset owners should have learned from that, and since I am very keen of testing things, I have decide to check that.
I designed a simple test to check the security posture from 13 big ICS vendors on communication security protection to their support web sites. (There is where the firmware upgrades used to be nowadays).

Mayor ICS Vendors I tested were:

• Siemens
• Schneider Electric
• Honeywell
• Rockwell Automation
• Yokogawa
• Moxa
• OSIsoft
• Phoenix Contact
• Advantech
• SEL
• ABB
• CODESYS
• MatrikonOPC

To make this checks, I used a simple but powerful tool called testssl.sh (that BTW, I recommend to you). This tool is able of testing, between other things, the following SSL vulnerabilities:

• Heartbleed (CVE-2014-0160)
• CCS (CVE-2014-0224)
• Secure Renegotiation (CVE-2009-3555)
• Secure Client-Initiated Renegotiation
• CRIME, TLS (CVE-2012-4929)
• BREACH (CVE-2013-3587)
• POODLE, SSL (CVE-2014-3566) 
• TLS_FALLBACK_SCSV (RFC 7507),
• FREAK (CVE-2015-0204)
• DROWN (2016-0800, CVE-2016-0703)
• LOGJAM (CVE-2015-4000)
• BEAST (CVE-2011-3389) 
• RC4 (CVE-2013-2566, CVE-2015-2808)

When using the tool against and SSL protected website, you can get "Vulnerable", "Probably" or "Not Vulnerable" results, that I associate with 2,1 and 0 values. That way, the most vulnerabilities the tool found for each SSL support web page, the higher risk I associate with the vendor.

These are the results for vulnerabilities:

First finding is that Advantech dosen't even have an SSL protected support Web page, So I associate the maximum risk value to it.

Second finding is that big ICS vendors were very similar in vulnerabilities, apart from ABB that scores much better with only one probably fixed vulnerability. Honeywell, on the other hand, shows the most potential problems to fix.

With these values on mind, I established another criteria to build a Heat Map (I love Heat Maps).
Seems logical thinking that this kind of risk is directly proportional with the use of the support site, so I searched for the Alexa Rank of those ICS support pages, and these are the values I found that day:

The lowest rank, the most accesses ...

Normalizing values from 1 to 5, I got the following Heat Map :

Seems like someone has some homework to do ....

Keep tune till the next revision.

Default passwords on control systems: SCADAPASS + DPE = Raise Awareness

When talking about responsible disclosure, everybody has an strong opinion. You can think publishing vulnerabilities is an irresponsible act, or you can think it is the only way of raise awareness on assets owners.
I am in the second category. As a control system Cybersecurity evangelist, I spend many time trying to show the hidden attack surface many organizations have, and the lack of visibility on the risks they are facing by managing their control systems totally apart from the Cybersecurity management framework other systems are in.
SAP servers, File servers, switches and firewalls on the corporate network have security policies, regular audits and security log and monitoring. Not doing that should be considered "irresponsible" and negligent.
The same corporation is using CCTV surveillance, access control systems, HVAC, fire detection system, lights and energy management systems and many other critical support systems, deployed by poorly security practices trained control integrators.  And now, both networks are connected. (And some times, Hyper-connected)

One of the worst (and common) practices you can find in control networks is installing by default. It means default usernames and passwords works in many devices inside the control network.

On December 2015, SCADA StrangeLove put in place a Default password publishing initiative, called SCADAPASS to rise awareness on control assets owners. Apart from your feelings or opinion on this, impact was huge in the ICS Cybersecurity world and it was included on Metasploit 4.11.6 .

Recently I remembered DPE project from ToolsWatch , and after getting familiar with dpeparser.py, I decided to integrate SCADA StrangeLove results on the XML db.
I wrote a quick-piggy script to convert CSV to DPE XML Schema and these are my results. All new entries are in the "sacda" type (-t scada).

As we love numbers, this is a summary from my work:

• About 47 new pure ICS vendors on DB (Siemens and Advantech were already there):
• 125 new elements (Without CPE, nor CVE since this is a harder process as you may know), but with their control role or description on the network. The list is too long to publish here.
• 203 new default usernames and passwords. Some devices has more than 50 default usernames !!!

I have contacted with the DPE development team, as I think it can be very useful to anyone in charge of assessing or managing control networks, but most important for assets owners of such networks.

New ICS DPE file can be downloaded here

Low cost SCADA Assessment: First internal approach

Best things in life are for free. That's what they say and sometimes I agree.

I read a post from EduHacker talking about Google fusion tables and it network graph functionality, so I decided give it a try.

I always try to quickly draw the traffic matrix from the network we are assessing since bandwidth between eyes and brain is the biggest you can get, and frankly most of the times I need it.

As ICS engineers we always use passive methods to assess through mirroring network traffic on switches. (By the way, this is a risky business since you have to be completely sure switch is not going to restart by configuring that).

Once you have your mirror port you can then plug your laptop and begin gathering IP traffic. It is amazing what you can get in a few minutes.

I took one ten minutes pcap file from one of our last OASyS SCADA assessments and follow the instructions from the post I read.

First of all you have to convert your pcap file to comma separated values (CSV) to import it in a Google fusion table.

To do this I made a quick and dirty csh script (I know, but I am a classic sir):

#!/bin/csh -f

###########################################################

#

# (c) 2015 - Kaostopper

#

# pcap_to_csv.csh: Quick and dirty way of getting a CSV file

#                  from a pcap file

#

#

# Enrique Martin Garcia

#

###########################################################

if ( $#argv != 1 ) then

  echo "Usaage: $0 pcap_file"

  exit 1

endif

set fichero_pcap = "$1"

set TSHARK = /usr/local/bin/tshark

echo "src_ip,src_port,dst_ip,dst_port,ip_proto"

$TSHARK -n -r $fichero_pcap -T fields -E separator=, -e ip.src -e tcp.srcport -e ip.dst -e tcp.dstport -e _ws.col.Protocol | grep -v ",," | sort -u

$TSHARK -n -r $fichero_pcap -T fields -E separator=, -e ip.src -e udp.srcport -e ip.dst -e udp.dstport -e _ws.col.Protocol | grep -v ",," | sort -u

exit 0

You can use any other way to convert your pcap files and you can choose any other fields, of course.

You have to be careful after uploading your CSV file to a fusion table since it is necessary to fix the type of value for your src_ip and dst_ip columns to text.

To do so, you have to click on the column header to access the change option:

You have to fix the type to text on those columns (src_ip and dst_ip)

With those two changes done, you can create your network graph:

As you can see, it was easy to detect a big misconfiguration in the Operation Workstation that was trying to connect to Google DNS servers. (8.8.8.8 and 8.8.4.4). This came from a temporary movement of this station due to civil works on the control room, but .....

You can play a lot with the other different types of graph, but this one is very easy to get and you don't need any other tool.

Beware with the data you upload and do like me: I have changed everything on the CSV file except public Google addresses and the SCADA System name.

No more excuses to update a very simple but useful network diagram for your SCADA Network.

Cyber physical attacks to critical infrastructure (Part I: Critical Infrastructure Systems domain)

Is it just another commercial trick from ICS Cyber Security sector or something to take care about ?

From the Aurora experiment cyber attack on a power generator in 2007, which was intended to demonstrate the ability to produce physical damage to assets remotely, to this day, this type of attack has materialized twice. (As far as we know).

The first cyber-physical attack in history detected, documented and widely known in the field of industrial cyber security professionals was STUXNET (2010), which marked the beginning of the development of this discipline and most standards for critical infrastructure, as it demonstrated the enormous destructive power of malware aimed at the destruction of the uranium enrichment centrifuges that Iran would use in its production of nuclear weapons.

The second cyber attack with physical consequences occurred recently (end of 2014) in a German steel plant, in which a cyber attack triggered after accessing control network from the business network, did not allow a graceful shutdown for a blast furnace, although the details and effects thereof have not been studied with the same detail as in the case of STUXNET.

In 2015 the interest on such attacks focus in altering the physical behavior of the environment through cyber attacks has increased through experiments carried out on cars, medical instruments and numerous automation devices connected to the Internet.

In the last Black Hat 2015 USA and DEF CON 23, there were very interesting presentations about these Cyber Physical Attacks from Marina Krotofil and Jason Larsen describing many of them.

This post, and the next ones, studies the higher impact (and therefore riskier) attacks on cyber-physical systems in critical infrastructure control networks and propose protection by making some changes on organizations structures and procedures and new technologies of intrusion detection based on analysis behavior of control protocols and correlation of operational events.

Critical sector and critical infrastructure

To put into context the domain to protect from such attacks, we describe the characteristics considered critical infrastructure in Europe and in Spain.

In January 2009 it came into effect Directive 2008/114 / EC of the Council of the European Union which established the need to identify Europe's critical infrastructures in order to design strategies to protect them.

In this Directive the need to identify infrastructure sectors of energy and transport, leaving open the possibility that all member states identify additional critical sectors.

As of December 2014 the European Agency for Network Security and Information Agency (ENISA) published a guide for the identification of critical assets.

This guide showed critical sectors  already identified by the member countries of the Union and can be seen in the following table:

Spain has identified twelve critical sectors:

• Energy (With three subsectors: Electricity, Oil and Gas)
• Nuclear
• Economics (Finance and Tax Administration)
• Water
• Transportation (With three subsectors: air, sea and land)
• Food
• Information Technologies and Communications
• Chemical
• Health
• Space
• Public administration
• Research

In each of these sectors they have been appointed or will be appointed in the near future, a set of Critical Operators (OC), which are those owners or operators of infrastructures which provide essential services and whose attack could lead to damages broad sectors of the population. This set of infrastructures will shape our domain to protect and share a number of common technical characteristics.

Technical characteristics. 

Many classified as critical infrastructure have a hybrid architecture in which there are networks of classical information technology (IT Network) and industrial control networks (OT Network) managing the elements that interact with the physical environment (cyber-physical systems). A basic scheme of this type of infrastructure could be the following:

  

Cyber-physical systems control a particular process and are managed by network systems, operate according to the following basic scheme:

The sensors measure the current process values on fixed intervals and send them to the control units assessing the need to seek concrete solutions to the actuators orders that the process remains within the values for which it was created and behave according to the original design.

Today all this traffic control has been migrating to TCP networks and conventional operating systems, which has made no earlier existing attack surfaces appear.

The key characteristics of the OT networks can be summarized as follows:

• Less number of devices and services than IT networks. 
• They should never be directly connected to Internet. 
• Execute repetitive operations between its nodes and systems.
• Very sensitive to delays or communication problems.

But these classes of networks also have strong weakness as:

• Use insecure or unauthenticated protocols. 
• Often not segmented logically or physically. 
• No possibility of installing third party software on some systems. 
• No possibility of patching or update certain systems

These features and constraints make protection of such critical networks very special and, as discussed below, using specific strategies and technologies for this type of environment.

In the next post we will dive on that.

Cyber physical attacks to critical infrastructure (Part II: Attack and defense)

Although the number and nature of cyber attacks on control systems that could have effects on the physical environment is very broad, we will only consider those that have been studied by various stakeholders in the industrial Cyber Security.

In particular, recent studies define the following categories depending on the purpose of physical cyber attack:

• Damage to equipment 
• Damage to Production 
• Deterioration of compliance

Let’s see each of them in detail:

Damage to equipment

Such cyber attacks are intended to produce permanent failures and breakdowns industrial equipment interacting with the physical environment. In particular, attacks have been studied on the following elements:

• Pipes and pipelines: The valve opening and closing quickly, and sometimes coordinated, is capable of causing a physical phenomenon called "water hammer" consisting of an increase in pressure inside the pipe can be higher than the structural strength thereof, causing breakage and subsequent discharge of fluid (liquid or gas) to drive.
• Tanks: In many cases Tanks are designed to withstand very high internal pressures, but at very low internal pressure (or vacuum), collapse. Sudden changes in the temperature inside a tank can lead to abrupt changes in internal pressure, which could eventually collapse it.
• Generators: As demonstrated in the Aurora experiment, opening and closing off phase switches from a generator connected to an electrical substation produce kinetic effects that just physically breaking it.
• Engines: Stuxnet cyber attack in the last phase tended to accelerate the engines of uranium centrifuges for long periods of time causing material fatigue and subsequent failure.
• Chemical Reactors: The most common chemical reactions typically occur at high temperatures, so a change in the conditions of reaction control may be associated with a significant increase in temperature would cause thermal damage to the reactor structure, reaching its total destruction.

It is also possible to combine two or more of these attacks each other, so that the power loss is associated with a loss of control of some element or its inlet in an unstable operating condition.

Although we have been considering these as attacks, there are historical examples on great industrial accidents caused by abnormal functions in control systems.

The following points will demonstrate how these detection technologies can help on detecting some operational failures that could lead to serious industrial accidents as well.

Damage to Production

The purpose of this type of cyber attacks to the process is altering the financial results of the organization that operates such processes. Among them they have been studied the following:

• Decrease the amount of final product: By changing certain variables control the process at specific points, you can alter the amount of product obtained. A clear example of this is built on the production of vinyl acetate monomer [7] Black Hat in Las Vegas in August 2015.
• Decrease in product purity: If the alterations made to the process control variables do change the purity of the final product, you can produce a significant devaluation of the same. A concrete example is the Paracetamol, whose purity can alter the price by several orders of magnitude.
• Increase in operating and maintenance costs: Cyber ​​attacks can cause alarm processes intentionally to force recalibration of the field elements as often as desired attackers, thereby increasing the costs of the targeted organization. Moreover, repeated attacks on processes with different values ​​is one of the most common practices of hiding them, because that way the suspicions maintenance teams move the organization.

Deterioration of compliance

Legal and regulatory frameworks to be met by organizations, makes certain commitments made by them can have very significant penalties for breach thereof. Among this kind of commitment we can find the following:

• Safety regulations: Altering a security parameter of the industrial plant may entail a violation of any rules of physical security which in turn is liable to a major fine if inspection.
• Impact on the environment: discharges into rivers or waste production values of certain compounds above the permissible threshold are punished with significant financial penalties.
• Contractual breaches: The purity or quantity alteration of the product can make certain clauses of the contracts do not meet preventing accorded billing and causing significant economic losses to the organization.

All these cyber attacks studied in the past year, have a number of common characteristics:

• Semantic attacks: They are necessary depth knowledge of the environment, the process and the variables to be altered to produce the desired effects.
• Targeted to the control network: Using "legitimate” users and systems, over unauthenticated control protocols and "valid" commands, and executed with appropriate permissions.
• Conducted by multidisciplinary teams: Composed by an IT team (Network and Systems), an OT team (SCADA) and process engineers (of the attacked sector)
• In view of the nature of cyber-physical and processes attacks, and the above on the technical characteristics of the control networks, critical infrastructure protection presents a number of problems that can only be addressed using the solutions that describe the next point.

It might seem that this type of attack is too complicated or exceptional to take into account in our risk analysis, but do not forget that:

1. They are targeted attacks intended to cause physical damage and could be executed or sponsored by state organizations.
2. Already they materialized before and were not mere theoretical laboratory studies.
3. In both cases the cyber attack had an external source to the facilities attacked even when isolated from the Internet is assumed. (The average number of connections found in control networks assessment is 11)
4. The success of these attacks could endanger human lives.
5. The PIC 8/2011 of Critical Infrastructure Protection Act explicitly mentions the need to consider in the risk assessment of this type of infrastructure events of very high impact, such as the case of these attacks.

Another common thinking when suppressing these cyber-physical attacks from risk analysis could be considering them covered by safety plans. As showed in the Mogford report after the Texas City refinery accident, there was a lack of preventative maintenance on safety critical systems. So once again, we can not relay on initial conditions to establish the actual security state of infrastructure, we need to assess it on a periodic basis.

Critical Infrastructure Protection

Cyber security is founded on three pillars: people, procedures and technologies. In this case it cannot be otherwise, so these sections formulate a series of recommendations to protect such infrastructure from cyber-physical attacks seen before.

People

As we saw earlier in this note such cyber attacks can only materialize through joint action of experts in different fields (IT Technology, OT technology and Industrial process to attack). It is necessary for critical infrastructure have multidisciplinary teams in their Cyber Security organizations working in a coordinated way in order to protect them.

This is one of the most common problems encountered in implementing the CIP law because the existing inertia in many organizations  the world of control and security have always been in different functional areas and with different officials and budgets.

The awareness of senior management of the infrastructure operator is required to make critical changes needed in the functional organizations to ensure a unique multidisciplinary team responsible for this Cyber Security.

Procedures

It is a priority to establish changes in the procurement procedures of the critical infrastructure operators requiring the inclusion of Cyber security requirements for solutions in automation and control, just as there are for safety on plants. Deploying controls and countermeasures in the control networks without this approach in design will be much more difficult and expensive  

Given the semantic nature of these attacks is necessary expand risk analysis for contemplating processes attacks. As seen above this is only possible with the participation of process control engineers in this activity where Cyber security and safety come to converge. (Hazard / Risk Analysis).

Technologies

For everything mentioned above, the security measures to be taken in such environments must take into account the importance of availability in such control networks. Any measure to be implemented should be as safe as possible in terms of the impact on the process to protect. According to the Department of Industrial CERT Homeland Security, the impact of the various protection technologies to consider when deploying in such networks is as follows:

  

As can be seen, intrusion detection systems are the technology with less impact on industrial control networks.

Within this technology, and considering the significant limitations that exist for installing third party software on the control systems (SCADA Servers, engineering work stations and operating positions or HMI) is indicated selecting NIDS technology (network Intrusion Detection System) since modification of the existing network architecture or reconfiguring any of the systems won’t be necessary.