Navigating the maze of Cyber Security Intelligence and Analytics

Quite recently, Mint reported that intelligence agencies of the three nations – India, UK and US – did not pull together all the strands gathered by their high-tech surveillance tools, which might have allowed them to disrupt the 2008 terror strike in Mumbai. Would data mining of the trove of raw information, along the lines of US National Security Agency’s PRISM electronic surveillance program, have helped in connecting the dots for the bigger picture to emerge? As leaked by Edward Snowden, NSA has been operating PRISM since 2007 to look for patterns across a wide range of data sources, spanning multiple gateways. PRISM’s Big Data framework aggregates structured and unstructured data including phone call records, voice and video chat sessions, photographs, emails, documents, financial transactions, internet searches, social media exchanges and smartphone logs, to gain real-time insights that could help avert security threats.

In the business world, information compiled in a relatively recent Verizon Data Breach Investigations Report points out that data is stolen within hours in 60% of breaches, but goes undetected for months in 54% of overall breaches. With the mounting pace of attacks, and increasing IT attack surface due to constant influx of new technologies such as cloud computing, BYOD and virtualization, CISOs are looking for real-time data analytics to improve threat defense through better controls, reliably detect an incident and quickly contain the breach before it inflicts an inordinate amount of damage, and also provide insight into extent of data exfiltration to quantify damages and potentially remediate the situation, without aggravating security staff shortages already faced by most organizations.

Figure – Timespan of Breach Detection (Source: Verizon)

The era of big data security analytics is already upon us with large organizations collecting and processing terabytes of internal and external security information. Internet businesses such as Amazon, Google, Netflix and Facebook who have been experimenting with analytics since early 2000s might contend that Big Data is just an old technology in a new disguise. In the past, ability to acquire and deploy high-performance computing systems was limited to large organizations with dire scaling needs. However, technological advances in the last decade – which have rapidly decreased the cost of compute and storage, increased the flexibility and cost-effectiveness of data centers and cloud computing for elastic computation and storage, and development of new Big Data frameworks which allow users to take advantage of distributed computing systems storing large quantities of data through flexible parallel processing – and major catalysts such as data growth and longer retention needs have made it attractive and imperative for many different types of organizations to invest in Big Data Analytics.

How do traditional Security Analysis tools fare?

Over the decades, security vendors have progressively innovated on multiple aspects to – (1) protect endpoints, networks, data centers, databases, content and other assets, (2) provide risk and vulnerability management that meet governance requirements, (3) ensure policy compliance with SOX, PCI, HIPAA, GLBA, DSS and other regulations, (4) offer identity/access/device management, while aiming (5) to provide a unified security management solution that allows for configuration of its security products, and offers visibility into Enterprise security through its reporting capability. While most individual security technologies have matured to the point of commoditization, security analysis remains a clumsy affair that requires many different tools (even in SIEM deployments, that I will elaborate on below), most of which do not interoperate due to the piecemeal approach of security innovation and multi-vendor security deployments.

Today, security analysts often rely on tools such as Log Management solutions and Security Information and Event Management (SIEM) systems for network security event monitoring, user monitoring and compliance reporting – both of which focus primarily on the collection, aggregation and analysis of real-time logs from network devices, operating systems, and applications. These tools allow for parsing and search of log data for trends, anomalies and other relevant information for forensics, though SIEMs are deemed to be more effective for forensics given its event reduction capability.

Log Management solutions from few prominent vendors are AlienVault USM, AlertLogic Log Manager, McAfee Enterprise Log Manager, LogRhythm, HP ArcSight ESM, Splunk, SolarWinds Log & Event Manager, Tenable Log Correlation Engine, ElasticSearch ELK, SawMill, Assuria Log Manager, BlackStratus Log Storm and EiQNetworks SecureVue. You’ll find a more  complete list along with details on these offerings here.

Let me now elaborate on how Log Management & SIEM solutions differ, and point out few misgivings of SIEM solutions (apart from its price, of course).

While a SIEM solution is a specialized tool for information security, it is certainly not a subset of Log Management. Beyond log management, SIEMs use correlation for real-time analysis through event reduction, prioritization, and real-time alerting by providing specific workflows to address security breaches as they occur. Another key feature of SIEM is the incorporation of non-event based data, such as vulnerability scanning reports, for correlation analysis. Let me decipher these features unique to SIEMs.

• Quite plainly, correlation is to look for common attributes and link events together into meaningful categories. Data correlation of real-time and historical events allows for identification of meaningful security events, among massive amount of raw event data with context information about users, assets, threats and vulnerabilities. Multiple forms of event correlation are available – e.g. a known threat described by correlation rules, abnormal behavior in case of deviation from baseline, statistical anomalies, and advanced correlation algorithms such as case-based reasoning, graph-based reasoning and cluster analysis for predictive analytics. In the simplest case, rules in SIEM are represented as rule based reasoning (RBR) and contain a set of conditions, triggers, counters and an action script. The effectiveness of SIEM systems can vary widely based on the correlation methods that are supported. With SIEMs – unlike in IDSs – it is possible to specify general description of symptoms and use baseline statistics to monitor deviations from common behavior of systems and traffic.
• Prioritization involves highlighting important security events over less critical ones based on correlation rules, or through inputs from vulnerability scanning reports which identify assets with known vulnerabilities given their software version and configuration parameters. With information about any vulnerability and asset severity, SIEMs can identify the vulnerability that has been exploited in case of abnormal behavior, and prioritize incidents in accordance with their severity, to reduce false positives.
• Alerting involves automated analysis of correlated events and production of alerts, based on configured event thresholds for incident management. This is usually seen as the primary task of a SIEM solution that differentiates it from a plain Log Management solution.

The below figure captures the complete list of SIEM capabilities.

Figure – SIEM Capabilities (Source: ManageEngine.com)

SIEM solutions available in the market are IBM Security’s QRadar, HP ArcSight, McAfee ESM, Splunk Enterprise, EMC RSA Security Analytics, NetIQ Sentinel, AlientVault USM, SolarWinds LEM, Tenable Network Security and Open Source SIM which goes by the name of OSSIM. Here is probably the best source of SIEM solutions out in the market, and how they rank.

Real-world challenges with SIEMs

• SIEMs provide the framework for analyzing data, but they do not provide the intelligence to analyze that data sensibly and detect or prevent threats in real-time. The intelligence has to be fed to the system by human operators in the form of correlation rules.
• The correlation rules look for a sequence of events based on static rule definitions. There is no dynamic rule generation based on current conditions. So, it takes immense effort to make and keep it useful. The need for qualified experts, who can configure and update these systems, increases the cost of its maintenance even if we were to assume that such expertise is widely available in the industry.
• SIEMs throw up a lot of false notifications when correlation rules are used initially, which prompt customers to even disable these detection mechanisms. SIEMs need to run advanced correlation algorithms using well-written specific correlation rules to reduce false positives. The sophistication of these algorithms also determines the capability to detect zero-day threats, which hasn’t been the forte of SIEM solutions in the market.

In spite of these challenges, a market does exist for these traditional tools, as both SIEM and Log Management play an important role in addressing organizational governance and compliance requirements, related to data retention for extended periods or incident reporting. So, these solutions are here to stay, and be in demand atleast by firms in specific industry verticals and business functions. Also, SIEM vendors have been exploring adjacent product categories – the ones I will cover below – to tackle the needs of the ever-so dynamic threat landscape.

Is Security Analytics a panacea for security woes?

With the current threat landscape increasingly featuring Advanced Persistent Threats (APTs), security professionals need intelligence-driven tools that are highly automated to discover meaningful patterns and deliver the highest level of security. And there we have all the key ingredients that make up a Security Analytics solution!

First off, Security Analytics (SA) is not a replacement for traditional security controls such as NGFW, IDS, IPS, AV, AMP, DLP or even SIEMs and other existing solutions, but can make them work better, by reconfiguring security controls based on SA outcome. For all of it to come together and a deeper truth on potential threats and current intrusions to emerge through Security Analytics, threat intelligence needs to be fused from multiple internal and external sources. Various levels of Indicators of Compromise (IOCs) need to be understood to capture earlier neglected artifacts and correlate behavior to detect potential threats and zero-day attacks, as traditional signature based approaches are no longer sufficient.

While use of Big Data technologies might be fundamental to any implementation of predictive analytics based on data science, it is not a sufficient condition to attain nirvana in the Security Analytics world. After wading through Big Data material, I’ve decided to keep it out of this discussion, as it doesn’t help define the essence of SA. One could as well scale up a SIEM or Log Management solution in a Big Data framework.

Now that you have a rough idea about the scope of this blog post, let me deep dive into various SA technical approaches, introduce the concepts of Threat Intelligence, IOCs and related standardization efforts, and finally present a Security Analytics architectural framework that could tie all of these together.

How have vendors implemented Security Analytics?

With Gartner yet to acknowledge that a market exists for Security Analytics – its consultants seem to be good with Threat Intelligence Management Platforms (TIMP) – I chose to be guided by the list of ‘Companies Mentioned’ here by ResearchAndMarkets, and additionally explore dominant security players (Palo Alto Networks, Fortinet) who were missing in this list, to check if there is anything promising cooking in this space. And, here is what I found. Less than a handful of these vendors have ventured into predictive analytics and self-learning systems, using advanced data science. Most others in the market offer enhanced SIEM and unified cyber forensic solutions – that can consume threat intelligence feeds and/or operate on L2-L7 captures – as a Security Analytics package, though it is certainly a step forward. The intent of this section is to offer a glimpse into technology developments in the Security Analytics space. I doubt that my research was exhaustive [as I mostly found product pitches online that promise the world to potential undiscerning customers], and would be glad to make the picture a little more complete, if you could educate me on what is going on in your firm in this space, if I’ve missed any game changer!

All SA products consume and manage data in some manner, be they log data, file data, full network packet captures, and/or behavioral data from systems or networks, by integrating with other security devices and solutions in the deployment. However, there must be intelligence in the system that helps render those data into something useful. Vendors have implemented this intelligence into SA via deep inspection of L2-L7 exchanges, anomaly sensors to spot behavior deviations, diamond model of intrusion analysis, game theory and/or advanced machine learning algorithms, potentially based on Campaigns which is a “collection of data, intelligence, incidents, events or attacks that all share some common criteria and can be used to track the attacker or attack tactic” [sic] [I’m going to wait for this concept to evolve and become commonplace, before I remark on this]. I’ve used sample vendor offerings to elaborate on each of these SA implementation approaches. I’ve come across categorization of SA offerings as data analyzers vs. behavior analyzers, but I view these as building blocks for SA and not alternative solutions.

Deep data inspection – Solera Networks’ DeepSee platform (now acquired by BlueCoat) offers an advanced cyber forensics solution that reconstructs and analyzes L2-L7 data streams (captured by integrating with industry standard and leading security vendor offerings for routers, switches, next-gen firewalls, IPSs, SIEMs et al) for application classification and metadata extraction, indexed storage and analysis to provide accurate visibility into activities, applications and personas on the network. It is built on the premise that full visibility, context and content are critical to react to security incidents in real-time or back-in-time. I scoured the net to understand how it detects a breach (beyond what a scaled-up SIEM can do with similar stream captures and batch data stores) and thus goes beyond Cyber Forensics, but didn’t find any material. CyberTap Security’s (acquired by IBM) Recon boasts of a similar solution with ability to reassemble captured network data to its original form, be they documents, web pages, pictures, mails, chat sessions, VOIP sessions or social media exchanges.

Behavioral anomaly sensors – With APT attacks consisting of multiple stages – intrusion, command-and-control communication, lateral movement, data exfiltration, cover tracks and persist [below figure has further details on APT lifecycle] – each action by the attacker provides an opportunity to detect behavioral deviations from the norm. Correlating these seemingly independent events can reveal evidence of the intrusion, exposing stealthy attacks that could not be identified through other methods. These detectors of behavioral deviations are referred to as “anomaly sensors,” with each sensor examining one aspect of the host’s or user’s activities within an enterprise’s network. Interset and PFP Cybersecurity are among the limited vendors who’ve built threat detection systems based on behavioral analytics, as reported here.

Figure – Lifecycle of Advanced Persistent Threats (Source: Sophos)

Cognitive Security (research firm funded by US Army, Navy and Air Force – now acquired by Cisco) stands out as it relies on advanced statistical modeling and machine learning to independently identify new threats, continuously learn what it sees, and adapt over time. This is a good solution to deep dive into, to understand how far any vendor has ventured into Security Analytics. It offers a suite of solutions that offer protection through the use of Network Behavior Analysis (NBA) and multi-stage Anomaly Detection (AD) methodology by implementing Cooperative Adaptive Mechanism for Network Protection (CAMNEP) algorithm for trust modeling and reputation handling. It employs Game Theory principles to ensure that hackers cannot predict or manipulate the system’s outcome, and compares current data with historical assessments called trust models to maintain a highly sensitive and low false positive detection engine. This platform utilizes standard NetFlow/IPFIX data and is said to deploy algorithms such as MINDS, Xu et al., Volume prediction, Entropy prediction and TAPS. It does not require supplementary information such as application data or user content and so ensures user data privacy and data protection throughout the security monitoring process. To reiterate, this is a passive self-monitoring and self-adapting system that complements existing security infrastructure. Cylance, among SINET’s Top 16 emerging Cybersecurity companies of 2014 as reported here, seems to have also made some headway in this artificial intelligence based SA approach.

Zipping past A-to-Z of Threat Intelligence

Definition – Here is how Gartner defines Cyber Threat Intelligence (CTI) or Threat Intelligence (TI) – “Evidence-based knowledge, including context, mechanisms, indicators, implications and actionable advice about an existing or emerging menace or hazard to assets that can be used to inform decisions regarding the subject’s response to that menace or hazard.”

How does TI help? – Clearly, TI is just not indicators, and can be gathered by human analysts or automated systems, and from internal or external sources. Such TI could help identify misuse of any corporate assets (say as botnets), detect data exfiltration and prevent leakage of further sensitive information, spot compromised systems that communicate with C2 (i.e. Command-and-Control aka CnC) servers hosted by malicious actors, detect targeted and persistent threats missed by other defenses and ultimately initiate remediation steps, or most optimistically – stop the attacker in his tracks.

How to gather TI? – A reliable source of TI is one’s own network – information from security tools such as firewalls, IPS et al, network monitoring and forensics tools, malware analysis through sandboxing and other detailed manual investigations of actual attacks. Additionally, it can be gleaned through server and client honeypots, spam and phishing email traps, monitoring hacker forums and social networks, Tor usage monitoring, crawling for malware and exploit code, open collaboration with research communities and within the industry for historical information and prediction based on known vulnerabilities.

TI external sources could either be (a) open-source or commercial, and (b) service or feed providers. These providers could also differ on how they glean raw security data or threat intelligence as categorized below:

• Those who have a large installed base of security or networking tools and can collect data directly from customers, anonymize it, and deliver it as threat intelligence based on real attack data. E.g. Blue Coat, McAfee Threat Intelligence, Symantec Deepsight, Dell SecureWorks, Palo Alto Wildfire, AlienVault OTX
• Those who rely heavily on network monitoring to understand attack data. These providers have access to monitoring tools that sit directly on the largest internet backbones or in the busiest data centers, and so they are able to see a wide range of attack data as it flows from source to destination. E.g. Verisign iDefense, Norse IPViking/Darklist, Verizon
• Few intelligence providers focus on the adversary, track what different attack groups are doing and closely monitor their campaigns and infrastructure. This type of intelligence can be invaluable because adversary focused intelligence can be proactive, knowing that a group is about to launch an attack allows their customers to prepare before the attack is launched. E.g. of these TI service providers who focus on manual intelligence gathering by employing human security experts are iSIGHT Partners, FireEye Mandiant, CrowdStrike
• Open source intelligence providers who typically crowd source. The best open source TI providers typically focus on a given threat type or malware family. E.g. Abuse.ch which tracks C2 servers for Zeus, SpyEye and Palevo malware while combining domain name blocklists. Other best open sources of TI are Blocklist.de, Emerging Threats, Spamhaus. ThreatStream OPTIC intelligence is community vetted but not open source.
• Academic/research communities such as Information Sharing and Analysis Centers (ISACs), Research and Education Networking (REN) ISAC, Defense Industrial Base Collaborative Information Sharing Environment (DCSIE)

Other TI sources of manual/cloud feeds include – malware data from VirusTotal, Malwr.com, VirusShare.com, ThreatExpert; National Vulnerability Database; Tor which provides a list of Tor node IP addresses; and others such as OSINT, SANS, CVEs, CWEs, OSVDB, OpenDNS. Few other commercial vendors include Vorstack, CyberUnited, Team Cymru and Recorded Future.

Indicators of Compromise – The Basics

Indicators of Compromise aka IOC denote any forensic artifact or remnant of an intrusion that can be identified on a host or network, with the term ‘artifact’ in the definition allowing for observational error. Indicators could take the form of – IP addresses of C2 servers, domain names, URLs, registry settings, email addresses, HTTP user agent, file mutex, file hashes, compile times, file size, name, path locations etc. Different types of indicators can be combined together in one IOC [as illustrated in the below figure].

Figure – Indicators of Compromise aka IOC (Source: Mandiant.com)

The below pyramid stacks up the various indicators one can use to detect an adversary’s activities and how much effort it would take for the adversary to pivot and continue with the planned attack, when indicators at each of these levels are denied.

Figure – Pyramid of Pain with IOCs (Image Source: AlienVault)

Starting at the base of the pyramid where the adversary’s pain is the lowest if detected and denied, we have Hash Values such as SHA1 or MD5 which are often used to uniquely identify specific malwares or malicious files involved in an intrusion. The adversary could potentially change an insignificant bit and cause a different hash to be generated, thus making our earlier detected hash IOC ineffective, unless we move to fuzzy hashes.

Next up in the pyramid are IP addresses. These again might not take long for the adversaries to recover from, as they can change the IP address with little effort. If they were to use an anonymous proxy service like Tor, then this indicator has no effect on the adversary. In comparison, Domain Names are slightly harder to change than IP addresses as they must be registered and visible in the Internet, but still doable though it might take a day or two for any adversary.

Looking at it from an IoC usage perspective in security deployments, the TTL of an IP address can be very low. Compromised hosts in legitimate networks could get patched, illicitly acquired hosting space might be turned off, malicious hosts are quickly identified and blocked, or the traffic might be black holed by the ISP. An IP address may have a TTL of 2 weeks, while domains and file hashes would have significantly longer TTLs.

Typical examples of Network Artifacts are URI patterns, C2 information embedded in network protocols, distinctive HTTP User-Agent or SMTP Mailer values. Host Artifacts could be registry keys or values known to be created by specific pieces of malware, files or directories dropped in certain places or using certain names, names or descriptions or malicious services or almost anything else that is distinctive. Detecting an attack using network/host artifacts can have some negative impact on the adversary, as it requires them to expend effort and identify which artifact has revealed their approach, fix and relaunch it.

Further up in the pyramid, we have Tools which would include utilities designed – say to create malicious documents for spearphishing, backdoors used to establish C2 communication, or password crackers and other host-based utilities they might want to use post their successful intrusion. Some examples of tool indicators include AV or YARA signatures, network aware tools with a distinctive communication protocol and fuzzy hashes. If the tool used by adversaries has been detected and the hole has been plugged, they have to find or create a new tool for the same purpose which halts their stride.

When we detect and respond to Tactics, Techniques and Procedures (TTPs), we operate at the level of the adversaries’s behavior and tendencies. By denying them any TTP, we force them to do the most time consuming thing possible – learn new behaviors. To quote a couple of examples – Spearphishing with a trojaned PDF file or with a link to a malicious .SCR file disguised as a ZIP, and dumping cached authentication credentials and reusing them in Pass-the-Hash attacks are TTPs.

There are a variety of ways of representing indicators – e.g. YARA signatures are usually used for identifying malicious executables, and Snort is used for identifying suspicious signatures in network traffic. Usually these formats specify not only ways to describe basic notions but also logical combinations using boolean operators. YARA is an open source tool used to create free form signatures that can be used to tie indicators to actors, and allows security analysts to go beyond the simple indicators of IP addresses, domains and file hashes. YARA also helps identify commands generated by the C2 infrastructure.

Sharing IOCs across organizational boundaries will provide access to actionable security information that is often peer group or industry relevant, support an intelligence driven security model in organizations, and force threat actors to change infrastructure more frequently and potentially slow them down.

Exchanging Threat Intelligence – Standards & Tools

Effective use of CTI is crucial to defend against malicious actors and thus important to ensure an organization’s security. To gain real value from this intelligence, it has to be delivered and used fairly quickly if not in real-time, as it has a finite shelf life with threat actors migrating to new attack resources and methods on an ongoing basis. In the last couple of years, there has been increased effort to enable CTI management and sharing within trusted communities, through standards for encoding and transporting CTI.

Generally, indicator based intelligence includes IP addresses, domains, URLs and file hashes. These are delivered as individual black lists or aggregated reports via emails or online portals, which are then manually examined and fed by analysts into the recipient organization’s security infrastructure. In certain cases, scripts are written to bring in data from VirusTotal and other OSINT platforms directly into heuristic network monitors such as Bro.

Let me touch upon various CTI sharing standards – OpenIOC, Mitre package (CybOX, STIX, TAXII), MILE package (IODEF, IODEF-SCI, RID) and VERIS – that are aimed at doing away with the above TI sharing inefficiencies.

OpenIOC is an open source, extensible and machine-digestible format to store IOC definitions as XML schema and share threat information within or across organizations. This standard provides the richest set of technical terms (over 500) for defining indicators and allows for nested logical structures, but is focused on tactical CTI. The standard was introduced and primarily used in Mandiant products, but can be extended by other organizations by creating and hosting an Indicator Term Document. There has been limited commercial adoption outside of Mandiant, with McAfee among the minority vendors with products that can consume OpenIOC files. MANDIANT IOC Editor, Mandiant IOC Finder and Redline are tools that can be used to work with OpenIOC.

Mitre has developed three standards that are designed to work together and enable CTI sharing – Cyber Observable eXpression (CybOX), Structured Threat Information Expression (STIX) and Trusted Automated eXchange of Indicator Information (TAXII). With STIX being accepted by industry leaders, STIX and TAXII are starting to see wide adoption. Common Attack Pattern Enumeration and Classification (CAPEC) and Malware Attribute Enumeration and Characterization (MAEC) are focused on attack patterns and malware analysis respectively.

Figure – Threat Intelligence Formats in Mitre Family (Source: Bit9.com)

• CybOX provides the ability to automate sharing of security intelligence by defining 70 objects (e.g. file, mutex, HTTP session, network flow) that can be used to define measurable events or stateful properties (e.g. file hashes, IPs, HTTP GET, registry keys). Objects defined in CybOX can be used in higher level schemas like STIX. While OpenIOC can effectively represent only CybOX objects, CybOX also understands the notion of events which enables it to specify event order or elapsed time, and bring in the notion of behaviors.
• STIX was designed to additionally provide context for the threat being defined through observable patterns, and thus covers the full range of cyber threat information that can be shared. It uses XML to define threat related constructs such as campaign, exploit target, incident, indicator, threat actor and TTP. In addition, extensions have been defined with other standards such as TLP, OpenIOC, Snort and YARA. The structured nature of the STIX architecture allows it to define relationship between constructs. E.g. the TTP used can be related to a specific threat actor.
• TAXII provides a transport mechanism to exchange CTI in a secure and automated manner, through its support for confidentiality, integrity and attribution. It uses XML and HTTP for message content and transport, and allows for custom formats and protocols. It supports multiple sharing models including variations of hub-and-spoke or peer-to-peer, and push or pull methods for CTI transfer.

Managed Incident Lightweight Exchange (MILE), an IETF group, works on the data format to define indicators and incidents, and on standards for exchanging data. This group has defined a package of standards for CTI which includes Incident Object Description and Exchange Format (IODEF), IODEF for Structured Cyber Security Information (IODEF-SCI), and Real-time Inter-network Defense (RID) which is used for communicating CTI over HTTP/TLS. IODEF is an XML based standard used to share incident information by Computer Security Incident Response Teams (CSIRTs) and has seen some commercial adoption e.g. from HP ArcSight. IODEF-SCI is an extension to the IODEF standard that adds support for attack pattern, platform information, vulnerability, weakness, countermeasure instruction, computer event log, and severity.

Vocabulary for Event Recording and Incident Sharing (VERIS) VERIS framework from Verizon has been designed for sharing strategic information and an aggregate view of incidents, but is not considered to be a good fit for sharing tactical data.

Many vendors and open source communities have launched platforms to share TI. e.g. AlienVault’s Open Threat Exchange (OTX), Collective Intelligence Framework (CIF), IID’s ActiveTrust. OTX is a publicly available sharing service of TI gleaned from OSSIM and AlienVault deployments. CIF is a client/server system for sharing TI which is internally stored in IODEF format, and provides feeds or allows searches via CLI and RESTFUL APIs. CIF is capable of exporting CTI for specific security tools. IID ActiveTrust platform is leveraged by government agencies and enterprises to confidently exchange TI and coordinate responses between organizations.

Unifying Security Intelligence & Analytics – The OpenSOC framework

So, how do organizations use Threat Intelligence that I’ve talked about at length? With Threat Intelligence coming in from a variety of sources and in multiple formats (even if each of these are standardized), a new solution being floated in the market is the Threat Intelligence Management platform (TIMP) or Threat Management Platform (TMP) – which has been tasked to parse incoming intelligence and translate it into formats as understood by various security solutions (e.g. malware IPs into NIDS signatures, email subjects into DLP rules, file hashes into ETDR/EPP/AV rules, Snort rules for IPS, block lists and watch lists for SIEMs/AS, signatures for AV/AM etc.), to make it suitable for dissemination. TI can also be uploaded into a SIEM for monitoring, correlation and alerting, or to augment any analysis with additional context data.

Now that I’ve tied up one dangling thread, what about Security Analytics? Having zoomed in early on in this blog post on what any SA operates on and how its output could better security controls in a deployment, I’ll provide a 30,000-40,000ft view (at cruising altitude?) this time around, by introducing the OpenSOC, an unified data-driven security platform that combines data ingestion, storage and analytics.

Figure – OpenSOC framework

The OpenSOC (Open Security Operations Center) framework provides the building blocks for Security Analytics to (1) capture, store, normalize and link various internal security data in real-time for forensics and remediation, (2) enrich, relate, validate and contextualize earlier processed data with threat intelligence and geolocation to create situational awareness and discover new threats in a timely manner, and (3) provide contextual real-time alerts, advanced search capabilities and full packet extraction tools, for a security engine that implements Predictive Modeling and Interactive Analytics.

The key to increasing the ability to detect, respond and contain targeted attacks is a workflow and set of tools that allows threat information to be communicated across the enterprise at machine speed. With OpenSOC being an open source solution, any organization can customize the sources and amount of security telemetry information to be ingested from within or outside the enterprise, and also add incident detection tools to suit its tailored Incident Management and Response workflow.

I’ve treaded on uncertain ground in navigating the product-market for Security Analytics, given that it is still nascent and the product category isn’t well delineated. Would welcome any views on this post, be they validating or contradicting mine.

How is the SDN landscape shaping up? (Part-3) – A security perspective

In a fast-changing technology landscape, security is a moving target as the sophisticated attacker unearths and takes advantage of new weaknesses. Cybercrime has moved beyond mere vandalism and emerged as a professional industry focused on exploit kit innovations and planned attacks, given its lucrativeness and with some countries around the world involved in extensive cyber-espionage. The global hackers’ economy is pegged conservatively at around $450B, while Gartner estimates the worldwide security market for 2014 to be just about $71B.

In the December 2013 breach of retail giant Target, data from as many as 40M credit cards and 70M user accounts were hijacked, while Home Depot data breach between April and September 2014 affected 56M debit and credit cards, data of which is said to have appeared within days on black-market sites. Per Checkpoint security report 2014, 88 percent of analyzed organizations experienced at least one potential data breach event in 2013, up from 54 percent in 2012. Verizon’s data breach investigation report identified nine major patterns that covered over 90% of security incidents in the past decade, with the dominant pattern varying across industry verticals. [Refer bottom-right of the following chart for a listing of these categories.] In the recent years, dangers of targeted attacks and advanced persistent threats (APTs) have garnered much of the attention in the information security world.

Figure – Data Breaches in organizations (Source: Verizon)

How can SDN improve security?

With the increase in network traffic, virtual machines, cloud computing, and malware threats, IT manpower is a major security bottleneck, as they simply can’t keep up with the increasing demand of sorting through incidents/alerts and fine-tuning security controls based upon latest threats. This situation can be expected to exacerbate as IoT applications gain momentum. The only way to bridge this growing response-capability gap is through intelligent incident detection and automated response.

While automated security is a key driver, the excitement with SDN enabled security is more so around the opportunity for intelligent response on a granular basis – be it on per flow, per application or per user basis – to provide Security-as-a-Service, while eliminating manual configuration errors and keeping down SecOps & NetOps staffing costs. No longer would the default system response to any severe security threat be ‘Fully Block’.

Enterprise SecOps teams are interested in using SDN to enable multiple usecases – to selectively block malicious traffic from endpoints while still allowing normal traffic flows, to centralize security policy and configuration management, and for network security policy auditing and conflict detection/resolution.

With deperimeterization, corporate networks no longer have a single point of entry or exit, resulting in the demand for network telemetry and flow-based security solutions. SDN could be used to act on any anomalies detected on flow capture data, by dynamically establishing rules to divert specific flows to either a centralized or distributed enforcement points. Additionally, SDN could be used for traffic engineering to direct discrete network application flows to specific combination of security services such as FWs, IDS/IPS and WAFs.

SDN’s capability to programmatically and dynamically insert customized security services in the network path, via service chaining of physical and virtual L2-L4 routers/switches/appliances [as illustrated in below figure], could help minimize performance impact of ‘bump in the wire’ security devices, while enhancing security by acting on gleaned threat intelligence.

Figure – SDN enabled network and application security (Source: devcentral.f5.com)

While SDN controller limits its visibility and programmatic focus to underlying physical/virtual network elements and thus directly implements network security, application layer security is provided in this architecture by working in tandem with orchestration components to control other L4-L7 data path elements.

In a nutshell, SDN can help improve security by aligning the right security service with the right flows.

Common Security Usecases of SDN

Having gone over SDN security benefits in the previous section, let’s now deep dive into a couple of well-defined SDN security usecases – DDoS mitigation, Advanced Malware Quarantine and Elephant Flow mitigation.

DDoS Mitigation

Distributed Denial of Service (DDoS) attacks that are typically launched from compromised systems (bots) bombard enterprise applications with fake service requests, with the intent of denying or degrading performance of legitimate service requests. In addition to hogging the application servers, such attacks consume SP network capacity and saturate inline security devices. Addressing DDoS attacks requires detection of fake requests and diverting traffic to a specialized cleaning device to remove fake packets from the traffic stream, and sending back legitimate packets to the enterprise application/service.

While DDoS defense can be implemented through Remotely Triggered Black Hole (RTBH) or Policy based routing (PBR), the former solution implements filters that block all traffic to the target application, thus making it unavailable for legitimate services too. On the other hand, while static PBR based solution provides granular flow control, it requires SPs to add manual filter configuration on their backbone/edge routers and thus prolongs the recovery time from such attacks.

In comparison, BGP FlowSpec which aligns with the SDN paradigm follows a more granular approach and automates distribution of filter lists that can match a particular flow via BGP NLRI to the backbone/edge routers. So, policy updates happen dynamically and only a specific flow traffic (based on L3/L4 information) is stopped instead of dropping all traffic to a server, as illustrated in the below figure. SDN can thus boost security by dynamically reprogramming and restructuring a network that is suffering a distributed denial-of-service attack.

Figure – DDoS Mitigation using BGP FlowSpec (Source: Alcatel-Lucent)

Automated Malware Quarantine (AMQ)

SDN can also offer security capabilities such as automatically quarantining an endpoint or network that has been infected with malware.

In the non-SDN architecture, AMQ is typically deployed as a proprietary standalone solution where each device performs its specified function autonomously, with limited awareness of other devices in the network. This closed approach is suitable only for static traffic flows, and inflexible in data center environments where server workloads are virtualized, traffic flows are highly dynamic and multiple simultaneous flows must be maintained.

In a SDN controller driven network, if detailed forensics by network security devices generate a high enough score to initiate a quarantine directive, the SDN controller translates this into a set of OpenFlow rules that is pushed down to OpenFlow enabled switches to cut off the infected host from the production network and display (via the Web Proxy Notifier in the below figure) a web page with corrective actions to be performed. Once the corrective actions are performed, the rules are changed to allow the end host back into the network. Such automated reconfigurations through SDN reduce the response time to security threats, while allowing user mobility at the network edge. Given that this AMQ implementation does not require any additional software or hardware support beyond OpenFlow enabled devices, this is a vendor agnostic solution and ripe for deployment.

Figure – Advanced Malware Quarantine using SDN (Source: ONF)

Elephant flow detection/mitigation

Short-lived flows referred to as mice tend to be bursty and are often latency sensitive. In contrast, long-lived flows termed as elephants normally perform transfers of large blocks of data and are typically packet latency insensitive. Without intelligent traffic engineering, elephant flows may fill up network pipes causing latency and/or service disruption of mice flows.

SDN applications could help select a marking action and instruct the SDN controller to push the action to OpenFlow routers/switches to assign a selected queue for traffic associated with elephant flows. Other applications include offloading elephant flows from L2/L3 network fabric to the optical circuit switched network i.e. a pure photonic layer 1 network, for better performance and scalability at lower cost. In an enterprise environment, elephant flows observed during network backup of a filesystem could be prevented from taxing the firewall and slowing down its performance without compromising on security, by implementing a firewall service bypass function dynamically via SDN, based on whitelisted flow parameters of “good” permitted elephant flows. The permitted elephant flows are now dynamically re-directed so that they are sent directly in/out of the campus and the firewall is no longer in the forwarding path.

Security-centric SDN offerings

While OpenFlow is a key technology enabler of SDN security usecases as we saw in the previous section, few firms offer enhanced SDN security solutions that go beyond OpenFlow.

Illumio – a promising startup with a veteran executive team from Cisco, McAfee, Nicira, Riverbed and VMware – is based on the idea that each workload must have its own defenses. Illumio’s Adaptive Security Platform (ASP) provides visibility and enforcement services to dynamically keep pace with the motion, change and automation of the underlying IT infrastructure and applications, by attaching fine-grained security at the level of individual workloads while being continuously aware of its context. [Refer below figure to understand workload context.] It allows enterprises to secure applications across private, public or hybrid cloud environments on any physical server or VM. Illumio has 25 customers including Morgan Stanley, Plantronics, Creative Artists Agency, UBS, Yahoo and NTT I³.

Figure – Workload context (Source: Illumio)

Illumio ASP provides two modes of operation: (1) illumination mode which security administrators can use to visualize workloads and traffic flows, perform policy analyses, assess security gaps and potential impacts to application flows, and even discover unused workloads. (2) enforcement mode which lets administrators write security policies using natural language to describe desired communications between workloads. Once the policies are enforced, workloads are locked down to interact only with the specifically allowed paths.

As illustrated in the below figure, Illumio ASP is architected as a distributed and asynchronous system with Virtual Enforcement Nodes (VENs) attached to individual workloads (running on any VM, physical server, or private/public/hybrid cloud) and a centralized policy engine (PCE). The VENs listen continuously for the context of their workload and relay the information to the PCE, which computes enforcement rules by combining the workload’s context with configured security policies. The enforcement policies are then sent to the VEN, which modifies the appropriate iptables or Windows Filtering Platform parameters on the workload.

Figure – Security-centric SDN – Illumio ASP architecture (Source: Illumio)

NetOptics, a provider of application and network visibility and monitoring solutions, offers Security-centric SDN by combining an SDN controller with Network Packet Brokers (NPBs) in an architecture that allows for intelligent orchestration of the customer’s existing security appliances and solutions. NPBs embody dynamic attack monitoring, ability to chain solutions and distribute traffic, while the SDN controller is used to assess the network and adapt the network behavior based on threats detected through security enforcement elements, to either divert suspicious traffic, change security devices’ responses, or block packets altogether, all with minimal-to-no human intervention.

Radware’s Defense4All, the first open SDN security application to be integrated into OpenDaylight, offers carriers and cloud providers DoS and DDoS detection and mitigation as a native network service. The application uses the programmability of SDN to collect statistics, analyze information and control the infrastructure layer to proactively defend against network flood attacks. This allows operators to provide DoS/DDoS protection service per virtual network segment or per customer.

In today’s dynamic cloud data centers, assets and their network configurations are subject to change yet compliance lacks automation and continues to rely on manual processes. Catbird, the leader in security policy automation and enforcement for private clouds and virtual infrastructure, offers CatBird vSecurity to bring automation and agility of the cloud to security automation and enforcement of compliance standards such as PCI, HIPAA and FISMA. By integrating with Cisco ACI, Catbird provides IT teams a highly standardized, automated approach to provisioning security as part of the network policy, through the asset lifecycle from asset inception to teardown.

Most vendors including Juniper (Altor Networks acquisition rebranded as Firefly Perimeter) and Cisco offer SDN controllers that seamlessly integrate with virtual and physical routers/switches/security devices to enable dynamic provisioning and service chaining of security services. Among the vendors that have already announced OpenDaylight Helium-based products is Brocade. Helium release has multiple security capabilities including Secure Network Bootstrapping Infrastructure (SNBI) and AAA. Alongside, we have OpenFlowSec, a community focused on developing OF security applications, kernel extensions and application frameworks, to assist OpenFlow practitioners in enhancing security capabilities of OpenFlow.

Is SDN secure enough, to offer security?

We’ve seen that good potential exists to enhance security through SDN, and there are ongoing industry efforts to help realize it. But, is SDN inherently secure enough, to augment network and application security? Or is SDN security an oxymoron as skeptics put it? Let me explore this aspect to try and calm down the naysayers to SDN from a security perspective, before I wind up this post.

Here are the top security concerns of the SDN architecture –

1. Single point of failure, given the central function of the SDN controller
2. Wide span of impact, when the network is opened up to applications
3. Need for secure communication between controller and end nodes/applications, to stem MITM attacks

Single Point of Failure

Because the control plane and thereby the SDN controller play such a central function in the SDN architecture, security strategies must focus on protecting the control plane to thwart any targeted attacks on the controller – be they to saturate the control plane or attempts to leverage policy configuration errors for infiltration and lateral movement – as access to the controller could potentially give complete control of the network to an attacker.

It is vital to secure the SDN controller by carefully designing access control policies, manage authorization, track and audit usage of the controller. Also, where it resides on the network is a big security concern. This concern of having a king that needs to be protected are being addressed in variants of the SDN architecture through High Availability in the controller, SSL communication between controller and network elements, extension to the OpenFlow data plane called connection migration, which dramatically reduces the amount of data-to-control-plane interactions that arise during such attacks, SIEM to log everything that comes out of the system, analytics to correlate logs from SIEM and alert the manager of any changes.

Wide span of impact

In contrast to pre-SDN one-by-one configuration process, where an error might only affect one part of the network, SDN now makes it easy to have one policy applied uniformly and in an automated way. However, opening up the network to its own applications require their own security policy framework, governance and management to be put in place.

Business applications are vulnerable to potential threats because of the powerful SDN programming model. Multiple SDN network services may interfere with one another, compromising the forwarding behavior of the network and such conflicts must be avoided. Security policies may be compromised at the controller, at one or more network devices, and/or at other places. As a result, security policies must be validated, along with the network configuration and behavior and performance.

While the upside is that SDN will demand clear policies, companies will need to spend time thinking about and designing those policies. These can be expected to be reside as pre-provisioned security policy logic in a policy management system.

Communication between controller and end nodes/applications

To thwart any security barriers to SDN adoption, OpenDaylight community launched a project to analyze current security capabilities of its SDN implementation, and provide recommendations for security enhancements. Below is a quick snapshot of the transport layer security of existing northbound (NB) and southbound (SB) plugins. For example, OpenFlow specifies the use of TLS or UDP/DTLS, which support authentication using certificates and encryption to secure the connection.

Figure – Transport Layer Security capabilities of SB & NB protocols (Source: ODL)

OpenDaylight recommendations for SDN controller security that are implemented in Helium and future releases include:

• AAA service for applications/users/devices
• Framework to securely and automatically discover, connect and configure the devices
• Secure mechanism to cluster the controller for scalability and availability
• Robust incident response support for controller/devices/user. E.g. A southbound syslog plugin to incorporate capture logs from devices in incident analysis
• Secure communication channel to connect to plugins, common trusted crypto key storage for all plugins, pluggable or built-in Certificate Authority

How is the scale tilted?

Securing networks is becoming more challenging to businesses, especially with BYOD and increased cloud adoption, if not yet due to other mass phenomena such as Internet-of-Everything. Organizations can certainly protect themselves better through automated and dynamic security solutions made possible through SDN, as it provides a centralized intelligence and control model that is well suited to provide much-needed flexibility to network security deployments.

With SDN, we can add agility to network intrusion responses and go beyond the network to protect what is actually of interest, namely applications. Focus on network security was more an interim arrangement as there existed a network visibility deficit at the higher layers. However, by focusing on the network all these years, we lost the most important context awareness which will help deduce if the application or user is doing what is allowed to be done. Security-centric SDN allows an organization to deploy a quick, decisive and deep enterprise-wide response to threat on all fronts. An integrated solution comprising both network and application-layer elements will ultimately provide the comprehensive ‘top-to-bottom of the stack’ security desperately needed to defend against attackers in the dynamic threat landscape.

The key to realizing self-defending networks, however, is “lower false positives” combined with “actionable-threat-intelligence”, given the lack of human element to weed out false alarms or manually correlate event logs to decide on the course of action, in the SDN driven security architecture.  The ecosystem of network security vendors, threat intelligence providers, and security professionals should strive for highly accurate intelligence so that automated security remediation decisions are made with a high-degree of confidence. Also, SDN technologies will need to continually evolve to enhance their inherent security capabilities to avoid being a dampener to adoption.

Adoption of SDN will force NetOps and SecOps to work together more closely, even if they aren’t merged into a single organization as few propose or atleast foresee to happen. This could be a bigger change than the driving technology per se, and one that could meet with opposition, given the organizational dynamics involved. Let’s wait and watch how these play out and if enterprises are able to tap into security-centric SDN benefits.

Meanwhile, how do you think the balance is tilted? Will security drive or hinder SDN adoption? Also, a clarion call to security professionals out there to critique my post. Look forward to learning from your feedback too, while I try to get a better handle on security.

A peek into Security in the context of BYOD, Cloud and IoE

Is IT team’s worst security nightmare unfolding with rampant BYOD due to unsurpassed mobility, adoption of cloud and exploding interconnections foreseen in the world of Internet of Everything? Gone are the days when the most effective policy was to build a controlled environment, and secure oneself through limited access to the external world.

Increasingly sophisticated attacks, dynamic nature of threats, advanced persistent threats (APTs), thriving underworld hacking industry, attackers innovating in lockstep with evolving technology and cloud-based solutions stress the need for enhanced, integrated and scalable security solutions focused on prevention, detection, mitigation and remediation of attacks, across the entire span of user and network touch points, without leaving any security gaps due to fragmented solutions stitched together using disparate products.

Integrated security solutions, aimed at protecting network resources and content, could be on-premises or cloud based offerings. In the new paradigm of SDN, these solutions should allow for central policy management and distributed policy enforcement.

Security solutions typically build an intelligence ecosystem by analyzing and aggregating extensive telemetry data including web requests, emails, malware samples and network intrusions, to protect networks, endpoints, mobile devices, virtual systems, web and email from known and emerging threats.

Protecting Network and Virtual Resources

Network Security Solutions have evolved beyond traditional firewalls that control ingress and egress traffic flows according to predefined rules that help enforce a given security policy, be it through packet filters or application proxies. Traditional firewalls resort to stateful packet inspection (SPI) as against deep packet inspection (DPI), and are not capable of distinguishing one kind of web traffic from another. DPI is helpful in managing application and data driven threats by looking deep into the packet and making granular access control decisions based on packet header and payload data.

Secure site-to-site and remote user connectivity can be enabled through IPSec, SSL, L2TP or PPTP based VPNs. User authentication for remote access VPNs are typically carried out through RADIUS, TACACS, LDAP, Active Directory, Kerberos or SmartCards.

While earlier-gen Intrusion Detection Systems (IDS) passively scan traffic and report on threats, Intrusion Prevention Systems (IPS) sit directly behind firewalls, actively analyze traffic, thwart any denial-of-service (DoS) attacks & application hijack attempts, by dropping malicious packets, blocking traffic from the source address, resetting the connection and additionally notifying the administrator as IDSs do.

Unified Threat Management (UTM) systems perform multiple security functions such as DPI, IPS, IDS, NAT etc. in a single platform. Given that this approach involves multiple devices and separate internal engines that examine a packet multiple times to perform individual security functions, it adds packet latency resulting in degraded network performance, apart from increasing operational management overhead.

Next-gen firewalls (NGFWs) – With security requirements being critical to businesses, IT managers had to sacrifice on network performance to achieve network security using UTMs, until the advent of next-gen firewall solutions. NGFWs are application-aware and so can differentiate traffic flows, even if they share a common protocol/port combination. They perform optimized DPI to detect anomalies and known malware, by examining the packet only once and thus ensuring performance. In addition to DPI enabled application-awareness/granular control, and traditional firewall functions of NAT and VPN, NGFWs come with integrated signature-based IPS engine and ability to integrate intelligence from outside the firewall such as directory based policy, blacklists and whitelists.

Most important of all, apart from identifying and controlling use of predefined applications based on their signatures, NGFWs can learn new applications by watching how applications behave and alert administrators if there is any deviation from base-lined normal behavior.

NGFWs perform packet inspection and SSL decryption in high-performance hardware and so can perform full packet inspection without introducing latency.

Network Access Control (NAC) – Traditionally, NAC can restrict what devices get on a network and thus were intended to work well in a closed static environment with company-owned devices. The phenomenon of BYOD has caused security to move up to the application layer with IT teams enforcing access controls through mobile app wrappers and installation of device management profiles.

In addition to solutions for physical resources, Sandboxing is increasingly being used in virtual environments, to improve security by isolating a given application to contain damages due to malware, intruders and other users, especially in virtual desktop infrastructure solutions.

Network Security Solutions – Sourced from opennetworking.org

Securing Content

Content Security Solutions protect users, email and data from inbound and outbound web security threats, and have evolved from standalone to hosted offerings.

Email Protection – Email security appliances keep critical business email safe from spams and malwares, with good spam capture rate, minimal false positives, fast blocking of new email transported viruses to avoid proliferation in the network, effective zero-hour antivirus solutions, and ability to scale threat analysis.

Web Security – In addition to effective malware protection, complete web security requires solutions to provide granular and nuanced policy knobs to control how end users access the internet, and implement proprietary and confidential data access controls through deep content analysis. Thus businesses can control access for specific features and applications such as messaging, audio, video based on user’s business requirements, without blocking access to entire websites or internet.

Security Market

While the overall security market opportunity is very strong, content security and traditional appliance/software markets are seeing a decline. Growth in hosted/SaaS solutions is offsetting the above downward trend to keep the overall security market flat.

Network security and content security market TAMs are at $6.5B and $2.8B respectively, with each growing at around 4% YoY.

Cisco leads the network security market with nearly 35% market share, followed by Check Point with 15% share and Fortinet, Palo Alto, Juniper and McAfee capturing between 5-8% of the market each.

Leaders in the content security market are Blue Coat, McAfee (Intel), Cisco, Websense and Symantec with each of these players having captured about 10-15% market share. While no single player currently dominates the market, top vendors have been extending their market reach through strategic partnerships and acquisitions.

Where do we go from here?

In the new world driven by SDN and Big Data analytics, security solutions will be evaluated on their ability to glean and integrate threat intelligence from the ever-growing ecosystem, dynamically update privileges and trust profiles of any user, device or application in real time to thwart or remediate any attack, and most importantly scale to actualize IoE, while hiding solution complexity for IT operations. Unlike other technology areas, majority of the security innovation is embedded deep inside the hardware/software offering with no inkling to the operations team or network user. Security is also a field where solution effectiveness is evaluated on the misses and not on instances of job well done, and so is quite often relegated to the back burner until a major flare-up.

What important aspects of security landscape have I missed out?

Can Enterprises and Service Providers fully mitigate personal data risks due to mobile apps, social networks and cloud hosting? If not, what measures do end users have to take, and what are the technology gaps?

What insights do you have into the mobile security market, or security needs of IoE?

Feel free to share your views in the comments section.