Suricata-vs-snort

From aldeid
Jump to navigation Jump to search

Description

For years, Snort (developed and maintained by SourceFire) has been the de facto standard for open source Intrusion Detection/Prevention Systems (IDS/IPS). Its engine combines the benefits of signatures, protocols, and anomaly-based inspection and has become the most widely deployed IDS/IPS in the world.

Suricata, a new and less widespread product developed by the Open Information Security Foundation (OISF), has recently appeared, and seems really promising. It is also based on signatures but integrates revolutionary techniques. This engine embeds a HTTP normalizer and parser (HTP library) that provides very advanced processing of HTTP streams, enabling the understanding of traffic on the 7th level of the OSI model.

Tests have been conducted against two platforms receiving the same payloads. Based on these tests, conclusions will be discussed to present the advantages and limitations of these two products.

I'd like to give a special thank to Joel Esler for his very constructive review on the write-up. I've been in touch with Joel to review Snort configuration files and make the results as accurate as possible.

Global overview

Param Suricata Snort
IPS feature optional while compiling (--enable-nfqueue) Snort_inline or snort used with -Q option
Rules
  • VRT::Snort rules
  • EmergingThreats rules
  • VRT::Snort rules
  • SO rules
  • EmergingThreats rules
Threads Multi-thread Single-thread
Ease of install Not available from packages. Manual installation. Relatively straightforward. Installation also available from packages.
Documentation Few resources on the Internet Well documented on the official website and over the Internet
Event logging Flat file, database, unified2 logs for barnyard
IPv6 support Fully supported Supported when compiled with --enable-ipv6 option.
Capture accelerators PF_RING, packet capture accelerator None, use of libpcap
Configuration file suricata.yaml, classification.config, reference.config, threshold.config snort.conf, threshold.conf
Offline analysis (pcap file) yes
Frontends Sguil, Aanval, BASE, FPCGUI (Full Packet Capture GUI), Snortsnarf

Environment & methodology

Platforms

Tests have been conducted against two identical platforms based on a Debian 5 Lenny distribution hosted on a ESX VMWare server. In addition, following prerequisites have been installed on the 2 test plateforms:

  • libpcap version 1.1.1
  • PCRE version: 7.6 2008-01-28
  • ZLIB version: 1.2.3.3

Versions

Last available versions in the time of this writing have been tested:

  • Suricata: 1.1beta1 (rev a96eae0)
  • Snort: 2.9.0.4 IPv6 (Build 111)

Rules

Three sets of rules have been used as follows:

Rule set Suricata Snort
VRT::Snort rules (free version) x x
SO rules (Shared Object) x
EmergingThreats rules (free version) x

All rules have been activated (even those commented out by default):

for x in `ls *.rules | grep -v 'local.rules'`
do
  sed -i -e 's/^#\s*alert/alert/i' $x
done

Configuration files

Configuration files used for the tests are available here:

Methodology

Scoring methodology

Following scoring has been used to evaluate test results:

  • 0: Does not meet criteria or meets them minimally
  • 1: Meets criteria
  • 2: Exceed criteria (essentially used in the shape of evasion tests)

In addition, a priority has been associated to each group of test. A total is computed by summing weighted totals (SUM[sum of the scores for a given group * priority for the group]).

Captures

Log files have been analyzed "live":

  • Suricata: tail -f /var/log/suricata/fast.log
  • Snort: tail -f /var/log/snort/alert

Tcpdump / Wireshark have been used to track malware & viruses.

Tools used for the tests

For the tests, following tools have been used:

  • Scapy: to forge packets from any type. Also Scapytain has been used to save payloads and for easy replay.
  • Netcat: to send payloads on a specific host/port in conjunction with echo
  • Hping: to perform DoS
  • Nmap: to perform ports scans from many types
  • tcpdump: to capture traffic and analyze it (e.g. filter)
  • tcpreplay: to replay capture files
  • Hydra: to brute force services

Test results

Synthesis

Warning
The results of these tests are currently being revised following Joel Esler's comments. Please consider current results with caution.

More than 300 unit tests have been conducted against Suricata and Snort, following a methodology enabling the calculation of scores. For more information about the testing platforms, configuration files, scoring methodology and tools used, please refer to this section.

Test Group Priority # of tests Suricata score Snort score
Test rules 3 8 6 8
Bad Traffic (non RFC compliant) 2 4 1 1
Fragmented packets 2 2 1 3
Multiple failed logins 3 1 1 0
Evasion techniques 2 15 21 29
Malware & viruses 3 14 9 7
Shellcodes 3 11 12 7
Denial of Service (DoS) 3 3 3 3
Client-side attacks 3 257 127 157
Performance 3 0 2 1
Inline / Prevention capabilities 2 0 1 1
TOTAL (unweighted sum) 315 184 217
TOTAL (weighted sum) 528 617

Test rules

These tests were aimed at testing the ability of the engines to trigger alerts based on rules (VRT::Snort, SO rules and EmergingThreats).

On a set of 8 attacks, both Suricata and Snort have shown their capabilities in terms of detection of basic attacks based on rules.

Nevertheless, some comments:

  • for some reasons that I can't explain, some rules are commented by default in the rules files and you will have to manually uncomment them.
  • Suricata won't load some rules due to unrecognized syntax (69 rule files processed. 11326 rules successfully loaded, 105 rules failed). Suricata is currently working on that point to integrate the missing keywords (e.g. file_data, http_raw_uri) in the engine.
Error code Error description # of rules
39 [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - Previous keyword has a fast_pattern:only; set. You can't have relative keywords around a fast_pattern only content 2
[ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - No preceding content or uricontent or pcre option 1
100 [ERRCODE: SC_ERR_RULE_KEYWORD_UNKNOWN(100)] - unknown rule keyword 'file_data' 86
[ERRCODE: SC_ERR_RULE_KEYWORD_UNKNOWN(100)] - unknown rule keyword 'http_raw_uri' 6
101 [ERRCODE: SC_ERR_FLAGS_MODIFIER(101)] - FLOW_PKT_ESTABLISHED flag is already set 1
102 [ERRCODE: SC_ERR_DISTANCE_MISSING_CONTENT(102)] - within needs two preceeding content or uricontent options 6
128 [ERRCODE: SC_ERR_INVALID_VALUE(128)] - invalid flow option "only_stream" 1
129 [ERRCODE: SC_ERR_UNKNOWN_REGEX_MOD(129)] - unknown regex modifier 'I' 2
  • Snort has a preprocessor called sfportscan that gives the advantage over Suricata to detect Nmap ports scans. Suricata has triggered alerts but none indicating a ports scan.

Bad traffic (non RFC compliant)

These tests were aimed at testing the behavior of the engines face to crafted packets that are non-RFC compliant.

On a set of 4 attacks, both Suricata and Snort have unsuccessfully detected bad traffic. I don't know whether it comes from the free rules, but the file bad-traffic.rules in empty.

Fragmented packets

These payloads are fragmented packets to test the ability of the engines to recompose and detect attack attempts.

On a set of 2 tests implying a ping of death and a nestea attack, Snort's spp_frag3 preprocessor has demonstrated its ability to recompose packets and successfully trigger appropriate alerts. On the other hand, Suricata has only triggered an alert for the second attack.

A bug has been filed for Suricata: https://redmine.openinfosecfoundation.org/issues/280. Suricata developers' team answered that "the defrag engine currently doesn't set events nor expose them to the signature language. Will be addressed soon."

Multiple failed logins

This tests the ability of the engine to detect multiple bad logins against a service (e.g. Brute force against FTP).

Suricata embeds such capabilities (e.g. flowint). This has been tested on Suricata with following manually crafted alerts, using flowint:

alert tcp any any -> any any (msg:"Counting Failed Logins"; content:"incorrect"; \
 flowint: username, notset; flowint:username, =, 1; noalert; sid:1;)
alert tcp any any -> any any (msg:"More than two Failed Logins!"; \
 content:"incorrect"; flowint: username, isset; flowint:username, +, 1; \
 flowint:username, >, 2; sid:2;)

Suricata has demonstrated its ability to detect multiple bad logins against a FTP service (vsFTPd). The alert has been triggered from the 3rd bad login accordingly to the rule.

On the other hand, Snort seems to base its detection of multiple bad logins on thresholds. Such rules enable to track the alerts:

#emerging-ftp.rules (from EmergingThreats rules):
alert tcp $HOME_NET 21 -> $EXTERNAL_NET any (msg:"GPL FTP FTP Bad login";

flow:from_server,established; content:"530 "; depth:4; pcre:"/^530\s+(Login| User)/smi"; classtype:bad-unknown; sid:491; rev:9;)

#ftp.rules (from VRT::Snort rules):
alert tcp $HOME_NET 21 -> $EXTERNAL_NET any (msg:"FTP Bad login";

flow:from_server,established; content:"530 "; fast_pattern:only; pcre:"/^530\s+ (Login|User)/smi"; classtype:bad-unknown; sid:491; rev:11;)

Notice that these rules are commented by default. In addition, Snort needs a threshold.conf that contains the counter. We haven't been able to test this feature.

The following tcpdump trace shows that the alert should be triggered (presence of "530 Login"):

13:57:04.924508 IP 192.168.100.36.21 > 192.168.100.18.32836: P
2351380887:2351380909(22) ack 1611369826 win 362 <nop,nop,timestamp 199400575
3066781>
0x0000: 4500 004a ad91 4000 4006 4395 c0a8 6424 E..J..@[email protected]$
0x0010: c0a8 6412 0015 8044 8c27 3997 600b 8d62 ..d....D.'9.`..b
0x0020: 8018 016a 49c4 0000 0101 080a 0be2 9c7f ...jI...........
0x0030: 002e cb9d 3533 3020 4c6f 6769 6e20 696e ....530.Login.in
0x0040: 636f 7272 6563 742e 0d0a                correct...

In addition, the PCRE engine has been successfully tested:

$ pcretest
PCRE version 7.6 2008-01-28
re> /^530\s+(Login|User)/smi
data> 530 Login incorrect
0: 530 Login
1: Login

At last, the rule itself has been isolated to the local.rules file and has been successfully loaded by Snort.

A bug has been reported to Snort to understand why the rule doesn't work.

Evasion techniques

These tests aim at testing the ability of the engines to detect attack attempts with evasion techniques such as fragmentation, encryption, ...

Fifteen evasion techniques have been tested.

Both Snort and Suricata have demonstrated their ability to detect the attacker on decoy attacks, even on the 7th position, as well as Nmap scans with fragmentation.

Snort seems to be better than Suricata at detecting certain evasion techniques, especially the following ones:

  • Random URI encoding (non-UTF8)
  • Directory self-reference (/./)
  • Prepend long random string
  • Fake parameter
  • Change the case of URL
  • Use Windows directory separator (\)
  • Use binary value 0x0b as a request spacer

In addition, JavaScript obfuscation hasn't been detected by Suricata in our test campaign.

Malware & viruses

These tests aim at testing the ability of the engines to detect malware and viruses.

The tests have been conducted on 14 malware and viruses. If Suricata has a better detection level than Snort, both Suricata and Snort have demonstrated their ability to detect viruses.

Shellcodes

These tests aim at testing the ability of the engines to detect shellcodes.

On a set of 11 shellcodes, Suricata has detected 9 shellcodes and Snort has detected 7 shellcodes.

Suricata is better at detecting shellcodes.

Denial of Service

These tests aim at testing statefull inspection capabilities of the engines face to Denial of Service attempts.

On a set of 3 tests, both Suricata and Snort have detected the 3 DoS attempts against SSH and MSSQL services. To notice that the alerts that have been triggered mainly come from Emerging Threats.

Client-side attacks

These tests consists in sending malicious documents commonly used for client-side attacks to test the ability of the engines to trigger alerts for client-side attacks.

The tests have demonstrated that Snort is better than Snort to detect client side attacks, with a detection rate of 82% against 49% for Suricata.

Performance

We did not test this due to not having any hardware that had multiple CPU's, but according to this article: http://holisticinfosec.blogspot.com/2010/08/suricata-in-toolsmith-meet-meerkat.html "Suricata has a noticeable performance improvement with hardware running multiple CPU's".

Inline/blocking capabilities

This point has not been tested. Nevertheless, according to AlienVault, both Suricata and Snort are compliant and have similar blocking capabilities.

Conclusion

More than 300 tests have been conducted against Suricata and Snort. Not every feature has been tested (IP/DNS reputation, performance, ...) but the tests were mainly aimed at testing the detection capabilities of the engines.

Both Snort and Suricata are based on sets of rules. Most of the tests have shown that VRT::Snort and EmergingThreats rules are complementary and are both needed to optimize the detection of all attack types. In addition, both Snort and Suricata have demonstrated their ability to detect attacks based on signatures from rules.

Suricata offers new features that Snort could implement in the future: multi-threading support, capture accelerators but suffers from a lack of documentation (few documentation on the Internet and outdated one on the official website). In addition, Suricata doesn't accept some rules from VRT::Snort and EmergingThreats due to incompatibilities (no support of certain keywords). The support of these missing keywords should be implemented in future versions of Suricata.

On the other hand, Snort is mature. It remains a very powerful IDS/IPS, very well documented over the Internet and that properly detects most of the malware and evasion techniques. Its preprocessors are very usefull for reassembling fragmented packets.

The comparison of stateful inspection features show that Snort and Suricata have different approaches. Snort bases the detection on rules and thresholds to track the number of time a rule is triggered whereas Suricata introduces session variables (e.g. via flowint) enabling to create counters. These variables can then be used by manual rules (local.rules file) to trigger events. One advantage Suricata has is its ability to understand level 7 of the OSI model, which enhances its ability of detecting malware. Suricata has demonstrated that it is far more efficient than Snort for detecting malware, viruses and shellcodes.

As a conclusion, Snort remains the de facto standard for IDS/IPS in production environments. It is stable, easily configurable and very well documented. Nevertheless, Suricata is an emerging IDS/IPS that could revolution the detection techniques and Snort will certainly implement some of these features (support of multi-threading) in future releases. We would still recommend Snort for production environments but keep a close eye to Suricata since this conclusion could quickly be updated in a very near future.

Comments

16:10, 12 April 2011 (CEST)
Good Write-up!
17:19, 12 April 2011 (CEST)
Interesting write up.

I'd like to note and know a few things.

  1. Ipv6 is completely supported.
  2. What exploits were used for the client side attacks? We love to know so that we can be sure we cover them.
  3. What configuration file was used (snort.conf). We'd love to know so that we can replicate your results.

Feel free to contact me at any time at joel [at] snort.org

Joel
18:15, 12 April 2011 (CEST)
Hi Joel,

Thanks for your support.

  1. IPv6: Updated.
  2. Client side attacks are detailed on this page.
  3. Configuration files are available on this page
19:20, 12 April 2011 (CEST)
  1. Snort DAQ supports PF_RING, so you can use that if you want.
  2. Would be nice to know what the detection is with the SO rules on.
17:00, 14 April 2011 (CEST)
I also notice that you have this in your snort.conf:

"include emerging.conf" What is this file?

Noticed that you have "DELETED" rules in your results, but your snort.conf file doesn't have deleted in it.

In addition you state that Snort needs a threshold.conf to increment counters and you couldn't test this feature, while this is not only not true, as Snort does not need a threshold.conf to increment counters, but you also /do/ use the threshold.conf in the snort.conf that you provide.

Snort does not need to be compiled with Inline support for it to work in inline mode. It works by default by using the -Q command line tag. The DAQ is responsible for the input method and tries to compile inline mode into DAQ by default.

Basically, it appears that your results are not matching up with your tests, and your tests are incomplete (as you are not running Shared Object rules)

18:27, 14 April 2011 (CEST)
Sebastien, interesting article. A couple of minor points you might want to correct:

- The IPv6 story is more complex than Joel notes. While both Suri and Snort inspect IPv6 traffic and write Unified2 alerts, I don't believe any of the frontends you discussed will see those alerts because the standard database-schema doesn't support them. Barnyard2 has a more nuanced description of IPv6 support that applies equally to Snort and Suri since they both output unified2: http://www.securixlive.com/barnyard2/index.php

- Regarding Acceleration: Both snort and suri support a variety of accelerators including pfring, endace capture-cards, napatech capture-cards, Intel X10 capture-cards, and myricom capture-cards. I would call this a draw between the two products.

- Regarding Multithreading: While suri is natively multi-threaded, snort can be "multi-process". All of the acceleration frameworks noted above support running multiple instances of snort on the same computer, each using a separate CPU. It's much more work up-front to configure, but this is how many big shops scale snort and it is well-tested.

Regarding Performance: Again, I think there's a more nuanced story than "suri is faster". Multi-thread suri can beat single-thread snort given enough hardware. Multi-process snort, is still quite a lot faster on equivalent hardware, though: http://lists.emergingthreats.net/pipermail/emerging-sigs/2010-August/008613.html

Thanks for the article!!!

- Mike Lococo
20:34, 14 April 2011 (CEST)
@Joel: Many thanks for your feedback. Here are some answers and comments:
  • include file: Default snort.conf file only contains VRT rules. Emerging Threat rules have been included in snort.conf with "include emerging.conf". I have added the file in this page so you can download it.
  • DELETED rules: Tests have been performed by 3 different teams (and I suspect 3 different snort.conf files). I will ensure tests are done again with the SAME configuration files and will update the write-up.
  • Threshold: Honestly, this part hasn't been deeply analyzed and certainly needs further investigations. Actually, I didn't fully understand how to increment counters. I'll analyze Snort docs more in the details and I'll update this test.
  • Snort Inline capabilities: I've updated the table accordingly to your comment.
  • SO rules/incomplete tests: I have seen your previous comment regarding SO rules. Currently analyzing it... (to be continued)
Thanks again for this review, it will help formalizing more accurate results.
20:47, 14 April 2011 (CEST)
@Mike: Many thanks for this very positive feedback. Your comments are really precise and constructive; I'll include them in my write-up.
19:21, 15 April 2011 (CEST)
After going through the 257 Client side samples that you have md5sums for, we have pulled 203 of them. When running these files through Snort we've alerted much much more than the post says we do. However, in order to replicate your results, we'd like to see if we can get copies of the other 54 samples from you. Is there some way I can provide you the list of the md5sum's that we don't have so that we may verify coverage?
Peter Manev
08:49, 27 July 2011 (MDT)
Hi,

Thank you for all the efforts to put this article together. If I might add a few things:

  1. s/^# alert/alert/ - re-activates a lot of rules, HOWEVER it does not reactivate "#alert" only "# alert" (with a white space between "#" and "alert"), when you do , that adds a good few hundred rules more (for both Suricata and Snort).
  2. There were no decoder rules included as well in the rule set - it makes a difference for Suricata(decoder-events.rules).

When all those are added and re-tested - you get responses from Suricata on a good few more things from the tests done - i.e Xmas scan,Nestea Attack, FullSynScan, Malformed Traffic, Land Attack, Nikto Random URI encoding, Client Side Attacks etc. , than originally reported that log no alerts.

example:

07/25/2011-14:29:34.228005  
[**] [1:2200011:1] SURICATA IPv4 wrong IP version 
[**] [Classification: (null)] [Priority: 3] 
[**] [Raw pkt: 00 50 56 00 00 FF 00 0C 29 5E 40 2A 08 00 32 00 00 1C 00 01 00 00
40 01 FA 47 C0 A8 89 24 C0 A8
07/25/2011-14:23:07.775713  
[**] [1:2000546:7] ET SCAN NMAP -f -sX 
[**] [Classification: Attempted Information Leak] [Priority: 2] {TCP} 192.168.137
.36:48018 -> 192.168.137.35:80
07/25/2011-14:23:07.775713  
[**] [1:1228:9] DELETED SCAN nmap XMAS 
[**] [Classification: Attempted Information Leak] [Priority: 2] {TCP} 192.168.137
.36:48018 -> 192.168.137.35:80
07/25/2011-14:32:12.577971  
[**] [1:2000537:8] ET SCAN NMAP -sS window 2048 
[**] [Classification: Attempted Information Leak] [Priority: 2] {TCP} 192.168.137
.36:41990 -> 192.168.137.35:80
07/25/2011-14:34:32.370835  
[**] [1:527:8] GPL SCAN same SRC/DST 
[**] [Classification: Potentially Bad Traffic] [Priority: 2] {TCP} 192.168.137.35
:135 -> 192.168.137.35:135
07/25/2011-14:34:32.370835  
[**] [1:527:10] DELETED BAD-TRAFFIC same SRC/DST 
[**] [Classification: Potentially Bad Traffic] [Priority: 2] {TCP} 192.168.137.35
:135 -> 192.168.137.35:135
07/25/2011-14:46:48.273792  
[**] [1:22000069:1] SURICATA FRAG IPv4 Fragmentation overlap 
[**] [Classification: (null)] [Priority: 3] {UDP} 192.168.137.36:0 -> 192.168.137
.35:0
07/25/2011-14:46:55.431430  
[**] [1:22000069:1] SURICATA FRAG IPv4 Fragmentation overlap 
[**] [Classification: (null)] [Priority: 3] {UDP} 192.168.137.36:0 -> 192.168.137
.35:0

Thanks

05:22, 17 November 2011 (MST)
You may have seen this already. If not, feel free to peruse the material. http://www.thinkmind.org/download.php?articleid=icds_2011_7_40_90007