- 1 Description
- 2 Global overview
- 3 Environment & methodology
- 4 Test results
- 5 Conclusion
- 6 Comments
For years, Snort (developed and maintained by SourceFire) has been the de facto standard for open source Intrusion Detection/Prevention Systems (IDS/IPS). Its engine combines the benefits of signatures, protocols, and anomaly-based inspection and has become the most widely deployed IDS/IPS in the world.
Suricata, a new and less widespread product developed by the Open Information Security Foundation (OISF), has recently appeared, and seems really promising. It is also based on signatures but integrates revolutionary techniques. This engine embeds a HTTP normalizer and parser (HTP library) that provides very advanced processing of HTTP streams, enabling the understanding of traffic on the 7th level of the OSI model.
Tests have been conducted against two platforms receiving the same payloads. Based on these tests, conclusions will be discussed to present the advantages and limitations of these two products.
I'd like to give a special thank to Joel Esler for his very constructive review on the write-up. I've been in touch with Joel to review Snort configuration files and make the results as accurate as possible.
|IPS feature||optional while compiling (--enable-nfqueue)||Snort_inline or snort used with -Q option|
|Ease of install||Not available from packages. Manual installation.||Relatively straightforward. Installation also available from packages.|
|Documentation||Few resources on the Internet||Well documented on the official website and over the Internet|
|Event logging||Flat file, database, unified2 logs for barnyard|
|IPv6 support||Fully supported||Supported when compiled with --enable-ipv6 option.|
|Capture accelerators||PF_RING, packet capture accelerator||None, use of libpcap|
|Configuration file||suricata.yaml, classification.config, reference.config, threshold.config||snort.conf, threshold.conf|
|Offline analysis (pcap file)||yes|
|Frontends||Sguil, Aanval, BASE, FPCGUI (Full Packet Capture GUI), Snortsnarf|
Environment & methodology
Tests have been conducted against two identical platforms based on a Debian 5 Lenny distribution hosted on a ESX VMWare server. In addition, following prerequisites have been installed on the 2 test plateforms:
- libpcap version 1.1.1
- PCRE version: 7.6 2008-01-28
- ZLIB version: 22.214.171.124
Last available versions in the time of this writing have been tested:
- Suricata: 1.1beta1 (rev a96eae0)
- Snort: 126.96.36.199 IPv6 (Build 111)
Three sets of rules have been used as follows:
|VRT::Snort rules (free version)||x||x|
|SO rules (Shared Object)||x|
|EmergingThreats rules (free version)||x|
All rules have been activated (even those commented out by default):
for x in `ls *.rules | grep -v 'local.rules'` do sed -i -e 's/^#\s*alert/alert/i' $x done
Configuration files used for the tests are available here:
Following scoring has been used to evaluate test results:
- 0: Does not meet criteria or meets them minimally
- 1: Meets criteria
- 2: Exceed criteria (essentially used in the shape of evasion tests)
In addition, a priority has been associated to each group of test. A total is computed by summing weighted totals (SUM[sum of the scores for a given group * priority for the group]).
Log files have been analyzed "live":
- Suricata: tail -f /var/log/suricata/fast.log
- Snort: tail -f /var/log/snort/alert
Tools used for the tests
For the tests, following tools have been used:
- Scapy: to forge packets from any type. Also Scapytain has been used to save payloads and for easy replay.
- Netcat: to send payloads on a specific host/port in conjunction with echo
- Hping: to perform DoS
- Nmap: to perform ports scans from many types
- tcpdump: to capture traffic and analyze it (e.g. filter)
- tcpreplay: to replay capture files
- Hydra: to brute force services
More than 300 unit tests have been conducted against Suricata and Snort, following a methodology enabling the calculation of scores. For more information about the testing platforms, configuration files, scoring methodology and tools used, please refer to this section.
|Test Group||Priority||# of tests||Suricata score||Snort score|
|Bad Traffic (non RFC compliant)||2||4||1||1|
|Multiple failed logins||3||1||1||0|
|Malware & viruses||3||14||9||7|
|Denial of Service (DoS)||3||3||3||3|
|Inline / Prevention capabilities||2||0||1||1|
|TOTAL (unweighted sum)||315||184||217|
|TOTAL (weighted sum)||528||617|
These tests were aimed at testing the ability of the engines to trigger alerts based on rules (VRT::Snort, SO rules and EmergingThreats).
On a set of 8 attacks, both Suricata and Snort have shown their capabilities in terms of detection of basic attacks based on rules.
Nevertheless, some comments:
- for some reasons that I can't explain, some rules are commented by default in the rules files and you will have to manually uncomment them.
- Suricata won't load some rules due to unrecognized syntax (69 rule files processed. 11326 rules successfully loaded, 105 rules failed). Suricata is currently working on that point to integrate the missing keywords (e.g. file_data, http_raw_uri) in the engine.
|Error code||Error description||# of rules|
|39||[ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - Previous keyword has a fast_pattern:only; set. You can't have relative keywords around a fast_pattern only content||2|
|[ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - No preceding content or uricontent or pcre option||1|
|100||[ERRCODE: SC_ERR_RULE_KEYWORD_UNKNOWN(100)] - unknown rule keyword 'file_data'||86|
|[ERRCODE: SC_ERR_RULE_KEYWORD_UNKNOWN(100)] - unknown rule keyword 'http_raw_uri'||6|
|101||[ERRCODE: SC_ERR_FLAGS_MODIFIER(101)] - FLOW_PKT_ESTABLISHED flag is already set||1|
|102||[ERRCODE: SC_ERR_DISTANCE_MISSING_CONTENT(102)] - within needs two preceeding content or uricontent options||6|
|128||[ERRCODE: SC_ERR_INVALID_VALUE(128)] - invalid flow option "only_stream"||1|
|129||[ERRCODE: SC_ERR_UNKNOWN_REGEX_MOD(129)] - unknown regex modifier 'I'||2|
- Snort has a preprocessor called sfportscan that gives the advantage over Suricata to detect Nmap ports scans. Suricata has triggered alerts but none indicating a ports scan.
Bad traffic (non RFC compliant)
These tests were aimed at testing the behavior of the engines face to crafted packets that are non-RFC compliant.
On a set of 4 attacks, both Suricata and Snort have unsuccessfully detected bad traffic. I don't know whether it comes from the free rules, but the file bad-traffic.rules in empty.
These payloads are fragmented packets to test the ability of the engines to recompose and detect attack attempts.
On a set of 2 tests implying a ping of death and a nestea attack, Snort's spp_frag3 preprocessor has demonstrated its ability to recompose packets and successfully trigger appropriate alerts. On the other hand, Suricata has only triggered an alert for the second attack.
A bug has been filed for Suricata: https://redmine.openinfosecfoundation.org/issues/280. Suricata developers' team answered that "the defrag engine currently doesn't set events nor expose them to the signature language. Will be addressed soon."
Multiple failed logins
This tests the ability of the engine to detect multiple bad logins against a service (e.g. Brute force against FTP).
Suricata embeds such capabilities (e.g. flowint). This has been tested on Suricata with following manually crafted alerts, using flowint:
alert tcp any any -> any any (msg:"Counting Failed Logins"; content:"incorrect"; \ flowint: username, notset; flowint:username, =, 1; noalert; sid:1;) alert tcp any any -> any any (msg:"More than two Failed Logins!"; \ content:"incorrect"; flowint: username, isset; flowint:username, +, 1; \ flowint:username, >, 2; sid:2;)
Suricata has demonstrated its ability to detect multiple bad logins against a FTP service (vsFTPd). The alert has been triggered from the 3rd bad login accordingly to the rule.
On the other hand, Snort seems to base its detection of multiple bad logins on thresholds. Such rules enable to track the alerts:
#emerging-ftp.rules (from EmergingThreats rules): alert tcp $HOME_NET 21 -> $EXTERNAL_NET any (msg:"GPL FTP FTP Bad login";
flow:from_server,established; content:"530 "; depth:4; pcre:"/^530\s+(Login| User)/smi"; classtype:bad-unknown; sid:491; rev:9;)
#ftp.rules (from VRT::Snort rules): alert tcp $HOME_NET 21 -> $EXTERNAL_NET any (msg:"FTP Bad login";
flow:from_server,established; content:"530 "; fast_pattern:only; pcre:"/^530\s+ (Login|User)/smi"; classtype:bad-unknown; sid:491; rev:11;)
Notice that these rules are commented by default. In addition, Snort needs a threshold.conf that contains the counter. We haven't been able to test this feature.
The following tcpdump trace shows that the alert should be triggered (presence of "530 Login"):
13:57:04.924508 IP 192.168.100.36.21 > 192.168.100.18.32836: P 2351380887:2351380909(22) ack 1611369826 win 362 <nop,nop,timestamp 199400575 3066781> 0x0000: 4500 004a ad91 4000 4006 4395 c0a8 6424 [email protected]@.C...d$ 0x0010: c0a8 6412 0015 8044 8c27 3997 600b 8d62 ..d....D.'9.`..b 0x0020: 8018 016a 49c4 0000 0101 080a 0be2 9c7f ...jI........... 0x0030: 002e cb9d 3533 3020 4c6f 6769 6e20 696e ....530.Login.in 0x0040: 636f 7272 6563 742e 0d0a correct...
In addition, the PCRE engine has been successfully tested:
$ pcretest PCRE version 7.6 2008-01-28 re> /^530\s+(Login|User)/smi data> 530 Login incorrect 0: 530 Login 1: Login
At last, the rule itself has been isolated to the local.rules file and has been successfully loaded by Snort.
A bug has been reported to Snort to understand why the rule doesn't work.
These tests aim at testing the ability of the engines to detect attack attempts with evasion techniques such as fragmentation, encryption, ...
Fifteen evasion techniques have been tested.
Both Snort and Suricata have demonstrated their ability to detect the attacker on decoy attacks, even on the 7th position, as well as Nmap scans with fragmentation.
Snort seems to be better than Suricata at detecting certain evasion techniques, especially the following ones:
- Random URI encoding (non-UTF8)
- Directory self-reference (/./)
- Prepend long random string
- Fake parameter
- Change the case of URL
- Use Windows directory separator (\)
- Use binary value 0x0b as a request spacer
Malware & viruses
These tests aim at testing the ability of the engines to detect malware and viruses.
The tests have been conducted on 14 malware and viruses. If Suricata has a better detection level than Snort, both Suricata and Snort have demonstrated their ability to detect viruses.
These tests aim at testing the ability of the engines to detect shellcodes.
On a set of 11 shellcodes, Suricata has detected 9 shellcodes and Snort has detected 7 shellcodes.
Suricata is better at detecting shellcodes.
Denial of Service
These tests aim at testing statefull inspection capabilities of the engines face to Denial of Service attempts.
On a set of 3 tests, both Suricata and Snort have detected the 3 DoS attempts against SSH and MSSQL services. To notice that the alerts that have been triggered mainly come from Emerging Threats.
These tests consists in sending malicious documents commonly used for client-side attacks to test the ability of the engines to trigger alerts for client-side attacks.
The tests have demonstrated that Snort is better than Snort to detect client side attacks, with a detection rate of 82% against 49% for Suricata.
We did not test this due to not having any hardware that had multiple CPU's, but according to this article: http://holisticinfosec.blogspot.com/2010/08/suricata-in-toolsmith-meet-meerkat.html "Suricata has a noticeable performance improvement with hardware running multiple CPU's".
This point has not been tested. Nevertheless, according to AlienVault, both Suricata and Snort are compliant and have similar blocking capabilities.
More than 300 tests have been conducted against Suricata and Snort. Not every feature has been tested (IP/DNS reputation, performance, ...) but the tests were mainly aimed at testing the detection capabilities of the engines.
Both Snort and Suricata are based on sets of rules. Most of the tests have shown that VRT::Snort and EmergingThreats rules are complementary and are both needed to optimize the detection of all attack types. In addition, both Snort and Suricata have demonstrated their ability to detect attacks based on signatures from rules.
Suricata offers new features that Snort could implement in the future: multi-threading support, capture accelerators but suffers from a lack of documentation (few documentation on the Internet and outdated one on the official website). In addition, Suricata doesn't accept some rules from VRT::Snort and EmergingThreats due to incompatibilities (no support of certain keywords). The support of these missing keywords should be implemented in future versions of Suricata.
On the other hand, Snort is mature. It remains a very powerful IDS/IPS, very well documented over the Internet and that properly detects most of the malware and evasion techniques. Its preprocessors are very usefull for reassembling fragmented packets.
The comparison of stateful inspection features show that Snort and Suricata have different approaches. Snort bases the detection on rules and thresholds to track the number of time a rule is triggered whereas Suricata introduces session variables (e.g. via flowint) enabling to create counters. These variables can then be used by manual rules (local.rules file) to trigger events. One advantage Suricata has is its ability to understand level 7 of the OSI model, which enhances its ability of detecting malware. Suricata has demonstrated that it is far more efficient than Snort for detecting malware, viruses and shellcodes.
As a conclusion, Snort remains the de facto standard for IDS/IPS in production environments. It is stable, easily configurable and very well documented. Nevertheless, Suricata is an emerging IDS/IPS that could revolution the detection techniques and Snort will certainly implement some of these features (support of multi-threading) in future releases. We would still recommend Snort for production environments but keep a close eye to Suricata since this conclusion could quickly be updated in a very near future.
16:10, 12 April 2011 (CEST)
17:19, 12 April 2011 (CEST)
18:15, 12 April 2011 (CEST)
19:20, 12 April 2011 (CEST)
17:00, 14 April 2011 (CEST)
18:27, 14 April 2011 (CEST)
20:34, 14 April 2011 (CEST)
20:47, 14 April 2011 (CEST)
19:21, 15 April 2011 (CEST)
08:49, 27 July 2011 (MDT)
05:22, 17 November 2011 (MST)