Hyperscan is a high performance multiple regular expression matching library. It follows the regular expression syntax of the commonly used libpcre library, but is a separate library with its own C API.

Hyperscan uses hybrid automata techniques to allow matching of large numbers (up to tens of thousands) of regular expressions simultaneously, and to allow matching of regular expressions across data streams. Hyperscan is commonly used in DPI library stacks.

Hyperscan runs on the x86 platform and provides support for Perl Compatible Regular Expression (PCRE) syntax, while supporting regular expression group matching and stream operations. Hyperscan provides a flexible C language API and some different operation modes to ensure its applicability in real network scenarios. Furthermore, the focus on efficient algorithms and the use of Intel Streaming SIMD Extensions (Intel SSE) enable Hyperscan to achieve high matching performance. It is suitable for usage scenarios such as deep packet inspection (DPI), intrusion detection system (IDS), intrusion prevention system (IPS) and firewall, and has been deployed in network security solutions all over the world. Hyperscan is also integrated into widely used open source IDS and IPS products such as Snort* and Suricata*.

characteristic:

Versatility

Hyperscan supports cross-compilation for multiple Intel processors, with specific optimizations for different instruction sets. There are no operating system restrictions, and both virtual machine and container scenarios are supported, covering most of the PCRE syntax, including “.*”, “[^>] *” and other complex expressions. Provide different operation modes (streaming, block and vectorization) to meet the needs of different scenarios. If requested by using each mode flag, Hyperscan can find matching data in the input stream The start and end positions. For details, see the current version of the Hyperscan Developer Reference Guide.

mass matching

Depending on complexity, Hyperscan can support matching a large number of rules. Unlike most regular matching engines, Hyperscan supports multi-pattern matching. After specifying a unique ID for each rule, Hyperscan can compile the rules into a database, and output all currently matched rule IDs during the matching process.

Streaming mode

Hyperscan supports three modes of operation:block mode, streaming mode and vectored mode.Block mode is the most straightforward, it scans a single contiguous block of data and returns a match to the caller if found. Streaming mode is designed for cross-packet matching in network scenarios, where the data to be scanned is split into multiple packets. In Streaming mode, Hyperscan can save the matching state of the current data block and use it as the initial matching state when a new data block arrives. As shown in the figure, no matter how the “xxxxabcxxxxxxxdefx” data is divided into packets over time, Streaming mode can guarantee the consistency of the final matching result. Additionally, Hyperscan can compress the saved match state to reduce the memory footprint of the application. Streaming mode operations provide an easy way to scan data arriving over a period of time without requiring you to buffer and rescan packets, or limit scanning to a fixed window of historical data. Finally, there is a vectored mode, which provides sequential scanning of a set of data blocks that are not contiguous in memory.

High Performance and Scalability

Hyperscan requires at least the Intel Streaming SIMD Extensions 3 instruction set and uses SIMD instructions to speed up matching performance.