OpenZFS has recently incorporated a new“uncached prefetch”A PR that purports to significantly optimize performance.

According to the developer who submitted the PR, this change introduces an “uncacheable buffer” for ARC (Adaptive Replacement Cache) – which is passed to arc_read() and arc_write() and stored in the ARC header. When remove_reference() removes the last reference on the ARC header, it either destroys it immediately, or if it was marked for prefetching, it puts it in a new arc_uncached state. The state is scanned every second, looking for stale buffers that don’t need to be read (in which case they are immediately evicted).

This change improves the speed of sequential single-threaded reads from wide NVMe pools (from 2049 to 3932 MiB/s) by enabling a combination of prefetch and avoiding memory copies. During writing, the profiling tool showed a 22% reduction in unhalted CPU cycles at the same throughput of 3653 MiB/s.

Another developer shared his benchmark results featuring this change, which shows a clear performance improvement:

See the PR description for more details.