Ten Gigabits of Fiber, and the Hard Drives Said No

Comparing throughput with various configurations of the Synology DS1813+ and Synology DS1823xs+.

The setup

I run a backup-repo NAS for a BDR appliance: a Proxmox backup server pushes VM backups to a Synology over a private point-to-point link, and the whole chain lives or dies on how fast that path can absorb writes. So before trusting any of it, I benchmark the real path: fio runs ON the backup server, over the same NFS mount the backup traffic uses. Never on the NAS itself, never over a synthetic protocol. The disk always says the truth; everything else is a press release.

Two NAS generations went under the meter. The old box: a DS1813+ running DSM 7.1.1, Atom D2700, 2 GB of RAM, four GbE ports, no expansion slot, on a direct 1 GbE private link. The new box: a DS1823xs+ running DSM 7.4.1, an 8-core AMD, a real PCIe slot, and two M.2 slots, with five 8 TB WD Red Plus CMR drives in RAID 6 on btrfs, and a 10G fiber link to the server: an Intel X520 on the server side, an Intel dual-port SFP+ card in the NAS, multimode optics, MTU 9000.

Same jobfile every phase, no exceptions: 30 GiB sequential write, 30 GiB sequential read at 1M blocks, queue depth 16; 60 seconds of 4K random write and read at queue depth 32; stonewall between every job so nothing runs concurrently. A few fio landmines cost me real time on the first pass and are now permanent rules: omit group_reporting (it merges the jobs and mangles the per-job JSON), wrap the whole suite in timeout (libaio random phases can hang at teardown and completed-job stats still survive), and direct=1 is mandatory when the client has 192 GB of RAM, or reads come back from cache and report RAM speeds as disk speeds.

Round one: the old box, mid-rebuild

The old NAS was mid-RAID-6 rebuild when the first numbers came off it, which turned out to be its own finding:

The write test crushed the rebuild: parity resync dropped from 37 MB/s to 16, roughly doubling the ETA, and it took about 8 minutes after I stopped the suite for the rebuild to claw back. That measurement is where the operating rule came from: no backup jobs against the array until the rebuild finishes. A client write wave can silently double a rebuild window.

For context, NAS-local reads on that box had measured 366 MB/s. The 1G wire cap is about 112 to 118 MB/s single-stream. Meaning: the array could push 3x what the wire could carry. The wire was the bottleneck, not the drives.

Round two: prove the wire first

New box, new link, and I refused to benchmark storage over an unproven wire. Jumbo pings (8,972 bytes) went through clean at 0.13 ms. DSM ships iperf2, not iperf3 (and no nc, and python 3.8), so iperf2 it was: 9.90 Gbit/s server to NAS with 8 streams, 9.45 Gbit/s back the other way. Line rate, both directions.

Then the wrong turn, which stays in because it's the good part. My first forward-direction test read 2.51 Gbit/s, and for about a minute the new fiber looked half dead. It wasn't the fiber. The "sink" was Python 3.8 on the NAS: about 78 MB/s per stream, times four streams, and the CPU was the ceiling. The same wire did 9.9 Gbit/s the moment a C-based tester sat on both ends.

A slow measurement is not a slow link. Before you blame the wire, look at what's doing the measuring.

Round three: 10G, cache off

5.8x the write throughput, 4.9x the read, 36x the random write IOPS. And the one number I never caught on the old box, random read, finally has a measured value: 1,391 IOPS, which is what five spindles behind RAID 6 do when you ask them for random 4K reads. That number matters later.

Before this run I wrote the prediction down: seq read 300 to 450 MB/s, seq write 250 to 400, array-bound, not wire-bound. It held. The 9.9 Gbit wire barely gets touched by any phase; a five-drive RAID 6 is the ceiling now.

One heart-stop moment during the write phase: the NAS load average hit 69 with 132 nfsd threads in uninterruptible wait. For a minute that looked like a runaway. It wasn't: it's the write-back pipeline, 132 worker threads queuing behind a five-spindle array doing parity math. Numbers don't lie, but out of context they mislead.

Round four: the cache

The new box has two M.2 slots, and they got a pair of 1 TB WD_BLACK SN7100s as a read-write cache, RAID 1, bound to the volume, with btrfs metadata pinned to the cache. The DSM wizard throws three warnings on the way in: drives not on the compatibility list (cosmetic for cache, any brand is accepted), cache fault tolerance lower than the pool's (with two slots, RAID 1 is the maximum that exists; I took it knowingly: UPS on the box, continuous destage, and the repo's data exists in three places), and metadata pinning costs 3.8 GB of 931 (cheap, take it).

Verified live under the hood: a new md RAID1 mirror, the cache layer flipped from a DUMMY pass-through to WRITE_BACK with 953 GB, and a skip-sequential threshold of 1024K, meaning the cache refuses to cache anything sequential over 1 MB. Remember that.

Then I ran the identical suite. Here's the whole result:

That's noise, all four lines. The honest headline: on bulk throughput, the cache did nothing.

The why is baked into the design. DSM's cache skips sequential IO over 1 MB by design (my 1M blocks sit exactly at the skip line), and a single-pass uniform random read over 30 GiB touches about 1% of the blocks, once, with zero revisits.

A single-pass random benchmark cannot see a cache, because a cache exists to serve the second pass.

So I gave it a second pass. Same 4K random read, but over a 2 GiB footprint, run twice: the first pass to fill, the second to serve. Cold pass: 25,510 IOPS at 1.3 ms. Warm pass: 74,129 IOPS at 0.4 ms. That's 53x the bulk random-read baseline, and it's what the cache is actually for: the small, hot, repeated reads that live underneath a backup repository (the dedup index, the metadata, the bookkeeping).

53 times faster. Same drives, same wire, same suite: the only thing that changed was the second pass.

The other thing the cache does never shows up in a throughput table at all. During the no-cache write phase, the NAS load sat at 69 with the whole nfsd queue backed up behind the array. With the cache bound, the same write phase ran at load 14.6. The write bursts land on the SSDs at link speed and drain to the hard drives in the background. Nothing in the benchmark suite scored that. A backup job running in the real world feels it constantly.

The ladders, slowest to fastest

Every measured number from all three runs, on two ladders: bulk throughput first, then the 4K random side.

The pattern is blunt: the wire class draws the only big break on this list, and the cache draws none of it. This ladder only carries bulk throughput in MB/s, so here is the other half: every 4K random number from all three runs, same rule, slowest to fastest, this time in IOPS.

The bottom of this ladder repeats the top of the other one: bulk random read never moves, because a single pass never revisits a block, so there is nothing for a cache to serve. The top belongs to the cache on its home turf: the warm pass over a 2 GiB working set, 53x the bulk baseline. Read the two ladders together and the machine's shape is obvious: the array owns bulk, the cache owns hot, and the wire owns nothing.

What an upgrade would buy

The ladders also show the link loafing: the final run's writes used 37% of the measured wire, its reads 46%. The per-spindle math falls straight out of the same measurements: RAID 6 with five drives writes across three data spindles at 443 MB/s, which is 148 MB/s per spindle, and reads across all five at 555, which is 111 per drive. Fill the remaining three bays and the shape is predictable: six drives put the array near 590 MB/s write and 660 read, eight drives land near 890 both ways, which is 72% of the wire on writes and 75% on reads.

Eight drives is the full chassis, so the fiber bought for this box already covers its entire spindle roadmap. Swapping the drive class instead (Red Plus for Red Pro or Exos) looks strong on a spec sheet, but the measured per-spindle numbers sit below the raw drive ceiling because parity math, NFS, and the NAS's own overhead take their cut first, so the realistic gain is 30 to 50% for the price of replacing all five drives and living through five rebuild windows. An all-SSD pool would finally saturate the link, at five to ten times the cost, which is the wrong trade for a backup repository.

The cheap moves come first: a larger NFS block size costs nothing to test, and the measurement that actually decides the spend is a real full backup job and a real restore, because incremental traffic is small-block work that already rides the cache, and bulk speed only matters in full-backup windows and restore sessions.

What I'd tell someone else

• Prove the wire before you benchmark storage on it. Jumbo pings, then a real throughput run, and cross-check the reverse direction before declaring any link slow.
• Python is not a link tester. If the number looks wrong, suspect the tool before the hardware.
• fio over NFS: stonewall every job, no group_reporting, direct=1 on big-RAM clients, and a timeout wrapper around the whole suite.
• Benchmark before you bind any cache, and write the cache state into every run's notes. A cache-bound run measures the cache path, not the array.
• The cache's real test is a hot set read twice, not a bulk sweep.
• When the array becomes the bottleneck, wire upgrades stop mattering. The next lever is spindles, not fiber.
• And the backup jobs themselves: after pointing the production repository at the fiber path, two live backup runs completed through it, clean. The disk always says the truth.

Test environment: Synology DS1813+ (DSM 7.1.1, Atom D2700, 2 GB RAM) and DS1823xs+ (DSM 7.4.1, AMD V1780B, 8 cores), five 8 TB WD Red Plus CMR in RAID 6 on btrfs, two WD_BLACK SN7100 1TB as a RAID 1 read-write SSD cache, Intel X520 and Intel dual-port SFP+ over multimode fiber at MTU 9000, fio 3.39 over NFSv3 (128K blocks), October 2026. Exposure note: all benchmarking ran against a dedicated throwaway export and a disposable 30 GiB test file; the production repository was untouched by the suites and was verified with live backup runs after the link flip.