gistfile1.txt

Summary:
Results for read-only are OK

Results for read-write are odd, nobarrier and "write through" have no impact on perf or hurt it. Results are in
two clusters. The cluster with better throughput is limited to:
1) ext4,wb - write_cache="write back", does not use nobarrier
2) ext4,dioread_nolock,wb -  write_cache="write back", dioread_nolock, does not use nobarrier
3) xfs,wb - write_cache="write back", does not use nobarrier

Results for write-only have two clusters but are not odd. Throughput is good with "write through". Throughput
is less good with "write back" unless nobarrier is used. The good cluster is:
1) ext4,wt
2) ext4,nobarrier,wt
3) ext4,nobarrier,wt
4) ext4,nobarrier,dioread_nolock,wt
5) ext4,nobarrier,wb
6) ext4,nobarrier,dioread_nolock,wb
7) xfs,nobarrier,wb
8) xfs,wt
9) xfs,nobarrier,wt

And the less good cluster is:
1) ext4,wb
2) ext4,dioread_nolock,wb
3) xfs,wb

Results from the i3 NUC
* details on servers at http://smalldatum.blogspot.com/2017/05/small-servers-for-database-performance.html
* NUC57i5bnh with 16gb RAM, Samsung 960 EVO SSD that uses NVMe interface
* OS is Ubuntu 16.04 -- 4.8.0-36 and 4.13.0-26. Meltdown fix is in 4.13.0-26 but not in 4.8.0-36. Tests
    for 4.13.0-26 are run with fix enabled (pti=on) and disabled (pti=off)
* Server CPU supports PCID. I think the OS uses it.

Tests are run:
* for 1 and 4 client threads
* for ext4 with and without nobarrier, with and without dioread_nolock
* for xfs with and without nobarrier
* with /sys/block/$dev/queue/write_cache set to "write back" and "write through"

Filesystem mount options:
UUID=...	/data	xfs	noatime,nodiratime,discard,noauto	0 1
UUID=...	/data	xfs	noatime,nodiratime,discard,noauto,nobarrier	0 1
UUID=... 	/data	ext4	data=ordered,discard,noauto	0 1
UUID=... 	/data	ext4	data=ordered,discard,noauto,dioread_nolock	0 1
UUID=... 	/data	ext4	nobarrier,data=ordered,discard,noauto	0 1
UUID=... 	/data	ext4	nobarrier,data=ordered,discard,noauto,dioread_nolock	0 1

Legend:
* numbers are MB/s throughput
* 1t, 4t - 1 and 4 client threads
* ext4, xfs - file system
* dioread_nolock, nobarrier - filesystem mount options
* wt, wb - write_cache set to "write through" or "write back"

- rndrd - read-only
4.13    4.13    4.8
pti=on  pti=off no-pti
148     151     154     1t,ext4,wt
149     151     153     1t,ext4,dioread_nolock,wt
148     151     152     1t,ext4,nobarrier,wt
148     151     153     1t,ext4,nobarrier,dioread_nolock,wt
148     150     153     1t,ext4,wb
148     149     152     1t,ext4,dioread_nlock,wb
150     151     153     1t,ext4,nobarrier,wb
148     149     152     1t,ext4,nobarrier,dio,wb
147     152     154     1t,xfs,wb
149     153     153     1t,xfs,nobarrier,wb
148     154     153     1t,xfs,wt
149     153     153     1t,xfs,nobarrier,wt
-
446     451     460     4t,ext4,wt
448     459     465     4t,ext4,dioread_nolock,wt
462     458     459     4t,ext4,nobarrier,wt
450     455     461     4t,ext4,nobarrier,dioread_nolock,wt
452     452     451     4t,ext4,wb
441     452     458     4t,ext4,dioread_nlock,wb
435     452     445     4t,ext4,nobarrier,wb
446     462     455     4t,ext4,nobarrier,dio,wb
464     477     471     4t,xfs,wb
371     463     457     4t,xfs,nobarrier,wb
435     374     434     4t,xfs,wt
382     368     369     4t,xfs,nobarrier,wt

- rndrw - read-write
4.13    4.13    4.8
pti=on  pti=off no-pti
 58      58      58     1t,ext4,wt
 65      58      64     1t,ext4,dioread_nolock,wt
 56      54      62     1t,ext4,nobarrier,wt
 62      56      64     1t,ext4,nobarrier,dioread_nolock,wt
115     122     125     1t,ext4,wb
123     124     126     1t,ext4,dioread_nolock,wb
 58      55      62     1t,ext4,nobarrier,wb
 57      74      61     1t,ext4,nobarrier,dioread_nolock,wb
124     126     132     1t,xfs,wb
124     128      59     1t,xfs,nobarrier,wb
 69      60      59     1t,xfs,wt
 59      57      56     1t,xfs,nobarrier,wt
-
 87      97     113     4t,ext4,wt
 83     100     131     4t,ext4,dioread_nolock,wt
 89      92     126     4t,ext4,nobarrier,wt
 87      83     122     4t,ext4,nobarrier,dioread_nolock,wt
204     216     239     4t,ext4,wb
213     215     269     4t,ext4,dioread_nolock,wb
 87      98     112     4t,ext4,nobarrier,wb
 92      99     137     4t,ext4,nobarrier,dioread_nolock,wb
203     202     212     4t,xfs,wb
197     199     119     4t,xfs,nobarrier,wb
147     131     116     4t,xfs,wt
124     124     124     4t,xfs,nobarrier,wt

- rndwr - write-only
4.13    4.13    4.8
pti=on  pti=off no-pti
457     465     463     1t,ext4,wt
457     464     462     1t,ext4,dioread_nolock,wt
455     464     462     1t,ext4,nobarrier,wt
453     463     465     1t,ext4,nobarrier,dioread_nolock,wt
200     208     203     1t,ext4,wb
205     217     206     1t,ext4,dioread_nolock,wb
459     464     464     1t,ext4,nobarrier,wb
460     463     465     1t,ext4,nobarrier,dioread_nolock,wb
207     217     221     1t,xfs,wb
203     218     463     1t,xfs,nobarrier,wb
441     480     464     1t,xfs,wt
439     467     461     1t,xfs,nobarrier,wt
-
456     463     467     4t,ext4,wt
458     469     469     4t,ext4,dioread_nolock,wt
453     464     462     4t,ext4,nobarrier,wt
456     468     472     4t,ext4,nobarrier,dioread_nolock,wt
248     262     254     4t,ext4,wb
362     364     364     4t,ext4,dioread_nolock,wb
457     462     464     4t,ext4,nobarrier,wb
464     466     471     4t,ext4,nobarrier,dioread_nolock,wb
269     277     280     4t,xfs,wb
269     278     473     4t,xfs,nobarrier,wb
464     473     474     4t,xfs,wt
464     474     475     4t,xfs,nobarrier,wt