mlock test on a well-controlled experiment environment

mlock_exp.md

About this article

Test the effect of mlock() with this special experiment branch: https://github.com/whisthq/whist/tree/yancey/udp_experiment

On this branch, udp recv is moved into a minimal dedicated thread. So that it's much clearer where to do mlock(). The experiment aims at help understand how mlock affects memory pressures.

Backgroud

difficulty in testing memory pressure related spikes, and workaround

The OS's memory scheduling is pretty mysterious, even if the running programs on the macbooks are the same, the memeory pressure can fluctuate alot:

In this graph of memroy pressure, I am using exactly the same program to stress the memory. But the memory pressure are still changed a lot. It's basically impossible to keep memory pressure at a stable level.

Workaround

In the following paragraphs I will do "ABAB.." test . Where A means "with mlock" B means "no mlock".

The tests are purposely done in an alternative way, one immediately after another, to reduce the chance of glitches of the OS's memeory shdeduling

Test 1

Push the memory pressure to a bit above "yellow". Decryption is turned off, to avoid noises in the experiment. (since SSL allocates memory that are not controlled by us)

A:

B:

A:

B:

Test 2

In this test, I allocated a much larger static buffer( so that memory swapping effect might be more severe, and you can see really crazy spikes, don't be surprised). And I pushed memory pressure to a higher level.

Decryption is turned on. So there might be some noises caused by the SSL allocated memory.

A

B

A

B

A

(not sure what happens with this round of test, but there are noticable spikes even if with mlock(). Maybe it's caused by SSL's memory are swapped onto disk)

Comments

1. there is a noticable difference with mlock according to the ABABA... tests. The difference is huge, you can see easily from graph, no need to do precise counting

2. Even if with same test condition (like with mlock() on + same program to stress memory), the result fluctuates alot. Test needs to be done for many times to get reliable conclusion. (that why i do ABAB..) The OS's memeory schedule is is pretty mysterious: 1. The memory pressure is hard to be kept at a stable level 2. even if with similar memeroy pressure, the spikes pattern changes a lot accross restart of whist program (i.e. run a new round of test).

3. Bc of 2 , some of the 20%~30% difference in https://www.notion.so/whisthq/mlock-data-d963748ce892484491f041136d3ca5d2 might not be reliable, since it only tests each case for one time. The 20%~30% might be fluctuations. But I am not suggesting to redo all the test. I don't suggest waste too much time on this mlock() topic. I am just trying to let you aware of the inaccuracy in the data, i.e, you can use the data to help understanding, but don't fully trust it!

4. the improvement in https://www.notion.so/whisthq/mlock-data-d963748ce892484491f041136d3ca5d2 seems much smaller than in this test. Possible reasons are mentioned in the previous gist https://gist.github.com/wangyu-/f943527dcdf9e8e0f67b9fede7443a26 .