Confusion about using $RANDOM to generate random numbers

random_notes.md

Random numbers in bash et al

There is a variable $RANDOM than you can read in bash or zsh to get a random number beteen 0 and 32767. Here in zsh on my (old) mac:

% echo $RANDOM
13757
% echo $RANDOM
16896

Logically you could make a bunch of random numbers with a loop

% for x in {1..10}; do echo $RANDOM; done
14020
14135
10150
15776
431
6192
6705
15111
27049
23618

zsh has an intentional quirk where the random same random state is passed to any subshell, which I didn't
know about. It was encountering this effect that first made me look more closely.

# rnd2.sh
#! /bin/bash

for ((i=1;i<=10;i++))
do
echo $(echo $RANDOM) # random runs in subshell
done

Running the above with zsh gives the same random numnber every time:

% zsh rnd2.sh
12958
12958
12958
12958
12958
12958
12958
12958
12958
12958

Running with bash gives what at first I thought was a random sequence:

% bash rnd2.sh
32075
16146
216
17054
1124
17963
2033
18871
2942
19780

In zsh it is possible to seed the RNG with RANDOM=$RANDOM to avoid this behavior, or you can just run in bash. I learned all this from Stack overflow after Kagi-ing why all my numbers were the same: https://stackoverflow.com/questions/63544826/unix-shell-why-are-the-same-random-numbers-repeated. I also know from the usual SO type comments that $RANDOM is not a great RNG, if for some reason you thought it was. It uses a simple modulus function for generating the numbers, and it generates numbers that end in 8 or 9 slighlty less often. I don't care about any of that, I just wanted something that wasn't obviously nonrandom.

So far so good. Then I wrote the following:

# rnd8.sh
#! /bin/bash

for ((i=1;i<=10;i++))
do
echo $(bc <<< $RANDOM)
done

% ./rnd8.sh # running in bash
8726
10075
11424
12773
14123
15472
16821
18170
19519
20869

The numbers all are in order and differ by either 1349 or occasionally 1350.

If I run the same code in bash on Linux it appears normally random:

$ ./rnd8.sh
14276
14526
24676
22391
13409
18486
8645
19573
14892
6701

I switched here to using a random number input into bc because that was closer to my real use case and it where I noticed this. If you look back at the output from rnd2.sh that uses echo:

% bash rnd2.sh
32075
16146
216
17054
1124
17963
2033
18871
2942
19780

You see this is actually two interleaved streams of number, counting up modulo 32768, i.e. it can be split into

32075 16146
216 17054
1124 17963
2033 18871
2942 19780

What is responsible for this? I assume it's something to do with the way the subshells are invoked. And is the difference between the mac and ubuntu because of the shell versions involved (bash 3.2.57(1)-release on the mac, 5.0.17(1)-release on ubuntu) or some other reason? Or most likely, am I overlooking some obvious thing? Either way, my conclusion is that I should look for a more reliable way to get random numbers.

python -c "from random import*; print(choice(range(2**15)))"

To answer my own question, it's the bash version. Running 3.2.57 under linux gives the same result.

If you look at the bash source code, the RNG is in variables.c:

static unsigned long rseed = 1;

...

static int
brand ()
{
  rseed = rseed * 1103515245 + 12345;
  return ((unsigned int)((rseed >> 16) & 32767));       /* was % 32768 */
}

And in a subshell:

int
get_random_number ()
{
  int rv;

  /* Reset for command and process substitution. */
  if (subshell_environment && seeded_subshell == 0)
    {
      sbrand (rseed + getpid() + NOW);
      seeded_subshell = 1;
    }

  do
    rv = brand ();
  while (rv == last_random_value);
  return rv;
}

I'd guess that NOW doesn't change fast enough, and pids are getting cycled in a way that the random state isn't getting re-seeded?

Turns out the time part may at least partially relate. If I run the code below, I get a single monotonic sequence of numbers as output, rather than the interleaved ones.

#! /bin/bash

for ((i=1;i<=10;i++))
do
echo $(echo $RANDOM)
sleep 2
done

I could do mpre experiments and try to definitelvely determine what's happening, bottom line, don't use $RANDOM, even for unimportant random numbers.,

Typo:

random same random state