Skip to content

Instantly share code, notes, and snippets.

@carlasouza carlasouza/bash.md Secret
Last active Nov 24, 2016

Embed
What would you like to do?

Title

One day during this week, I tried to open bash and I was suprised by this error:

-bash: xmalloc: cannot allocate 4000000016 bytes

Wat

Bash was trying to allocate memory 4gb of memory? Why?

Looking on my frankenstein .bashrc, I decided that it was time to try to understand what each variable actually does. I like to keep eveything I ever executed in my history, with timestamps and multi line support. I started cleaning it, and found out that this issue stopped happening once I removed the variables responsible for my bash history behavior.

These are the ones to blame:

  export HISTSIZE=500000000
  export HISTFILESIZE=500000000

I had both $HISTSIZE and $HISTFILESIZE set up to 500000000 lines. It's a reasonable number, right? What do they actually do?

  • HISTFILESIZE: the max size (inlines) of your bash_history. The maximum number of lines contained in the history file
  • HISTSIZE: the amount of lines that will be kept in memory, and flushed into disk (.bash_history) when you exit the shell. The maximum number of commands to remember on the history list (on a single current open shell). It will also load the last $HSITSIZE lines from your history file into memory when you open a new shell

The last $HISTSIZE lines are copied/appended to $HISTFILEfrom that "list"

Lets invoke our friend strace. I left only the important lines of the trace:

[pid 11677] open("/home/carla/.bash_history", O_RDONLY) = 3
[pid 11677] fstat(3, {st_mode=S_IFREG|0600, st_size=16755, ...}) = 0
[pid 11677] read(3, "ls\n#1478028902\n./i3-get-window-c"..., 16755) = 16755
[pid 11677] close(3)                    = 0
[pid 11677] chown("/home/carla/.bash_history", 1000, 100) = 0
[pid 11677] stat("/home/carla/.bash_history", {st_mode=S_IFREG|0600, st_size=16755, ...}) = 0
[pid 11677] open("/home/carla/.bash_history", O_RDONLY) = 3
[pid 11677] fstat(3, {st_mode=S_IFREG|0600, st_size=16755, ...}) = 0
[pid 11677] read(3, "ls\n#1478028902\n./i3-get-window-c"..., 16755) = 16755
[pid 11677] close(3)                    = 0
[pid 11677] mmap(NULL, 4000002048, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Cannot allocate memory)
[pid 11677] brk(0xf0e0a000)             = 0x2746000
[pid 11677] mmap(NULL, 4000137216, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Cannot allocate memory)
[pid 11677] mmap(NULL, 134217728, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7f9e145ae000
[pid 11677] munmap(0x7f9e145ae000, 61153280) = 0
[pid 11677] munmap(0x7f9e1c000000, 5955584) = 0
[pid 11677] mprotect(0x7f9e18000000, 135168, PROT_READ|PROT_WRITE) = 0
[pid 11677] mmap(NULL, 4000002048, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Cannot allocate memory)
[pid 11677] write(2, "-bash: xmalloc: cannot allocate "..., 49-bash: xmalloc: cannot allocate 4000000016 bytes
) = 49
[pid 11677] exit_group(2)               = ?
[pid 11677] +++ exited with 2 +++

Strace is telling me that as part of starting bash, it will ready my .bash_history and try to allocate memory right after.

Which ended up as 4gb of memory, and thanks to Google Chrome already using all of it already, I couldn't, crashing my dearly bash.

Until now, all the information I knew is that it was trying to allocate 500000000* of memory, resulting in 4GB of memory. Doing the math, it was 4000000016 bytes because the variable was set to remember 500000000 lines

My Hipotesis

Lets to some math. (4000000000 + 16) / 500000000 + 2=8` bytes per line, where this 2 is being added for some reason.

Strace can't help me any further at this time, so I got the source code. I made sure I had the same version as the one installed on my machine (4.4.0).

Now it's time to use GDB. The code needs to be compiled with extra flags to allow the debugging.

./configure
CFLAGS='-g -Wall -Wextra' make

I cleaned my .bashrc, leaving only this two lines:

export HISTSIZE=500000000
export HISTFILESIZE=500000000

After glancing into the code, I found the files I thought would lead me to why it was trying to allocate so much memory: lib/readline/history.c, specially the function: add_history (line_start)

The function add_history() is defined at lib/readline/history.c:276

Now we're good to go.

Digging into the code with gdb:

$ cd ~/bash/bash/
$# gdb ./bash
gnu gdb (GDB) 7.12
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./bash...done.
(gdb) break lib/readline/history.c:281
Breakpoint 1 at 0x4ba0e9: file history.c, line 281.
(gdb) run
Starting program: /home/carla/bash/bash-4.4/bash

Breakpoint 1, add_history (string=0x757008 "dmesg") at history.c:281
281  if (history_stifled && (history_length == history_max_entries))
(gdb) p history_stifled
$1 = 1
(gdb) p history_length
$2 = 0
(gdb) p history_max_entries
$3 = 500000000
(gdb) next
307      if (history_size == 0)
(gdb) p history_size
$4 = 0
(gdb) next
309  if (history_stifled && history_max_entries > 0)
(gdb) next
310    history_size = history_max_entries + 2;
(gdb) p history_size
$5 = 0
(gdb) p history_max_entries
$6 = 500000000
(gdb) next
313  the_history = (HIST_ENTRY **)xmalloc (history_size * sizeof (HIST_ENTRY *));
(gdb) p history_size
$7 = 500000002
(gdb) p sizeof (HIST_ENTRY *)
$8 = 8
(gdb) p history_size * sizeof (HIST_ENTRY *)
$9 = -294967280
(gdb) next
bash: xmalloc: cannot allocate 4000000016 bytes (299008 bytes allocated)
[Inferior 1 (process 4932) exited with code 02]

BOOM!

Let's try to understand what is happening here.

This is the data important for us ATM: history_max_entries equals 500000000, which is the value I have for $HISTSIZE on my .bashrc

It all starts when bash wants to load history: load_history() at bashhist.c:300 It fetches from $HISTSIZE available in your environment.

  0 void
  1 load_history ()
  2 {
  3   char *hf;
  4 
  5   /* Truncate history file for interactive shells which desire it.
  6      Note that the history file is automatically truncated to the
  7      size of HISTSIZE if the user does not explicitly set the size
  8      differently. */
  9   set_if_not ("HISTSIZE", "500");
 10   sv_histsize ("HISTSIZE");
 11 
 12   set_if_not ("HISTFILESIZE", get_string_value ("HISTSIZE"));
 13   sv_histsize ("HISTFILESIZE");
 14 
 15   /* Read the history in HISTFILE into the history list. */
 16   hf = get_string_value ("HISTFILE");
 17 
 18   if (hf && *hf && file_exists (hf))
 19     {
 20       read_history (hf);
 21       /* We have read all of the lines from the history file, even if we
 22 »······· read more lines than $HISTSIZE.  Remember the total number of lines
 23 »······· we read so we don't count the last N lines as new over and over
 24 »······· again. */
 25       history_lines_in_file = history_lines_read_from_file;
 26       using_history ();
 27       /* history_lines_in_file = where_history () + history_base - 1; */
 28     }
 29 }

Let's see what this function sv_histsize on line 10 does. It is defined at lib/readline/bind.c line 1825.

As a bonus we also can see that it has a default to 500 if the variable is not defined. Why is it set twice, I'm not sure.

   1 static int
1825 sv_histsize (value)
   1      const char *value;
   2 {
   3   int nval;
   4 
   5   nval = 500;
   6   if (value && *value)
   7     {
   8       nval = atoi (value);
   9       if (nval < 0)
  10 »·······{
  11 »·······  unstifle_history ();
  12 »·······  return 0;
  13 »·······}
  14     }
  15   stifle_history (nval);
  16   return 0;
  17 }

I see that nval will have my $HISTSIZE value. Now lets see what the function stifile_history (500000000)`.

  1 void
504 stifle_history (max)
  1      int max;
  2 {
  3   register int i, j;
  4 
  5   if (max < 0)
  6     max = 0;
  7 
  8   if (history_length > max)
  9     {
 10       /* This loses because we cannot free the data. */
 11       for (i = 0, j = history_length - max; i < j; i++)
 12 »·······free_history_entry (the_history[i]);
 13 
 14       history_base = i;
 15       for (j = 0, i = history_length - max; j < max; i++, j++)
 16 »·······the_history[j] = the_history[i];
 17       the_history[j] = (HIST_ENTRY *)NULL;
 18       history_length = j;
 19     }
 20 
 21   history_stifled = 1;
 22   max_input_history = history_max_entries = max;
 23 }

Remember the variable history_max_entries? We saw it in the GDB output, having the value being passed to xmalloc. The function stifle_history sets the variable history_max_entries as 500000000.

309    if (history_stifled && history_max_entries > 0)                                # history_max_entries => 500000000
310      history_size = history_max_entries + 2;                                      # history_size => 500000002
311    else
312      history_size = DEFAULT_HISTORY_INITIAL_SIZE;
313    the_history = (HIST_ENTRY **)xmalloc (history_size * sizeof (HIST_ENTRY *));   # sizeof (HIST_ENTRY *) => 8
                                                                                      # => xmalloc (4000000016) => 4GB

Bash crases on line 313 from the snipet above. the_history = (HIST_ENTRY **)xmalloc (history_size * sizeof (HIST_ENTRY *));

It is allocating memory, the amount of history_size (that comes from your .bashrc $HISTSIZE) and multiplying by 8.

This bug was introduced on September 15th (a0c0a0) and fixed on patch 1 on Novemeber 14th (8ddc8d6) by setting a max limit for memory allocation:

 $ git show 8ddc8d6e3a3d85eec6d4ba9b9ed2bc36bce56716
commit 8ddc8d6e3a3d85eec6d4ba9b9ed2bc36bce56716
Author: Chet Ramey <chet.ramey@case.edu>
Date:   Mon Nov 14 14:26:51 2016 -0500

    Bash-4.4 patch 1

diff --git a/lib/readline/history.c b/lib/readline/history.c
index 3b8dbc5..9ff25a7 100644
--- a/lib/readline/history.c
+++ b/lib/readline/history.c
@@ -57,6 +57,8 @@ extern int errno;
 /* How big to make the_history when we first allocate it. */
 #define DEFAULT_HISTORY_INITIAL_SIZE   502
 
+#define MAX_HISTORY_INITIAL_SIZE       8192
+
 /* The number of slots to increase the_history by. */
 #define DEFAULT_HISTORY_GROW_SIZE 50
 
@@ -307,7 +309,9 @@ add_history (string)
       if (history_size == 0)
        {
          if (history_stifled && history_max_entries > 0)
-           history_size = history_max_entries + 2;
+           history_size = (history_max_entries > MAX_HISTORY_INITIAL_SIZE)
+                               ? MAX_HISTORY_INITIAL_SIZE
+                               : history_max_entries + 2;
          else
            history_size = DEFAULT_HISTORY_INITIAL_SIZE;
          the_history = (HIST_ENTRY **)xmalloc (history_size * sizeof (HIST_ENTRY *));
diff --git a/patchlevel.h b/patchlevel.h
index 1cd7c96..40db1a3 100644
--- a/patchlevel.h
+++ b/patchlevel.h
@@ -25,6 +25,6 @@
    regexp `^#define[   ]*PATCHLEVEL', since that's what support/mkversion.sh
    looks for to find the patch level (for the sccs version string). */
 
-#define PATCHLEVEL 0
+#define PATCHLEVEL 1
 
 #endif /* _PATCHLEVEL_H_ */

What I learned

So there is no point into setting the HISTSIZSE more than 8192, since its limit was now set on the bug fix. (Unless it is being used elsewhere, which I didn't look for)

So I changed it to

  export HISTSIZE=8192
  export HISTFILESIZE=500000000

I read somewhere that we could improve this. Instead of keeping in memory the commands to later be added to history once we close the terminal (and if you crash your terminal emulator, you'll lose it), we can flush the commands as soon as we run them. For that, we just need to add to .bashrc:

  shopt -s histappend
  export PROMPT_COMMAND='history -a'

Now we could decrease even more the $HISTSIZE because we dont need to keep in memory.

But theres two caveauts:

  • if you work with multiple terminals, and is used to press arrow up for the previous commands in a certain terminal, it will be all mixed together with all you other terminals and ordered by timestamp.
  • it will increase disk IO, because it is flushing into the $HISTFILE at every command

So I changed my mind back to keep at the max 8192 and removed the append feature.

Source:

[1] https://www.gnu.org/software/bash/manual/html_node/Bash-Variables.html

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.