lslezak/README.md

## README.md

      
    Raw
  

              README.md
            
          
    YaST Installer Memory Consumption

Ruby 2.1+ allows tracing object allocations, see more details in
this article.
I enabled it and dumped the Ruby memory usage data in a SLES12-SP2 Beta3
installation
Notes


Tested in a x86_64 VirtualBox VM
In a graphical (Qt) installation
The dump was created at the installation proposal when all data needed for
installation were collected.
I just left the defaults without going into details, loading/displaying details
could allocate some more objects
The registration has been skipped
The dump obviously contains the data only from the Ruby YaST part, it does not
include the other parts (e.g. SCR agents, Perl modules) or external libraries
(libzypp, Qt, X11,...)

Content


File memory_dump.json.xz contains the collected memory data,
run unxz memory_dump.json.xz to uncompress it before analyzing it.


File dump_collector.rb contains a simple Ruby script which reads a dump file
in JSON format and prints the places which create the largest objects (in total).

Just run ./dump_collector.rb memory_dump.json command.


Feel free to modify the script to collect a different statistics.
The Result

A quick scan found out that the place which consumes most memory is
/usr/share/YaST2/lib/installation/ssh_config_file.rb:45:STRING: count: 4, total size: 274056

which loads the SSH keys and configuration from the previous installation and
needs about 270kB memory.
The question is we can optimize it better, the SSH keys actually need to be
stored somewhere as the target partition will be reformatted...
The only optimization could be probably possible when the user selects to not
copy the keys.
In that case we could drop the loaded keys when the installation starts. But that's
not the default so this improvement would actually help only in some cases...
TODO

You can collect a different statistics from the data, e.g. the place which creates
the most objects, which methods, check the object "age" (the GC generation
value), etc...

  
## dump_collector.rb
#! /usr/bin/env ruby

# This simple script collects some statistics from a JSON memory dump from
# a Ruby process.
# Usage: ./dump_collector.rb memory_dump.json

require "json"

exit 1 unless ARGV[0]

puts "Reading #{ARGV[0]}..."

counts = Hash.new(0)
sizes = Hash.new(0)

File.foreach(ARGV[0]) do |line|
  data = JSON.parse(line)
  if data["file"] && data["line"]
    id = "#{data["file"]}:#{data["line"]}:#{data["type"]}"
    counts[id] += 1
    sizes[id] += data["memsize"].to_i
  end
end

# print the locations which created objects bigger than 10000 bytes in total
sizes.sort_by{ |file, size| size }.reverse!.each do |pair|
  file = pair.first
  size = pair.last
  puts "#{file}: count: #{counts[file]}, total size: #{size}" if size > 10000
end

## memory_dump.json.xz

      
    Raw
  

              memory_dump.json.xz
            
          
      This file has been truncated, but you can view the full file.
    

            View raw
	#! /usr/bin/env ruby

	# This simple script collects some statistics from a JSON memory dump from
	# a Ruby process.
	# Usage: ./dump_collector.rb memory_dump.json

	require "json"

	exit 1 unless ARGV[0]

	puts "Reading #{ARGV[0]}..."

	counts = Hash.new(0)
	sizes = Hash.new(0)

	File.foreach(ARGV[0]) do \|line\|
	data = JSON.parse(line)
	if data["file"] && data["line"]
	id = "#{data["file"]}:#{data["line"]}:#{data["type"]}"
	counts[id] += 1
	sizes[id] += data["memsize"].to_i
	end
	end

	# print the locations which created objects bigger than 10000 bytes in total
	sizes.sort_by{ \|file, size\| size }.reverse!.each do \|pair\|
	file = pair.first
	size = pair.last
	puts "#{file}: count: #{counts[file]}, total size: #{size}" if size > 10000
	end