Instantly share code, notes, and snippets.

Embed
What would you like to do?
List local memcached keys using Ruby
#!/usr/bin/env ruby
# List all keys stored in memcache.
# Credit to Graham King at http://www.darkcoding.net/software/memcached-list-all-keys/ for the original article on how to get the data from memcache in the first place.
require 'net/telnet'
headings = %w(id expires bytes cache_key)
rows = []
localhost = Net::Telnet::new("Host" => "localhost", "Port" => 11211, "Timeout" => 3)
matches = localhost.cmd("String" => "stats items", "Match" => /^END/).scan(/STAT items:(\d+):number (\d+)/)
slabs = matches.inject([]) { |items, item| items << Hash[*['id','items'].zip(item).flatten]; items }
longest_key_len = 0
slabs.each do |slab|
localhost.cmd("String" => "stats cachedump #{slab['id']} #{slab['items']}", "Match" => /^END/) do |c|
matches = c.scan(/^ITEM (.+?) \[(\d+) b; (\d+) s\]$/).each do |key_data|
cache_key, bytes, expires_time = key_data
rows << [slab['id'], Time.at(expires_time.to_i), bytes, cache_key]
longest_key_len = [longest_key_len,cache_key.length].max
end
end
end
row_format = %Q(|%8s | %28s | %12s | %-#{longest_key_len}s |)
puts row_format%headings
rows.each{|row| puts row_format%row}
localhost.close
@jwoffindin

This comment has been minimized.

Show comment
Hide comment
@jwoffindin

jwoffindin Mar 2, 2012

Great, very useful - thanks!

jwoffindin commented Mar 2, 2012

Great, very useful - thanks!

@ulfmagnetics

This comment has been minimized.

Show comment
Hide comment
@ulfmagnetics

ulfmagnetics Mar 6, 2012

+1 extremely helpful for debugging. thank you.

ulfmagnetics commented Mar 6, 2012

+1 extremely helpful for debugging. thank you.

@bkimble

This comment has been minimized.

Show comment
Hide comment
@bkimble

bkimble Mar 6, 2012

No problem guys! It's super silly that there is no easy way to do this out of the box. I try to use Redis as much as I can but sometimes memcached is the right tool :)

Also because it wasn't mentioned anywhere, Graham King, whose article at http://www.darkcoding.net/software/memcached-list-all-keys/ helped out here.

Owner

bkimble commented Mar 6, 2012

No problem guys! It's super silly that there is no easy way to do this out of the box. I try to use Redis as much as I can but sometimes memcached is the right tool :)

Also because it wasn't mentioned anywhere, Graham King, whose article at http://www.darkcoding.net/software/memcached-list-all-keys/ helped out here.

@hr1383

This comment has been minimized.

Show comment
Hide comment
@hr1383

hr1383 Mar 19, 2012

very useful !!

hr1383 commented Mar 19, 2012

very useful !!

@fajarmf

This comment has been minimized.

Show comment
Hide comment
@fajarmf

fajarmf May 30, 2012

very useful! thank you :D

fajarmf commented May 30, 2012

very useful! thank you :D

@chriskk

This comment has been minimized.

Show comment
Hide comment
@chriskk

chriskk Jul 24, 2012

Thanks for the code! Used it for a simple command line tool to list and get the value for specific keys. https://gist.github.com/3168935

chriskk commented Jul 24, 2012

Thanks for the code! Used it for a simple command line tool to list and get the value for specific keys. https://gist.github.com/3168935

@reubent

This comment has been minimized.

Show comment
Hide comment
@reubent

reubent Jan 18, 2013

Thank you!

reubent commented Jan 18, 2013

Thank you!

@eabbott

This comment has been minimized.

Show comment
Hide comment
@eabbott

eabbott Feb 15, 2013

The telnet lib for me was (inconsistently) calling the code block several times, updating the block as follows did the trick for me.

slab_ids = []
# The code block will get called for each <insert technical expl>
localhost.cmd("String" => "stats items", "Match" => /^END/) do |c|
  slab_ids = slab_ids + c.scan(/STAT items:(\d+):/).flatten.uniq
end
slab_ids = slab_ids.flatten.uniq

eabbott commented Feb 15, 2013

The telnet lib for me was (inconsistently) calling the code block several times, updating the block as follows did the trick for me.

slab_ids = []
# The code block will get called for each <insert technical expl>
localhost.cmd("String" => "stats items", "Match" => /^END/) do |c|
  slab_ids = slab_ids + c.scan(/STAT items:(\d+):/).flatten.uniq
end
slab_ids = slab_ids.flatten.uniq
@kryptek

This comment has been minimized.

Show comment
Hide comment
@kryptek

kryptek Apr 25, 2013

Very useful little snippet, but I wanted the output to be a little more verbose, as well as retrieve the exact amount of items in a slab. I added the terminal-table gem in order to tabulate the data nicely.

#!/usr/bin/env ruby
require 'net/telnet'
require 'terminal-table'

tbl       = Terminal::Table.new headings: %w(id expires cache_key bytes)
localhost = Net::Telnet::new("Host" => "localhost", "Port" => 11211, "Timeout" => 3)
matches   = localhost.cmd("String" => "stats items", "Match" => /^END/).scan(/STAT items:(\d+):number (\d+)/)

slabs = matches.inject([]) { |items, item| items << Hash[*['id','items'].zip(item).flatten]; items }

slabs.each do |slab|
  localhost.cmd("String" => "stats cachedump #{slab['id']} #{slab['items']}", "Match" => /^END/) do |c|
    matches = c.scan(/^ITEM (.+?) \[(\d+) b; (\d+) s\]$/).each do |key_data|
      cache_key, bytes, expires_time = key_data
      tbl << [slab['id'], Time.at(expires_time.to_i), cache_key, bytes]
    end
  end
end

puts tbl
localhost.close

kryptek commented Apr 25, 2013

Very useful little snippet, but I wanted the output to be a little more verbose, as well as retrieve the exact amount of items in a slab. I added the terminal-table gem in order to tabulate the data nicely.

#!/usr/bin/env ruby
require 'net/telnet'
require 'terminal-table'

tbl       = Terminal::Table.new headings: %w(id expires cache_key bytes)
localhost = Net::Telnet::new("Host" => "localhost", "Port" => 11211, "Timeout" => 3)
matches   = localhost.cmd("String" => "stats items", "Match" => /^END/).scan(/STAT items:(\d+):number (\d+)/)

slabs = matches.inject([]) { |items, item| items << Hash[*['id','items'].zip(item).flatten]; items }

slabs.each do |slab|
  localhost.cmd("String" => "stats cachedump #{slab['id']} #{slab['items']}", "Match" => /^END/) do |c|
    matches = c.scan(/^ITEM (.+?) \[(\d+) b; (\d+) s\]$/).each do |key_data|
      cache_key, bytes, expires_time = key_data
      tbl << [slab['id'], Time.at(expires_time.to_i), cache_key, bytes]
    end
  end
end

puts tbl
localhost.close
@rodolfo42

This comment has been minimized.

Show comment
Hide comment
@rodolfo42

rodolfo42 Jul 26, 2013

Nice script! Very useful indeed.

Thanks

rodolfo42 commented Jul 26, 2013

Nice script! Very useful indeed.

Thanks

@bkimble

This comment has been minimized.

Show comment
Hide comment
@bkimble

bkimble Jul 30, 2013

Thanks for the updates to this old gist of mine. It's amazing how well social coding works sometimes, and how this started from a blurb from Graham King's website!

@kryptek : I like what you've done, sans the gem dependency. I see this guy as a quick and dirty thing you want to throw on a system to figure out what the heck is going on with cache, and it's a distraction to install a gem. I updated the gist with your revision and with baked-in formatting instead :)

If the code block is still getting called multiple times with the new version, please let me know and I'll dig in to it further

Owner

bkimble commented Jul 30, 2013

Thanks for the updates to this old gist of mine. It's amazing how well social coding works sometimes, and how this started from a blurb from Graham King's website!

@kryptek : I like what you've done, sans the gem dependency. I see this guy as a quick and dirty thing you want to throw on a system to figure out what the heck is going on with cache, and it's a distraction to install a gem. I updated the gist with your revision and with baked-in formatting instead :)

If the code block is still getting called multiple times with the new version, please let me know and I'll dig in to it further

@jrumbut

This comment has been minimized.

Show comment
Hide comment
@jrumbut

jrumbut Aug 20, 2013

@bkimble Thanks for the script! I've added some very simple and inefficient content grabbing to it. For my purposes I only want to see the content when it's ASCII encoded, but I've commented that line out.

# List all keys stored in memcache.
# Credit to Graham King at http://www.darkcoding.net/software/memcached-list-all-keys/ for the original article on how to get the data from memcache in the first place.

require 'net/telnet'

headings = %w(id expires bytes cache_key content)
rows = []

localhost = Net::Telnet::new("Host" => "localhost", "Port" => 11211, "Timeout" => 3)
matches   = localhost.cmd("String" => "stats items", "Match" => /^END/).scan(/STAT items:(\d+):number (\d+)/)

slabs = matches.inject([]) { |items, item| items << Hash[*['id','items'].zip(item).flatten]; items }

longest_key_len = 0
longest_content_len = 0
slabs.each do |slab|
  localhost.cmd("String" => "stats cachedump #{slab['id']} #{slab['items']}", "Match" => /^END/) do |c|
    matches = c.scan(/^ITEM (.+?) \[(\d+) b; (\d+) s\]$/).each do |key_data|
      cache_key, bytes, expires_time = key_data
      content = localhost.cmd("String" => "get #{cache_key}", "Match" => /^END/) {|c| c}
      #content = content.ascii_only? ? content : "Not ASCII, whatever it is" #make it a little more pipeline friendly
      rows << [slab['id'], Time.at(expires_time.to_i), bytes, cache_key, content]
      longest_content_len = [longest_content_len, content.length].max
      longest_key_len = [longest_key_len,cache_key.length].max
    end
  end
end

row_format = %Q(|%8s | %28s | %12s | %-#{longest_key_len}s | %-#{longest_content_len}s |)
puts row_format%headings
rows.each{|row| puts row_format%row}

localhost.close

jrumbut commented Aug 20, 2013

@bkimble Thanks for the script! I've added some very simple and inefficient content grabbing to it. For my purposes I only want to see the content when it's ASCII encoded, but I've commented that line out.

# List all keys stored in memcache.
# Credit to Graham King at http://www.darkcoding.net/software/memcached-list-all-keys/ for the original article on how to get the data from memcache in the first place.

require 'net/telnet'

headings = %w(id expires bytes cache_key content)
rows = []

localhost = Net::Telnet::new("Host" => "localhost", "Port" => 11211, "Timeout" => 3)
matches   = localhost.cmd("String" => "stats items", "Match" => /^END/).scan(/STAT items:(\d+):number (\d+)/)

slabs = matches.inject([]) { |items, item| items << Hash[*['id','items'].zip(item).flatten]; items }

longest_key_len = 0
longest_content_len = 0
slabs.each do |slab|
  localhost.cmd("String" => "stats cachedump #{slab['id']} #{slab['items']}", "Match" => /^END/) do |c|
    matches = c.scan(/^ITEM (.+?) \[(\d+) b; (\d+) s\]$/).each do |key_data|
      cache_key, bytes, expires_time = key_data
      content = localhost.cmd("String" => "get #{cache_key}", "Match" => /^END/) {|c| c}
      #content = content.ascii_only? ? content : "Not ASCII, whatever it is" #make it a little more pipeline friendly
      rows << [slab['id'], Time.at(expires_time.to_i), bytes, cache_key, content]
      longest_content_len = [longest_content_len, content.length].max
      longest_key_len = [longest_key_len,cache_key.length].max
    end
  end
end

row_format = %Q(|%8s | %28s | %12s | %-#{longest_key_len}s | %-#{longest_content_len}s |)
puts row_format%headings
rows.each{|row| puts row_format%row}

localhost.close
@tobijb

This comment has been minimized.

Show comment
Hide comment
@tobijb

tobijb commented Aug 26, 2013

Thanks!

@theo-bittencourt

This comment has been minimized.

Show comment
Hide comment
@theo-bittencourt

theo-bittencourt Jan 11, 2014

Someone had to do this in a remote server, like Memcachier?

theo-bittencourt commented Jan 11, 2014

Someone had to do this in a remote server, like Memcachier?

@rajagopals

This comment has been minimized.

Show comment
Hide comment
@rajagopals

rajagopals Jan 28, 2014

Any help on doing this in a remote server would be great!

rajagopals commented Jan 28, 2014

Any help on doing this in a remote server would be great!

@bkimble

This comment has been minimized.

Show comment
Hide comment
@bkimble

bkimble Apr 16, 2014

To do this on a remote server, you'd have a few options. Either deploy this script to your memcache servers and run it there, or just change

localhost = Net::Telnet::new("Host" => "localhost", "Port" => 11211, "Timeout" => 3)

to be the actual host/port of your memcache server. Keep in mind that firewall rules on your server could limit access to memcache in this manner and you may wind up having to put the script on the server anyway.

Owner

bkimble commented Apr 16, 2014

To do this on a remote server, you'd have a few options. Either deploy this script to your memcache servers and run it there, or just change

localhost = Net::Telnet::new("Host" => "localhost", "Port" => 11211, "Timeout" => 3)

to be the actual host/port of your memcache server. Keep in mind that firewall rules on your server could limit access to memcache in this manner and you may wind up having to put the script on the server anyway.

@gfa

This comment has been minimized.

Show comment
Hide comment
@gfa

gfa May 27, 2014

does it list all the keys or most of them?

gfa commented May 27, 2014

does it list all the keys or most of them?

@saxxi

This comment has been minimized.

Show comment
Hide comment
@saxxi

saxxi Jan 5, 2015

As I didn't know about this snippet I created a similar one based on the same source. Please have a look if you find it useful.

require 'net/telnet'
memcached = Net::Telnet::new("Host" => "localhost", "Port" => 11211)
memcached.puts 'stats items'
slab_ids = memcached.waitfor(/./).split("\n").map { |line| x = line[/([^\:]+)\d+/, 0].to_i ; x > 0 ? x : nil }.uniq.compact
slab_ids.inject([]) do |items, slab_id|
  memcached.puts "stats cachedump #{slab_id} 100"
  items << memcached.waitfor(/./).split("\n")[0].split(' ', 3)[1]
  items
end

saxxi commented Jan 5, 2015

As I didn't know about this snippet I created a similar one based on the same source. Please have a look if you find it useful.

require 'net/telnet'
memcached = Net::Telnet::new("Host" => "localhost", "Port" => 11211)
memcached.puts 'stats items'
slab_ids = memcached.waitfor(/./).split("\n").map { |line| x = line[/([^\:]+)\d+/, 0].to_i ; x > 0 ? x : nil }.uniq.compact
slab_ids.inject([]) do |items, slab_id|
  memcached.puts "stats cachedump #{slab_id} 100"
  items << memcached.waitfor(/./).split("\n")[0].split(' ', 3)[1]
  items
end
@saxxi

This comment has been minimized.

Show comment
Hide comment
@saxxi

saxxi Jan 5, 2015

@gfa from the first couple of lines of the source in http://www.darkcoding.net/software/memcached-list-all-keys/:
"In the general case, there is no way to list all the keys that a memcached instance is storing. You can, however, list something like the first 1Meg of keys, which is usually enough during development."

saxxi commented Jan 5, 2015

@gfa from the first couple of lines of the source in http://www.darkcoding.net/software/memcached-list-all-keys/:
"In the general case, there is no way to list all the keys that a memcached instance is storing. You can, however, list something like the first 1Meg of keys, which is usually enough during development."

@ginjo

This comment has been minimized.

Show comment
Hide comment
@ginjo

ginjo Jan 30, 2015

@bkimble thanks for this!

Here's yet another fork of your original gist:
https://gist.github.com/ginjo/aaa4a4395fcdbbad3d29

This revision adds a couple of features for working with cache item rows as ruby objects.

  • Isolates the cache item query and enumeration into a method that yields to an optional block (passing all elements of cache item. You can do anything with the block - print rows, return rows as an array of hashes, return a single 'matching' row, etc..
  • Accepts a list of servers and runs the above enumerator for all items found on every server.

I've included a method that mimics your original formatted printing, plus a method that returns all found cache items as an array of hashes.

ginjo commented Jan 30, 2015

@bkimble thanks for this!

Here's yet another fork of your original gist:
https://gist.github.com/ginjo/aaa4a4395fcdbbad3d29

This revision adds a couple of features for working with cache item rows as ruby objects.

  • Isolates the cache item query and enumeration into a method that yields to an optional block (passing all elements of cache item. You can do anything with the block - print rows, return rows as an array of hashes, return a single 'matching' row, etc..
  • Accepts a list of servers and runs the above enumerator for all items found on every server.

I've included a method that mimics your original formatted printing, plus a method that returns all found cache items as an array of hashes.

@AlBundyLoves69

This comment has been minimized.

Show comment
Hide comment
@AlBundyLoves69

AlBundyLoves69 May 3, 2015

Unfortunately it does not work when the server has several 100 thousand keys...

AlBundyLoves69 commented May 3, 2015

Unfortunately it does not work when the server has several 100 thousand keys...

@rafanasiev

This comment has been minimized.

Show comment
Hide comment
@rafanasiev

rafanasiev Apr 14, 2017

excellent, thanks a lot!

rafanasiev commented Apr 14, 2017

excellent, thanks a lot!

@javanux

This comment has been minimized.

Show comment
Hide comment
@javanux

javanux Aug 18, 2017

Nice 😮 . I have customized this script for dump and restore using dalli gem.
https://gist.github.com/javanux/dbd005ef50ae4b8cea213af058abf771

javanux commented Aug 18, 2017

Nice 😮 . I have customized this script for dump and restore using dalli gem.
https://gist.github.com/javanux/dbd005ef50ae4b8cea213af058abf771

@jsqu99

This comment has been minimized.

Show comment
Hide comment
@jsqu99

jsqu99 Sep 5, 2017

I'm fairly certain this does not work on memcachier.

I successfully telneted in to our production servers and tried issues the stats cachedump command and got:

CLIENT_ERROR unsupported stats command

jsqu99 commented Sep 5, 2017

I'm fairly certain this does not work on memcachier.

I successfully telneted in to our production servers and tried issues the stats cachedump command and got:

CLIENT_ERROR unsupported stats command

@vkdimitrov

This comment has been minimized.

Show comment
Hide comment
@vkdimitrov

vkdimitrov Apr 11, 2018

bingo! 👍

vkdimitrov commented Apr 11, 2018

bingo! 👍

@ralphjay24

This comment has been minimized.

Show comment
Hide comment
@ralphjay24

ralphjay24 Apr 13, 2018

Thank you! This help me to solve problem of infinite caching(no expiration).

ralphjay24 commented Apr 13, 2018

Thank you! This help me to solve problem of infinite caching(no expiration).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment