Skip to content

Instantly share code, notes, and snippets.

View 1tylermitchell's full-sized avatar

Tyler Mitchell 1tylermitchell

View GitHub Profile
@1tylermitchell
1tylermitchell / Rakefile
Created December 8, 2015 18:19
Partial serverspec Rake file that loops through all nodes in a HDFS cluster automatically
# Two approaches to getting the list of nodes automatically.
# Some stuff re: roles from serverspec-example repository
...
def hdfsnodes()
# Requires special access to HDFS reporting tool...
clusterreport,nodesreport = `sudo su hdfs -c 'hdfs dfsadmin -report'`.split("\n\n-------------------------------------------------\n")
nodesreplist=nodesreport.strip.split("\n")
nodenamelist = nodesreplist.grep(/Name/)
@1tylermitchell
1tylermitchell / hdfs_cluster_spec.rb
Last active May 9, 2018 23:03
Spec file for checking various Hadoop/HDFS cluster settings using serverspec platform
require 'spec_helper'
require 'nokogiri' # For XML parsing of config files
describe "Processor count" do
# I hate calling out directly from Ruby like this
# but the function: "command('nproc')" was returning strings I couldn't cast
describe `nproc`.to_i do
it { should >= 12 }
end
@1tylermitchell
1tylermitchell / transfer_threads_spec.rb
Last active November 18, 2015 00:43
Serverspec check Hadoop conf xml file (ruby)
require 'spec_helper'
require 'nokogiri'
describe file('/etc/hadoop/conf/hdfs-site.xml') do
it {should be_file}
it {should contain /dfs.datanode.max.transfer.threads/}
it do
doc = Nokogiri::XML(File.read("/etc/hadoop/conf/hdfs-site.xml"))
val=doc.xpath("//property[name='dfs.datanode.max.transfer.threads']/value").text.to_i
expect(val).to be >= 4096
@1tylermitchell
1tylermitchell / memory_spec.rb
Created October 8, 2015 22:10
serverspec test for total memory on system
require 'spec_helper'
describe "Node Memory Availability" do
describe host_inventory['memory']['total'].delete('kB').to_i do
it { should > 0 }
end
end
@1tylermitchell
1tylermitchell / graphDensity.rq
Last active August 29, 2015 14:15
SPARQL query for Graph Density ratio
SELECT (?nrEdges/(?nrNodes *(?nrNodes - 1.0)) AS ?graphDensity)
FROM <graphName>
WHERE {
{ SELECT (COUNT (*) AS ?nrEdges) (COUNT (DISTINCT ?person) AS ?nrNodes)
WHERE { ?person <knows> ?anotherPerson . }
}
}