Skip to content

Instantly share code, notes, and snippets.

Tyler Mitchell 1tylermitchell

Block or report user

Report or block 1tylermitchell

Hide content and notifications from this user.

Learn more about blocking users

Contact Support about this user’s behavior.

Learn more about reporting abuse

Report abuse
View GitHub Profile
@1tylermitchell
1tylermitchell / Rakefile
Created Dec 8, 2015
Partial serverspec Rake file that loops through all nodes in a HDFS cluster automatically
View Rakefile
# Two approaches to getting the list of nodes automatically.
# Some stuff re: roles from serverspec-example repository
...
def hdfsnodes()
# Requires special access to HDFS reporting tool...
clusterreport,nodesreport = `sudo su hdfs -c 'hdfs dfsadmin -report'`.split("\n\n-------------------------------------------------\n")
nodesreplist=nodesreport.strip.split("\n")
nodenamelist = nodesreplist.grep(/Name/)
@1tylermitchell
1tylermitchell / hdfs_cluster_spec.rb
Last active May 9, 2018
Spec file for checking various Hadoop/HDFS cluster settings using serverspec platform
View hdfs_cluster_spec.rb
require 'spec_helper'
require 'nokogiri' # For XML parsing of config files
describe "Processor count" do
# I hate calling out directly from Ruby like this
# but the function: "command('nproc')" was returning strings I couldn't cast
describe `nproc`.to_i do
it { should >= 12 }
end
@1tylermitchell
1tylermitchell / transfer_threads_spec.rb
Last active Nov 18, 2015
Serverspec check Hadoop conf xml file (ruby)
View transfer_threads_spec.rb
require 'spec_helper'
require 'nokogiri'
describe file('/etc/hadoop/conf/hdfs-site.xml') do
it {should be_file}
it {should contain /dfs.datanode.max.transfer.threads/}
it do
doc = Nokogiri::XML(File.read("/etc/hadoop/conf/hdfs-site.xml"))
val=doc.xpath("//property[name='dfs.datanode.max.transfer.threads']/value").text.to_i
expect(val).to be >= 4096
@1tylermitchell
1tylermitchell / memory_spec.rb
Created Oct 8, 2015
serverspec test for total memory on system
View memory_spec.rb
require 'spec_helper'
describe "Node Memory Availability" do
describe host_inventory['memory']['total'].delete('kB').to_i do
it { should > 0 }
end
end
@1tylermitchell
1tylermitchell / graphDensity.rq
Last active Aug 29, 2015
SPARQL query for Graph Density ratio
View graphDensity.rq
SELECT (?nrEdges/(?nrNodes *(?nrNodes - 1.0)) AS ?graphDensity)
FROM <graphName>
WHERE {
{ SELECT (COUNT (*) AS ?nrEdges) (COUNT (DISTINCT ?person) AS ?nrNodes)
WHERE { ?person <knows> ?anotherPerson . }
}
}
You can’t perform that action at this time.