Skip to content

Instantly share code, notes, and snippets.

@justfalter

justfalter/install_cifv1_centos6_with_selinux.md

Created January 3, 2014 23:05
Show Gist options
  • Star 2 You must be signed in to star a gist
  • Fork 1 You must be signed in to fork a gist
  • Save justfalter/8248433 to your computer and use it in GitHub Desktop.
Save justfalter/8248433 to your computer and use it in GitHub Desktop.
Download ZIP
install_cifv1_centos6_with_selinux
Raw
install_cifv1_centos6_with_selinux.md

  1. Install dependencies

    sudo yum install sudo rng-tools postgresql-server httpd httpd-devel mod_ssl gcc make expat expat-devel mod_perl mod_perl-devel perl-Digest-SHA perl-Digest-SHA libxml2 libxml2-devel perl-XML-LibXML uuid-perl perl-DBD-Pg bind bind-utils perl-JSON rsync perl-Unicode-String perl-Config-Simple perl-Module-Pluggable perl-MIME-Lite perl-CPAN perl-Class-Accessor perl-YAML perl-XML-Parser uuid uuid-devel uuid-perl perl-Net-DNS perl-DateTime-Format-DateParse perl-IO-Socket-INET6 openssl-devel perl-Module-Install wget perl-Net-SSLeay perl-Class-Trigger perl-Date-Manip libuuid-devel

wget http://dl.fedoraproject.org/pub/epel/6/x86_64/libapreq2-2.13-1.el6.x86_64.rpm
wget http://dl.fedoraproject.org/pub/epel/6/x86_64/libapreq2-devel-2.13-1.el6.x86_64.rpm
wget http://dl.fedoraproject.org/pub/epel/6/x86_64/perl-libapreq2-2.13-1.el6.x86_64.rpm
sudo rpm -i libapreq2-2.13-1.el6.x86_64.rpm libapreq2-devel-2.13-1.el6.x86_64.rpm perl-libapreq2-2.13-1.el6.x86_64.rpm

  2. Install ZeroMQ

    sudo yum install glib2-devel e2fsprogs-devel libuuid-devel gcc-c++
wget http://download.zeromq.org/zeromq-2.1.9.tar.gz
tar xvzf zeromq-2.1.9.tar.gz
cd zeromq-2.1.9
sed -i "s/libzmq_werror=\"yes\"/libzmq_werror=\"no\"/g" configure
rm -fv $(find foreign -type f | grep -v Makefile)
./configure --disable-static --prefix=/usr --libdir=/usr/lib64
make
sudo make install
cd ..

  3. Install perl dependencies

    sudo PERL_MM_USE_DEFAULT=1 perl -MCPAN -e 'install Net::Abuse::Utils, Linux::Cpuinfo, Google::ProtocolBuffers, Iodef::Pb::Simple, Compress::Snappy, Net::Abuse::Utils::Spamhaus, Net::DNS::Match, Snort::Rule, Parse::Range, Log::Dispatch, ZeroMQ, Sys::MemInfo, JSON::XS, File::Type, LWP::UserAgent, Class::Trigger, LWP::Protocol::https, Class::DBI, Net::Patricia, Text::Table, Regexp::Common::net, Regexp::Common::net::CIDR, Text::CSV, XML::RSS, Time::HiRes, LWPx::ParanoidAgent'

  4. Install Net::SSLeay 1.49

    wget http://search.cpan.org/CPAN/authors/id/M/MI/MIKEM/Net-SSLeay-1.49.tar.gz
tar -zxvf Net-SSLeay-1.49.tar.gz
cd Net-SSLeay-1.49
PERL_MM_USE_DEFAULT=1 perl Makefile.PL
sudo make install
cd ../

  5. Install IO::Socket::INET6 2.69

    wget http://search.cpan.org/CPAN/authors/id/S/SH/SHLOMIF/IO-Socket-INET6-2.69.tar.gz
tar -zxvf IO-Socket-INET6-2.69.tar.gz
cd IO-Socket-INET6-2.69
PERL_MM_USE_DEFAULT=1 perl Makefile.PL && sudo make install
cd ../

  6. Configure eth0 startup to use local resolver 127.0.0.1

    echo "DNS1=127.0.01" >> /etc/sysconfig/network-scripts/ifcfg-eth0
sudo service network restart

  7. Configure Postgresql

    sudo service postgresql initdb
sudo mkdir -p /etc/postgresql/8.4/main
sudo chown -R postgres:postgres /etc/postgresql
sudo chmod 760 -R /etc/postgresql
sudo ln -sf /var/lib/pgsql/data/postgresql.conf /etc/postgresql/8.4/main/postgresql.conf
sudo ln -sf /var/lib/pgsql/data/pg_hba.conf /etc/postgresql/8.4/main/pg_hba.conf
sudo service postgresql start

  8. Add cif user

    sudo adduser cif
sudo chmod 770 /home/cif

  9. Apache Stuff

    sudo usermod -a -G cif apache

  10. Set everything to start up on boot.

sudo chkconfig --levels 345 postgresql on
sudo chkconfig --levels 345 named on
sudo chkconfig --levels 345 rngd on
sudo chkconfig --levels 345 httpd on

  1. Configure bind https://code.google.com/p/collective-intelligence-framework/wiki/BindSetup_v1

  2. Set up postgresql following https://code.google.com/p/collective-intelligence-framework/wiki/PostgresInstall_v1

mkdir /var/lib/pgsql/data/cif
mkdir /var/lib/pgsql/data/cif/archive
mkdir /var/lib/pgsql/data/cif/index
sudo chown postgres:postgres /var/lib/pgsql/data/cif
  1. Edit postgresql authentication config
sudo vi /etc/postgresql/8.4/main/pg_hba.conf
  1. Performance configuration options for postgresql.
export page_size=`getconf PAGE_SIZE`
export phys_pages=`getconf _PHYS_PAGES`
export shmall=`expr $phys_pages / 2`
export shmmax=`expr $shmall \* $page_size`
echo kernel.shmmax = $shmmax | sudo tee -a /etc/sysctl.conf
echo kernel.shmall = $shmall | sudo tee -a /etc/sysctl.conf
echo vm.overcommit_memory = 2 | sudo tee -a /etc/sysctl.conf
echo vm.swappiness = 0 | sudo tee -a /etc/sysctl.conf
sudo /sbin/sysctl -p

sudo sed -i 's/shared_buffers/#shared_buffers/' /etc/postgresql/8.4/main/postgresql.conf
sudo sed -i 's/max_connections/#max_connections/' /etc/postgresql/8.4/main/postgresql.conf

export total_ram_b=`expr $page_size \* $phys_pages`
export total_ram_kb=`expr $total_ram_b / 1024`
export total_ram_mb=`expr $total_ram_kb / 1024`
export ten_percent_total_ram=`expr $total_ram_mb / 10`
export work_mem=`expr $total_ram_mb / 8`
export shared_buffers=$ten_percent_total_ram
export effective_cache_size=`expr $ten_percent_total_ram \* 6`

echo ""| sudo tee -a /etc/postgresql/8.4/main/postgresql.conf
echo ""| sudo tee -a /etc/postgresql/8.4/main/postgresql.conf
echo "#------------------------------------------------------------------------------"| sudo tee -a /etc/postgresql/8.4/main/postgresql.conf
echo "# CIF Setup                                                                    "| sudo tee -a /etc/postgresql/8.4/main/postgresql.conf
echo "#------------------------------------------------------------------------------"| sudo tee -a /etc/postgresql/8.4/main/postgresql.conf
echo "# Rough estimates on how to configured postgres to work with large data sets"| sudo tee -a /etc/postgresql/8.4/main/postgresql.conf
echo "# See the following URL for proper postgres performance tuning"| sudo tee -a /etc/postgresql/8.4/main/postgresql.conf
echo "# http://wiki.postgresql.org/wiki/Tuning_Your_PostgreSQL_Server"| sudo tee -a /etc/postgresql/8.4/main/postgresql.conf
echo "wal_buffers = 12MB" " # recommended range for this value is between 2-16MB"| sudo tee -a /etc/postgresql/8.4/main/postgresql.conf
echo "work_mem = $work_mem""MB" " # minimum 512MB needed for cif_feed"| sudo tee -a /etc/postgresql/8.4/main/postgresql.conf
echo "shared_buffers = $shared_buffers""MB" "# recommended range for this value is 10% on shared db server"| sudo tee -a /etc/postgresql/8.4/main/postgresql.conf
echo "checkpoint_segments = 10" " # at least 10, 32 is a more common value on dedicated server class hardware"| sudo tee -a /etc/postgresql/8.4/main/postgresql.conf
echo "effective_cache_size = $effective_cache_size""MB" " # recommended range for this value is between 60%-80% of your total available RAM"| sudo tee -a /etc/postgresql/8.4/main/postgresql.conf
echo "max_connections = 8" " # limiting to 8 due to high work_mem value"| sudo tee -a /etc/postgresql/8.4/main/postgresql.conf

sudo service postgresql restart
  1. Building and installing CIF
wget "https://github.com/collectiveintel/cif-v1/releases/download/v1.0.1-FINAL/cif-v1-v1.0.1-FINAL.tar.gz" -O cif-v1-v1.0.1-FINAL.tar.gz
tar xvzf cif-v1-v1.0.1-FINAL.tar.gz
cd cif-v1-v1.0.1-FINAL
./configure  --with-db-index-location=/var/lib/pgsql/data/cif/index --with-db-archive-location=/var/lib/pgsql/data/cif/archive
make testdeps
make install
make initdb
  1. Put this into /etc/cif.conf
# the simple stuff
# cif_archive configuration is required by cif-router, cif_feed (cif-router, libcif-dbi)
[cif_archive]
# if we want to enable rir/asn/cc, etc... they take up more space in our repo
# datatypes = infrastructure,domain,url,email,search,malware,cc,asn,rir
datatypes = infrastructure,domain,url,email,search,malware

# if you're going to enable feeds
# feeds = infrastructure,domain,url,email,search,malware

# enable your own groups is you start doing data-sharing with various groups
#groups = everyone,group1.example.com,group2.example.com,group3.example.com

# client is required by the client, cif_router, cif_smrt (libcif, cif-router, cif-smrt)
[client]
# the apikey for your client
apikey = XXXXXX-XXX-XXXX

[client_http]
host = https://localhost:443/api
verify_tls = 0

# cif_smrt is required by cif_smrt
[cif_smrt]
# change example.com to your local domain and hostname respectively
# this identifies the data in your instance and ties it to your specific instance in the event
# that you start sharing with others
#name = example.com
#instance = cif.example.com
name = localhost
instance = cif.localhost

# the apikey for cif_smrt
apikey = XXXXXX-XXX-XXXX 

# advanced stuff
# db config is required by cif-router, cif_feed, cif_apikeys (cif-router, libcif-dbi)
[db]
host = 127.0.0.1
user = postgres
password =
database = cif

# if the normal IODEF restriction classes don't fit your needs
# ref: https://code.google.com/p/collective-intelligence-framework/wiki/RestrictionMapping_v1
# restriction map is required by cif-router, cif_feed (cif-router, libcif-dbi)

[restriction_map]
#need-to-know = amber
#private = red
#default = amber
#public = green     

# logging
# values 0-4
[router]
# set to 0 if it's too noisy and reload the cif-router (apache), only on for RC2
debug = 1
  1. Create your cif user, and cif_smrt users, and add their apikeys to /etc/cif.conf
cif_apikeys -u mike -a -g everyone -G everyone
cif_apikeys -u cif_smrt -a -g everyone -G everyone -w
  1. Configure Apache 2 according to:h ttps://code.google.com/p/collective-intelligence-framework/wiki/ServerInstall_CentOS6_v1#CIF_Router_Configuration_(Apache)

  • Allow Apache CGI to establish network connections to the database

    setsebool -P httpd_can_network_connect_db on

  • Make this change to /etc/httpd/conf.d/cif.conf so that it works with SELinux

    PerlSetVar CIFRouterConfig "/etc/cif.conf"

  • Create /home/cif/.cif (just a link to /etc/cif.conf)

    ln -s /etc/cif.conf /home/cif/.cif
chown cif:cif /home/cif/.cif
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment