mongodb replica set practise on AWS EC2

gistfile1.txt

Mongo replica set on  EC2 directly

Reference

Reference

https://eladnava.com/deploy-a-highly-available-mongodb-replica-set-on-aws/

https://gist.github.com/leommoore/309de7c0042ed697ee84

Other references

1 on single EC2 instance

https://mongodb-documentation.readthedocs.io/en/latest/ecosystem/tutorial/install-mongodb-on-amazon-ec2.html

http://www.tothenew.com/blog/how-to-setup-replica-sets-in-mongo-db-aws-ec2/

2 Quick Start with AWS CloudFormation

Access MongoDB replica set
https://docs.mongodb.com/manual/core/read-preference/
http://api.mongodb.com/python/current/api/pymongo/read_preferences.html
http://api.mongodb.com/python/current/examples/high_availability.html

reference marketplace cases or others

Todo

1. Deploy

• Distribute 3 EC2 nodes on 3 zones of one VPC for production environment
• Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
  
  See http://dochub.mongodb.org/core/prodnotes-filesystem

2. Security:

• DB User and password, authority. Access control is not enabled for the database. Read and write access to data and configuration is unrestricted.
  
• Customized EC2 instance security group
• Customized EC2 instance IAM group
• Let Mongo listen to instance's INTERNAL IP address if portal and other clusters (in Docker container) can access it
  
• Review private and public DNS. At present DNS name are created by ' elastic ip ' .
  
  Use customized DNS and change hostname by ec2 metadata if DNS can be customized in advance.
• The security group is open by far. Adjust security group. Maintain Mongo replica set from' sshhop' EC2 instance only.

3. Storage:

• Able to add more volume

4. Monitor: need to set up tooling to constantly monitor replication lag, CPU usage, disk space utilization, Network and Performance
5. Upgrade
6. Scalable your - Add more member to the replica set

• Upgrade to sharding cluster

7. Backup and restore
   periodically back up databases to prevent data loss

8. migration

9. Automation deployment. Note those no-idempotency update operation
   Options

• Bash command and Restful API
• Terraform/ AWS Cloud platform/ Ansible
• ?

Issues

The root reason of losing data after reboot EC2 instance

Deploy using Mongo Community Edition 3.4.4

Deploy using Mongo Community Edition 3.4.4 on EC2 instances

Step A> Create EC2 instances

Issues:

• With the aws ec2 command to create a security group, the created security group is not in Oregon;
  
• With the aws ec2 command to create a EC2 instance, failed to find the VPC in Oregon.
  
  Workaround: from the page to create 4 EC2 instances as Primary, second-1, second-2, arbitrator.
  
• lost data after reboot.
  
  Note DO NOT reboot EC2, As OS configurations and the Mongo and its configurations, business data should not be lost after reboot.
  
  
  

Configuration options:

• OS:
  
  Amazon Linux AMI 2017.03.0 (HVM), SSD Volume Type - ami-4836a428. It is an EBS-backed,
  
  Type: M3 or M4 would be a good choice since it provides steady compute resources, or even go for the memory-optimized r3.*
  
• Storage:
  
  Prepare 3 extra volumes for Mongo data 100 G, journal 25 G and log 9.8 G for for beta environment
  
• Configure EC2 never to terminate. terminal will lose everything, include pem file. These EC2 instances should be stateful.
  

Todo :

• Review private and public DNS. At present DNS name are created by ' elastic ip ' .
  
  If DNS can be customized in advance. use customized DNS and change hostname by ec2 metadata
  

* Security group is open by far. Adjust security group. Maintain Mongo replica set from' sshhop' EC2 instance only.
Reference

"A common approach is to create a MongoDB security group that contains the nodes of your cluster (replica set members or sharded cluster members), followed by the creation of a separate security group for your app servers or clients.
Create a rule in your MongoDB security group with the “source” field set to the Security Group name containing your app servers and the port set to 27017 (or whatever port you use for your MongoDB). This will ensure that only your app servers have permission to connect to your MongoDB instances.
Remember that Security Groups only control ingress traffic."
https://docs.mongodb.com/ecosystem/platforms/amazon-ec2/
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-network-security.html

* Customized IAM group Reference
https://docs.mongodb.com/ecosystem/platforms/amazon-ec2/#deploy-mongodb-ec2
https://docs.mongodb.com/manual/tutorial/install-mongodb-on-amazon/

Step B> Configuration on each EC2 Instance

1> Install Mongo 3.4.4 for each EC2 instance

$ echo "[mongodb-org-3.4] 
name=MongoDB Repository 
baseurl=https://repo.mongodb.org/yum/amazon/2013.03/mongodb-org/3.4/x86_64/
gpgcheck=1
enabled=1
gpgkey=https://www.mongodb.org/static/pgp/server-3.4.asc" | sudo tee  /etc/yum.repos.d/mongodb-org-3.4.repo
$ sudo yum install -y mongodb-org

verify mongod user and group and mongod configuration path

$ cat /etc/passwd |grep mongod
$ cat /etc/mongod.conf 

2> Configure each EC2 instance - Storage

Issues :

The device name is not same as the volume name

Reference:

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-using-volumes.html

http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/device_naming.html

http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance-booting-from-wrong-volume.html

$ sudo mkdir /data /log /journal
$ sudo chmod -R 755  /data /log /journal
$ sudo chown -R mongod:mongod /data /journal /log

$ sudo mkfs.ext4 /dev/sdb
$ sudo mkfs.ext4 /dev/sdc
$ sudo mkfs.ext4 /dev/sdd
 
$ echo '/dev/sdb /data/db ext4 defaults,auto,noatime,noexec 0 0
/dev/sdc /journal ext4 defaults,auto,noatime,noexec 0 0
/dev/sdd /log ext4 defaults,auto,noatime,noexec 0 0' | sudo tee -a  /etc/fstab
# this is not idempotent 

$ sudo mount /data
$ sudo mount /journal
$ sudo mount /log

$ sudo chmod -R 755  /data /log /journal
$ sudo chown -R mongod:mongod /data /journal /log

$ sudo ln -s /journal /data/journal
$ sudo touch /log/mongod.log && sudo chmod  755  /log/mongod.log && sudo chown   mongod:mongod   /log/mongod.log 
$ sudo ln -s /log/mongod.log   /var/log/mongodb/mongod.log 

Verify

$ ll /data/
lrwxrwxrwx 1 mongod mongod     8 May 23 16:35 journal -> /journal
$ ll /var/log/mongodb/mongod.log 
lrwxrwxrwx 1 root root 15 May 23 22:20 /var/log/mongodb/mongod.log -> /log/mongod.log
$ df -f 

Issue:

• Find many /journal path.
  
  Not clear the root reason
  it does not always occur
  
• Wrong error report information about ' /data/db' in log of Mongod. But there is not this issue when Mongo does not use ESB.
  
  The root reason is /log/mongod.log.
  
  Fixed by mount the device to default log file path. and link the log file to the one specified in Mongod configuration file.
  
• Reboot EC2 and lost data: 2 nodes lost all configuration data. 1 node only lost the Mongo DB.
  
  Not clear the root reason
  

3> Configure each EC2 instance - configure Mongod storage

editing the /etc/mongod.conf

sudo vim /etc/mongod.conf

# where to write logging data.
systemLog:
  destination: file
  logAppend: true
  path: /var/log/mongodb/mongod.log
 
# Where and how to store data.
storage:
#  dbPath: /var/lib/mongo
   dbPath: /data 

4> Configure each EC2 instance - ulimit settings

Reference https://access.redhat.com/solutions/199993

$ echo '* soft nofile 64000
* hard nofile 64000
* soft nproc 64000
* hard nproc 64000' | sudo tee /etc/security/limits.d/90-mongodb.conf
# verify
$ cat  /etc/security/limits.d/90-mongodb.conf 
* soft nofile 64000
* hard nofile 64000
* soft nproc 64000
* hard nproc 64000

5> Configure each EC2 instance - read ahead settings

Reference https://www.percona.com/blog/2016/08/12/tuning-linux-for-mongodb/

sudo blockdev --setra 32  /dev/sdb  
sudo blockdev --setra 32  /dev/sdc 
sudo blockdev --setra 32  /dev/sdd  
echo 'ACTION=="add|change", KERNEL=="sdb", ATTR{bdi/read_ahead_kb}="16"' | sudo tee -a /etc/udev/rules.d/85-ebs.rules  &&
echo 'ACTION=="add|change", KERNEL=="sdc", ATTR{bdi/read_ahead_kb}="16"' | sudo tee -a /etc/udev/rules.d/85-ebs.rules  &&
echo 'ACTION=="add|change", KERNEL=="sdd", ATTR{bdi/read_ahead_kb}="16"' | sudo tee -a /etc/udev/rules.d/85-ebs.rules

6> Configure each EC2 instance - keep alive settings

#update keep alive 

$ sudo sysctl -w net.ipv4.tcp_keepalive_time=3000

7> Configure each EC2 instance - set hostname

#set hostname 
 
$ echo `hostname -A`  | sudo tee -a /etc/hostname && sudo hostname -F /etc/hostname  

8> Configure each EC2 instance - Disable Transparent Huge Pages

echo '#!/bin/sh
### BEGIN INIT INFO
# Provides:          disable-transparent-hugepages
# Required-Start:    $local_fs
# Required-Stop:
# X-Start-Before:    mongod mongodb-mms-automation-agent
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Disable Linux transparent huge pages
# Description:       Disable Linux transparent huge pages, to improve
#                    database performance.
### END INIT INFO
case $1 in
  start)
    if [ -d /sys/kernel/mm/transparent_hugepage ]; then
      thp_path=/sys/kernel/mm/transparent_hugepage
    elif [ -d /sys/kernel/mm/redhat_transparent_hugepage ]; then
      thp_path=/sys/kernel/mm/redhat_transparent_hugepage
    else
      return 0
    fi

    echo 'never' > ${thp_path}/enabled
    echo 'never' > ${thp_path}/defrag

    unset thp_path
    ;;
esac' | sudo tee /etc/init.d/disable-transparent-hugepages && sudo chmod 755 /etc/init.d/disable-transparent-hugepages && sudo chkconfig --add disable-transparent-hugepages

echo 'never' | sudo tee   /sys/kernel/mm/transparent_hugepage/enabled && echo 'never' | sudo tee   /sys/kernel/mm/transparent_hugepage/defrag 

9> Configure each EC2 instance - configure Mongo replica se name and bindIp

sudo nano /etc/mongod.conf

 
#  bindIp: 127.0.0.1  # Listen to local interface only, comment to listen on all interfaces.
#replication:
replication:
 replSetName: "ft-mongo-replica-set"

By default the database will now be listening on port 27017.

The web administrative UI will be on port 28017.

bindIp: 127.0.0.1 -> bindIp: 0.0.0.0 enable access Mongo using the public IP of your ec2 instance.

Todo :

replace 127.0.0.1 with instance's INTERNAL IP address if portal and other cluster (in Docker container) can access it

10> Configure each EC2 instance - start Mongod

$ sudo service mongod restart

Step C> Configure Mongo replica set

Login any EC2 instance and run Mongo shell

[ec2-user@ip-172-31-4-46 /]$ mongo

or from another EC2 instance run mongo shell to connect to any Mongo EC2 instance

ubuntu@ip-172-31-8-191:~$ mongo 172.31.4.46

 > rs.initiate()
{
	"info2" : "no configuration specified. Using a default configuration for the set",
	"me" : "ip-172-31-4-46.us-west-2.compute.internal:27017",
	"ok" : 1
}
 > rs.add("ip-172-31-13-124.us-west-2.compute.internal")
{ "ok" : 1 }
 > rs.add("ip-172-31-13-173.us-west-2.compute.internal")
{ "ok" : 1 }
 > rs.add("ip-172-31-2-3.us-west-2.compute.internal",true)
{ "ok" : 1 }
 > rs.status()
 > rs.config().members[0].priority = 10
 > rs.config()

Detail ouput http://paste.openstack.org/show/611328/

Reference

https://mongodb-documentation.readthedocs.io/en/latest/ecosystem/tutorial/install-mongodb-on-amazon-ec2.html

https://docs.mongodb.com/manual/administration/replica-sets/

https://docs.mongodb.com/manual/replication/

Step D> Test Mongo replica set

//Todo