Note: At the time of writing this, I have used MongoDB v4.2
- Create and run MongoDB in EC2 Instance with Basic Config
- Making EC2 proper for MongoDB Database
- Adding Replicas
- Create EC2 instance. (As per this link - C4, M4, and D2 instance family types will perform better for MongoDB Server choices which needs fast storage unlike the other compute type use cases.)
- While creating EC2 instance, create separate EBS volumes for storing data, logs and the journal for improved performance. [As per this and this link references].
Copying from here:
Add three additional EBS volumes to setup the correct file system for MongoDB
- Device:
/dev/sdf, Size: x GB (put desired size) [For data]- Device:
/dev/sdg, Size: x GB (put desired size) [For journal]- Device:
/dev/sdh, Size: x GB (put desired size) [For log]
It’s recommended that journal is 1/5 of data size and log is 1/2 of journal size. You can always increase the EBS size later as needed but decrease. Put rest in default or to your needs.
- Create or use existing key value pair.
- Store the key value pair .pem file in a secured directory.
- Modify permission for the .pem file
To allow Only read and restrict write and execute permissions.
chmod 400 KeyValuPairName.pem
-
Connect with EC2 instance through SSH Client
ssh -i "KeyValuPairName.pem" ec2-user@ipaddress -
Install MongoDB in EC2 instance Follow the steps from MongoDB documentation to install MongoDB in EC2
-
Start MongoDB & Also, configure MongoDB to start automatically on reboot
sudo chkconfig mongod on
- Connect through Compass
- We can see the list of databases we have and we can create database, collection, document from Compass.
- We do not need to make any changes to config, admin, local databases. We can create a new database.
- Connect programmatically:
To connect programmatically using NodeJS,
- Just create a new simple node js project by creating a new folder,
npm init, install Mongoose npm package & run use the followign code snippet to connect with MongoDB from NodeJS.
const mongoose = require('mongoose');
const uri = 'mongodb://ipaddress:27017/databasename'
mongoose.connect(uri, {
useNewUrlParser: true,
useUnifiedTopology: true
}).then(() => {
console.log('done')
})
// logging mongoDb connection
mongoose.connection.on('connected', async function() {
console.log('MongoDB connected @ ' + uri);
});
mongoose.connection.on('error', function(err) {
console.log('MongoDB error: ' + err);
});
- Create Security Group
- Before we run the above snippet, we have to configure the security group for port 27017. Otherwise, we will get an error
MongoTimeoutError: Server selection timed out after 30000 ms
- Modify mongod.config to allow programmatic access
- After creating the security group, we have to make a change in
/etc/mongod.conffile. Otherwise we will get an errorMongoNetworkError: connect ECONNREFUSED **.**.**.**:27017 - Make changes to the
/etc/mongod.conffile usingsudo nano /etc/mongod.conf - Modify the
net:configuration to allow all ip address to access.
Note: This has to be changed later to specific ip addresses.
- Now we will be able to access the db from our NodeJS code without authentication.
- Create an admin user To allow only authenticated users to read and write into database, we have to create admin user first & we have to create database user.
- To create admin user with root permission, Go to mongo shell and
use admin& create admin user using the following code,db.createUser({ user: "admin", pwd: "password", roles: ["root"] }) - Come out from the mongo shell.
- Enable Security in mongod.config
After creating the admin user in admin database, we can go ahead and change the security config in
/ect/mongod.conffile. The settings should like the following,
Only Update security in MongoDB config after successfully created an admin user and if you don’t plan to use replica set. If we plan to use replica set, then we need a different configuration to be added here.
- Create database user
- Now we can cerate a database specific user by going to mongo shell and
use databasename& by using the following code,
db.createUser(
{
user: "username",
pwd: password,
roles: [ { role: "readWrite", db: "databasename" }]
}
)
- After creating a database user, in our NodeJS code we can use the following connection string to connect with the database to do read and write.
const uri = 'mongodb://username:password@ipaddress:27017/databasename'
- Also, we can use the following settings to access the database using username and password through Compass.
As we have added additional three EBS volumes to setup the correct file system for MongoDB in the EC2 Creation step (The first step), Now we have to Configure the file system,
Mount each volume and set ownership, (Copied from here)
sudo mkfs.xfs -L mongodata /dev/sdf
sudo mkfs.xfs -L mongojournal /dev/sdg
sudo mkfs.xfs -L mongolog /dev/sdh
sudo mkdir /data
sudo mkdir /journal
sudo mkdir /log
sudo mount -t xfs /dev/sdf /data
sudo mount -t xfs /dev/sdg /journal
sudo mount -t xfs /dev/sdh /log
sudo ln -s /journal /data/journal
sudo chown mongod:mongod /data
sudo chown mongod:mongod /log/
sudo chown mongod:mongod /journal/
sudo vi /etc/fstab
/dev/sdf /data xfs defaults,auto,noatime,noexec 0 0
/dev/sdg /journal xfs defaults,auto,noatime,noexec 0 0
/dev/sdh /log xfs defaults,auto,noatime,noexec 0 0
sudo vi /etc/security/limits.conf
* soft nofile 64000
* hard nofile 64000
* soft nproc 32000
* hard nproc 32000
sudo vi /etc/security/limits.d/90-nproc.conf
Paste the following lines into the file:
* soft nproc 32000
* hard nproc 32000
Confirm disks have mounted properly
df -h
You should see something similar like below:
Filesystem Size Used Avail Use% Mounted on
devtmpfs 479M 0 479M 0% /dev
tmpfs 494M 0 494M 0% /dev/shm
tmpfs 494M 13M 482M 3% /run
tmpfs 494M 0 494M 0% /sys/fs/cgroup
/dev/xvda1 8.0G 1.3G 6.8G 16% /
tmpfs 99M 0 99M 0% /run/user/1000
/dev/xvdf 20G 33M 20G 1% /data
/dev/xvdg 4.0G 33M 4.0G 1% /journal
/dev/xvdh 2.0G 33M 2.0G 2% /log
Create the following file
sudo vi /etc/init.d/disable-transparent-hugepages
Copy/paste the following code to the file
#!/bin/bash
### BEGIN INIT INFO
# Provides: disable-transparent-hugepages
# Required-Start: $local_fs
# Required-Stop:
# X-Start-Before: mongod mongodb-mms-automation-agent
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: Disable Linux transparent huge pages
# Description: Disable Linux transparent huge pages, to improve
# database performance.
### END INIT INFOcase $1 in
start)
if [ -d /sys/kernel/mm/transparent_hugepage ]; then
thp_path=/sys/kernel/mm/transparent_hugepage
elif [ -d /sys/kernel/mm/redhat_transparent_hugepage ]; then
thp_path=/sys/kernel/mm/redhat_transparent_hugepage
else
return 0
fiecho 'never' > ${thp_path}/enabled
echo 'never' > ${thp_path}/defragre='^[0-1]+$'
if [[ $(cat ${thp_path}/khugepaged/defrag) =~ $re ]]
then
# RHEL 7
echo 0 > ${thp_path}/khugepaged/defrag
else
# RHEL 6
echo 'no' > ${thp_path}/khugepaged/defrag
fiunset re
unset thp_path
;;
esac
Make it executable
sudo chmod 755 /etc/init.d/disable-transparent-hugepages
Create log rotation rule
sudo vi /etc/logrotate.d/mongodb
Copy/paste the following code to the file
/var/log/mongodb/*.log {
daily
rotate 30
compress
dateext
missingok
notifempty
sharedscripts
postrotate
/bin/kill -SIGUSR1 `cat /media/data/mongodb/mongod.lock 2> /dev/null` 2> /dev/null || true
endscript
For log rotation best practice - Refer this link
Tip: You can test log rotation after MongoDB is setup by running: _sudo logrotate -v -f /etc/logrotate.d/mongodb_
Update MongoDB config
sudo vi /etc/mongod.conf
Update the values
systemLog:
logRotate: reopen # add this line
path: /log/mongod.log # update
Steps invloved:
- Repeat the EC2 creation and configurations for 2 more instances
- Make sure, all instances are using the same security group
- Make sure, in the mongod.conf file - we are not using bindIpAll: true under net: (Remaining steps we can follow from the reference link of "Steps to Install MongoDB on AWS EC2 Instance & Adding Replicas")