Mongo Cluster Migration Assistant in Go

retrieval_script.md

Mongodb Node IP Retrieval Tool in Go

I was using the migrate-mongo npm package to create indexes on a 3 node mongo cluster running in AWS. The package required IP addresses to be placed in the config.js file for the migration to locate the remote cluster. To accomplish this I created a go script to retrieve the IP addresses and update the config file for migrations. The script will use the .aws credential file on the machine its running on.

About The Migration Library

Migration Package Structure

The migration package has the following structure

MigrationDirectory
├── DB_A
│   ├── config.js
│   └── migrations
│   	└── migrationFoo.js
├── DB_B
│   ├── config.js
│   └── migrations
│   	└── migrationBar.js

There is a Directory for each DB to be migrated, and in each DB directory there is a config file and a migrations directory.

Migrations Directory

The migration directory contains files with the migration to be performed, indexes to be created.

Config.js

Is the file we need to modify to perform migrations, this file looks something like

'use strict';

// In this file you can configure migrate-mongo

module.exports = {

  mongodb: {

    // TODO Change (or review) the url to your MongoDB:
	url: 'mongodb://{user}:{password}@{Node1IP}:27017,{Node2IP}:27017,{Node3IP}:27017/{DB}?replicaSet={replicaSetIdentifier}',

    // TODO Change this to your database name:
    databaseName: "{databaseName}",

    options: {
      useNewUrlParser: true, // removes a deprecation warning when connecting
    }
  },

  // The migrations dir, can be an relative or absolute path. Only edit this when really necessary.
  migrationsDir: 'migrations',

  // The mongodb collection where the applied changes are stored. Only edit this when really necessary.
  changelogCollectionName: 'changelog',

};

My Script

package main

import (
	"fmt"
	"io/ioutil"
	"log"
	"os"
	"strings"

	"github.com/aws/aws-sdk-go/aws"
	"github.com/aws/aws-sdk-go/aws/session"
	"github.com/aws/aws-sdk-go/service/cloudformation"
)

func readFile(fileToRead string) []byte {
	file, err := ioutil.ReadFile(fileToRead)
	if err != nil {
		log.Fatalln(err)
		os.Exit(1)
	}
	return file
}

func fileParser(file []byte, primaryIP string, secondaryIP0 string, secondaryIP1 string, replSet string) string {
	lines := strings.Split(string(file), "\n")

	for i, line := range lines {
		if strings.Contains(line, "url:") {
			lines[i] = fmt.Sprintf("url: 'mongodb://XXXUSERNAMEXXX:XXXPASSXXX@%s:27017,%s:27017,%s:27017/admin?replicaSet=%s',", primaryIP,
				secondaryIP0,
				secondaryIP1,
				replSet)
		}
	}
	return strings.Join(lines, "\n")
}

func fileWriter(updatedFile string) {
	err := ioutil.WriteFile("config.js", []byte(updatedFile), 0644)

	if err != nil {
		log.Fatalln(err)
	}
}

func getStackOutputValue(stack *cloudformation.Stack, outputKey string) (*string, error) {
	for _, o := range stack.Outputs {
		if *o.OutputKey == outputKey {
			return o.OutputValue, nil
		}
	}

	return nil, fmt.Errorf("Key=%s is not part of the stack output values", outputKey)
}

func main() {

	args := os.Args[1:]
	stackName := args[0]
	outputKey := [4]string{"PrimaryReplicaNodeIp", "SecondaryReplicaNode0Ip", "SecondaryReplicaNode1Ip", "ReplicaSetID"}

	fmt.Printf("Looking up \"%s\" for \"%s\"...\n", outputKey[0], stackName)

	sess := session.Must(session.NewSessionWithOptions(session.Options{
		SharedConfigState: session.SharedConfigEnable,
	}))

	svc := cloudformation.New(sess)
	resp, _ := svc.DescribeStacks(&cloudformation.DescribeStacksInput{StackName: aws.String(stackName)})

	var stack *cloudformation.Stack
	for _, s := range resp.Stacks {
		if *s.StackName == stackName {
			stack = s
		}
	}

	if stack == nil {
		fmt.Printf("Stack=%s wasn't found\n", stackName)
		os.Exit(1)
	}

	var ips [4]*string
	var err error
	for i := 0; i < len(outputKey); i++ {
		ips[i], err = getStackOutputValue(stack, outputKey[i])
		if err != nil {
			log.Fatal(err)
			os.Exit(1)
		}
	}

	file := readFile("config.js")
	changedFile := fileParser(file, *ips[0], *ips[1], *ips[2], *ips[3])
	fileWriter(changedFile)

}

Usage:

go run {script}.go {cloud_formation_stack_name}

NOTE: You will need to replace XXXUSERNAMEXXX and XXXPASSWORDXXX with your mongo clusters Username and Password