Skip to content

Instantly share code, notes, and snippets.

@PowellTravis

PowellTravis/API Data to MongoDB using Mongoose and Axios.md

Last active February 9, 2023 20:42
  • Star 8 You must be signed in to star a gist
  • Fork 3 You must be signed in to fork a gist
Star You must be signed in to star a gist
Embed
What would you like to do?
Learn more about clone URLs
Download ZIP
Using Mongoose and Axios Together to Upload API Data
Raw
API Data to MongoDB using Mongoose and Axios.md

BANNER

Using Mongoose and AXIOS to upload APU data to MongoDB

Uploading data using Axios and Mongoose can be hard at first. But when you really understand what you are doing you will be glad you are not reaching out to the API each time someone loads your webpage.

Installing the Packages

  1. Make sure you have nodejs and NPM installed, NPM comes with nodejs

  2.  npm install axios mongoose

  3. Require the packages in your index.js file, and assign a vairable to db.

    const nomgoose = require('mongoose');
const axios = require('axios');
const db = mongoose.connection;

  4. Get the API token and request URL (I will be using my repository API URL)

  5. Connect to the DB, and get the data from GitHub

Connecting to the Database

First off we need to connect to the Database

mongoose.connect(`mongodb+srv://${user}:${password}@${URL}`, {useNewUrlParser: true }, function (err) {
 
   if (err) throw err;

   console.log('Successfully connected');
 
});

Fetching API Data

So now that we are connecting to the database we need to get going on requesting the data from GitHub.

axios.get('https://api.github.com/users/PowellTravis/repos?per_page=100&page=1')
  .then(function (response) {
    onSuccess(response)
  })
  .catch(function (error) {
    console.log(error);
  });

With this we can output the response to a function that we can use to call the individual data points. Outputting the response to a different function keeps the code split up and allows for more flexibility in the use of the response variable.

Making the Schema

Next we need to start making the schema that will be the key to uploading the data in a specific format. So we need to define our Schema:

var repSchema = mongoose.Schema({
    name: String,
    description: String,
    url: String
});

After that we need to create a model with our Schema:

var Data = mongoose.model('Data', repSchema);

Creating our Success function

Now we need to create our OnSuccess function that will allow us to gain all of our variables to upload into the database.

function onSuccess(response) {
    var array = response;
  var arrayLength = Object.keys(array).length 
    console.log(arrayLength)
   for(var i = 0; i <= arrayLength; i++) {
      var name = array.data[i].name;
      var desc = array.data[i].description;
      var url = array.data[i].html_url;
      console.log( name + " " + desc + " " + url);

      assignDataValue(name, desc, url)

    }
}

Assigning Data Values

Here we are assigning the variable response to array, then getting the object length of array to be able to accuratle get all of the names and what ever all you want. In the very bottom of the file I have listed all of the types of objects you can call to store on the database. Now as you see inside the onSuccess function there is another function that takes in 3 variables name, desc, and url. Those are passed on to another function that is required to successfully upload the data to the database.

function assignDataValue(name, desc, url) {

   var upData = new Data()
      upData.name = name;
      upData.description = desc;
      upData.url = url;

      upData.save();
}

So here we are taking in the three variables that we want to use to upload, assigning them to the schema points we defined earlier. Then we use the .save() funciton from mongoose to upload the data to the mongo database. Down below I have the full code for uploading data to the database.

I have a file down below that explains how to fetch data from the API. I also have another file down below showing how to wipe a collection in the database.

Full Code

var mongoose = require("mongoose");
var axios = require("axios");
var db = mongoose.connection;

//All calls out of the server
mongoose.connect(`mongodb+srv://${user}:${password}@${url}?retryWrites=true`, {useNewUrlParser: true }, function (err) {
 
   if (err) throw err;

   console.log('Successfully connected');
 
});

axios.get('https://api.github.com/users/PowellTravis/repos?per_page=100&page=1')
  .then(function (response) {
    onSuccess(response)
  })
  .catch(function (error) {
    console.log(error);
  });


var repSchema = mongoose.Schema({
    name: String,
    description: String,
    url: String
});
var Data = mongoose.model('Data', repSchema);

function onSuccess(response) {
    var array = response;
  //  var arraytobe = response;
  var arrayLength = Object.keys(array).length 
    console.log(arrayLength)
   for(var i = 0; i <= arrayLength; i++) {
      var name = array.data[i].name;
      var desc = array.data[i].description;
      var url = array.data[i].html_url;
      console.log( name + " " + desc + " " + url);

      assignDataValue(name, desc, url)

    }
}

function assignDataValue(name, desc, url) {

   var upData = new Data()
      upData.name = name;
      upData.description = desc;
      upData.url = url;

      upData.save();
}

All of the objects you can pull from the GitHub API

Full code using all of the methods is down below.

** - Have to make an additional API request
.name is the name of the repository
.full_name is the full name of the repository (Github UN/Repo Name)
.html_url is the url that people can use to visit your repo on your accoun t
.descripton is the description of the repository
.url is an api link for the repository**
.forks_url is an api link to your forks on the repository**
.collaborators_url is an api link to the collaborators**
.teams_url is a link to the teams that are assigned to the repository**
.hooks_url is an api link to all of the hooks you have attatched to the repository**
.issue_events_url is an api link that allows you to view all events that are caused by an issue post**
.events_url is an api link that shows you all of the repository events**
.branches_url is an api link that shows all of the branches you have made**
.tags_url is an api link that shows the tags applied to the repository**
.comments_url an api link that shows regular comments on the repository**
.issue_comment_url an api link that shows issue comments**
.merges_url is an api link that shows all of the branch merges of a repository**
.downloads_url is an api link that shows all of the downloads of the repository**
.issues_url shows current issues on the repository**
.pulls_url shows how mant pulls there are on the repository**
.milestones_url is an api url that shows the milestones that the repository has reached**
.labels_url is an api url that will show the labels that were applied to the repository**
.releases_url is an api url that shows all of the releases**
.deployments_url is an api url that shows your deployments**
.created_at is an object that shows when the repository was created
.updated_at is an object that shows when the repository was last updated
.git_url is an object that has your url (github.com/username/repository.git
.ssh_url is an object that gives you your ssh link git@github.com:username/repository.git
.clone_url is the clone url that github gives you for people to git clone or put into the GitHub desktop app
.homepage gives you more information about the repository itself
.stargazers_count is the number tha
.watchers_count gives the number of watchers
.language gives the language most used
.has_issues is a boolean statement
.has_projects is a boolean statement
.has_downloads is a boolean statement
.has_wiki is a boolean statement
.has_pages is a boolean statement
.forks_count is a number of how many forks there are
.archived is a boolean statement
.open_issues_count is a number of how many open issues there are.
.license is a statement that tells what license you have assigned to the repository
.default_branch usually master or
Raw
Fetch API Data.md

Fetching API data from Database

Fetching data stored in the database is not difficult at all, infact it can be one of the easiest parts of your mongoose setup.

Assigning Variables

var mongoose = require("mongoose");
var db = mongoose.connection;

There are two ways you are able to search the database. You are able to connect individually or through a function that you can call in your main file. The way I am going to show you is calling the function through exports.

exports.dbSearch = function(Data) {

Data.find({}, function(err, users) {
    if (err) throw err;
  
    // object of all the users
    console.log(users);
  });

}

So what we are doing here is using the node exports module to create an export named dbSearch. We are passing a function that requires Data which is a defined variable in our main file. then we are finding all of the objects within the database. if we want to search for a specific object you can do

Data.find({name}, function(err, users) {
    if (err) throw err;
  
    // object of all the users
    console.log(users);
  });

to search for a specific object in your database/ collection. That is how easy it is to get all of the information in your database.

Full code using all of the methods is down below.

Raw
Wipe Collection.md

Wiping a collection in a database

You might be wanting to wipe the collection for testing, or want to wipe a collection when you are uploading the same content over and over again.

Assigning Variables

var mongoose = require("mongoose");
var db = mongoose.connection;

Removing the Collection

Here we need to first connect to the database then use db.dropCollection. So when we use this it wreases the whole collection within the database. Also here you are able to use exports.USENAME = fuction() {} to exxport it as a funciton you can call outside of the file in node.

var mongoose = require("mongoose");
var db = mongoose.connection;

mongoose.connect(`mongodb+srv://${user}:${password}@${ur;}?retryWrites=true`, {useNewUrlParser: true }, function (err) {
 
   if (err) throw err;
 
   console.log('Successfully connected');
 
});

db.dropCollection(`${collection}`, function (err, result) {

        if (err) {

            console.log(`Couldn't delete the collection ${collection}`);

        } else {

            console.log(`${collection} was deleted`);

        }

    });

Here we are defining that once the drop colleciton is run you can log in the console that the collection was wiped successfully, or that it was not able to be wiped. Again you can call this as a seprate function to be used everytime a files is called.

Full code using all of the methods is down below.

Raw
You Code.md 
var mongoose = require("mongoose");
var axios = require("axios");
var db = mongoose.connection;
var dbSearch = require("./dbSearch").dbSearch;
var dbErase = require("./dbErase").dbErase;

//All calls out of the server
mongoose.connect(`mongodb+srv://admin:Z@j7$6x&EET&d9D9xF7@cluster0-h8ddm.azure.mongodb.net/Production?retryWrites=true`, {useNewUrlParser: true }, function (err) {
 
   if (err) throw err;

   console.log('Successfully connected');
 
});

dbErase();

axios.get('https://api.github.com/users/PowellTravis/repos?per_page=100&page=1')
  .then(function (response) {
    onSuccess(response)
  })
  .catch(function (error) {
    console.log(error);
  });


var repSchema = mongoose.Schema({
    name: String,
    description: String,
    url: String
});
var Data = mongoose.model('Data', repSchema);

function onSuccess(response) {
    var array = response;
  //  var arraytobe = response;
  var arrayLength = Object.keys(array).length 
    console.log(arrayLength)
   for(var i = 0; i <= arrayLength; i++) {
      var name = array.data[i].name;
      var desc = array.data[i].description;
      var url = array.data[i].html_url;
      console.log( name + " " + desc + " " + url);

      assignDataValue(name, desc, url)

    }
}

function assignDataValue(name, desc, url) {

   var upData = new Data()
      upData.name = name;
      upData.description = desc;
      upData.url = url;

      upData.save();
}

dbSearch(Data);
@PowellTravis
Copy link
Author

banner

@GiacomoR90
Copy link

the code works, the problem that it consumes a lot of RAM memory, do you have another way to load very many objects without using the for?

@PowellTravis
Copy link
Author

the code works, the problem that it consumes a lot of RAM memory, do you have another way to load very many objects without using the for?

Unfortunately I have not looked at this code in quite a while. I believe you could instead use something like

function onSuccess(response) {
    content = response.data
    content.forEach(data => {
        assignDataValue(data.name, data.description, data.html_url)
    })
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment