Get only new items from Firebase

1_query_timestamp.js

// assumes you add a timestamp field to each record (see Firebase.ServerValue.TIMESTAMP)
// pros: fast and done server-side (less bandwidth, faster response), simple
// cons: a few bytes on each record for the timestamp
var ref = new Firebase(...);
ref.orderByChild('timestamp').startAt(Date.now()).on('child_added', function(snapshot) {
  console.log('new record', snap.key());
});

2_once_value.js

// pros: works?
// cons: fetches entire record set, requires discarding the first record
// credits: http://stackoverflow.com/questions/12850789/is-there-a-way-to-know-in-what-context-child-added-is-called-particularly-page/12851236#12851236
var ref = new Firebase(...);
ref.once("value", function(snap) {
  //TODO: display initial state...
  // Object.keys not supported in IE 8, but has a polyfill: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Object/keys
  var keys = Object.keys(snap.val()||{});
  var lastIdInSnapshot = keys[keys.length-1]

  ref.orderByKey().startAt(lastIdInSnapshot).on("child_added", function(newMessSnapshot) {
    if( snap.key() === lastIdInSnapshot ) { return; }
    console.log('new record', snap.key());
  }
}

3_limit_endat.js

// pros: does not fetch entire record set
// cons: triggers child_removed events when last item changes, grabs the last record which needs to be ignored
var first = true;
var ref = new Firebase(...);
ref.limitToLast(1).on("child_added", function(snap) {
   if( first ) {
       first = false; 
   } 
   else {
       console.log('new record', snap.key());
   }
});

This doesn't work if you add children in this script directly after you put on the listener. What happens is the cb fires for the new items first before ever sending the exiting one. This means assuming that the first call to the callback will be the one that should get tossed is a bad assumption. I am looking for a solution but haven't found an elegant one.

I'm referring to the endAt().limit(1) method

I am confused why Firebase do not support this natively. This is somehow a hacky way, though.

Also, in the second method, you are still going to pull all previous data from server, which is not efficient.

Updated for 2.x queries.

How would I use orderByChild('timestamp') with AngularFire's $watch?

Very useful! Thanks

@katowulf, any thoughts on why 3_limit_endat.js doesn't work well on larger datasets? It triggers as expected on a node with 10 records, but does not trigger on a node with 400,000 records.

I think I like the 1_query_timestamp.js method best. But is there a way to not use the client side date but the server side date at startAt?

@ngerritsen firebase.database.ServerValue.TIMESTAMP

Edit: this does not work on startAt
http://stackoverflow.com/q/39437845/1325646

The problem with 1_query_timestamp.js is that client time may be out of sync with server time. As mentioned firebase.database.ServerValue.TIMESTAMP cannot be used with startAt.

There is an approach to get the servertime, see: http://stackoverflow.com/questions/23128027/retrieve-firebase-server-time-without-setting-it-first.

Kato, on 3 you remark 'triggers child_removed events when last item changes'

can you explain that ? what does child_removed has to do with it ?

also isn't 3 missing orderByKey() ?

@kofifus I think he means that when you first run the code, it returns the last already-existing item in the database. Whereas, you actually only want to know about new items that will be added in the future. Hence, it requires that if statement in the handler to ignore the first entry.

Thank you ! That worked for me.

It works! Awesome! Respect 🙇‍♂️

how can i use timestamp with python? I am going to save image from python to database and want to retrieve the latest image uploaded from database

Thanks for your solution ! It's work for me
🥇

dont forget to add an index for the timestamp to make the filtration quicker https://www.firebase.com/docs/security/guide/indexing-data.html

This is also one solution,
ref.startAt(null,ref.push().key).on('child-added,...'
works only when keys are generated by push()

This is also one solution,
ref.startAt(null,ref.push().key).on('child-added,...'
works only when keys are generated by push()

This is interesting, do you know why this works? Are keys generated in order by RTDB?

On 2), ref.orderByKey().startAt(lastIdInSnapshot).on("child_added"..., I found out that instead of throwing away the first value, you can add a '-' to the lastIdSnapshot. It's just an character with a low ASCII value, so any push'ed item will go after the previous one + '-'.

This is interesting, do you know why this works?

if we were to use ref.push().key to put a new entry on ref, it would be stored as the last child.So, using second parameter of startAt function, we retrive childs with key greater than or equal to ref.push().key
https://firebase.google.com/docs/reference/js/firebase.database.Query#startat

Are keys generated in order by RTDB?

yes, https://firebase.google.com/docs/database/admin/save-data#push-vs-transaction