Hull - Processor: Unified data via last change

README.md

Hull - Processor: Unified data via last change

The code in index.js can be copied directly into a (User) Processor in Hull and configured via variables to unify attributes based on the last change of various input attributes. The unified attribute will be calculated based on the last change to any related input attribute. The code is also able to handle the unification of top-level attributes such as first_name and last_name to properly display that information in user profiles in Hull.

Example Scenarios

Unified last_name (top-level attribute)

Assuming you have various systems such as Salesforce and Hubspot connected to Hull, incoming data is stored as grouped attributes, so you most likely have the following attributes on a user profile:

• salesforce_lead.last_name
• salesforce_contact.last_name
• hubspot.last_name

Now your team decides to pick up that the information and since all sources are equal in priority use the value of the source which changed last determine the unified attribute. So if someone makes a change on the Salesforce Contact, this should update the current unified value in Hull. If a day later someone updates the Hubspot Contact, that value should update the unified attribute in Hull as well.

Now all you have to do is to create a new variable in the Settings tab of the Processor under Custom Variables to indicate that you want to apply a fallback strategy for the top-level attribute last_name. So you enter the key fallbacks_last__last_name and a value of salesforce_contact.last_name;salesforce_lead.last_name;hubspot.last_name;. That's it, now the Processor will execute the logic to unify the top-level attribute last_name for you.

Unified job_title

Assuming you have various systems such as Salesforce and Hubspot connected to Hull, incoming data is stored as grouped attributes, so you most likely have attributes similar to the following on a user profile:

• salesforce_lead.title
• salesforce_contact.title
• hubspot.job_title

Now your team decides to pick up that the information and since all sources are equal in priority use the value of the source which changed last determine the unified attribute. So if someone makes a change on the Salesforce Contact, this should update the current unified value in Hull. If a day later someone updates the Hubspot Contact, that value should update the unified attribute in Hull as well.

Now all you have to do is to create a new variable in the Settings tab of the Processor under Custom Variables to indicate that you want to apply a fallback strategy for the unified attribute job_title. So you enter the key unified_last__job_title and a value of salesforce_contact.title;hubspot.job_title;salesforce_lead.title. That's it, now the Processor will execute the logic to unify the attribute job_title and store the output in a group called unified. To consolidate your systems, you should use in the outgoing data section of all connectors the attribute unified/job_title instead of a cross-reference mapping from a particular other source.

Configuration

Configuration is entirely done via variables in the Settings of the Processor. The keys of the variables define the type of unification:

• keys following the convention fallbacks_last__<ATTRIBUTE_NAME> define top-level attribute unification; naturally there are only two values allowed for <ATTRIBUTE_NAME> which are first_name and last_name. It is not recommended to unify the email with that code even you can do it.
• keys following the convention unified_last__<ATTRIBUTE_NAME> define grouped attribute unification; you can define any name that makes, but it is recommended to use a snake_cased attribute name.

The value for each variable takes an unordered list devided by semicolons. The code will parse that list and apply the first matching changed value as the value for the unified attribute.

Remarks

If you want to prioritize input attributes in a certain order you might want to check out the Gist for priority rules here: https://gist.github.com/SMK1085/11b5a43a7e7363477351706add5bc932

You can also combine them into one Processor, to do so, copy all the function in the UTILITY FUNCTIONS block except splitVariableToList from this Gist into the corresponding block of the priority rules Gist. Then add everything from the YOUR LOGIC block into the corresponding block of the priority rules Gist.

The naming conventions for the variables are designed in a way they won't clash, so you can use both strategies for unification in one Processor.

index.js

/*
 * ---------------------------------------------------------------------------------
 * Temporary objects to diff against, no need to check if something has 
 * changed, it will be done at the bottom automatically.
 *
 * Usage: 
 * `_.set(attribs, <NAME>, <VALUE>);` to set a new attribute value.
 * 
 * Examples:
 * `_.set(attribs, 'first_name', user.hubspot.first_name);` to set a top-level attribute
 * `_.set(attribs, 'unified/first_name', user.hubspot.first_name);` to set an attribute in a group`
 * `_.set(attribs, 'first_name', { 
 *    value: user.hubspot.first_name,
 *    operation: 'setIfNull'
 *  });` to set a top-level attribute, but only if it is not set
 *
 * ---------------------------------------------------------------------------------
 */
let attribs = {};
/*
 * ---------------------------------------------------------------------------------
 * START YOUR LOGIC
 * ---------------------------------------------------------------------------------
 */
 
 const userChanges = _.get(changes, "user", {});
 
 // Handle the top-level attribute changes
 const tlAttribsLast = determineTopLevelAttributesLast(userChanges);
 attribs = _.merge(attribs, tlAttribsLast);
 
 // Handle unified attribute changes
 const unifiedAttribsLast = determineUnifiedAttributesLast(userChanges);
 attribs = _.merge(attribs, unifiedAttribsLast);

/*
 * ---------------------------------------------------------------------------------
 * END YOUR LOGIC
 * ---------------------------------------------------------------------------------
 */

/*
 * ---------------------------------------------------------------------------------
 * START UTILITY FUNCTIONS
 * ---------------------------------------------------------------------------------
 */
 
 
 // If you have any utility functions, add them in this block, this helps your 
 // code staying tidy and everything is in a separate section.
 // NOTE: Do not use const myFunction = (foo) => { ... };
 //       but use function myFunction(foo) { ... }
 
  
 function determineTopLevelAttributesLast(usrChanges) {
   const changedAttribs = {};
   _.forIn(variables, (v, k) => {
     if (k.startsWith("fallbacks_last__")) {
       const fallbacksList = splitVariableToList(v, ";");

       _.forEach(fallbacksList, f => {
           const tlAttribName = `${k.replace("fallbacks_last__", "")}`;
           const changesObj = _.get(usrChanges, getChangesKey(f), []);
           if (changesObj.length === 2) {
             const changedVal = _.last(changesObj);
             _.set(changedAttribs, tlAttribName, changedVal);
             console.log(`Fallbacks: Setting value for '${tlAttribName}' based on a change in '${f}'.`);
             return false;  
           }
       });
     }
   });
   
   return changedAttribs;
 }
 
 function determineUnifiedAttributesLast(usrChanges) {
   const changedAttribs = {};
   _.forIn(variables, (v, k) => {
     if (k.startsWith("unified_last__")) {
       const fallbacksList = splitVariableToList(v, ";");

       _.forEach(fallbacksList, f => {
           const tlAttribName = `unified/${k.replace("unified_last__", "")}`;
           const changesObj = _.get(usrChanges, getChangesKey(f), []);
           if (changesObj.length === 2) {
             const changedVal = _.last(changesObj);
             _.set(changedAttribs, tlAttribName, changedVal);
             console.log(`Unified: Setting value for '${tlAttribName}' based on a change in '${f}'.`);
             return false;  
           }
       });
     }
   });
   
   return changedAttribs;
 }
 
 function splitVariableToList(stringVal, separator) {
   return _.compact(stringVal.split(separator));
 }
 
 function getChangesKey(attributeName) {
   // Attribute format is with dots hubspot.job_title
   // vs change key is traits_hubspot/job_title
   if(attributeName.includes(".")) {
     return `traits_${attributeName.replace(".", "/")}`;
   }
   
   return attributeName;
 }
 
/*
 * ---------------------------------------------------------------------------------
 * END UTILITY FUNCTIONS
 * ---------------------------------------------------------------------------------
 */


/*
 * ---------------------------------------------------------------------------------
 * Hull API calls
 * 
 * Diffs automatically attribs against all attributes of the user 
 * and only performs calls to the API if needed. This helps you reduce billable
 * incoming requests and prevents loops.
 *
 * A friendly reminder: Unless you know what you are doing, you should NOT 
 * change the code below. ;-)
 *
 * _.omitBy() == [].filter()
 * ---------------------------------------------------------------------------------
 */
const diffedAttribs = _.omitBy( attribs, ( value, index ) => {
  // `folder_name/attribute_name` > `user.folder_name.attribute_name`
  const path = _.replace( index, '/', '.' );
  if (path.endsWith("_date") || path.endsWith("_at")) {
   return moment(_.get( user, path )).isSame(moment(value), "second"); 
  } else {
    return _.isEqual( _.get( user, path ), value );
  }
});

if ( Object.keys( diffedAttribs ).length !== 0 ) {
  hull.traits( diffedAttribs );
}