Hull - Processor: Unified data via Priority rules

README.md

Hull - Processor: Unified data via Priority rules

The code in index.js can be copied directly into a (User) Processor in Hull and configured via variables to unify attributes based on various input attributes. The input attributes are considered in the order defined and the first matching input attribute will determine the value of the unified attribute. The code is also able to handle the unification of top-level attributes such as first_name and last_name to properly display that information in user profiles in Hull.

Example Scenarios

Unified last_name (top-level attribute)

Assuming you have various systems such as Salesforce and Hubspot connected to Hull, incoming data is stored as grouped attributes, so you most likely have the following attributes on a user profile:

• salesforce_lead.last_name
• salesforce_contact.last_name
• hubspot.last_name

Now your team decides to pick up that the information and prioritize the data. You might end up defining that information about a customer which is represented as a Salesforce Contact has probably the most accurate data since it has been vetted, so salesforce_contact.last_name takes top priority. If that data is not present, your team may decide that leads vetted by a sales person are still more accurate than marketing data from Hubspot, so you define that salesforce_lead.last_name takes precedence over hubspot.last_name.

Now all you have to do is to create a new variable in the Settings tab of the Processor under Custom Variables to indicate that you want to apply a fallback strategy for the top-level attribute last_name. So you enter the key fallbacks__last_name and a value of salesforce_contact.last_name;salesforce_lead.last_name;hubspot.last_name;. That's it, now the Processor will execute the logic to unify the top-level attribute last_name for you.

Unified job_title

Assuming you have various systems such as Salesforce and Hubspot connected to Hull, incoming data is stored as grouped attributes, so you most likely have attributes similar to the following on a user profile:

• salesforce_lead.title
• salesforce_contact.title
• hubspot.job_title

Now your team decides to pick up that the information and prioritize the data. You might end up defining that information about a customer which is represented as a Salesforce Contact has probably the most accurate data since it has been vetted, so salesforce_contact.title takes top priority. If that data is not present, your team may decide that enriched marketing data in Hubspot is still more accurate than Sales data from leads, so you define that hubspot.job_title takes precedence over salesforce_lead.title.

Now all you have to do is to create a new variable in the Settings tab of the Processor under Custom Variables to indicate that you want to apply a fallback strategy for the unified attribute job_title. So you enter the key unified__job_title and a value of salesforce_contact.title;hubspot.job_title;salesforce_lead.title. That's it, now the Processor will execute the logic to unify the attribute job_title and store the output in a group called unified. To consolidate your systems, you should use in the outgoing data section of all connectors the attribute unified/job_title instead of a cross-reference mapping from a particular other source.

Configuration

Configuration is entirely done via variables in the Settings of the Processor. The keys of the variables define the type of unification:

• keys following the convention fallbacks__<ATTRIBUTE_NAME> define top-level attribute unification; naturally there are only two values allowed for <ATTRIBUTE_NAME> which are first_name and last_name. It is not recommended to unify the email with that code even you can do it.
• keys following the convention unified__<ATTRIBUTE_NAME> define grouped attribute unification; you can define any name that makes, but it is recommended to use a snake_cased attribute name.

The value for each variable takes an ordered list devided by semicolons. The code will parse that list and apply the first matching attribute value as the value for the unified attribute

index.js

/*
 * ---------------------------------------------------------------------------------
 * Temporary objects to diff against, no need to check if something has 
 * changed, it will be done at the bottom automatically.
 *
 * Usage: 
 * `_.set(attribs, <NAME>, <VALUE>);` to set a new attribute value.
 * 
 * Examples:
 * `_.set(attribs, 'first_name', user.hubspot.first_name);` to set a top-level attribute
 * `_.set(attribs, 'unified/first_name', user.hubspot.first_name);` to set an attribute in a group`
 * `_.set(attribs, 'first_name', { 
 *    value: user.hubspot.first_name,
 *    operation: 'setIfNull'
 *  });` to set a top-level attribute, but only if it is not set
 *
 * ---------------------------------------------------------------------------------
 */
let attribs = {};


/*
 * ---------------------------------------------------------------------------------
 * START YOUR LOGIC
 * ---------------------------------------------------------------------------------
 */
 
 // Handle the top-level attribute changes
 const tlAttribs = determineTopLevelAttributes(user);
 attribs = _.merge(attribs, tlAttribs);
 
 // Handle unified attribute changes
 const unifiedAttribs = determineUnifiedAttributes(user);
 attribs = _.merge(attribs, unifiedAttribs);

/*
 * ---------------------------------------------------------------------------------
 * END YOUR LOGIC
 * ---------------------------------------------------------------------------------
 */

/*
 * ---------------------------------------------------------------------------------
 * START UTILITY FUNCTIONS
 * ---------------------------------------------------------------------------------
 */
 
 
 // If you have any utility functions, add them in this block, this helps your 
 // code staying tidy and everything is in a separate section.
 // NOTE: Do not use const myFunction = (foo) => { ... };
 //       but use function myFunction(foo) { ... }
 
  
 function determineTopLevelAttributes(usr) {
   const changedAttribs = {};
   _.forIn(variables, (v, k) => {
     if (k.startsWith("fallbacks__")) {
       const fallbacksList = splitVariableToList(v, ";");

       _.forEach(fallbacksList, f => {
         if (_.get(usr, f, null) !== null) {
           const tlAttribName = k.replace("fallbacks__", "");
           if (typeof(_.get(usr, f, null)) === "string") {
            _.set(changedAttribs, tlAttribName, _.trim(_.get(usr, f)));
           } else {
            _.set(changedAttribs, tlAttribName, _.get(usr, f));
           }
           console.log(`Fallbacks: Setting value for '${tlAttribName}' based on '${f}'.`);
           return false;
         }
       });
     }
   });
   
   return changedAttribs;
 }
 
 function determineUnifiedAttributes(usr) {
   const changedAttribs = {};
   _.forIn(variables, (v, k) => {
     if (k.startsWith("unified__")) {
       const fallbacksList = splitVariableToList(v, ";");

       _.forEach(fallbacksList, f => {
         if (_.get(usr, f, null) !== null) {
           const tlAttribName = `unified/${k.replace("unified__", "")}`;
           if (typeof(_.get(usr, f, null)) === "string") {
            _.set(changedAttribs, tlAttribName, _.trim(_.get(usr, f)));
           } else {
            _.set(changedAttribs, tlAttribName, _.get(usr, f));
           }
           console.log(`Unified: Setting value for '${tlAttribName}' based on '${f}'.`);
           return false;
         }
       });
     }
   });
   
   return changedAttribs;
 }
 
 function splitVariableToList(stringVal, separator) {
   return _.compact(stringVal.split(separator));
 }
 
/*
 * ---------------------------------------------------------------------------------
 * END UTILITY FUNCTIONS
 * ---------------------------------------------------------------------------------
 */


/*
 * ---------------------------------------------------------------------------------
 * Hull API calls
 * 
 * Diffs automatically attribs against all attributes of the user 
 * and only performs calls to the API if needed. This helps you reduce billable
 * incoming requests and prevents loops.
 *
 * A friendly reminder: Unless you know what you are doing, you should NOT 
 * change the code below. ;-)
 *
 * _.omitBy() == [].filter()
 * ---------------------------------------------------------------------------------
 */
const diffedAttribs = _.omitBy( attribs, ( value, index ) => {
  // `folder_name/attribute_name` > `user.folder_name.attribute_name`
  const path = _.replace( index, '/', '.' );
  if (path.endsWith("_date") || path.endsWith("_at")) {
   return moment(_.get( user, path )).isSame(moment(value), "second"); 
  } else {
    return _.isEqual( _.get( user, path ), value );
  }
});

if ( Object.keys( diffedAttribs ).length !== 0 ) {
  hull.traits( diffedAttribs );
}