pseudocode-of-migration-script.md

It depends on how big is the migration task like whether we only need to migrate contacts and its activities OR with contacts migrate all associated data like address, contributions, events, notes etc. In my opinion if the migration job is small and we only need to import a single entity, then I would prefer to use the PHP script techinique:

1. PHP script which would be like:

class MigrationScript {

public function process($batchTotal, $batchsize) {
  $offset = 0;
  $limit = $batchsize;
  while ($limit < $batchTotal) {
    $limit += $batchsize;
    
    // fetch 100 records
    $results = Third_Party_Connector()->query("SELECT records FROM table LIMIT $offset, $limit");

    foreach ($results as $result) {
      try {
          // your code to run migration / import code using $result data, 1 at a time 
        }
        catch (API_Exception $e) {
          // if any error occur, record the external identifier and reason, so you can later process the missing data
          recordError($result['id'], $e->getmessage());
        }
    }
    $offset += $limit;
  }
 }
  
$tool = new MigrationScript();
$totalRecords = $args[0]
$batchSize = $args[1];
$tool->process($totalRecords, $batchSize); // migrate 10k records in 10 batches, each contain 100 records from 3rd party application

You can run the script in seperate screen as php migration.script 1000 100

2. Using API - On the otherhand if the migration job is huge as in second case, I would prefer to do it via APIs and you can run each APIs (where each API is responsible to do a single job, let say migrate all contacts) from terminal in seperate screen as

$ cd <civicrm-directory>
<civicrm-directory>$ bin/cli.php -e RaiserMigration -a CreateContact

Follow this doc to see what are the criteria to build a migration extension that rely on APIs to migrate each entity. I have tried to explain with a help of civi extension called biz.jmaconsulting.raisersedgemigration which migrates the RaiserEdge data in CiviCRM.

At last, there are five basic component a migration script must have:

1. Third-party connector - This is a wrapper class which is responsbile to connect with third party application and fetch data in civiCRM. e.g. RaiserEdge SQL connector

2. Field Mapper - This is responsible to map the data fetched from third party application into respective attributes of CiviCRM e.g. RaiserEdge Field Mapper

3. Data Formatter - This is responsible to format the data. Say, a location types in the remote application is termed as billingaddress, warehousebilling and in Civi you need to map all the addresses of both of these types into Billing location type, here the formatter will help to replace both the billing type values with respective billing location type ID in CiviCRM.

4. Importer - function to import a single entity say contacts in other word perform a single subtask as mentioned in the doc with a createContact function.

5. Error Handler - is responsbile to handle error messages and re import missing data after correction.