Migrate Statamic v3 file-based users to a database

README.md

Statamic v3: Migrate file-based users to a database

Ok, so you're using Statamic v3. It's going well. You've got loads of users and now you need more power. You need to put your users into a database!

There are loads of good reasons why you should do this, but there isn't an awful lot of advice out there on how to do it. So here's some. (Whether it can be called 'good' advice is left as an exercise for the reader.)

So you're looking at the docs and you're following all the steps... super easy - edit this line, comment that out - simples. But then you get to Step 8:

Run a command to migrate your file based users into the database.

Oh sweet! It's that easy!? I love you Statamic!!! ... wait, where's the command? It's here! 🙃

So, in true Blue Peter fashion, here's one I made earlier:

php artisan users:migrate [--force]

Just pop MigrateUsers.php into your app/Console/Commands folder (create that if it doesn't exist).

NB: You'll need to revert the changes in Step 3 of the Statamic docs' instructions - disabling the Users Stache store - as this is going to need access to that. (You can disable it again once you're done.)

It's just a stub

This command is just a stub to get you going, feel free to copy and modify it till your heart's content.

The main thing you'll probably want to modify is the migrate method - this is where you'll map your file-based user's attributes to the your database model's fields.

You can modify the namespace of the model easily at the top, just replace App\Models\User with the appropriate Eloquent model import - but be sure to leave the UserModel alias so things don't break.

By default, this command will reference users by their email, both in the database ($searchColumn) and user YAML files ($searchField). Feel free to change one or both of these values depending on your configuration, e.g. if in your database you use a UUID, change the value of the $searchColumn property to the name of that column, and change the value of the $searchField property to the relevant field in your users' YAML files.

What about Roles and Groups?

Yeh, I didn't get to that. This took me a few hours to write and then write this up, so you know... take what you can get 🙃

Update: Oh alright, I added Roles support.

Output

The output is very minimal by default. If you want more detail, you can use verbosity flags -v|vv, but if you have lots of user files (e.g. thousands) then it's going to be quite long and might exceed your terminal's buffer. You may want to pipe it to a file. (Note that errors will get logged too.)

The command is safe and will not overwrite user records in your database by default. If you wish to force it to do so, use the --force or -f flag.

This means you should be able to keep on running the command until you get everything just right without creating loads of duplicates.

Errors

This script is simple so you shouldn't bump into any errors using it as-is, but depending on how old some of those YAML files are, you might bump into issues if you modify the migrate command to start normalising your data.

There's not much this command can do to help in those scenarios; you're on your own! I can only recommend a bevy of if-else and try-catch blocks to catch all of those little data oddities.

However, when you do encounter an error, you'll likely want to capture that and move on instead of stopping the process. In that case, you can use the helpful logError() method, which will keep a track of all the errors you report for a given user and log/output a useful summary at the end.

License

MIT.

Please leave the credits at the top wherever you use it.

MigrateUsers.php

<?php
/**
 * Statamic v3 User Migrator
 * @author Simon Hamp <simon.hamp@me.com>
 * @copyright Copyright (c) 2021, Simon Hamp
 * @license MIT
 */
namespace App\Console\Commands;

use Exception;
use TypeError;
use Carbon\Carbon;
use Statamic\Facades\User;
use Illuminate\Support\Str;
use Illuminate\Console\Command;
use App\Models\User as UserModel;
use Illuminate\Support\Facades\Log;
use Illuminate\Database\QueryException;
use Statamic\Auth\File\User as FileUser;
use Statamic\Auth\UserRepositoryManager;
use Statamic\Auth\Eloquent\User as DatabaseUser;
use Symfony\Component\Console\Output\OutputInterface;
use Statamic\Stache\Repositories\UserRepository as StacheUserRepository;

class MigrateUsers extends Command
{
    /**
     * The name and signature of the console command.
     *
     * @var string
     */
    protected $signature = 'users:migrate
        {--f|--force : Overwrite users in the database if they exist there already}';

    /**
     * The console command description.
     *
     * @var string
     */
    protected $description = 'Migrate your users from the filesystem to the database';

    /**
     * The column to search on to see if a user already exists in the database.
     *
     * @var string
     */
    protected $searchColumn = 'email';

    /**
     * The field on the file-based user record to find in the database $searchColumn.
     *
     * @var string
     */
    protected $searchField = 'email';

    protected $errors = [];

    protected $success = 0;

    protected $skipped = 0;

    /**
     * Execute the console command.
     *
     * @return int
     */
    public function handle()
    {
        $force = $this->option('force');

        User::swap(app(StacheUserRepository::class));

        try {
            $fileUsers = User::all();
        } catch (TypeError $e) {
            $this->error("Make sure your 'users' Stache store is configured in config/statamic/stache.php");
            return 1;
        }

        User::swap(app(UserRepositoryManager::class)->repository());

        $total = $fileUsers->count();

        $this->line("Found {$total} user records to migrate...");
        $bar = false;

        if ($this->getOutput()->getVerbosity() <= OutputInterface::VERBOSITY_VERBOSE) {
            $bar = $this->output->createProgressBar($total);

            $bar->start();
        }

        $fileUsers->each(function (FileUser $fileUser) use ($force, $bar) {
            $databaseUser = $this->prepDbUser($fileUser);

            if ($databaseUser->model()->exists) {
                $identifier = $this->searchValue($fileUser);

                if ($this->getOutput()->isVeryVerbose()) {
                    $this->warn("User [{$identifier}] already exists!" . ($force ? ' Overwriting...' : ''));
                }

                if (! $force) {
                    $this->skipped++;
                    return;
                }
            }

            if ($this->getOutput()->isVeryVerbose()) {
                $this->line('Migrating user: ' . $fileUser->email());
            }

            if ($this->migrate($fileUser, $databaseUser) && $this->getOutput()->isVeryVerbose()) {
                $this->info('User migrated! ID: ' . $databaseUser->getAuthIdentifier());
            }

            if ($bar) {
                $bar->advance();
            }
        });

        if ($bar) {
            $bar->finish();
            $this->newLine();
        }

        $this->newLine();
        $this->info("Successfully migrated {$this->success} out of {$total} users.");
        $this->line("Skipped {$this->skipped} users.");

        if (! empty($this->errors)) {
            $count = count($this->errors);

            $this->newLine();

            $this->warn("However, there were {$count} errors!");

            if ($this->getOutput()->isVerbose()) {
                $table = collect($this->errors)
                    ->flatMap(fn ($errors, $index) => collect($errors)->transform(fn ($error) => [$index, Str::limit($error)]));

                $this->table(['ID', 'Errors'], $table->all());

                $this->newLine();
            } else {
                $this->line("Use the `-v` option to see more details or check your log.");
            }

            return 1;
        }

        return 0;
    }

    protected function migrate(FileUser $fileUser, DatabaseUser $databaseUser): bool
    {
        // Standard fields
        /**
         * =============
         * A note on IDs
         * =============
         *
         * In case you need to keep referring to your Statamic users by their UUIDs, you'll need
         * to import this value too bu uncommenting the following line.
         *
         * You will need to have adjusted the migration to support this column _before_ migrating:
         * $table->uuid('statamic_id')->unique()
         *
         * And you may want to modify your User model to set this as the primary key:
         *
         * @see https://laravel.com/docs/8.x/eloquent#primary-keys
         */
        //$databaseUser->set('statamic_id', $fileUser->id());
        $databaseUser->set('name', $fileUser->name());
        $databaseUser->set('email', $fileUser->email());
        $databaseUser->set('super', (bool) $fileUser->isSuper());
        $databaseUser->model()->password = $fileUser->passwordHash();

        // Your custom fields here...

        // Roles
        foreach ($fileUser->roles() as $role) {
            $databaseUser->assignRole($role);
        }

        try {
            $databaseUser->save();

            $this->success++;

            return true;
        } catch (QueryException $e) {
            $this->logError($fileUser, $e->getMessage());
        }

        return false;
    }

    protected function prepDbUser(FileUser $fileUser): DatabaseUser
    {
        $model = UserModel::where($this->searchColumn, $this->searchValue($fileUser))->firstOrNew();

        /** @var DatabaseUser $user */
        $user = DatabaseUser::fromModel($model);

        return $user;
    }

    protected function searchValue(FileUser $fileUser): string
    {
        return $fileUser->fluentlyGetOrSet($this->searchField)->args([]);
    }

    protected function logError(FileUser $fileUser, string $error): void
    {
        $identifier = $this->searchValue($fileUser);
        $this->errors[$identifier][] = $error;
        if ($this->getOutput()->isVeryVerbose()) {
            $this->error("Failed to migrate user [{$identifier}]!");
        }
        Log::error("Failed to migrate user [{$identifier}]: {$error}");
    }
}

I changed line 205 en 206 to:

  $user = new DatabaseUser();
  $user =  $user->model($model);

Call to undefined method Statamic\Stache\Repositories\UserRepository::fromModel()

I tried to replace it with fromUser but it doesn't work.

See: https://gist.github.com/simonhamp/a2b9113c100e5194db53298162f1dde0?permalink_comment_id=4728402#gistcomment-4728402