Migrate Statamic v3 file-based users to a database

README.md

Statamic v3: Migrate file-based users to a database

Ok, so you're using Statamic v3. It's going well. You've got loads of users and now you need more power. You need to put your users into a database!

There are loads of good reasons why you should do this, but there isn't an awful lot of advice out there on how to do it. So here's some. (Whether it can be called 'good' advice is left as an exercise for the reader.)

So you're looking at the docs and you're following all the steps... super easy - edit this line, comment that out - simples. But then you get to Step 8:

Run a command to migrate your file based users into the database.

Oh sweet! It's that easy!? I love you Statamic!!! ... wait, where's the command? It's here! 🙃

So, in true Blue Peter fashion, here's one I made earlier:

php artisan users:migrate [--force]

Just pop MigrateUsers.php into your app/Console/Commands folder (create that if it doesn't exist).

NB: You'll need to revert the changes in Step 3 of the Statamic docs' instructions - disabling the Users Stache store - as this is going to need access to that. (You can disable it again once you're done.)

It's just a stub

This command is just a stub to get you going, feel free to copy and modify it till your heart's content.

The main thing you'll probably want to modify is the migrate method - this is where you'll map your file-based user's attributes to the your database model's fields.

You can modify the namespace of the model easily at the top, just replace App\Models\User with the appropriate Eloquent model import - but be sure to leave the UserModel alias so things don't break.

By default, this command will reference users by their email, both in the database ($searchColumn) and user YAML files ($searchField). Feel free to change one or both of these values depending on your configuration, e.g. if in your database you use a UUID, change the value of the $searchColumn property to the name of that column, and change the value of the $searchField property to the relevant field in your users' YAML files.

What about Roles and Groups?

Yeh, I didn't get to that. This took me a few hours to write and then write this up, so you know... take what you can get 🙃

Update: Oh alright, I added Roles support.

Output

The output is very minimal by default. If you want more detail, you can use verbosity flags -v|vv, but if you have lots of user files (e.g. thousands) then it's going to be quite long and might exceed your terminal's buffer. You may want to pipe it to a file. (Note that errors will get logged too.)

The command is safe and will not overwrite user records in your database by default. If you wish to force it to do so, use the --force or -f flag.

This means you should be able to keep on running the command until you get everything just right without creating loads of duplicates.

Errors

This script is simple so you shouldn't bump into any errors using it as-is, but depending on how old some of those YAML files are, you might bump into issues if you modify the migrate command to start normalising your data.

There's not much this command can do to help in those scenarios; you're on your own! I can only recommend a bevy of if-else and try-catch blocks to catch all of those little data oddities.

However, when you do encounter an error, you'll likely want to capture that and move on instead of stopping the process. In that case, you can use the helpful logError() method, which will keep a track of all the errors you report for a given user and log/output a useful summary at the end.

License

MIT.

Please leave the credits at the top wherever you use it.

MigrateUsers.php

<?php
/**
 * Statamic v3 User Migrator
 * @author Simon Hamp <simon.hamp@me.com>
 * @copyright Copyright (c) 2021, Simon Hamp
 * @license MIT
 */
namespace App\Console\Commands;

use Exception;
use TypeError;
use Carbon\Carbon;
use Statamic\Facades\User;
use Illuminate\Support\Str;
use Illuminate\Console\Command;
use App\Models\User as UserModel;
use Illuminate\Support\Facades\Log;
use Illuminate\Database\QueryException;
use Statamic\Auth\File\User as FileUser;
use Statamic\Auth\UserRepositoryManager;
use Statamic\Auth\Eloquent\User as DatabaseUser;
use Symfony\Component\Console\Output\OutputInterface;
use Statamic\Stache\Repositories\UserRepository as StacheUserRepository;

class MigrateUsers extends Command
{
    /**
     * The name and signature of the console command.
     *
     * @var string
     */
    protected $signature = 'users:migrate
        {--f|--force : Overwrite users in the database if they exist there already}';

    /**
     * The console command description.
     *
     * @var string
     */
    protected $description = 'Migrate your users from the filesystem to the database';

    /**
     * The column to search on to see if a user already exists in the database.
     *
     * @var string
     */
    protected $searchColumn = 'email';

    /**
     * The field on the file-based user record to find in the database $searchColumn.
     *
     * @var string
     */
    protected $searchField = 'email';

    protected $errors = [];

    protected $success = 0;

    protected $skipped = 0;

    /**
     * Execute the console command.
     *
     * @return int
     */
    public function handle()
    {
        $force = $this->option('force');

        User::swap(app(StacheUserRepository::class));

        try {
            $fileUsers = User::all();
        } catch (TypeError $e) {
            $this->error("Make sure your 'users' Stache store is configured in config/statamic/stache.php");
            return 1;
        }

        User::swap(app(UserRepositoryManager::class)->repository());

        $total = $fileUsers->count();

        $this->line("Found {$total} user records to migrate...");
        $bar = false;

        if ($this->getOutput()->getVerbosity() <= OutputInterface::VERBOSITY_VERBOSE) {
            $bar = $this->output->createProgressBar($total);

            $bar->start();
        }

        $fileUsers->each(function (FileUser $fileUser) use ($force, $bar) {
            $databaseUser = $this->prepDbUser($fileUser);

            if ($databaseUser->model()->exists) {
                $identifier = $this->searchValue($fileUser);

                if ($this->getOutput()->isVeryVerbose()) {
                    $this->warn("User [{$identifier}] already exists!" . ($force ? ' Overwriting...' : ''));
                }

                if (! $force) {
                    $this->skipped++;
                    return;
                }
            }

            if ($this->getOutput()->isVeryVerbose()) {
                $this->line('Migrating user: ' . $fileUser->email());
            }

            if ($this->migrate($fileUser, $databaseUser) && $this->getOutput()->isVeryVerbose()) {
                $this->info('User migrated! ID: ' . $databaseUser->getAuthIdentifier());
            }

            if ($bar) {
                $bar->advance();
            }
        });

        if ($bar) {
            $bar->finish();
            $this->newLine();
        }

        $this->newLine();
        $this->info("Successfully migrated {$this->success} out of {$total} users.");
        $this->line("Skipped {$this->skipped} users.");

        if (! empty($this->errors)) {
            $count = count($this->errors);

            $this->newLine();

            $this->warn("However, there were {$count} errors!");

            if ($this->getOutput()->isVerbose()) {
                $table = collect($this->errors)
                    ->flatMap(fn ($errors, $index) => collect($errors)->transform(fn ($error) => [$index, Str::limit($error)]));

                $this->table(['ID', 'Errors'], $table->all());

                $this->newLine();
            } else {
                $this->line("Use the `-v` option to see more details or check your log.");
            }

            return 1;
        }

        return 0;
    }

    protected function migrate(FileUser $fileUser, DatabaseUser $databaseUser): bool
    {
        // Standard fields
        /**
         * =============
         * A note on IDs
         * =============
         *
         * In case you need to keep referring to your Statamic users by their UUIDs, you'll need
         * to import this value too bu uncommenting the following line.
         *
         * You will need to have adjusted the migration to support this column _before_ migrating:
         * $table->uuid('statamic_id')->unique()
         *
         * And you may want to modify your User model to set this as the primary key:
         *
         * @see https://laravel.com/docs/8.x/eloquent#primary-keys
         */
        //$databaseUser->set('statamic_id', $fileUser->id());
        $databaseUser->set('name', $fileUser->name());
        $databaseUser->set('email', $fileUser->email());
        $databaseUser->set('super', (bool) $fileUser->isSuper());
        $databaseUser->model()->password = $fileUser->passwordHash();

        // Your custom fields here...

        // Roles
        foreach ($fileUser->roles() as $role) {
            $databaseUser->assignRole($role);
        }

        try {
            $databaseUser->save();

            $this->success++;

            return true;
        } catch (QueryException $e) {
            $this->logError($fileUser, $e->getMessage());
        }

        return false;
    }

    protected function prepDbUser(FileUser $fileUser): DatabaseUser
    {
        $model = UserModel::where($this->searchColumn, $this->searchValue($fileUser))->firstOrNew();

        /** @var DatabaseUser $user */
        $user = DatabaseUser::fromModel($model);

        return $user;
    }

    protected function searchValue(FileUser $fileUser): string
    {
        return $fileUser->fluentlyGetOrSet($this->searchField)->args([]);
    }

    protected function logError(FileUser $fileUser, string $error): void
    {
        $identifier = $this->searchValue($fileUser);
        $this->errors[$identifier][] = $error;
        if ($this->getOutput()->isVeryVerbose()) {
            $this->error("Failed to migrate user [{$identifier}]!");
        }
        Log::error("Failed to migrate user [{$identifier}]: {$error}");
    }
}

Hello,
I was wondering if you minded confirming something? I have swapped over to using sqlite, migration done, I reverted step 3 (only) to enable the user stache and ran your script, the following exception occurs when trying to get User::all(), I can't figure it out as it definately finds the first user in the list (verbose error writes out the YAML):

Illuminate\Contracts\Container\BindingResolutionException

Target [Statamic\Contracts\Auth\User] is not instantiable.

Not sure if something has changed in Statamic?

Two small issues:

https://statamic.dev/knowledge-base/storing-users-in-a-database

does not lead to anything - 404

I'm not sure if installing a starterkit is a clean installation or not, but

executing this:

php artisan users:migrate

generate this error:

   Illuminate\Contracts\Container\BindingResolutionException

  Target [Statamic\Contracts\Auth\User] is not instantiable.

  at vendor/laravel/framework/src/Illuminate/Container/Container.php:1091
    1087▕         } else {
    1088▕             $message = "Target [$concrete] is not instantiable.";
    1089▕         }
    1090▕
  ➜ 1091▕         throw new BindingResolutionException($message);
    1092▕     }
    1093▕
    1094▕     /**
    1095▕      * Throw an exception for an unresolvable primitive.

      +11 vendor frames
  12  [internal]:0
      Statamic\Stache\Stores\Store::Statamic\Stache\Stores\{closure}("90a8caf2-0195-4a15-b725-999e1d89284a")

      +7 vendor frames
  20  app/Console/Commands/MigrateUsers.php:74
      Illuminate\Support\Facades\Facade::__callStatic("all", [])

I believe that I have read the doc above, but any help would be appreciated

@rabol looks like that link has changed to https://statamic.dev/tips/storing-users-in-a-database

@netnakgraham both of you seem to be bumping into an issue with importing the wrong class. I've not bumped into this myself, so it would be helpful if you could share the code you're using

I am using the code from this page, just saved it to my app/Console/Commands folder

I'm running on a complete new installation.
I installed like this:

statamic new mysite studio1902/statamic-peak

Statamic\Contracts\Auth\User is a interface which is why it cannot be instantiated :)

What code would you like to see ?

@rabol perhaps you need to clear your stache?

I've fixed the Target [Statamic\Contracts\Auth\User] is not instantiable.-error by manually selecting the file repository.
Here's the beginning of my handle-method.

public function handle(\Statamic\Auth\UserRepositoryManager $userRepositoryManager): int
{
    $force = $this->option('force');

    /** @var \Statamic\Stache\Repositories\UserRepository $result */
    $stacheUserRepository = $userRepositoryManager->repository('file');

    try {
        $fileUsers = $stacheUserRepository->all();
    } catch (TypeError $e) {
        $this->error("Make sure your 'users' Stache store is configured in config/statamic/stache.php");
        return 1;
    }

    $total = $fileUsers->count();

    $this->line("Found {$total} user records to migrate...");
    $bar = false;

    if ($this->getOutput()->getVerbosity() <= OutputInterface::VERBOSITY_VERBOSE) {
        $bar = $this->output->createProgressBar($total);

        $bar->start();
    }

    // ...

}

I've fixed the Target [Statamic\Contracts\Auth\User] is not instantiable.-error by manually selecting the file repository. Here's the beginning of my handle-method.

Thanks @stefanzweifel, I ran into this as well.

Thanks @stefanzweifel 🙏

My code is attempting to do that, but maybe I've missed something. I will test this all again at some point and incorporate your fix

Call to undefined method Statamic\Stache\Repositories\UserRepository::fromModel()

I tried to replace it with fromUser but it doesn't work.

Call to undefined method Statamic\Stache\Repositories\UserRepository::fromModel()

I tried to replace it with fromUser but it doesn't work.

I have the same issue

Call to undefined method Statamic\Stache\Repositories\UserRepository::fromModel()

  at vendor/laravel/framework/src/Illuminate/Support/Facades/Facade.php:353
    349▕         if (! $instance) {
    350▕             throw new RuntimeException('A facade root has not been set.');
    351▕         }
    352▕ 
  ➜ 353▕         return $instance->$method(...$args);
    354▕     }
    355▕ }
    356▕ 

      +1 vendor frames 

  2   app/Console/Commands/MigrateUsers.php:203
      Statamic\Auth\User::__callStatic("fromModel")

  3   app/Console/Commands/MigrateUsers.php:92
      App\Console\Commands\MigrateUsers::prepDbUser(Object(Statamic\Auth\File\User))

I changed line 205 en 206 to:

  $user = new DatabaseUser();
  $user =  $user->model($model);

Call to undefined method Statamic\Stache\Repositories\UserRepository::fromModel()

I tried to replace it with fromUser but it doesn't work.

See: https://gist.github.com/simonhamp/a2b9113c100e5194db53298162f1dde0?permalink_comment_id=4728402#gistcomment-4728402