Migrates a Rasa tracker store (Redis, Mongo, SQL) to Rasa X.

migrate_tracker_store_to_rasa_x.py

# This gist is deprecated.
# Please follow the guide here 
# https://rasa.com/docs/rasa-x/installation-and-setup/existing-deployment/#import-existing-conversations-from-rasa-open-source
# to migrate from Rasa Open Source to Rasa X

Hi, I'm getting this error:

Traceback (most recent call last):
File "migrate_tracker_store_to_rasa_x.py", line 172, in
_migrate_tracker_store_to_rasa_x(path_to_endpoints_file, args.max_trackers)
File "migrate_tracker_store_to_rasa_x.py", line 66, in _migrate_tracker_store_to_rasa_x
db_session = db_utils.get_database_session(True, create_tables=True)
TypeError: get_database_session() got an unexpected keyword argument 'create_tables'

Thanks
Emannuel Roque

im getting the same error

In addition to above, I attempted to run this against my running Mongo tracking store for the bot and got the following error: [Errno -3] Temporary failure in name resolution

The mongo server is running and I confirmed via Mongo Express that my conversations are being logged.

TypeError: get_database_session() got an unexpected keyword argument 'create_tables'

will work on fixing this asap

Temporary failure in name resolution

seems you are using a host name, which can't be resolved by your dns server. What's your setup?

@Btibert3 @EmannuelRoque Fixed the script. Please let me know if you run into any more errors.

@wochinge It's working! Perfect! Thanks....

Getting this when using a sqlite backend.

(rasabot) Brocks-MBP-2:pqe-chatbot btibert$ python migrate_tracker_store_to_rasa_x.py 
INFO:apscheduler.scheduler:Scheduler started
Welcome to Rasa X 🚀 

This script will migrate your old tracker store to the new SQL based Rasa X tracker store.
Let's start!

? Please provide the path to your endpoints configuration which specified the credentials for your old tracker store endpoints.y
ml                                                                                                                              
Start migrating 0 trackers.
0it [00:00, ?it/s]
None
Traceback (most recent call last):
  File "migrate_tracker_store_to_rasa_x.py", line 172, in <module>
    _migrate_tracker_store_to_rasa_x(path_to_endpoints_file, args.max_trackers)
  File "migrate_tracker_store_to_rasa_x.py", line 95, in _migrate_tracker_store_to_rasa_x
    set_latest_event_id(db_session, rasa_x_tracker_store)
  File "migrate_tracker_store_to_rasa_x.py", line 154, in set_latest_event_id
    existing.latest_event_id = max_event_id
AttributeError: 'NoneType' object has no attribute 'latest_event_id'

@Btibert3 This seems weird. That could only happen if zero events were migrated. Maybe check your database logs if anything indicates errors there.

That was it, but strange that my local testing was creating the database, I was talking to it, but no dice. Grabbed my dev db and it worked great. Thanks!

I have a question regarding the use of this python script:
I run a mongdb docker container using the following command on Win10:
docker run -p 27017:27017 -v ./mongodb-clone:/data/db -d mongo

The mongodb-clone directory exists, it is a copy of the directory coming from the production server:
scp username@host:/my/own/datadir mongodb-clone

when I ran this migrating script:
python migrate_tracker_store_to_rasa_x.py

and got the following error:

Any hints???

Thanks.

Hello,

When I run the script, I receive this:

My new endpoint is like this:

tracker_store:
  type: sql
  dialect: "oracle"
  url: host
  port: 5555
  username: user
  password: pass
  db: db

What does this message mean, is it an error, warning, bug? The tables were created, but I don't know whether the process was correctly finished.
Can anybody help me?

Finally I managed to run "migrate_tracker_store_to_rasa_x.py", passing "new_endpoints.yml" like this:

tracker_store:
  type: "sql"
  dialect: oracle
  url: "oracle://admin:teste@127.0.0.1:1521/rasa"

But I take this:

sqlalchemy.exc.DatabaseError: (cx_Oracle.DatabaseError) ORA-00906: left parenthese not found
[SQL:
CREATE TABLE entity_synonym_value (
        id INTEGER NOT NULL,
        entity_synonym_id INTEGER,
        name VARCHAR2,
        PRIMARY KEY (id),
        FOREIGN KEY(entity_synonym_id) REFERENCES entity_synonym (id) ON DELETE cascade
)

]
(Background on this error at: http://sqlalche.me/e/4xp6)

@DiegoProtec Your error is a bug in Rasa X which makes in incompatible with Oracle. We already fixed it, and it will be part of the next release (some other things have to go in this as well, which is why I can't do that right now). Sorry for the inconvenience!

@FrancisThibaultNRC Fixed your error. The migration should have worked anyway, but this step is not longer necessary so I removed it.

@wochinge thank you so much, now after logger printed, I can understand the flow. I'm passing a new tracker store for converstaions at environment variable "DB_URL", it migrate my events and conversations but can't migrate the values to the tables, the tables stay empties.

I retried the new script and even if I specified the username and password in the endpoints.yml file, I got the following message while running the following command (sudo docker run -p 27017:27017 -v ~/git/aida/agents/agents/cybersickness/mongodb/data:/data/db -d mongo) on linux:
SASL SCRAM-SHA-1 authentication failed for rasa on admin from client 172.17.0.1:48740 ; UserNotFound: Could not find user "rasa" for db "admin"

This is the right username but not the right database name as specified in the following endpoints.yml:

tracker_store:
type: mongod
url: mongodb://localhost:27017
db: cybersickness
username: rasa
password: example

Any ideas?

@wochinge when you said this step is no longer necessary and you removed it, so which step were you referring to?

Thanks for your collaboration.

Hello!

When I run the "migrate.py", with environment variable "DB_URL":

DB_URL=oracle://admin:admin@localhost:1521/db
occurs this:

sqlalchemy.exc.DatabaseError: (cx_Oracle.DatabaseError) ORA-00936: expression not found
[SQL: select name, synonym from entity_synonym]
(Background on this error at: http://sqlalche.me/e/4xp6)

@DiegoProtec That's an issue with Rasa X and compatible. Can you please create an issue in the rasa repository (I believe you have already done that, didn't you)?

Hi @wochinge, here is there, issue.
Ty!

Hi @wochinge,

I have been able to migrate conversations from rasa tracker store into sql databases, thank you so much for your script, worked nicely. Could it be possible to extract conversations by given a range (456-700). I know that now we can extract a maximum number of conversations (--max-trackers 10), but it could be nice to extract based on a given range.

Thanks.

the script doesn't work in rasa-x 0.24.1

@nicolasfarina Which error are you getting?

@nicolasfarina Which error are you getting?

Version of Rasa X is 24.6 in Rasa x 23.5 it is work.

@nicolasfarina I believe that's because you have an older version of Rasa you're running the migration with, but you have a conversation on the database that has a document from Rasa 1.6 and up. I removed the document with the event session_started from my Mongo database and was able to run the migrations fine.

@wochinge Any solution or should we delete the event from the tracker.db?

I'm using Rasa 1.7.0 with Redis 5.0.7.
I get this error:

File "migrate_tracker_store_to_rasa_x.py", line 259, in
args.max_trackers,
File "migrate_tracker_store_to_rasa_x.py", line 89, in _migrate_tracker_store_to_rasa_x
old_tracker_store = _get_tracker_store_from_endpoints_config(old_endpoints_file)
File "migrate_tracker_store_to_rasa_x.py", line 170, in _get_tracker_store_from_endpoints_config
tracker_store = TrackerStore.find_tracker_store(
AttributeError: type object 'TrackerStore' has no attribute 'find_tracker_store'

Please use rasa export now which is officially maintained and doesn't break between releases :-)