Skip to content

Instantly share code, notes, and snippets.

@jethrokuan

jethrokuan/migrate.el

Last active Jul 19, 2021
Embed
What would you like to do?
Migration script
;;; THIS MIGRATION SCRIPT IS NOW DEPRECATED.
;;; PLEASE USE M-x org-roam-migrate-wizard, FOUND WITHIN THE ORG-ROAM REPO ITSELF.
@sunzhe96

This comment has been minimized.

Copy link

@sunzhe96 sunzhe96 commented Apr 28, 2021

the parenthesis on line 31 does not match

@meliache

This comment has been minimized.

Copy link

@meliache meliache commented May 5, 2021

Thanks for the script. I had some issues that required manual intervention

1. Handling of links with backslashes in link text (e.g. for pretty symbols)

When migrating I got the error

progn: Invalid use of ‘\’ in replacement text

It seems the error was cause by roam file links with backslashes in the description, like the following:

 [[file:~/org/roam/20201208145501-b_to_mu_nu.org][B \to \mu\nu]]

Maybe it was also the target file #+title, which also had the same backslash string. I fixed both to be sure and it worked.

2. Wrong quotes in alias conversion

I e.g. get

:ROAM_ALIASES: "NN \"neural network\""

where the outer quotes were not present in the original string, it should be NN "neural network".

3. Aliases added to wrong property drawer

This happened in files where I already had a property ID drawer at the top of the file and the #+ROAM_ALIAS: definition below it. Then, the aliases were added to the property drawer in the next org headline instead of the property drawer at the top of the file.

@d12frosted

This comment has been minimized.

Copy link

@d12frosted d12frosted commented May 7, 2021

This also should cover tags migration according to org-roam-tag-sources.

@d12frosted

This comment has been minimized.

Copy link

@d12frosted d12frosted commented May 10, 2021

Seems like (kill-line) should be (kill-line 1) to kill through line and not just the rest of line.

You should get aliases using (split-string-and-unquote (cadar (org-collect-keywords '("roam_alias")))) to avoid the described issue with multi-word aliases. And asumming that the alias value is valid, you can simply get it and set it as is - the same with ref I think.

This is what I used for migration (requires d12frosted/vulpea#92):

(defun vulpea-migrate-buffer ()
  "Migrate current buffer note to `org-roam' v2."
  ;; Create file level ID if it doesn't exist yet
  (org-with-point-at 1
    (org-id-get-create))

  ;; update title (just to make sure it's lowercase)
  (vulpea-buffer-title-set (vulpea-buffer-prop-get "title"))

  ;; move roam_key into properties drawer roam_ref
  (when-let* ((ref (vulpea-buffer-prop-get "roam_key")))
    (org-set-property "ROAM_REFS" ref)
    (let ((case-fold-search t))
      (org-with-point-at 1
        (while (re-search-forward "^#\\+roam_key:" (point-max) t)
          (beginning-of-line)
          (kill-line 1)))))

  ;; move roam_alias into properties drawer roam_aliases
  (when-let* ((aliases (vulpea-buffer-prop-get-list "roam_alias")))
    (org-set-property "ROAM_ALIASES"
                      (combine-and-quote-strings aliases))
    (let ((case-fold-search t))
      (org-with-point-at 1
        (while (re-search-forward "^#\\+roam_alias:" (point-max) t)
          (beginning-of-line)
          (kill-line 1)))))

  ;; move roam_tags into filetags
  (let* ((roam-tags (vulpea-buffer-prop-get-list "roam_tags"))
         (file-tags (vulpea-buffer-prop-get-list "filetags"))
         (path-tags (seq-filter
                     (lambda (x) (not (string-empty-p x)))
                     (split-string
                      (string-remove-prefix
                       org-roam-directory
                       (file-name-directory (buffer-file-name)))
                      "/")))
         (tags (seq-map
                (lambda (tag)
                  (setq tag (replace-regexp-in-string
                             ;; see `org-tag-re'
                             "[^[:alnum:]_@#%]"
                             "_"        ; use any valid char - _@#%
                             tag))
                  (if (or
                       (string-prefix-p "status" tag 'ignore-case)
                       (string-prefix-p "content" tag 'ignore-case)
                       (string-equal "Project" tag))
                      (setq tag (downcase tag)))
                  tag)
                (seq-uniq (append roam-tags file-tags path-tags)))))
    (when tags
      (apply #'vulpea-buffer-tags-set tags)
      (let ((case-fold-search t))
        (org-with-point-at 1
          (while (re-search-forward "^#\\+roam_tags:" (point-max) t)
            (beginning-of-line)
            (kill-line 1))))))

  (save-buffer))

(defun vulpea-migrate-db ()
  "Migrate all notes."
  (dolist (f (org-roam--list-all-files))
    (with-current-buffer (find-file f)
      (message "migrating %s" f)
      (vulpea-migrate-buffer)))

  ;; Step 2: Build cache
  (org-roam-db-sync 'force))

Update on 2021-05-18: the script doesn't override previous filetags anymore.

Update on 2021-06-09: the script handles tags much better by properly sanitizing names according to org-tag-re.

@junwei-wang

This comment has been minimized.

Copy link

@junwei-wang junwei-wang commented May 14, 2021

Bugs when "#+roam_alias" exists:

$ git diff sha_2.org                
diff --git a/roam/sha_2.org b/roam/sha_2.org
index 1f07744..9155df8 100644
--- a/roam/sha_2.org
+++ b/roam/sha_2.org
@@ -1,8 +1,11 @@
+:PROPERTIES:
+:ID:       c20ceaae-7d3d-4f06-83a6-ba149b98f53f
+:ROAM_ALIASES: "Secure Hash Algorithm 2"
+:END:
 #+TITLE: SHA-2
 #+CREATED: [2021-04-23 Fri 22:06]
-#+LAST_MODIFIED: [2021-04-24 Sat 20:29]
-#+ROAM_ALIAS: "Secure Hash Algorithm 2"
-#+ROAM_TAGS: crypto hash
+#+LAST_MODIFIED: [2021-05-14 Fri 14:18]
+#+ROAM_ALIAS:#+FILETAGS: crypto hash

 * Overview

@junwei-wang

This comment has been minimized.

Copy link

@junwei-wang junwei-wang commented May 14, 2021

I know org-roam-bibtex does not support v2 yet, but there is also a small bug in your migration script, for example

diff --git a/roam/biblio/AC:BCGHKR11.org b/roam/biblio/AC:BCGHKR11.org
index ece11a4..6e035e4 100644
--- a/roam/biblio/AC:BCGHKR11.org
+++ b/roam/biblio/AC:BCGHKR11.org
@@ -1,7 +1,10 @@
+:PROPERTIES:
+:ID:       3c8b23c9-1942-45d6-b1a1-ee66d24a98a6
+:ROAM_REFS: cite:AC:BCGHKR11
+:END:
 #+TITLE: 2011 - [AC:BCGHKR11] - Program obfuscation with leaky hardware -- Bitansky, Canetti, Goldwasser, Halevi, Kalai, Rothblum
 #+CREATED: [2021-01-21 Thu 17:37]
-#+LAST_MODIFIED: [2021-01-21 Thu 17:37]
-#+ROAM_KEY: cite:AC:BCGHKR11
-#+ROAM_ALIAS:
-#+ROAM_TAGS: paper crypto obfuscation
+#+LAST_MODIFIED: [2021-05-14 Fri 14:17]
+#+ROAM_KEY:#+ROAM_ALIAS:
+#+FILETAGS: paper crypto obfuscation
 #+PROPERTY: authors Bitansky, N., Canetti, R., Goldwasser, S., Halevi, S., Kalai, Y. T., & Rothblum, G. N.

@yosevu

This comment has been minimized.

Copy link

@yosevu yosevu commented Jun 14, 2021

M-x-evaluate-buffer should be M-x eval-buffer.

@JonathanReeve

This comment has been minimized.

Copy link

@JonathanReeve JonathanReeve commented Jul 5, 2021

Correct me if I'm wrong, but this won't work, since org-roam-db-sync appears to be a v2 function, and other functions, like org-roam--list-all-files are v1. Unless I'm missing something?

@rodelrod

This comment has been minimized.

Copy link

@rodelrod rodelrod commented Jul 10, 2021

@JonathanReeve: to make it work, I've ran step 1 in V1 and then steps 2 and 3 in V2.
Then I had to fix all of the files with ROAM_ALIASES by hand because they were borked. I didn't have many so I did it manually.
It was transforming something like:

#+title: PostgreSQL
#+created: [2020-04-03 17:22]
#+roam_alias: Postgres
- related :: [[file:databases.org][Databases]]

into something similar to:

:PROPERTIES:
:ID:       b6d673e3-d117-4bbe-806c-817effcb6050
:ROAM_ALIASES: [[file:databases.org][Databases]]
:END:
#+title: PostgreSQL
#+created: [2020-04-03 17:22]
#+roam_alias: 

Same thing with ROAM_KEY/ ROAM_REFS.

@rodelrod

This comment has been minimized.

Copy link

@rodelrod rodelrod commented Jul 10, 2021

M-x-evaluate-buffer should be M-x eval-buffer.

Confirmed

@liebkne

This comment has been minimized.

Copy link

@liebkne liebkne commented Jul 14, 2021

When I eval step 1, I got user-error: Before first headline at position 1 in buffer conditional_random_fields.org. Any idea?

@japhir

This comment has been minimized.

Copy link

@japhir japhir commented Jul 19, 2021

I get some errors with titles that use entities, such as \delta or \endash.

Here's the error message I get:

Debugger entered--Lisp error: (error "Invalid use of ‘\\’ in replacement text")
  vulpea-buffer-prop-set("title" "dosSantos2010: Glacial\\textendash interglacial var...")
  vulpea-buffer-title-set("dosSantos2010: Glacial\\textendash interglacial var...")
  vulpea-migrate-buffer()
  (save-current-buffer (set-buffer (find-file f)) (message "migrating %s" f) (vulpea-migrate-buffer))
  (with-current-buffer (find-file f) (message "migrating %s" f) (vulpea-migrate-buffer))
  (while --dolist-tail-- (setq f (car --dolist-tail--)) (with-current-buffer (find-file f) (message "migrating %s" f) (vulpea-migrate-buffer)) (setq --dolist-tail-- (cdr --dolist-tail--)))
  (let ((--dolist-tail-- (org-roam--list-all-files)) f) (while --dolist-tail-- (setq f (car --dolist-tail--)) (with-current-buffer (find-file f) (message "migrating %s" f) (vulpea-migrate-buffer)) (setq --dolist-tail-- (cdr --dolist-tail--))))
  (dolist (f (org-roam--list-all-files)) (with-current-buffer (find-file f) (message "migrating %s" f) (vulpea-migrate-buffer)))
  vulpea-migrate-db()
  eval-expression((vulpea-migrate-db) nil nil 127)
  funcall-interactively(eval-expression (vulpea-migrate-db) nil nil 127)
  command-execute(eval-expression)
@japhir

This comment has been minimized.

Copy link

@japhir japhir commented Jul 19, 2021

(if I just replace the offending entity with the corresponding utf-8 symbol and run it again, it appears to continue nicely).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment