Skip to content

Instantly share code, notes, and snippets.

Last active July 19, 2021 10:18
Show Gist options
  • Save jethrokuan/02f41028fb4a6f81787dc420fb99b6e4 to your computer and use it in GitHub Desktop.
Save jethrokuan/02f41028fb4a6f81787dc420fb99b6e4 to your computer and use it in GitHub Desktop.
Migration script
;;; PLEASE USE M-x org-roam-migrate-wizard, FOUND WITHIN THE ORG-ROAM REPO ITSELF.
Copy link

ghost commented Apr 28, 2021

the parenthesis on line 31 does not match

Copy link

meliache commented May 5, 2021

Thanks for the script. I had some issues that required manual intervention

1. Handling of links with backslashes in link text (e.g. for pretty symbols)

When migrating I got the error

progn: Invalid use of ‘\’ in replacement text

It seems the error was cause by roam file links with backslashes in the description, like the following:

 [[file:~/org/roam/][B \to \mu\nu]]

Maybe it was also the target file #+title, which also had the same backslash string. I fixed both to be sure and it worked.

2. Wrong quotes in alias conversion

I e.g. get

:ROAM_ALIASES: "NN \"neural network\""

where the outer quotes were not present in the original string, it should be NN "neural network".

3. Aliases added to wrong property drawer

This happened in files where I already had a property ID drawer at the top of the file and the #+ROAM_ALIAS: definition below it. Then, the aliases were added to the property drawer in the next org headline instead of the property drawer at the top of the file.

Copy link

This also should cover tags migration according to org-roam-tag-sources.

Copy link

d12frosted commented May 10, 2021

Seems like (kill-line) should be (kill-line 1) to kill through line and not just the rest of line.

You should get aliases using (split-string-and-unquote (cadar (org-collect-keywords '("roam_alias")))) to avoid the described issue with multi-word aliases. And asumming that the alias value is valid, you can simply get it and set it as is - the same with ref I think.

This is what I used for migration (requires d12frosted/vulpea#92):

(defun vulpea-migrate-buffer ()
  "Migrate current buffer note to `org-roam' v2."
  ;; Create file level ID if it doesn't exist yet
  (org-with-point-at 1

  ;; update title (just to make sure it's lowercase)
  (vulpea-buffer-title-set (vulpea-buffer-prop-get "title"))

  ;; move roam_key into properties drawer roam_ref
  (when-let* ((ref (vulpea-buffer-prop-get "roam_key")))
    (org-set-property "ROAM_REFS" ref)
    (let ((case-fold-search t))
      (org-with-point-at 1
        (while (re-search-forward "^#\\+roam_key:" (point-max) t)
          (kill-line 1)))))

  ;; move roam_alias into properties drawer roam_aliases
  (when-let* ((aliases (vulpea-buffer-prop-get-list "roam_alias")))
    (org-set-property "ROAM_ALIASES"
                      (combine-and-quote-strings aliases))
    (let ((case-fold-search t))
      (org-with-point-at 1
        (while (re-search-forward "^#\\+roam_alias:" (point-max) t)
          (kill-line 1)))))

  ;; move roam_tags into filetags
  (let* ((roam-tags (vulpea-buffer-prop-get-list "roam_tags"))
         (file-tags (vulpea-buffer-prop-get-list "filetags"))
         (path-tags (seq-filter
                     (lambda (x) (not (string-empty-p x)))
                       (file-name-directory (buffer-file-name)))
         (tags (seq-map
                (lambda (tag)
                  (setq tag (replace-regexp-in-string
                             ;; see `org-tag-re'
                             "_"        ; use any valid char - _@#%
                  (if (or
                       (string-prefix-p "status" tag 'ignore-case)
                       (string-prefix-p "content" tag 'ignore-case)
                       (string-equal "Project" tag))
                      (setq tag (downcase tag)))
                (seq-uniq (append roam-tags file-tags path-tags)))))
    (when tags
      (apply #'vulpea-buffer-tags-set tags)
      (let ((case-fold-search t))
        (org-with-point-at 1
          (while (re-search-forward "^#\\+roam_tags:" (point-max) t)
            (kill-line 1))))))


(defun vulpea-migrate-db ()
  "Migrate all notes."
  (dolist (f (org-roam--list-all-files))
    (with-current-buffer (find-file f)
      (message "migrating %s" f)

  ;; Step 2: Build cache
  (org-roam-db-sync 'force))

Update on 2021-05-18: the script doesn't override previous filetags anymore.

Update on 2021-06-09: the script handles tags much better by properly sanitizing names according to org-tag-re.

Copy link

Bugs when "#+roam_alias" exists:

$ git diff                
diff --git a/roam/ b/roam/
index 1f07744..9155df8 100644
--- a/roam/
+++ b/roam/
@@ -1,8 +1,11 @@
+:ID:       c20ceaae-7d3d-4f06-83a6-ba149b98f53f
+:ROAM_ALIASES: "Secure Hash Algorithm 2"
 #+CREATED: [2021-04-23 Fri 22:06]
-#+LAST_MODIFIED: [2021-04-24 Sat 20:29]
-#+ROAM_ALIAS: "Secure Hash Algorithm 2"
-#+ROAM_TAGS: crypto hash
+#+LAST_MODIFIED: [2021-05-14 Fri 14:18]
+#+ROAM_ALIAS:#+FILETAGS: crypto hash

 * Overview

Copy link

I know org-roam-bibtex does not support v2 yet, but there is also a small bug in your migration script, for example

diff --git a/roam/biblio/ b/roam/biblio/
index ece11a4..6e035e4 100644
--- a/roam/biblio/
+++ b/roam/biblio/
@@ -1,7 +1,10 @@
+:ID:       3c8b23c9-1942-45d6-b1a1-ee66d24a98a6
 #+TITLE: 2011 - [AC:BCGHKR11] - Program obfuscation with leaky hardware -- Bitansky, Canetti, Goldwasser, Halevi, Kalai, Rothblum
 #+CREATED: [2021-01-21 Thu 17:37]
-#+LAST_MODIFIED: [2021-01-21 Thu 17:37]
-#+ROAM_TAGS: paper crypto obfuscation
+#+LAST_MODIFIED: [2021-05-14 Fri 14:17]
+#+FILETAGS: paper crypto obfuscation
 #+PROPERTY: authors Bitansky, N., Canetti, R., Goldwasser, S., Halevi, S., Kalai, Y. T., & Rothblum, G. N.

Copy link

yosevu commented Jun 14, 2021

M-x-evaluate-buffer should be M-x eval-buffer.

Copy link

Correct me if I'm wrong, but this won't work, since org-roam-db-sync appears to be a v2 function, and other functions, like org-roam--list-all-files are v1. Unless I'm missing something?

Copy link

rodelrod commented Jul 10, 2021

@JonathanReeve: to make it work, I've ran step 1 in V1 and then steps 2 and 3 in V2.
Then I had to fix all of the files with ROAM_ALIASES by hand because they were borked. I didn't have many so I did it manually.
It was transforming something like:

#+title: PostgreSQL
#+created: [2020-04-03 17:22]
#+roam_alias: Postgres
- related :: [[][Databases]]

into something similar to:

:ID:       b6d673e3-d117-4bbe-806c-817effcb6050
:ROAM_ALIASES: [[][Databases]]
#+title: PostgreSQL
#+created: [2020-04-03 17:22]

Same thing with ROAM_KEY/ ROAM_REFS.

Copy link

M-x-evaluate-buffer should be M-x eval-buffer.


Copy link

When I eval step 1, I got user-error: Before first headline at position 1 in buffer Any idea?

Copy link

japhir commented Jul 19, 2021

I get some errors with titles that use entities, such as \delta or \endash.

Here's the error message I get:

Debugger entered--Lisp error: (error "Invalid use of ‘\\’ in replacement text")
  vulpea-buffer-prop-set("title" "dosSantos2010: Glacial\\textendash interglacial var...")
  vulpea-buffer-title-set("dosSantos2010: Glacial\\textendash interglacial var...")
  (save-current-buffer (set-buffer (find-file f)) (message "migrating %s" f) (vulpea-migrate-buffer))
  (with-current-buffer (find-file f) (message "migrating %s" f) (vulpea-migrate-buffer))
  (while --dolist-tail-- (setq f (car --dolist-tail--)) (with-current-buffer (find-file f) (message "migrating %s" f) (vulpea-migrate-buffer)) (setq --dolist-tail-- (cdr --dolist-tail--)))
  (let ((--dolist-tail-- (org-roam--list-all-files)) f) (while --dolist-tail-- (setq f (car --dolist-tail--)) (with-current-buffer (find-file f) (message "migrating %s" f) (vulpea-migrate-buffer)) (setq --dolist-tail-- (cdr --dolist-tail--))))
  (dolist (f (org-roam--list-all-files)) (with-current-buffer (find-file f) (message "migrating %s" f) (vulpea-migrate-buffer)))
  eval-expression((vulpea-migrate-db) nil nil 127)
  funcall-interactively(eval-expression (vulpea-migrate-db) nil nil 127)

Copy link

japhir commented Jul 19, 2021

(if I just replace the offending entity with the corresponding utf-8 symbol and run it again, it appears to continue nicely).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment