Skip to content

Instantly share code, notes, and snippets.

@tillahoffmann
Last active October 6, 2023 21:30
Show Gist options
  • Star 6 You must be signed in to star a gist
  • Fork 3 You must be signed in to fork a gist
  • Save tillahoffmann/ac583f40d7898da410284c5b5aba8399 to your computer and use it in GitHub Desktop.
Save tillahoffmann/ac583f40d7898da410284c5b5aba8399 to your computer and use it in GitHub Desktop.
Readcube-Overleaf integration: Adds an "Update Library" button to Overleaf that allows you to import your Readcube library.
// ==UserScript==
// @name Readcube-Overleaf integration
// @namespace https://tillahoffmann.github.io/
// @version 0.1
// @description Adds an "Update Library" button to Overleaf that allows you to import your Readcube library.
// @author Till Hoffmann
// @match https://www.overleaf.com/*
// @connect readcube.com
// @grant GM_xmlhttpRequest
// ==/UserScript==
function formatPageNumbers(x) {
// Remove spaces
x = x.replaceAll(/\s/g, '');
// Replace single dashes with double dashes
x = x.replaceAll(/(?<=\d)-(?=\d)/g, '--');
return x;
}
const formattingLookup = {
'title': 'article/title',
'journal': 'article/journal',
'pages': {
'path': 'article/pagination',
'format': formatPageNumbers,
},
'volume': 'article/volume',
'year': 'article/year',
'doi': 'ext_ids/doi',
};
function formatItem(item, usedKeys) {
var citekey = item.user_data.citekey;
// Cite key is available
if (citekey) {
citekey = citekey.replaceAll("'", "");
}
// Generate a cite key based on the author
else if (item.article.authors.length) {
var author = item.article.authors[0].split(/\s+/);
// Get the last name of the author (naively, anyway)
citekey = author[author.length - 1];
// Add the year if it's available
if (item.article.year) {
citekey = citekey + item.article.year;
}
// Add a suffix if it's not unique
if (usedKeys[citekey]) {
usedKeys[citekey] += 1;
citekey += String.fromCharCode(95 + usedKeys[citekey]);
} else {
usedKeys[citekey] = 1;
}
}
// Just generate something random
else {
var randomInt = Math.floor(Math.random() * 0xffffffff);
citekey = randomInt.toString(16);
}
var lines = [
'@' + item.item_type + '{' + citekey + ',',
' author = {' + (item.article.authors || []).join(' and ') + '},',
];
for (var [key, value] of Object.entries(formattingLookup)) {
if (typeof value === "string") {
value = {'path': value};
}
var x = item;
for (var subpath of value.path.split('/')) {
x = x[subpath];
if (x === undefined) {
break;
}
}
if (x) {
if (value.format) {
x = value.format(x);
}
lines.push(' ' + key + ' = {' + x + '},');
}
}
lines.push('}');
return lines.join('\n');
}
function fetchItems(config) {
// Construct the url
var url = config.baseUrl;
if (config.scrollId) {
url = url + '?scroll_id=' + config.scrollId;
}
GM_xmlhttpRequest({
method: "GET",
url: url,
onload: function(response) {
unsafeWindow.config = config;
var data = JSON.parse(response.responseText);
if (data.items.length > 0) {
config.items = (config.items || []).concat(data.items);
// Continue recursively
fetchItems(Object.assign({}, config, {scrollId: data.scroll_id}));
} else if (config.callback) {
config.callback(config);
}
}
});
}
function updateLibrary() {
// Get the first editor and fetch the code
var editor = null;
for (let element of document.getElementsByClassName('ace_editor')) {
editor = ace.edit(element);
break;
}
var code = editor.getValue();
// Try to match the URL pattern and load the items
var pattern = /%% https?:\/\/app.readcube.com\/library\/([\w-]+)\/list\/([\w-]+)/gi;
for (var match of code.matchAll(pattern)) {
console.log('library: ', match[1]);
console.log('list: ', match[2]);
var url = 'https://sync.readcube.com/collections/' + match[1] + '/lists/' + match[2] + '/items';
console.log('fetching ' + url);
fetchItems({
'baseUrl': url,
'match': match,
'editor': editor,
'callback': function(config) {
// Format the code and add it to the editor
console.log(config);
// Sort the items for consistent cite key generation
config.items.sort(function(a, b) {
if (a.id < b.id) {
return -1;
}
return 1;
});
var lines = [config.match[0]];
var usedKeys = {};
for (var item of config.items) {
lines.push(formatItem(item, usedKeys));
}
editor.setValue(lines.join('\n\n'));
},
});
// We only process one library (for now anyway)
break;
}
}
(function() {
'use strict';
// Inject the update button in the toolbar
setInterval(function() {
var parent = document.querySelector('.toolbar-editor .toolbar-right');
if (parent.getAttribute('readcube')) {
return;
}
var child = document.createElement('a');
child.innerText = 'Update Library';
child.onclick = function() {
updateLibrary();
}
parent.insertBefore(child, parent.firstChild);
parent.setAttribute('readcube', 'injected');
}, 1000);
})();
@tillahoffmann
Copy link
Author

Readcube - Overleaf integration

Copy a ReadCube library url (preceded by %% ) into your bibliography file on Overleaf and hit the "Update Library" button to import your library as bibtex. Here's an example file.

%% https://app.readcube.com/library/4cfcacca-b4d3-4bc3-9fad-e3bb3b9e25fd/list/0bd2089a-c080-474b-8d48-dc527cd441d0

[your library will be populated here automatically]

Installation

Make sure you have Tampermonkey installed and click here to install the user script.

@PaulVanSchayck
Copy link

Thanks for this! Really nice workaround for having an easy way to synchronise between Overleaf and ReadCube/Papers.

I ran into a small issue regarding the presence of authors in the data. And also made the "Update Library" button a bit nicer. See my revision here.

@tillahoffmann
Copy link
Author

Nice, thanks for the improvements!

@ryanringle
Copy link

This looks great! I've just installed Tampermonkey for Safari, but don't see the button. Where is it supposed to appear? I've added a library URL to a test project, and when I'm in the Overleaf project I can see that the script is enabled.

@PaulVanSchayck
Copy link

PaulVanSchayck commented Jan 7, 2022

image

This is how it looks with my revision. In this original version, button is in the same location. But the text is a bit mangled, also doesn't really look like a button.

@ryanringle
Copy link

Hmm...doesn't seem to be working on my end (Safari 15.2 on 14" MacBook Pro).
Screen Shot 2022-01-07 at 9 12 09 AM

@PaulVanSchayck
Copy link

Right click and inspect, check the console after a reload. There is probably an error somewhere.

@mathkaren
Copy link

Thanks, this looks nice and helpful. I got a similar problem not seeing the "update library" bottom, and my console showed an error on line 164 (see attached). Could you help and take a look at it? Cheers~
Screen Shot 2022-06-13 at 4 31 26 PM

@tillahoffmann
Copy link
Author

Hi @mathkaren, I'm afraid I'm no longer using readcube and have resorted back to standard bibtex because it is lacking some features I needed (like proper handling of first and last names). Contributions in the form of revisions are more than welcome, however.

@PaulVanSchayck
Copy link

Hi @mathkaren . I ran into the same issue, it's due to Readcube having updated their layout.

You can use the forked version from @MichalPt from here. It has a fixed for this.

https://gist.github.com/MichalPt/0d08085321d03a63aca07720785849c8

@mathkaren
Copy link

mathkaren commented Jun 18, 2022 via email

@PaulVanSchayck
Copy link

My fork now also contains code to escape Latex, in case a title contains for example the ampersand (&) character. The changes from @MichalPt are also included.

@martinfell
Copy link

Hi all, I just tried to get this running but so far I also have the problem of not seeing the Update library button. I checked the console but I'm afraid I don't understand it. Can anyone help with this? Of note, I'm using Paul's forked version.
Screenshot 2022-11-16 at 11 30 22

@PaulVanSchayck
Copy link

It still works for me. Note that I'm using the "Source (legacy)" editor. It doesn't work with the Rich Text editor, or the new "Source" Latex editor.

@martinfell
Copy link

Good news! Readcube/Papers is finally integrated into Overleaf:
https://www.papersapp.com/write-more-productively-in-overleaf-using-papers/

@jaepillee0315
Copy link

It still works for me. Note that I'm using the "Source (legacy)" editor. It doesn't work with the Rich Text editor, or the new "Source" Latex editor.

Hello Paul, as the legacy source editor becomes obsolete, I wonder if there is still a way to use this in the new source latex editor. Do you know why this code does not work in the new Source Latex editor?

@MichalPt
Copy link

MichalPt commented May 3, 2023

Hi @jaepillee0315 , I updated the original code to support the new editor used by Overleaf. You can find it in my branch: https://gist.github.com/MichalPt/0d08085321d03a63aca07720785849c8#file-readleaf-user-js

@jaepillee0315
Copy link

Hi @jaepillee0315 , I updated the original code to support the new editor used by Overleaf. You can find it in my branch: https://gist.github.com/MichalPt/0d08085321d03a63aca07720785849c8#file-readleaf-user-js

This is awesome. Thank you so much!

@PaulVanSchayck
Copy link

@MichalPt Thanks a lot for figuring this out. I also had the issue with the new editor, but only had time to build an ugly hack to do the conversion and offer it as file download (which I then manually copied into Overleaf again). This is much better. Thanks for figuring out the obfuscated Overleaf JS code.

@ScedasticQ
Copy link

I have a likely very stupid question. How is everybody managing to get the library URL? For me, the URL is hidden (I just see the papers normal website). And if I share my library as public, it gives me a link that looks nothing like the one people here have. I'd appreciate any help!

@jaepillee0315
Copy link

I have a likely very stupid question. How is everybody managing to get the library URL? For me, the URL is hidden (I just see the papers normal website). And if I share my library as public, it gives me a link that looks nothing like the one people here have. I'd appreciate any help!

I can directly see the URL when I access the personal library in the search bar on Google Chrome. It looks like "app.readcube.com/library/[bunch of letters and numbers for the library]". It looks like Google Chrome also has a setting for seeing the full URLs. Check the bar below---you can get this window by right-clicking the search bar.

image

image

@ScedasticQ
Copy link

@jaepillee0315 Wow thank you for the quick response! Unfortunately, I'm still confused. On the webapp, if I click nothing, this is all I get "https://app.readcube.com/" regardless of "Always show full URLs." If I click the "My papers" folder, I get "https://app.readcube.com/library/0a5458f1-92f7-4b8d-91cb-33723708d367/all" and if I click a specific paper, I get "https://app.readcube.com/library/0a5458f1-92f7-4b8d-91cb-33723708d367/all/(sidepanel:details)?item_id=430c97a7-5c3b-4322-a36f-36de89796df9&collection_id=0a5458f1-92f7-4b8d-91cb-33723708d367"

None of the links resemble the ones here. The first number is there, but the number after "list" isn't, and the word list itself too.

If I use any of those links, the file in Overleaf doesn't fill up with anything when I click the Update ReadCube button.

@jaepillee0315
Copy link

@jaepillee0315 Wow thank you for the quick response! Unfortunately, I'm still confused. On the webapp, if I click nothing, this is all I get "https://app.readcube.com/" regardless of "Always show full URLs." If I click the "My papers" folder, I get "https://app.readcube.com/library/0a5458f1-92f7-4b8d-91cb-33723708d367/all" and if I click a specific paper, I get "https://app.readcube.com/library/0a5458f1-92f7-4b8d-91cb-33723708d367/all/(sidepanel:details)?item_id=430c97a7-5c3b-4322-a36f-36de89796df9&collection_id=0a5458f1-92f7-4b8d-91cb-33723708d367"

None of the links resemble the ones here. The first number is there, but the number after "list" isn't, and the word list itself too.

If I use any of those links, the file in Overleaf doesn't fill up with anything when I click the Update ReadCube button.

You need to create a list. Click the gear button next to "My Papers," click "Create List," and give a name to the list. Then, assign the papers you want to cite to the list (When you click a paper, you will see there are "TAGS", "LISTS", "CITEKEY", etc. Use "LIST" to add the paper to the list). Then, use the list's URL on Overleaf. Make sure you put double percent sign (%%) in front of the URL. I hope this is useful.

@MichalPt
Copy link

Hi, just to let you all know, I've put together a new version of the code. Now, the imported bib entries can be sorted according to the name of the first author. This is necessary in order to have the multi-citatations with natbib properly sorted in the PDF. I've added a toggle to use this option (set as default) or not which appears after hovering over the import button for 1.5 s. Besides, a minor bug stemming from the Overleaf new text editor behaviour, which prevented re-importing bib entries if scrolled down far enough, is bypassed by automatic scroll-to-the-top in the editor before the import itself.
img1
img2
The revision is available in my repository: https://gist.github.com/MichalPt/0d08085321d03a63aca07720785849c8

@PaulVanSchayck
Copy link

Thanks @MichalPt . Nice workaround with the editor.

Another thing I recently added to my version is to rewrite the ReadCube type to the Bibtex type. Unfortunately, they don't always match:

function formatType(x) {
    const types = {
        'conference_paper' : 'inproceedings',
        'poster': 'inproceedings',
        'webpage': 'misc'
    }

    if( x in types ) {
        return types[x]
    } else {
        return x
    }
}

And call formatType(item.item_type)

@PaulVanSchayck
Copy link

I've updated my revision again. Changes:

  • Merged changes from @MichalPt (scrolling to top, sorting)
  • Added the formatType() function from above
  • Added a formatRaw() for URL and DOI to prevent from escaping special chars in those fields.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment