Skip to content

Instantly share code, notes, and snippets.

@muhdiboy
Last active October 1, 2024 02:19
Show Gist options
  • Save muhdiboy/a293cbff355af750e3b8f45ec816d1f1 to your computer and use it in GitHub Desktop.
Save muhdiboy/a293cbff355af750e3b8f45ec816d1f1 to your computer and use it in GitHub Desktop.
LastFM automated duplicate scrobble deletion script

Why would I need this?

Your scrobbler might have decided to scrobble every song hundreds of times, and you can't really remove those scrobbles efficiently. Or you might have accidentally installed multiple scrobbler extensions at the same time - wondering why multiple scrobbles appear for every song played at a time - and you want to clear them after finding the issue.

Using this script still doesn't necessarily make the process quick since Last.fm only displays a limited number of scrobbles that can be removed on each page of your library. However unlike the implementation of @sk22 and its forks, this UserScript, which is derived from those scripts, is run once. The rest of the process is automated and the script will stop at the page you have set using the prompt.

Installation

Prerequisites

You will need some form of UserScript interpreter/injector plugin and a compatible browser.

  • For Google Chrome and Chromium-based browsers (e.g. Vivaldi, Opera or Brave), you need the Tampermonkey plugin. Just click on the link or search for it yourself on the Chrome Web Store and add the extension.
  • Currently not supported: For Firefox, you also need the Tampermonkey plugin (or alternatively Greasemonkey). Just click on one of the desired links or search for it yourself on the Firefox Browser Add-Ons page.

How to install the UserScript

Just click on the "Raw" button in the upper right-hand corner of the script file or by clicking here. Your extension should open an installation window where you confirm the installation of the UserScript. If that is not working, try to add the script manually to your extension by copying and pasting it.

Using the script - step by step

  • Open your Last.fm account's library while being logged in (https://www.last.fm/user/_/library). Navigate to the page from which you want to start removing duplicates by clicking through the pages at the bottom of the page or by entering the page number in the URL (e.g. https://www.last.fm/user/_/library?page=123).
    • Alternatively, you can choose a specific date range instead and navigate to the last or desired page.
  • Manually reload the page if the button doesn't appear.
  • You have the option to change the behaviour of the comparison process. By unchecking the checkbox, you can make the script ignore timestamps and compare over n titles. You'll be prompted to enter a number after clicking the button.
  • Click the button and enter the page number at which you want the script to stop.
    • If you have unchecked the checkbox, you will be prompted to enter a number for how many tracks you want to compare.
  • Now enjoy the purge :)
    • You can continue with other tasks on your PC. Just leave the window active in the background. I advise you to run the script in a separate window while doing other work. Using the same browser in another window is also fine. Avoid activating another tab, though.
    • It takes approximately 5 minutes to go through 100 pages (in my own case).
    • If you want to stop the script, you can click the Cancel button on the left-hand side.
  • When it reaches the page you have set at the prompt, the script will automatically stop.

Updating the script

You can set up the script to auto-update in the settings of the UserScript. Open up the plugin settings and select the "LastFM automated duplicate scrobble deletion script." Then, go to the script settings in the upper left-hand corner, check the "Search for updates" checkbox and save.

Alternatively, you can also update it manually by following the installation instructions above or searching for updates in the plugin overview.

Possible future updates

Done

  • Adding a button on the last.fm history page to initiate the script instead of activating and deactivating the Userscript.
    • added in v1.3
  • Implementing a "stop at page x" dialogue window.
    • added in v1.3
  • Implementing a "check x tracks for duplicates" dialogue window instead of editing "num" in the script
    • added in v1.3
  • Eliminate the need to manually edit the script by incorporating a checkbox or another method for configuring options.
    • added in v1.4

Thank-yous

Based on https://gist.github.com/sk22/39cc280840f9d82df574c15d6eda6629#gistcomment-3046698 Thanks to previous contributors @CennoxX, @mattsson, @gms8994, @huw and @sk22 Thanks to new contributions from @Eiron

Feel free to post any problems, fixes, improvements and feedback. I will try to help in any form. To anyone having more experience in Javascript or OpenUserJS/UserScript: your help is also needed to improve the script by implementing the above-mentioned possible future updates as I do not have the necessary skills to contribute.

// ==UserScript==
// @name LastFM automated duplicate scrobble deletion script
// @namespace https://gist.github.com/muhdiboy
// @version 1.5.0
// @downloadURL https://gist.github.com/muhdiboy/a293cbff355af750e3b8f45ec816d1f1/raw/lastfm-automated-remove-duplicates.user.js
// @updateURL https://gist.github.com/muhdiboy/a293cbff355af750e3b8f45ec816d1f1/raw/lastfm-automated-remove-duplicates.user.js
// @description Based on https://gist.github.com/sk22/39cc280840f9d82df574c15d6eda6629#gistcomment-3046698, thanks to previous contributors (@CennoxX, @mattsson, @gms8994, @huw and @sk22) and new contributors (@Eiron).
// @author muhdiboy
// @match https://www.last.fm/*user/*
// @icon data:image/gif;base64,R0lGODlhAQABAAAAACH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==
// @grant none
// ==/UserScript==
(function() {
// Variables
let runScript = localStorage.getItem('runScript') === 'true';
let rmOnlyRealDup = localStorage.getItem('rmOnlyRealDup') !== 'false'; // Set checkbox to checked/true as default
let numTracks = localStorage.getItem('numTracks') || "5";
let stopAtPage = localStorage.getItem('stopAtPage') || "0";
let currentPage = extractQueryParam('page', window.location.href) || 1;
// Update currentPage when a click event occurs on the page
window.addEventListener('click', function () {
currentPage = extractQueryParam('page', window.location.href) || 1;
});
// Functions
let pageURLCheckInterval = setInterval (function () {
// Check the URL at regular intervals to add or remove buttons
if ( this.lastPathStr !== location.pathname || this.lastQueryStr !== location.search) {
this.lastPathStr = location.pathname;
this.lastQueryStr = location.search;
if (shouldMatchPage()) createButtons(); else removeButtons();
}
}, 125);
function replaceQueryParam(param, newval, search) {
// Function to replace a query parameter in the URL
const regex = new RegExp("([?;&])" + param + "[^&;]*[;&]?");
const query = search.replace(regex, "$1").replace(/&$/, '');
return (query.length > 2 ? query + "&" : "?") + (newval ? param + "=" + newval : '');
}
function extractQueryParam(name, url) {
// Function to extract a query parameter from the URL
if (!url) url = window.location.href;
name = name.replace(/[\[]/, "\\[").replace(/[\]]/, "\\]");
const regexS = "[\\?&]" + name + "=([^&#]*)";
const regex = new RegExp(regexS);
const results = regex.exec(url);
return results == null ? null : results[1];
}
function shouldMatchPage() {
// Function to determine if the current page should be matched
const url = window.location.href;
const excludeUrlRegex = /^https:\/\/www\.last\.fm\/.*user\/.*\/library\/(albums|artists|tracks|music).*/;
const matchUrlRegex = /^https:\/\/www\.last\.fm\/.*user\/.*\/library.*/;
return matchUrlRegex.test(url) && !excludeUrlRegex.test(url);
}
function removeButtons() {
// Function to remove all buttons from the page
const cancelButton = document.getElementById('cancelButton');
const buttonContainer = document.getElementById('buttonContainer');
if (cancelButton) cancelButton.remove();
if (buttonContainer) buttonContainer.remove();
}
// Buttons
function createButtons() {
// Create the cancel button to stop the script
const cancelButton = document.createElement('button');
cancelButton.id = 'cancelButton';
cancelButton.innerHTML = 'Cancel<br>Script';
cancelButton.style.position = 'fixed';
cancelButton.style.bottom = '50%';
cancelButton.style.left = '0';
cancelButton.style.zIndex = '9999';
cancelButton.style.fontFamily = 'monospace';
cancelButton.style.padding = '3px';
cancelButton.style.border = 'dashed';
if (runScript) {
cancelButton.style.color = 'white';
cancelButton.style.background = 'red';
cancelButton.style.borderColor = 'firebrick';
cancelButton.addEventListener('click', function() {
localStorage.setItem('runScript', 'false');
localStorage.removeItem('savedUrl');
location.reload();
});
} else {
cancelButton.style.color = 'gray';
cancelButton.style.background = 'lightgray';
cancelButton.style.borderColor = 'lightgray';
cancelButton.style.cursor = 'default';
}
document.body.appendChild(cancelButton);
if (!runScript) {
// Create the main button to trigger the script
const runButton = document.createElement('button');
runButton.textContent = 'Remove Duplicates';
runButton.style.position = 'relative';
runButton.style.display = 'block';
runButton.style.marginBottom = '5px';
runButton.style.fontFamily = 'sans-serif';
runButton.style.fontSize = 'initial';
runButton.style.border = 'outset';
runButton.style.borderColor = 'crimson';
runButton.style.color = 'white';
runButton.style.background = 'firebrick';
runButton.style.padding = '5px';
runButton.addEventListener('click', function () {
let stopLoop = false;
while (!stopLoop) {
const inputStopAtPage = prompt("Stop at which page?", stopAtPage);
if (inputStopAtPage === null) return;
if (!/^\d+$/.test(inputStopAtPage)) {
alert("Invalid input. Please enter a valid number.");
continue;
}
if (inputStopAtPage > currentPage && !(currentPage === 0 && inputStopAtPage === 1)) {
alert("Please enter a page less than or equal to your current page: " + currentPage);
continue;
}
localStorage.setItem('stopAtPage', inputStopAtPage);
stopLoop = true;
}
if (!rmOnlyRealDup) {
stopLoop = false;
while (!stopLoop) {
const inputNumTracks = prompt("Enter the number of tracks to check for duplicates\n(between 2 and 50):", numTracks);
if (inputNumTracks === null) return;
if (!/^\d+$/.test(inputNumTracks) || inputNumTracks < 2 || inputNumTracks > 50) {
alert("Invalid input. Please enter a valid number between 2 and 50.");
continue;
}
localStorage.setItem('numTracks', inputNumTracks);
stopLoop = true;
}
}
localStorage.setItem('runScript', 'true');
location.reload();
});
// Create the checkbox element
const rmOnlyRealDupCheckbox = document.createElement('input');
rmOnlyRealDupCheckbox.type = 'checkbox';
rmOnlyRealDupCheckbox.id = 'rmOnlyRealDupCheckbox';
rmOnlyRealDupCheckbox.style.position = 'relative';
rmOnlyRealDupCheckbox.style.marginLeft = '10px';
rmOnlyRealDupCheckbox.checked = rmOnlyRealDup;
rmOnlyRealDupCheckbox.addEventListener('change', function () {
localStorage.setItem('rmOnlyRealDup', rmOnlyRealDupCheckbox.checked);
});
// Create the label for the checkbox
const rmOnlyRealDupCheckboxLabel = document.createElement('label');
rmOnlyRealDupCheckboxLabel.style.position = 'relative';
rmOnlyRealDupCheckboxLabel.style.marginLeft = '5px';
rmOnlyRealDupCheckboxLabel.style.fontFamily = 'sans-serif';
rmOnlyRealDupCheckboxLabel.style.fontSize = 'medium';
rmOnlyRealDupCheckboxLabel.setAttribute('for', 'rmOnlyRealDupCheckbox');
rmOnlyRealDupCheckboxLabel.textContent = 'Only compare real duplicates?';
// Create a container element to hold the main button and checkbox
const buttonContainer = document.createElement('div');
buttonContainer.id = 'buttonContainer';
buttonContainer.style.position = 'absolute';
buttonContainer.style.top = '370px';
buttonContainer.style.left = '50%';
buttonContainer.style.transform = 'translate(-50%, -50%)';
buttonContainer.appendChild(runButton);
buttonContainer.appendChild(rmOnlyRealDupCheckboxLabel);
buttonContainer.appendChild(rmOnlyRealDupCheckbox);
const metaElement = document.querySelector('.content-top-has-nav');
metaElement.insertAdjacentElement('afterend', buttonContainer);
}
}
// Automated duplicate scrobble deletion and navigation logic
if (shouldMatchPage()) {
window.addEventListener('load', function () {
// Event listener for when the page has finished loading
if (runScript) {
let found = 0;
const initialRefreshTimeout = 5000; // Interval for reloading the page
let refreshTimeout = parseInt(localStorage.getItem('refreshTimeout')) || initialRefreshTimeout;
const sections = Array.from(document.getElementsByTagName("tbody"));
const savedUrl = localStorage.getItem('savedUrl') || window.location.href;
// If the URL contains "delete", go back to last functioning site
if (window.location.toString().includes("delete")) window.location.href = savedUrl;
// If the current page doesn't contain tracks, go back to last functioning site (useful to detect rate-limiting page)
if (!document.querySelector('.chartlist')) {
setTimeout(function() {
window.location.href = savedUrl;
}, refreshTimeout);
localStorage.setItem('refreshTimeout', refreshTimeout + 5000);
return;
} else {
refreshTimeout = initialRefreshTimeout;
localStorage.setItem('refreshTimeout', initialRefreshTimeout);
localStorage.setItem('savedUrl', window.location.href);
}
sections.forEach(function (section) {
// Loop through each section
const els = Array.from(section.rows);
const names = els.map(function (el) {
const nmEl = el.querySelector('.chartlist-name');
const artEl = el.querySelector('.chartlist-artist');
if (rmOnlyRealDup) {
const tstEl = el.querySelector('.chartlist-timestamp');
// Construct the name string including track name, artist and timestamp
return nmEl && artEl && tstEl && nmEl.textContent.replace(/\s+/g, ' ').trim() + ':' + artEl.textContent.replace(/\s+/g, ' ').trim() + ':' + tstEl.textContent.replace(/\s+/g, ' ').trim();
} else {
// Construct the name string including only track name and artist
return nmEl && artEl && nmEl.textContent.replace(/\s+/g, ' ').trim() + ':' + artEl.textContent.replace(/\s+/g, ' ').trim();
}
});
names.forEach(function (name, i) {
// Loop through each name in the section
if (!names.slice(i + 1, i + 1 + parseInt(numTracks)).includes(name)) return;
// Check if the current name has duplicates within the specified range
const delBtn = els[i].querySelector('[data-ajax-form-sets-state="deleted"]');
// If a delete button is found, click it and increment the counter
if (delBtn) { delBtn.click(); found++; }
});
});
if (found > 0) {
// If duplicates were found, reload the page after refreshTimeout
setTimeout(function() {
location.reload();
}, refreshTimeout);
return;
}
if (currentPage <= stopAtPage) {
// Stop the Script and message user
localStorage.setItem('runScript', 'false');
localStorage.removeItem('savedUrl');
alert("LastFM duplicate deletion complete.");
location.reload();
}
// Go up one page
else window.location.href = window.location.pathname + replaceQueryParam('page', extractQueryParam('page', window.location.href) - 1, window.location.search);
}
});
}
})();
@muhdiboy
Copy link
Author

muhdiboy commented Feb 3, 2022

@Aethraaa
Copy link

i still couldn't figure it out, if it's possible can you make a tutorial video?

@muhdiboy
Copy link
Author

For everyone following this Gist, please update the script (with the updated How-to).

@Aethraaa, I've made it easier to install the script. Just click on the link in the How-to under How to install the script.
Let me know if you have issues with the rest of the usage.

@leahrr
Copy link

leahrr commented Mar 7, 2022

I leave mine on for a bit and I have years and years of scrobbles to delete and I come back and it didn't actually delete anything, its moving from page to page without doing anything

@muhdiboy
Copy link
Author

muhdiboy commented Mar 8, 2022

Hi @leahrr,

thank you for your feedback. I will look into this tomorrow. Until then please make sure that you really have duplicates and are using the latest version of the script. Also please tell me your Browser with version number.

@leahrr
Copy link

leahrr commented Mar 8, 2022

I have visible duplicates
I am using Chrome version 99.0.4844.51

@muhdiboy
Copy link
Author

muhdiboy commented Mar 11, 2022

@leahrr I couldn't reproduce it. Maybe you are not logged in? I'm sorry though. Maybe try out a different browser without the possibility of conflicting add-ons.

@hash-khat
Copy link

hash-khat commented Apr 2, 2022

Thanks for bringing this to the level it's at, muhdiboy. I'm sure there are untold many out there who appreciate it.

I used a manual version last year that worked fine. Using this on my usual browser (Edge) did nothing: moved quickly through the pages but deleted nothing (99.0.1150.55 (Official build) (64-bit)).

Tried on a fresh install of Chrome and the same behaviour: moves quick but deletes nothing (Version 100.0.4896.60 (Official Build) (64-bit)).

Hope this isn't too difficult to figure out because it's well beyond me to understand (I am a manual labourer). Even if you don't, thanks for the effort thus far!

@Eiron
Copy link

Eiron commented Apr 6, 2022

I made a couple of tweaks below.

One is to include an EventListener for the page load (lines 76 & 108).

The other is to also include the '.chartlist-timestamp' (lines 82-83).

In theory, this should wait for the page to load completely before checking for things to delete, and not remove those single-track binge sessions where you genuinely have repeated a song. It should catch, then, only those identical scrobbles with the same timestamp.

Code
// ==UserScript==
// @name         LastFM automated duplicate scrobble deletion script
// @namespace    https://gist.github.com/muhdiboy
// @version      1.1-u5
// @downloadURL  https://gist.github.com/muhdiboy/a293cbff355af750e3b8f45ec816d1f1/raw/lastfm-automated-remove-duplicates.user.js
// @updateURL    https://gist.github.com/muhdiboy/a293cbff355af750e3b8f45ec816d1f1/raw/lastfm-automated-remove-duplicates.user.js
// @description  Based on https://gist.github.com/sk22/39cc280840f9d82df574c15d6eda6629#gistcomment-3046698, thanks to previous contributors: @CennoxX, @mattsson, @gms8994, @huw and @sk22
// @author       muhdiboy
// @include      /^https:\/\/www\.last\.fm\/user\/.*\/library.*$/
// @exclude      /^https:\/\/www\.last\.fm\/user\/.*\/library\/(albums|artists|tracks).*$/
// @icon         data:image/gif;base64,R0lGODlhAQABAAAAACH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==
// @grant        none
// ==/UserScript==

(function() {
    var found = 0;
    var num = 5;
    var sections = Array.from(document.getElementsByTagName("tbody"));

    function exit( status ) {
        // source: https://web.archive.org/web/20091002005151/http://kevin.vanzonneveld.net:80/techblog/article/javascript_equivalent_for_phps_exit
        // http://kevin.vanzonneveld.net
        // +   original by: Brett Zamir (http://brettz9.blogspot.com)
        // +      input by: Paul
        // +   bugfixed by: Hyam Singer (http://www.impact-computing.com/)
        // +   improved by: Philip Peterson
        // +   bugfixed by: Brett Zamir (http://brettz9.blogspot.com)

        var i;

        if (typeof status === 'string') {
            alert(status);
        }

        window.addEventListener('error', function (e) {e.preventDefault();e.stopPropagation();}, false);

        var handlers = [
            'copy', 'cut', 'paste',
            'beforeunload', 'blur', 'change', 'click', 'contextmenu', 'dblclick', 'focus', 'keydown', 'keypress', 'keyup', 'mousedown', 'mousemove', 'mouseout', 'mouseover', 'mouseup', 'resize', 'scroll',
            'DOMNodeInserted', 'DOMNodeRemoved', 'DOMNodeRemovedFromDocument', 'DOMNodeInsertedIntoDocument', 'DOMAttrModified', 'DOMCharacterDataModified', 'DOMElementNameChanged', 'DOMAttributeNameChanged', 'DOMActivate', 'DOMFocusIn', 'DOMFocusOut', 'online', 'offline', 'textInput',
            'abort', 'close', 'dragdrop', 'load', 'paint', 'reset', 'select', 'submit', 'unload'
        ];

        function stopPropagation (e) {
            e.stopPropagation();
            // e.preventDefault(); // Stop for the form controls, etc., too?
        }
        for (i=0; i < handlers.length; i++) {
            window.addEventListener(handlers[i], function (e) {stopPropagation(e);}, true);
        }

        if (window.stop) {
            window.stop();
        }

        throw '';
    }

    function replaceQueryParam(param, newval, search) {
        var regex = new RegExp("([?;&])" + param + "[^&;]*[;&]?");
        var query = search.replace(regex, "$1").replace(/&$/, '');
        return (query.length > 2 ? query + "&" : "?") + (newval ? param + "=" + newval : '');
    }

    function gup( name, url ) {
        if (!url) url = location.href;
        name = name.replace(/[\[]/,"\\\[").replace(/[\]]/,"\\\]");
        var regexS = "[\\?&]"+name+"=([^&#]*)";
        var regex = new RegExp( regexS );
        var results = regex.exec( url );
        return results == null ? null : results[1];
    }



    window.addEventListener('load', function () {
        sections.forEach(function (section) {
            var els = Array.from(section.rows);
            var names = els.map(function (el) {
                var nmEl = el.querySelector('.chartlist-name');
                var artEl = el.querySelector('.chartlist-artist');
                var tstEl = el.querySelector('.chartlist-timestamp');
                return nmEl && artEl && tstEl && nmEl.textContent.replace(/\s+/g, ' ').trim()+':'+artEl.textContent.replace(/\s+/g, ' ').trim()+':'+tstEl.textContent.replace(/\s+/g, ' ').trim();
            });

            names.forEach(function (name, i, names) {
                if (!names.slice(i + 1, i + 1 + num).includes(name)) return;
                var delBtn = els[i].querySelector('[data-ajax-form-sets-state="deleted"]');
                if (delBtn) { delBtn.click(); found++; };
            });
        });


        if (window.location.toString().includes("delete")) {
            history.go(-1);
            exit();
        }

        if (found > 0) {
            setInterval(function() {
                location.reload();
            }, 5000);
        }

        if (window.location.toString().includes("page=-")) alert("LastFM duplicate deletion complete. Disable the UserScript!");

        else window.location.href = window.location.pathname + replaceQueryParam('page', gup('page', window.location.href) - 1, window.location.search);
    });

})();

@muhdiboy
Copy link
Author

muhdiboy commented Apr 7, 2022

Hello @Eiron ,

thank you very much for your input! It makes the script much more stable.
I released a new version with your additions and also added a variable to enable or disable removing only true duplicates. The reason for my addition is because I have a bug with my wireless earbuds that seem to play the last track on loop while they are not being used. The option is there for people who want to change the (now) default behaviour.

@alxjms92, also thank you for your feedback. I'm still not sure why that happens, as I've tested the script on fresh Firefox and Chrome installations. Maybe this new addition will be helpful. Test it out and report your experience please. Thank you!

@Eiron
Copy link

Eiron commented Apr 7, 2022

@muhdiboy: I had the same experience as @leahrr and @alxjms92. It is what prompted me to make the adjustments above. Hopefully, this new version will work for them!

Edit: Should also add that I really appreciate this tool. Removed over 1000 duplicates from over the years in a matter of minutes - it would have been an impossible task else.

@noChillGrandma
Copy link

April 30th, 2022... this worked like a charm! Thank you for making this!!!

@muhdiboy
Copy link
Author

muhdiboy commented Jun 7, 2022

I used it again today and noticed a warning from Tampermonkey about Chromium moving to Manifest V3 next year. So I changed the include parameter to match. Also some small changes to the comments and arrangement in the script.

Also I noticed that the script didn't run well and stopped after a few reloads when the tab ran in background. This didn't happen before and might be because of other running websites (Youtube, Twitch). Moving the tab to a seperate window solved the issue. So just a suggestion if people have the same difficulty to move the script running tab to a seperate window.

@muhdiboy
Copy link
Author

Hi everyone,

I've added many improvements to the script and updated the Readme. A button to start the script along with prompts to customize the script has been added.
Be sure to test it out. 😁

@bloodclot
Copy link

This does not work for me on a clean install of Chrome and FFX. This is what happens on chrome when any number of pages are entered in https://imgur.com/a/MzeToEo

On FFX, the button never appears

@muhdiboy
Copy link
Author

muhdiboy commented Jun 23, 2023

Thank you for providing feedback and putting in the effort to setup a new environment.

If I didn't see wrong you've put in page 5 to stop the script at while being on page 2. That won't work, because the script will stop when hitting that page number or when it has already passed it.

So a working example would be starting at page 5, enter a number lower than the current page, for example 3, and the script will go until it will be at page 3.

I‘ll look into why the button doesn't appear on Firefox, however it did on my testing before publishing.

@muhdiboy
Copy link
Author

I've updated the Gist to version 1.4 with many changes and a complete overhaul including a new checkbox, a Cancel button, prompts catching wrong input and many more things working in the background to make it more stable, reliable and easier to use.
It is no longer necessary to modify the script, everything is done via the buttons/checkbox.

However, I can confirm, that Firefox isn't behaving the same as Chromium browsers. It often stops after one or two reloads. Manual refresh can help it go further, but that is not efficient.
The Readme has also been updated accordingly.

Feel free to test it out and post your feedback.

@muhdiboy
Copy link
Author

Just a small info: Because of Last.fm's dynamic page design the buttons may appear in the wrong pages. This occurs sometimes when changing e.g. from "Library" to "Artists". Pressing them won't do anything because the script will always check before doing something. Also reloading the page will always remove them, if they shouldn't be there.

@bmrs
Copy link

bmrs commented Jul 31, 2023

Hi,

Thank you for this very enjoyable script. The process is not very smooth on my library with 160,000+ scrobbles on 3267 pages: it stops after a dozen of pages each time. It is running in a separate window from the last page to the page #1. Help appreciated!

@muhdiboy
Copy link
Author

I can confirm, that the script sometimes has issues to load the next page. I think the issue might be because the event listener (`window.addEventListener('load', function ()`, [line 178](https://gist.github.com/muhdiboy/a293cbff355af750e3b8f45ec816d1f1#file-lastfm-automated-remove-duplicates-user-js-L178)) is currently getting loaded after a few checks and inside a function, which might add a miniscule amount of delay.

To test this I added a timeout of 3-5 seconds to the listener, which would result in no reloads. After lowering it to 250 ms the issue could be reproduced more frequently and would almost disappear when reducing it to 25 ms.

I also checked how long it needs to reach the listener and measured ~200 ms. After that it is waiting for the page to load and that takes ~50-100 ms. That means that the script only has ~50 ms of time to reach that point, otherwise it will not run because the listener won't trigger as the website has already been loaded. So I have to minimize that 200 ms time window.

Fixed it by untying the run function from the URL checker function, et voilà, the script is reaching that spot instantly and only has to wait for the webpage to load which triggers the event listener.


Hi @bmrs, thank you for noticing and giving feedback. Please try again with the new version (1.4.3). Enjoy!

@bmrs
Copy link

bmrs commented Sep 25, 2023

Hi @bmrs, thank you for noticing and giving feedback. Please try again with the new version (1.4.3). Enjoy!

Yes, it works well now, and way faster… Thank you!

@reticivis-net
Copy link

Found a bug. On line 201, you add "num", but num is a string and needs to be cast to an int, or else it's concatenated, not added.

@pleebus
Copy link

pleebus commented Nov 28, 2023

hey dude, loving this script, is helping to do something I thought I was going to have to do semi-manually...

however, I'm hitting the "405 - Method Not Allowed" error every few pages. is this just because my last.fm history is so huge (4,000+ pages going back almost 20 years) with a ridiculous amount of duplicates? I had some bug a few years ago that caused some tracks to be scrobbled literally 50+ times for a single play and have only recently decided to do something about that. Thought this could be the solution, and it's working, but it's also not so stable and crashes after maximum 60 seconds each time I run it. am I doing something wrong, or is it just the size of the task I'm asking it to do?!

EDIT: If it helps to add some more context, it does seem that the redirect to /library/delete happens whenever it hits a block of duplicate songs. I'm using a fresh install of Brave that I downloaded just to run this script. But it's also happening in a fresh Chrome install.

@Jeboose
Copy link

Jeboose commented Dec 27, 2023

hey dude, loving this script, is helping to do something I thought I was going to have to do semi-manually...

however, I'm hitting the "405 - Method Not Allowed" error every few pages. is this just because my last.fm history is so huge (4,000+ pages going back almost 20 years) with a ridiculous amount of duplicates? I had some bug a few years ago that caused some tracks to be scrobbled literally 50+ times for a single play and have only recently decided to do something about that. Thought this could be the solution, and it's working, but it's also not so stable and crashes after maximum 60 seconds each time I run it. am I doing something wrong, or is it just the size of the task I'm asking it to do?!

EDIT: If it helps to add some more context, it does seem that the redirect to /library/delete happens whenever it hits a block of duplicate songs. I'm using a fresh install of Brave that I downloaded just to run this script. But it's also happening in a fresh Chrome install.

You're using tampermonkey with lastfm pro? I'm having trouble getting this script to do much of anything it seems to just endlessly scroll through my scrobbles without actually removing any. Tried firefox, chrome, and brave with no results.

@muhdiboy
Copy link
Author

muhdiboy commented Jan 9, 2024

@reticivis-net

Found a bug. On line 201, you add "num", but num is a string and needs to be cast to an int, or else it's concatenated, not added.

Thank you, I updated it to parse it to int. Version 1.4.4 onwards includes this change.


@pleebus

am I doing something wrong, or is it just the size of the task I'm asking it to do?!
I'm using a fresh install of Brave that I downloaded just to run this script. But it's also happening in a fresh Chrome install.

Sorry, that the script didn't work for you as expected. I've tried on multiple installations (fresh/old + Windows/Ubuntu(Flatpak, DEB)) and couldn't seem to reproduce the mentioned errors.

EDIT: If it helps to add some more context, it does seem that the redirect to /library/delete happens whenever it hits a block of duplicate songs.

Yes, after some retesting, that was what I've experienced too. And it was just that the line "if (window.location.toString().includes("delete")) history.go(-1);" was not in the right place.
Now it should go back in history and retry, if it hits the "405 - Methos Not Allowed" Page. Tested again after this change and it ran flawlessly again.

Change is included in version 1.4.5

@muhdiboy
Copy link
Author

muhdiboy commented Jan 9, 2024

@Jeboose
Can you retest with the newest version? If you have a different issue, please retest on a single page where you are sure multiple duplicates exist and report your findings. I will try to help as possible.

Just to be sure, if you didn't know the feature: the script is differentiating duplicates, ones that are scrobbled at exactly the same time (to the minute) and ones where you can choose the amount to be checked. First one is the true duplicate, enabled by the checkmark. Second one is active when the checkmark is unchecked. When unchecked, another input box will appear after hitting the "Remove duplicates" button. There you can input the number of scrobbles to compare to.
This is described in the doc above: Using the script - step by step

@RudeySH
Copy link

RudeySH commented Sep 4, 2024

Since June 2024, Last.fm has added rate-limiting. From my own testing I concluded that you can request about 20 pages per minute. Eventually you'll run into this:

image

Page not available
You’re requesting too many pages
You're requesting a lot of pages which may slow down the site for other users. Why not take a break and enjoy a cup of tea?

This messes up the automated script, because it triggers the rate-limiting very fast, and then the script will continuously skip pages until you cancel it.

@muhdiboy
Copy link
Author

muhdiboy commented Sep 9, 2024

Hey @RudeySH, thanks for your findings!

I can confirm the script's misbehavior. This issue only started occurring recently.
Unless you plan to monitor the entire process manually and stop/restart the script (which temporarily resets the problem), we need a fix to handle this.

Potential solutions:

  1. Increase Delay Before Reloading:

    • A longer delay between requests could prevent triggering the rate/request limit.
    • This might reduce the chance of encountering the rate-limit page, but may not fully solve the issue.
  2. Detect Text on the Page:

    • The current script relies on URL-based detection, which is not working in this case. It detects the rate-limit page as a normal library page.
    • I should implement a method to search for specific text on each reload and stop the script when it finds the rate-limit text (or something similar).

Also, I couldn’t find any clear documentation on LastFM’s rate-limiting rules, which makes this trickier.
If we had more information, it could help refine the script and avoid hitting the limit.


Something different I wanted to write down:
I've noticed an issue when deleting duplicates: sometimes the "405 - Method Not Allowed" page appears.
Currently, the method used to handle this is by navigating back to the previous page in the browser history.
However, this only works if the "405" page doesn't reappear consecutively, as it may cause the history to point to an incorrect page (e.g., a wrong library page).

To improve this, the method should be updated to store the active page's URL (or at least the page number) and return to this stored page whenever a "405" page is detected.
This would ensure that the script can always return to the last functioning page and continue its operation smoothly.

I'll work on fixing these issues over the following days and will update you via a comment to keep you informed.
Any further suggestions or ideas are welcome and greatly appreciated.

@RudeySH
Copy link

RudeySH commented Sep 9, 2024

I'm also maintaining a Last.fm script and have dealt with this issue myself.

I think the best course of action is to make the script detect the rate-limiting page, and automatically pauses for 10/20/30/etc. seconds before automatically trying again.

I advice against detecting the text on the page. The text might differ based on the user language. Instead, I would simply test if the page contains a <table> (or if you want to be really specific, a table with the chartlist CSS class).

A Last.fm script I wrote stores the amount of seconds to wait in a variable. The variable starts at 10 seconds. So, my Last.fm script only waits 10 seconds the first time it gets rate-limited. If it can successfully resume (without immediately getting rate-limited again) after 10 seconds, my script will continue loading pages. The next time it gets rate limited, it will still only wait 10 seconds. However, as soon as it detects rate-limiting immediately after waiting 10 seconds, it increases the variable by 10 seconds. From that moment onwards, the script will wait 20 seconds every time it gets rate-limited. When the script detects rate-limiting immediately after waiting 20 seconds, the variable is increased to 30 seconds. It goes on like that until the variable reaches 60 seconds. This approach might sound convoluted but it allows my script to load pages relatively fast without knowing the exact details of Last.fm's undocumented rate-limiting.

@muhdiboy
Copy link
Author

muhdiboy commented Oct 1, 2024

Hi @RudeySH,

thank you so much for giving tips. I've decided to go with a 5 second delay and increase it by another 5 seconds with each consecutive rate-limit.


New Release: Version 1.5.0!
Changes:

  • Syntax Adjustments: Improvements made for better readability and performance.
  • Type Fixes: Many variables now have corrected types for better functionality.
  • Functional Changes: Fixed the page reload function and removed clutter during execution.
  • Error Handling Improvement: Enhanced the script to correctly navigate back in case of errors.
  • Rate-Limiting Fix: The awaited solution for rate-limiting issues is now implemented.

If you encounter any issues or have suggestions for further improvements, just let me know!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment