So, it has been an interesting journey, but time to remove git-lfs. Here follows a summary of the approach I used to safely remove git-lfs,
- commit & push everything
 - create a branch, something like fix/remove-lfs
 - remove hooks 
git lfs uninstall - remove lfs stuff from .gitattributes (open file, delete content - don't delete the file!)
 - list all lfs files, 
git lfs ls-files - run 
git rm --cachedfor each file- if your list is big, copy the contents into a file.txt
 
 - make sure you remove the number and asterik on each line, you only want the paths to the files