Sources : brkbeatjunkie and Nvidia's guide
You'll need Nvidia Profile Inspector to enable these settings : http://www.techspot.com/downloads/5077-nvidia-inspector.html There should be a similar alternative for AMD, but I haven't researched it.
WORKING EXAMPLES
If you just want working values, follow the pictures below. The values are explained in more detail in the AA and AO sections. The values in the screenshots were tested for a stable 60 FPS on a GTX 1080, at 4K. Depending on your performance, you can try higher values of "Antialiasing - Setting" and "Antialiasing - Transparency Supersampling".
ME1 : https://i.imgur.com/My9p7bb.png
ME2 : https://i.imgur.com/QW9fp1a.png For ME2, make sure to add the ME2Game.exe file to your Mass Effect 2 profile in Nvidia Profile Inspector. Otherwise the settings won't apply. If you don't know how to do that : https://i.imgur.com/TAWF7iD.png https://i.imgur.com/FLpWSfF.png
ME3 : https://i.imgur.com/9M4iKVF.png
ANTI ALIASING (AA)
AA reduces the "staircase" artifacts on lines. Like this : https://international.download.nvidia.com/geforce-com/international/comparisons/mass-effect-3-tweak-guide/Mass-Effect-3-Tweak-Guide-AntiAliasing-Comparison-AAOff-SGSSAA.html
Read this article to enable SGSSAA anti aliasing (works for ME1, ME2, ME3) : https://www.geforce.com/whats-new/guides/mass-effect-3-tweak-guide#8 For other games, check this document : https://docs.google.com/spreadsheets/d/1ekUZsK2YXgd5XjjH1M7QkHIQgKO_i4bHCUdPeAd6OCo/pub?output=html There's a good explanation of the differences between AA techniques, too.
AMBIENT OCCLUSION (AO)
To force HBAO use this document to find the HBAO flag for the game you want to add AO to : https://docs.google.com/spreadsheets/d/1sq0vgFlerS-YT093TS2jEpx6xjZP7X1x1BcAs7EHXYY/edit#gid=0 Then follow this picture to add the flag in Nvidia inspector : https://imgur.com/a/E2J7DFM Lastly, disable AO in the game's settings so that only HBAO is running, not the default game's AO (ME2, ME3 only).
There are some glitches in-game when using AO in the Mass Effect games. I wouldn't recommend having it on all the time, but it can be useful for screenshots. Here are some screenshots showing what the glitches look like : https://imgur.com/a/wfcgMiy (thanks roivasff14)
I'm afraid those AA bits still don't cut it (at least in ME1).
Even if they are better than the default with bugged lens flare, I found they still screw up lights.
This is the default "application-controlled" mode, while this is with msaa and it has a quite stupid blue tint cranked up to 11.
I have tried all the compatibility bits I could find on the net, to no avail.
Idk then if it couldn't just be the last nvidia drivers are bugged, or something with my gpu.