Sources : brkbeatjunkie and Nvidia's guide
You'll need Nvidia Profile Inspector to enable these settings : http://www.techspot.com/downloads/5077-nvidia-inspector.html There should be a similar alternative for AMD, but I haven't researched it.
WORKING EXAMPLES
If you just want working values, follow the pictures below. The values are explained in more detail in the AA and AO sections. The values in the screenshots were tested for a stable 60 FPS on a GTX 1080, at 4K. Depending on your performance, you can try higher values of "Antialiasing - Setting" and "Antialiasing - Transparency Supersampling".
ME1 : https://i.imgur.com/My9p7bb.png
ME2 : https://i.imgur.com/QW9fp1a.png For ME2, make sure to add the ME2Game.exe file to your Mass Effect 2 profile in Nvidia Profile Inspector. Otherwise the settings won't apply. If you don't know how to do that : https://i.imgur.com/TAWF7iD.png https://i.imgur.com/FLpWSfF.png
ME3 : https://i.imgur.com/9M4iKVF.png
ANTI ALIASING (AA)
AA reduces the "staircase" artifacts on lines. Like this : https://international.download.nvidia.com/geforce-com/international/comparisons/mass-effect-3-tweak-guide/Mass-Effect-3-Tweak-Guide-AntiAliasing-Comparison-AAOff-SGSSAA.html
Read this article to enable SGSSAA anti aliasing (works for ME1, ME2, ME3) : https://www.geforce.com/whats-new/guides/mass-effect-3-tweak-guide#8 For other games, check this document : https://docs.google.com/spreadsheets/d/1ekUZsK2YXgd5XjjH1M7QkHIQgKO_i4bHCUdPeAd6OCo/pub?output=html There's a good explanation of the differences between AA techniques, too.
AMBIENT OCCLUSION (AO)
To force HBAO use this document to find the HBAO flag for the game you want to add AO to : https://docs.google.com/spreadsheets/d/1sq0vgFlerS-YT093TS2jEpx6xjZP7X1x1BcAs7EHXYY/edit#gid=0 Then follow this picture to add the flag in Nvidia inspector : https://imgur.com/a/E2J7DFM Lastly, disable AO in the game's settings so that only HBAO is running, not the default game's AO (ME2, ME3 only).
There are some glitches in-game when using AO in the Mass Effect games. I wouldn't recommend having it on all the time, but it can be useful for screenshots. Here are some screenshots showing what the glitches look like : https://imgur.com/a/wfcgMiy (thanks roivasff14)
Nope, definitively something with the game itself. It's the same with SSAA on my R9 290.
My uneducate guess is that lights being objects (or at least that's what I read around the 3D depth communities) may be a culprit.
Shepard at the center of the comm room is probably at just about the right distance to make any anomaly pop out.
Idk why sampling-based anti-aliasing would trigger this, but after extensive testing with LOD bias independently I can say it's not that.
For the moment, the only safe bet for quality improvement is VSR/DSR/GeDoSaTo.