Sources : brkbeatjunkie and Nvidia's guide
You'll need Nvidia Profile Inspector to enable these settings : http://www.techspot.com/downloads/5077-nvidia-inspector.html There should be a similar alternative for AMD, but I haven't researched it.
WORKING EXAMPLES
If you just want working values, follow the pictures below. The values are explained in more detail in the AA and AO sections. The values in the screenshots were tested for a stable 60 FPS on a GTX 1080, at 4K. Depending on your performance, you can try higher values of "Antialiasing - Setting" and "Antialiasing - Transparency Supersampling".
ME1 : https://i.imgur.com/My9p7bb.png
ME2 : https://i.imgur.com/QW9fp1a.png For ME2, make sure to add the ME2Game.exe file to your Mass Effect 2 profile in Nvidia Profile Inspector. Otherwise the settings won't apply. If you don't know how to do that : https://i.imgur.com/TAWF7iD.png https://i.imgur.com/FLpWSfF.png
ME3 : https://i.imgur.com/9M4iKVF.png
ANTI ALIASING (AA)
AA reduces the "staircase" artifacts on lines. Like this : https://international.download.nvidia.com/geforce-com/international/comparisons/mass-effect-3-tweak-guide/Mass-Effect-3-Tweak-Guide-AntiAliasing-Comparison-AAOff-SGSSAA.html
Read this article to enable SGSSAA anti aliasing (works for ME1, ME2, ME3) : https://www.geforce.com/whats-new/guides/mass-effect-3-tweak-guide#8 For other games, check this document : https://docs.google.com/spreadsheets/d/1ekUZsK2YXgd5XjjH1M7QkHIQgKO_i4bHCUdPeAd6OCo/pub?output=html There's a good explanation of the differences between AA techniques, too.
AMBIENT OCCLUSION (AO)
To force HBAO use this document to find the HBAO flag for the game you want to add AO to : https://docs.google.com/spreadsheets/d/1sq0vgFlerS-YT093TS2jEpx6xjZP7X1x1BcAs7EHXYY/edit#gid=0 Then follow this picture to add the flag in Nvidia inspector : https://imgur.com/a/E2J7DFM Lastly, disable AO in the game's settings so that only HBAO is running, not the default game's AO (ME2, ME3 only).
There are some glitches in-game when using AO in the Mass Effect games. I wouldn't recommend having it on all the time, but it can be useful for screenshots. Here are some screenshots showing what the glitches look like : https://imgur.com/a/wfcgMiy (thanks roivasff14)
Interesting...
Yesterday this link https://i.imgur.com/QW9fp1a.png was giving 404 for me. Today it works.
Is it really necessary to disable gamma correction for AA?
If so, then what about the line gamma, should that be disabled too?