It's a bad GPU for mining bitcoins.
TL;DR
HardOCP have this nice article dealing with the technical part of mining bitcoin. You can extract from it that your RAM have close to nothing to do with how much hash you can compute per second. The bottleneck is your cpu. Or more exactly your gpu as those have the good instruction more or less hardwired inside of them.
And from the graph, it look like nvidia have never hardwired the exact instruction required to compute a hash and wire less and less of it. They provide a lot of software solution for game and I assume they build their GPU for these solution. On the other hand, ATI GPU seems to be the right tool for this job.
From what you tell about your gaming rig, their should be three open slot for gpu. If you aim to do profitable mining, you can fill some or all of these slot with gpu offering better performance than a GTX 560. As a bonus, you will be able to play whi