"Whenever I start thinking about CORS, my intuition about which site hosts the headers is incorrect, just as you described in your question. For me, it helps to think about the purpose of the same origin policy.
The purpose of the same origin policy is to protect you from malicious JavaScript on siteA.com accessing private information you've chosen to share only with siteB.com. Without the same origin policy, JavaScript written by the authors of siteA.com could make your browser make requests to siteB.com, using your authentication cookies for siteB.com. In this way, siteA.com could steal the secret information you share with siteB.com.
Sometimes you need to work cross domain, which is where CORS comes in. CORS relaxes the same origin policy for
domainA.com, using the Access-Control-Allow-Origin header to list other domains (domainB.com) that are trusted
to run JavaScript that can interact with domainA.com.
To understand which domain should serve the CORS headers, consider this: You visit malicious.com, which contains some JavaScript that tries to make a cross domain request to mybank.com. It should be up to mybank.com, not malicious.com, to decide whether or not it sets CORS headers that relax the same origin policy allowing the JavaScript from malicious.com to interact with it. If malicous.com could set its own CORS headers allowing its own JavaScript access to mybank.com, this would completely nullify the same origin policy.
I think the reason for my bad intuition is the point of view I have when developing a site. It's my site, with all my JavaScript, therefore it isn't doing anything malicious and it should be up to me to specify which other sites my JavaScript can interact with. When in fact I should be thinking which other sites JavaScript are trying to interact with my site and should I use CORS to allow them?"