A simple python script to scrape comments from one or many BBC News articles. This was tested to collect the Brexit related discourse posted on the BBC News website.
-
Go to a BBC news article with comments you're interested in, and view the source of the webpage. Alternateively, in Chrome, right-click on the 'View Comments' button and select 'Inspect'. For example: http://www.bbc.co.uk/news/uk-politics-eu-referendum-36306681
-
Look within the HTML and for the string 'forumId=CPS' (without the Single Quotes). After 'CPS' there will be a number, copy this number. For example '__CPS__43250035', then copy '43250035'
-
Edit the 'articles' list in the python scraper code, add the relevant Article title and number as required. For example: