Skip to content

Instantly share code, notes, and snippets.

@akashnimare

akashnimare/Instructions

Last active March 21, 2017 20:56
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save akashnimare/995fab3d5158b031d454cbe684db8296 to your computer and use it in GitHub Desktop.
Save akashnimare/995fab3d5158b031d454cbe684db8296 to your computer and use it in GitHub Desktop.
Download ZIP
Scarpe IMDB top 250 movies review
Raw
imdb-reviews.py
import requests
import json
from BeautifulSoup import BeautifulSoup
all_review = []
urls = ['http://www.imdb.com/title/tt0978762/', 'http://www.imdb.com/title/tt0042876/', 'http://www.imdb.com/title/tt0047478/', 'http://www.imdb.com/title/tt0055630/', 'http://www.imdb.com/title/tt0089881/', 'http://www.imdb.com/title/tt0245712/', 'http://www.imdb.com/title/tt0032976/', 'http://www.imdb.com/title/tt0036342/', 'http://www.imdb.com/title/tt0038787/', 'http://www.imdb.com/title/tt0040746/', 'http://www.imdb.com/title/tt0044079/', 'http://www.imdb.com/title/tt0047396/', 'http://www.imdb.com/title/tt0046912/', 'http://www.imdb.com/title/tt0052357/', 'http://www.imdb.com/title/tt0053125/', 'http://www.imdb.com/title/tt0054215/', 'http://www.imdb.com/title/tt0338564/', 'http://www.imdb.com/title/tt0910970/', 'http://www.imdb.com/title/tt0266543/', 'http://www.imdb.com/title/tt0079944/', 'http://www.imdb.com/title/tt0454876/', 'http://www.imdb.com/title/tt1832382/', 'http://www.imdb.com/title/tt0095953/', 'http://www.imdb.com/title/tt1024648/', 'http://www.imdb.com/title/tt0036775/', 'http://www.imdb.com/title/tt0043014/', 'http://www.imdb.com/title/tt0046359/', 'http://www.imdb.com/title/tt0051201/', 'http://www.imdb.com/title/tt0053291/', 'http://www.imdb.com/title/tt0053604/', 'http://www.imdb.com/title/tt0382932/', 'http://www.imdb.com/title/tt0086250/', 'http://www.imdb.com/title/tt0094226/', 'http://www.imdb.com/title/tt0114814/', 'http://www.imdb.com/title/tt0041959/', 'http://www.imdb.com/title/tt0364569/', 'http://www.imdb.com/title/tt0015864/', 'http://www.imdb.com/title/tt0021749/', 'http://www.imdb.com/title/tt0032553/', 'http://www.imdb.com/title/tt0048424/', 'http://www.imdb.com/title/tt0012349/', 'http://www.imdb.com/title/tt0027977/', 'http://www.imdb.com/title/tt0209144/', 'http://www.imdb.com/title/tt0372784/', 'http://www.imdb.com/title/tt0482571/', 'http://www.imdb.com/title/tt0468569/', 'http://www.imdb.com/title/tt1375666/', 'http://www.imdb.com/title/tt1345836/', 'http://www.imdb.com/title/tt0105695/', 'http://www.imdb.com/title/tt0327056/', 'http://www.imdb.com/title/tt0405159/', 'http://www.imdb.com/title/tt1205489/', 'http://www.imdb.com/title/tt0017925/', 'http://www.imdb.com/title/tt0119488/', 'http://www.imdb.com/title/tt0117951/', 'http://www.imdb.com/title/tt1010048/', 'http://www.imdb.com/title/tt0180093/', 'http://www.imdb.com/title/tt1125849/', 'http://www.imdb.com/title/tt0947798/', 'http://www.imdb.com/title/tt0114369/', 'http://www.imdb.com/title/tt0137523/', 'http://www.imdb.com/title/tt0050212/', 'http://www.imdb.com/title/tt0056172/', 'http://www.imdb.com/title/tt0080678/', 'http://www.imdb.com/title/tt1201607/', 'http://www.imdb.com/title/tt0892769/', 'http://www.imdb.com/title/tt0044081/', 'http://www.imdb.com/title/tt0047296/', 'http://www.imdb.com/title/tt0477348/', 'http://www.imdb.com/title/tt0013442/', 'http://www.imdb.com/title/tt0047528/', 'http://www.imdb.com/title/tt0056801/', 'http://www.imdb.com/title/tt0317248/', 'http://www.imdb.com/title/tt0405094/', 'http://www.imdb.com/title/tt0053198/', 'http://www.imdb.com/title/tt0078788/', 'http://www.imdb.com/title/tt0068646/', 'http://www.imdb.com/title/tt0071562/', 'http://www.imdb.com/title/tt0025316/', 'http://www.imdb.com/title/tt0031679/', 'http://www.imdb.com/title/tt0036613/', 'http://www.imdb.com/title/tt0038650/', 'http://www.imdb.com/title/tt0111161/', 'http://www.imdb.com/title/tt0120689/', 'http://www.imdb.com/title/tt0401792/', 'http://www.imdb.com/title/tt0070511/', 'http://www.imdb.com/title/tt0044706/', 'http://www.imdb.com/title/tt0017136/', 'http://www.imdb.com/title/tt0022100/', 'http://www.imdb.com/title/tt0101414/', 'http://www.imdb.com/title/tt1291584/', 'http://www.imdb.com/title/tt0076759/', 'http://www.imdb.com/title/tt0064115/', 'http://www.imdb.com/title/tt0070735/', 'http://www.imdb.com/title/tt0095765/', 'http://www.imdb.com/title/tt0325980/', 'http://www.imdb.com/title/tt0457430/', 'http://www.imdb.com/title/tt0119217/', 'http://www.imdb.com/title/tt0120735/', 'http://www.imdb.com/title/tt0208092/', 'http://www.imdb.com/title/tt0107048/', 'http://www.imdb.com/title/tt0087544/', 'http://www.imdb.com/title/tt0092067/', 'http://www.imdb.com/title/tt0096283/', 'http://www.imdb.com/title/tt0119698/', 'http://www.imdb.com/title/tt0245429/', 'http://www.imdb.com/title/tt0347149/', 'http://www.imdb.com/title/tt0042546/', 'http://www.imdb.com/title/tt0029947/', 'http://www.imdb.com/title/tt0038355/', 'http://www.imdb.com/title/tt0050976/', 'http://www.imdb.com/title/tt0050986/', 'http://www.imdb.com/title/tt0060827/', 'http://www.imdb.com/title/tt0080684/', 'http://www.imdb.com/title/tt0095327/', 'http://www.imdb.com/title/tt0796366/', 'http://www.imdb.com/title/tt0088247/', 'http://www.imdb.com/title/tt0090605/', 'http://www.imdb.com/title/tt0103064/', 'http://www.imdb.com/title/tt0434409/', 'http://www.imdb.com/title/tt0211915/', 'http://www.imdb.com/title/tt0107207/', 'http://www.imdb.com/title/tt0116282/', 'http://www.imdb.com/title/tt0118715/', 'http://www.imdb.com/title/tt0084787/', 'http://www.imdb.com/title/tt0032551/', 'http://www.imdb.com/title/tt0056217/', 'http://www.imdb.com/title/tt0056218/', 'http://www.imdb.com/title/tt0075148/', 'http://www.imdb.com/title/tt0033870/', 'http://www.imdb.com/title/tt0040897/', 'http://www.imdb.com/title/tt0114709/', 'http://www.imdb.com/title/tt0095016/', 'http://www.imdb.com/title/tt0057115/', 'http://www.imdb.com/title/tt0102926/', 'http://www.imdb.com/title/tt0042192/', 'http://www.imdb.com/title/tt0069281/', 'http://www.imdb.com/title/tt0848228/', 'http://www.imdb.com/title/tt1305806/', 'http://www.imdb.com/title/tt0401383/', 'http://www.imdb.com/title/tt0374546/', 'http://www.imdb.com/title/tt0435761/', 'http://www.imdb.com/title/tt0020629/', 'http://www.imdb.com/title/tt0110413/', 'http://www.imdb.com/title/tt0167404/', 'http://www.imdb.com/title/tt0075314/', 'http://www.imdb.com/title/tt0081398/', 'http://www.imdb.com/title/tt0099685/', 'http://www.imdb.com/title/tt0112641/', 'http://www.imdb.com/title/tt0407887/', 'http://www.imdb.com/title/tt1130884/', 'http://www.imdb.com/title/tt0113247/', 'http://www.imdb.com/title/tt0112573/', 'http://www.imdb.com/title/tt0077416/', 'http://www.imdb.com/title/tt0034583/', 'http://www.imdb.com/title/tt0113277/', 'http://www.imdb.com/title/tt0338013/', 'http://www.imdb.com/title/tt1655442/', 'http://www.imdb.com/title/tt0061184/', 'http://www.imdb.com/title/tt0061722/', 'http://www.imdb.com/title/tt0073486/', 'http://www.imdb.com/title/tt0086879/', 'http://www.imdb.com/title/tt1136608/', 'http://www.imdb.com/title/tt0061811/', 'http://www.imdb.com/title/tt0363163/', 'http://www.imdb.com/title/tt0091763/', 'http://www.imdb.com/title/tt1675434/', 'http://www.imdb.com/title/tt0033467/', 'http://www.imdb.com/title/tt0052311/', 'http://www.imdb.com/title/tt0052561/', 'http://www.imdb.com/title/tt0440963/', 'http://www.imdb.com/title/tt0469494/', 'http://www.imdb.com/title/tt1049413/', 'http://www.imdb.com/title/tt0198781/', 'http://www.imdb.com/title/tt0120737/', 'http://www.imdb.com/title/tt0167261/', 'http://www.imdb.com/title/tt0167260/', 'http://www.imdb.com/title/tt0903624/', 'http://www.imdb.com/title/tt0120382/', 'http://www.imdb.com/title/tt0105236/', 'http://www.imdb.com/title/tt0110912/', 'http://www.imdb.com/title/tt0266697/', 'http://www.imdb.com/title/tt0361748/', 'http://www.imdb.com/title/tt1187043/', 'http://www.imdb.com/title/tt0083987/', 'http://www.imdb.com/title/tt0246578/', 'http://www.imdb.com/title/tt0086190/', 'http://www.imdb.com/title/tt0078748/', 'http://www.imdb.com/title/tt0083658/', 'http://www.imdb.com/title/tt0172495/', 'http://www.imdb.com/title/tt0092005/', 'http://www.imdb.com/title/tt0093779/', 'http://www.imdb.com/title/tt0056592/', 'http://www.imdb.com/title/tt0054997/', 'http://www.imdb.com/title/tt0088763/', 'http://www.imdb.com/title/tt0109830/', 'http://www.imdb.com/title/tt0118799/', 'http://www.imdb.com/title/tt0110357/', 'http://www.imdb.com/title/tt0063522/', 'http://www.imdb.com/title/tt0071315/', 'http://www.imdb.com/title/tt0253474/', 'http://www.imdb.com/title/tt0268978/', 'http://www.imdb.com/title/tt0015648/', 'http://www.imdb.com/title/tt0169547/', 'http://www.imdb.com/title/tt0065214/', 'http://www.imdb.com/title/tt0758758/', 'http://www.imdb.com/title/tt0058461/', 'http://www.imdb.com/title/tt0059578/', 'http://www.imdb.com/title/tt0060196/', 'http://www.imdb.com/title/tt0064116/', 'http://www.imdb.com/title/tt0087843/', 'http://www.imdb.com/title/tt0050083/', 'http://www.imdb.com/title/tt0072890/', 'http://www.imdb.com/title/tt0074958/', 'http://www.imdb.com/title/tt0045152/', 'http://www.imdb.com/title/tt0049406/', 'http://www.imdb.com/title/tt0050825/', 'http://www.imdb.com/title/tt0057012/', 'http://www.imdb.com/title/tt0062622/', 'http://www.imdb.com/title/tt0066921/', 'http://www.imdb.com/title/tt0072684/', 'http://www.imdb.com/title/tt0081505/', 'http://www.imdb.com/title/tt0093058/', 'http://www.imdb.com/title/tt1659337/', 'http://www.imdb.com/title/tt0073195/', 'http://www.imdb.com/title/tt0082971/', 'http://www.imdb.com/title/tt0097576/', 'http://www.imdb.com/title/tt0108052/', 'http://www.imdb.com/title/tt0107290/', 'http://www.imdb.com/title/tt0120815/', 'http://www.imdb.com/title/tt0061512/', 'http://www.imdb.com/title/tt0395169/', 'http://www.imdb.com/title/tt0114746/', 'http://www.imdb.com/title/tt0071853/', 'http://www.imdb.com/title/tt0079470/', 'http://www.imdb.com/title/tt0133093/', 'http://www.imdb.com/title/tt0319061/', 'http://www.imdb.com/title/tt1504320/', 'http://www.imdb.com/title/tt1139797/', 'http://www.imdb.com/title/tt0120586/', 'http://www.imdb.com/title/tt0032138/', 'http://www.imdb.com/title/tt0031381/', 'http://www.imdb.com/title/tt0040522/', 'http://www.imdb.com/title/tt0070047/', 'http://www.imdb.com/title/tt0046250/', 'http://www.imdb.com/title/tt0052618/', 'http://www.imdb.com/title/tt1220719/', 'http://www.imdb.com/title/tt0082096/', 'http://www.imdb.com/title/tt0075686/', 'http://www.imdb.com/title/tt0079522/']
movie_name = ['Mary and Max', 'Rashomon', 'Seven Samurai', 'Yojimbo', 'Ran', 'Amores Perros', 'Rebecca', 'Shadow of a Doubt', 'Notorious', 'Rope', 'Strangers on a Train', 'Rear Window', 'Dial M for Murder', 'Vertigo', 'North by Northwest', 'Psycho', 'Infernal Affairs', 'WALL\xc2\xb7E', 'Finding Nemo', 'Stalker', 'Life of Pi', 'A Separation', 'Rain Man', 'Argo', 'Double Indemnity', 'Sunset Blvd.', 'Stalag 17', 'Witness for the Prosecution', 'Some Like It Hot', 'The Apartment', 'Ratatouille', 'Scarface', 'The Untouchables', 'The Usual Suspects', 'The Third Man', 'Oldboy', 'The Gold Rush', 'City Lights', 'The Great Dictator', 'The Night of the Hunter', 'The Kid', 'Modern Times', 'Memento', 'Batman Begins', 'The Prestige', 'The Dark Knight', 'Inception', 'The Dark Knight Rises', 'Unforgiven', 'Mystic River', 'Million Dollar Baby', 'Gran Torino', 'The General', 'L.A. Confidential', 'Trainspotting', 'Slumdog Millionaire', 'Requiem for a Dream', 'The Wrestler', 'Black Swan', 'Se7en', 'Fight Club', 'The Bridge on the River Kwai', 'Lawrence of Arabia', 'The Elephant Man', 'Harry Potter and the Deathly Hallows: Part 2', 'How to Train Your Dragon', 'A Streetcar Named Desire', 'On the Waterfront', 'No Country for Old Men', 'Nosferatu', 'La strada', '8\xc2\xbd', 'City of God', 'The Lives of Others', 'The 400 Blows', 'Apocalypse Now', 'The Godfather', 'The Godfather: Part II', 'It Happened One Night', 'Mr. Smith Goes to Washington', 'Arsenic and Old Lace', "It's a Wonderful Life", 'The Shawshank Redemption', 'The Green Mile', 'Sin City', 'Papillon', 'High Noon', 'Metropolis', 'M', 'Beauty and the Beast', 'Warrior', 'Star Wars', 'Butch Cassidy and the Sundance Kid', 'The Sting', 'Cinema Paradiso', 'Pirates of the Caribbean: The Curse of the Black Pearl', "Pan's Labyrinth", 'Good Will Hunting', 'Lock, Stock and Two Smoking Barrels', 'Snatch.', 'Groundhog Day', 'Nausica\xc3\xa4 of the Valley of the Wind', 'Castle in the Sky', 'My Neighbor Totoro', 'Princess Mononoke', 'Spirited Away', "Howl's Moving Castle", 'Harvey', 'Bringing Up Baby', 'The Big Sleep', 'The Seventh Seal', 'Wild Strawberries', 'Persona', 'Star Wars: Episode V - The Empire Strikes Back', 'Grave of the Fireflies', 'Star Trek', 'The Terminator', 'Aliens', 'Terminator 2: Judgment Day', 'V for Vendetta', 'Am\xc3\xa9lie', 'In the Name of the Father', 'Fargo', 'The Big Lebowski', 'The Thing', 'The Grapes of Wrath', 'The Man Who Shot Liberty Valance', 'The Manchurian Candidate', 'Rocky', 'The Maltese Falcon', 'The Treasure of the Sierra Madre', 'Toy Story', 'Die Hard', 'The Great Escape', 'The Silence of the Lambs', 'All About Eve', 'Sleuth', 'The Avengers', 'The Secret in Their Eyes', 'The Diving Bell and the Butterfly', 'Spring, Summer, Fall, Winter... and Spring', 'Toy Story 3', 'All Quiet on the Western Front', 'L\xc3\xa9on: The Professional', 'The Sixth Sense', 'Taxi Driver', 'Raging Bull', 'Goodfellas', 'Casino', 'The Departed', 'Shutter Island', 'La Haine', 'Braveheart', 'The Deer Hunter', 'Casablanca', 'Heat', 'Eternal Sunshine of the Spotless Mind', 'The Artist', "Who's Afraid of Virginia Woolf?", 'The Graduate', "One Flew Over the Cuckoo's Nest", 'Amadeus', 'District 9', 'In the Heat of the Night', 'Downfall', 'Platoon', 'The Intouchables', 'Citizen Kane', 'Touch of Evil', 'Anatomy of a Murder', 'The Bourne Ultimatum', 'There Will Be Blood', 'Up', 'Monsters, Inc.', 'The Lord of the Rings: The Fellowship of the Ring', 'The Lord of the Rings: The Two Towers', 'The Lord of the Rings: The Return of the King', 'The Hobbit: An Unexpected Journey', 'The Truman Show', 'Reservoir Dogs', 'Pulp Fiction', 'Kill Bill: Vol. 1', 'Inglourious Basterds', '3 Idiots', 'Gandhi', 'Donnie Darko', 'Star Wars: Episode VI - Return of the Jedi', 'Alien', 'Blade Runner', 'Gladiator', 'Stand by Me', 'The Princess Bride', 'To Kill a Mockingbird', 'The Hustler', 'Back to the Future', 'Forrest Gump', 'Life Is Beautiful', 'The Lion King', "Rosemary's Baby", 'Chinatown', 'The Pianist', 'A Beautiful Mind', 'Battleship Potemkin', 'American Beauty', 'The Wild Bunch', 'Into the Wild', 'A Fistful of Dollars', 'For a Few Dollars More', 'The Good, the Bad and the Ugly', 'Once Upon a Time in the West', 'Once Upon a Time in America', '12 Angry Men', 'Dog Day Afternoon', 'Network', "Singin' in the Rain", 'The Killing', 'Paths of Glory', 'Dr. Strangelove or: How I Learned to Stop Worrying and Love the Bomb', '2001: A Space Odyssey', 'A Clockwork Orange', 'Barry Lyndon', 'The Shining', 'Full Metal Jacket', 'The Perks of Being a Wallflower', 'Jaws', 'Raiders of the Lost Ark', 'Indiana Jones and the Last Crusade', "Schindler's List", 'Jurassic Park', 'Saving Private Ryan', 'Cool Hand Luke', 'Hotel Rwanda', 'Twelve Monkeys', 'Monty Python and the Holy Grail', 'Life of Brian', 'The Matrix', 'Big Fish', "The King's Speech", 'Let the Right One In', 'American History X', 'The Wizard of Oz', 'Gone with the Wind', 'Bicycle Thieves', 'The Exorcist', 'Roman Holiday', 'Ben-Hur', 'Ip Man', 'Das Boot', 'Annie Hall', 'Manhattan']
for i in xrange(len(urls)) :
r = requests.get(urls[i] + 'reviews')
html = r.content
soup = BeautifulSoup(html)
reviews = soup.find('div', attrs={'id':'tn15content'})
reviews = reviews.find('p')
all_review.append(reviews)
with open("result.json", "w") as outfile:
for i in range(len(urls)):
a = movie_name[i]
b = ' '.join(map(str, all_review[i]))
json.dump({
'Movie_name': a,
'review': b
}, outfile, indent=4, sort_keys=True)
Raw
Instructions
* Python - v2.7.13
* pip
## Installation
* pip install -r requirements.txt
* python imdb-reviews.py
* All reviews will be saved in result.json
Raw
requirements.txt
BeautifulSoup
requests
json
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment