Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Use selenium in Colab
# install chromium, its driver, and selenium
!apt update
!apt install chromium-chromedriver
!pip install selenium
# set options to be headless, ..
from selenium import webdriver
options = webdriver.ChromeOptions()
options.add_argument('--headless')
options.add_argument('--no-sandbox')
options.add_argument('--disable-dev-shm-usage')
# open it, go to a website, and get results
wd = webdriver.Chrome(options=options)
wd.get("https://www.website.com")
print(wd.page_source) # results
# divs = wd.find_elements_by_css_selector('div')
# I create my own library to make it even easier
!pip install kora -q
from kora.selenium import wd
wd.get("https://www.website.com")
print(wd.page_source) # results
# I add a few helpers
divs = wd.select("div") # css selecter
div = divs[0]
span = div.select1("span") # return the first result
wd # screenshot
@ahmadahmadi72
Copy link

ahmadahmadi72 commented Jan 23, 2019

I want to use Gecko Driver and Firefox in selenium. How should I do?

@ninok8
Copy link

ninok8 commented Sep 16, 2019

Thank you very helpful!

@Liza-Kadetova
Copy link

Liza-Kadetova commented Dec 12, 2019

thanks a lot!

@hugonbgg
Copy link

hugonbgg commented Feb 6, 2020

Thanks mate!

@hyunsikhwang
Copy link

hyunsikhwang commented Apr 4, 2020

thanks a lot!

@chetan7116desai
Copy link

chetan7116desai commented Apr 19, 2020

As I run the code ,after completion its not directed to the provided url ,however the page source is printed,can help me on this?

@Sagargajare
Copy link

Sagargajare commented May 9, 2020

As I run the code ,after completion its not directed to the provided url ,however the page source is printed,can help me on this?

this chrome-driver is running in headless mode so it will run in background You can disable it

@murgado
Copy link

murgado commented Jul 16, 2020

Thank you very much mate! I was mad looking for this

@ashish25ece
Copy link

ashish25ece commented Jul 25, 2020

After running the above code, it throws this error. Could you please help me on this?

Message: unknown error: Chrome failed to start: crashed.
(unknown error: DevToolsActivePort file doesn't exist)

@hun-park
Copy link

hun-park commented Sep 3, 2020

I really appreciate a your work!

@guiraojpg
Copy link

guiraojpg commented Oct 8, 2020

You are the best!

@rohanaggarwal45
Copy link

rohanaggarwal45 commented Nov 23, 2020

How can we download files using Selenium in Google Colab Notebooks..??

@rohanaggarwal45
Copy link

rohanaggarwal45 commented Nov 23, 2020

Please help

@swankyshahir
Copy link

swankyshahir commented May 11, 2021

As I run the code ,after completion its not directed to the provided url ,however the page source is printed,can help me on this?

this chrome-driver is running in headless mode so it will run in background You can disable it

could you show the code how to disable headless in it so that it shows the url page. thanks

@murgado
Copy link

murgado commented May 11, 2021

As I run the code ,after completion its not directed to the provided url ,however the page source is printed,can help me on this?

this chrome-driver is running in headless mode so it will run in background You can disable it

could you show the code how to disable headless in it so that it shows the url page. thanks

I will reply to this, hope it helps you.

While using "google collaborate" you must run selenium in headless mode because it can't display new browsers. Therefore, the only way to run it without headless mode would be through your own device, declaring your webdriver not specifying the headless option.

If you still want to run it in headless mode either because you are using collab or because you plan to use it in a server-side environment, there's still a way to print the page's url:

# Consider wb as your webdriver
wb = webdriver.Chrome(executable_path = driverPath , options=chrome_options)

# Display webdriver's current url
print(wb.current_url)

# If you want to display the title of the page your webdriver is visting
print(wb.title)

# If you want to display the html of the page your webdriver is visting
print(wb.page_source)

@abdoo13
Copy link

abdoo13 commented Jun 12, 2021

Useful. That really worked for me.

@Shivani29sheth
Copy link

Shivani29sheth commented Jul 12, 2021

As I run the code ,after completion its not directed to the provided url ,however the page source is printed,can help me on this?

this chrome-driver is running in headless mode so it will run in background You can disable it

could you show the code how to disable headless in it so that it shows the url page. thanks

I will reply to this, hope it helps you.

While using "google collaborate" you must run selenium in headless mode because it can't display new browsers. Therefore, the only way to run it without headless mode would be through your own device, declaring your webdriver not specifying the headless option.

If you still want to run it in headless mode either because you are using collab or because you plan to use it in a server-side environment, there's still a way to print the page's url:

# Consider wb as your webdriver
wb = webdriver.Chrome(executable_path = driverPath , options=chrome_options)

# Display webdriver's current url
print(wb.current_url)

# If you want to display the title of the page your webdriver is visting
print(wb.title)

# If you want to display the html of the page your webdriver is visting
print(wb.page_source)

Thank you, that was really helpful!

@JorgeSantosJ
Copy link

JorgeSantosJ commented Jul 25, 2021

How to add others web driver options using kora?

@korakot
Copy link
Author

korakot commented Jul 26, 2021

You can create your own wd

from selenium import webdriver
options = webdriver.ChromeOptions()
options.add_argument('--headless')
options.add_argument('--no-sandbox')
options.add_argument('--disable-dev-shm-usage')
# create a webdriver instance, ready to use
wd = webdriver.Chrome('chromedriver',options=options)

Kora just help with installation and create a default wd for you.

@jiahaoli57
Copy link

jiahaoli57 commented Sep 24, 2021

Nice job, btw, for those who cannot use the code provided by korakot, use:

from IPython.display import Javascript
def open_web():
  url = 'https://github.com/jiahaoli57'
  display(Javascript('window.open("{url}");'.format(url=url)))

@VictorFaraon
Copy link

VictorFaraon commented Nov 30, 2021

Hi Korakot!
I'm trying tu use selenium in Google Colab, but I'm having trouble to refert to de chromedriver.exe file.
If I put it in the Colaboratory space, I get the message "'chromedriver.exe' executable may have wrong permissions."
And if I refer to the file in my computer, the error is: Message: "'C:\chromedriver\chromedriver.exe' executable needs to be in PATH."
I also failed to use webdriver-manager in the Colab environment.
Can you help me on this?

@Ezra-Cohen
Copy link

Ezra-Cohen commented Jan 13, 2022

I just want to say thank you, I was trying everything I could for a project I was working on, nothing worked until I came across this, you are genuinely amazing for making this

@korakot
Copy link
Author

korakot commented Apr 28, 2022

For reference, this method is first discovered here in Dec 2018.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment