The first error you encountered was indeed caused by a mismatch in binaries version between Chrome and the WebDriver.
But you've already figured this out, so onto the issue you are facing now:
org.openqa.selenium.NoSuchSessionException: no such session
This error implies that the ChromeDriver was unable to communicate with the existing Browsing Context i.e. Chrome Browser session.
Custom profiles on selenium-webdriver are sometimes fiddly. Try setting the preferences directly.
Here's the updated code for your example:
import {Builder, WebDriver} from 'selenium-webdriver';
import {Options} from "selenium-webdriver/firefox";
const customUA = "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.143 Safari/537.36";
const options = new Options().setPreference('general.useragent.override', customUA);
As @Grubshka mentionned, it's always a good practice to make your request look like it's coming from a web browser by setting "common" headers. More infos here.
In your case though, it looks like the website you are targeting is rate-limiting based on IP address, and that 25 requests coming from the same IP in 10 minutes is exceeding their limit.
The easiest way to get around this would be to use a [residential proxy](https://brightdata.com/blog/proxy-101/what-is-a-proxy-server?kw=&cpn=14745430544&cam=aw_all_products-all_geos-search_dsa_blog-kw_en-desktop_blog-proxy-101__612826834975&utm_term=&utm_campaign=all_products-all_geos-search_dsa_blog-kw_en-desktop&utm_source=adwords&utm_medium=ppc&utm_content=blog-proxy-101&hsa_acc=1393175403&hsa_cam=14745430544&hsa_grp=136943772153&hsa_ad=612826834975&hsa_src=g&hsa_tgt=dsa-1649388331144&hsa_kw=&hsa_mt=&hsa_net=adwords&hsa_ver=3&cq_src=google_ads&cq_cmp=14745430544&cq_term=&cq_plac=&cq_net=g&cq_plt=gp&
There are a few potential reasons for this error.
One is that the web server is not configured to allow outgoing connections on the port that the web scraper is trying to use (8443).
Another possibility is that the web scraper is trying to connect to a website that is blocking connections from the web server's IP address as part of an anti-scraping protection.
| @import"https://fonts.googleapis.com/css2?family=Inter:wght@400;500;600;700&display=swap";@import"https://fonts.googleapis.com/css2?family=Lexend+Deca:wght@300;400;500;600;700&display=swap";.dynamite-editable,[data-d-ignore]{outline-offset:-1.5px;pointer-events:all}[data-d-ignore] *{pointer-events:none}body:not(.dynamite-metakey) .dynamite-editable,body:not(.dynamite-metakey) .dynamite-editable *{cursor:text!important}.dynamite-editable:hover{outline:1.5px dotted #2c5ffa}.dynamite-editable:focus{outline:1.5px solid rgba(44,95,250,1);caret-color:#2c5ffa}.dynamite-editable:empty:before{content:attr(placeholder);color:#878787}[data-d-ignore]:hover{outline:1.5px dotted rgb(220,220,220)}.dynamite-editable.dynamite-diff{outline:1.5px dotted rgba(44,95,250,1);color:#2c5ffa}@keyframes fadeIn{0%{opacity:0}to{opacity:1}}.dynamite-tooltip{padding:4px 6px;background-color:#000;color:#fff;border-radius:4px;position:absolute;font-size:9px;top:100%;z-index:9999;font-weight:300;animation:fadeIn 50ms ease .5s backwards}.dynam |
If you're getting blocked every 4 requests, it's likely that your IP address is being flagged. One way to work around this is to use a proxy server that will automatically rotate your IP address with each request. This way, you're less likely to get flagged and blocked.
There are a few different ways to do this, but one example is to use residential proxies. These are IP addresses that are associated with real physical locations (homes, businesses, etc.), so they're less likely to be flagged as suspicious.
Some allow scoping to a particular geo region and use "sticky" IPs (use the same IP for a set amount of time)
There are a number of different providers that offer this service, so you'll need to do some research to find one that meets your needs. Once you have a proxy server set up, you can use it in your code just like you would any other proxy.
Here's how it would work in your case:
| window.onload = function load() { | |
| window.close('','_parent',''); | |
| }; |
| async function get(type) { | |
| const rottentomatoesResponse = await fetch(`https://www.rottentomatoes.com/api/private/v2.0/browse?sortBy=popularity&type=${type}`) | |
| const json = await rottentomatoesResponse.json() | |
| const titles = json.results.map(element => { | |
| return { | |
| url: element.url, | |
| title: element.title.split(':')[0] | |
| } | |
| }); |