I’ve been testing out Selenium with Chromedriver and I noticed that some pages can detect that you’re using Selenium even though there’s no automation at all. Even when I’m just browsing manually just using Chrome through Selenium and Xephyr I often get a page saying that suspicious activity was detected. I’ve checked my user agent, and my browser fingerprint, and they are all exactly identical to the normal Chrome browser.
When I browse to these sites in normal Chrome everything works fine, but the moment I use Selenium I’m detected.
In theory, chromedriver and Chrome should look literally exactly the same to any web server, but somehow they can detect it.
If you want some test code try out this:
from pyvirtualdisplay import Display from selenium import webdriver display = Display(visible=1, size=(1600, 902)) display.start() chrome_options = webdriver.ChromeOptions() chrome_options.add_argument('--disable-extensions') chrome_options.add_argument('--profile-directory=Default') chrome_options.add_argument("--incognito") chrome_options.add_argument("--disable-plugins-discovery"); chrome_options.add_argument("--start-maximized") driver = webdriver.Chrome(chrome_options=chrome_options) driver.delete_all_cookies() driver.set_window_size(800,800) driver.set_window_position(0,0) print 'arguments done' driver.get('http://stubhub.com')
If you browse around stubhub you’ll get redirected and ‘blocked’ within one or two requests. I’ve been investigating this and I can’t figure out how they can tell that a user is using Selenium.
How do they do it?
I installed the Selenium IDE plugin in Firefox and I got banned when I went to stubhub.com in the normal Firefox browser with only the additional plugin.
When I use Fiddler to view the HTTP requests being sent back and forth I’ve noticed that the ‘fake browser’s’ requests often have ‘no-cache’ in the response header.
Results like this Is there a way to detect that I’m in a Selenium Webdriver page from JavaScript? suggest that there should be no way to detect when you are using a webdriver. But this evidence suggests otherwise.
The site uploads a fingerprint to their servers, but I checked and the fingerprint of Selenium is identical to the fingerprint when using Chrome.
This is one of the fingerprint payloads that they send to their servers:
{"appName":"Netscape","platform":"Linuxx86_64","cookies":1,"syslang":"en-US","userlang":"en- US","cpu":"","productSub":"20030107","setTimeout":1,"setInterval":1,"plugins": {"0":"ChromePDFViewer","1":"ShockwaveFlash","2":"WidevineContentDecryptionMo dule","3":"NativeClient","4":"ChromePDFViewer"},"mimeTypes": {"0":"application/pdf","1":"ShockwaveFlashapplication/x-shockwave- flash","2":"FutureSplashPlayerapplication/futuresplash","3":"WidevineContent DecryptionModuleapplication/x-ppapi-widevine- cdm","4":"NativeClientExecutableapplication/x- nacl","5":"PortableNativeClientExecutableapplication/x- pnacl","6":"PortableDocumentFormatapplication/x-google-chrome- pdf"},"screen":{"width":1600,"height":900,"colorDepth":24},"fonts": {"0":"monospace","1":"DejaVuSerif","2":"Georgia","3":"DejaVuSans","4":"Trebu chetMS","5":"Verdana","6":"AndaleMono","7":"DejaVuSansMono","8":"LiberationM ono","9":"NimbusMonoL","10":"CourierNew","11":"Courier"}}
It’s identical in Selenium and in Chrome.
VPNs work for a single use, but they get detected after I load the first page. Clearly some JavaScript code is being run to detect Selenium.
Advertisement
Answer
Replacing cdc_
string
You can use Vim or Perl to replace the cdc_
string in chromedriver
. See the answer by @Erti-Chris Eelmaa to learn more about that string and how it’s a detection point.
Using Vim or Perl prevents you from having to recompile source code or use a hex editor.
Make sure to make a copy of the original chromedriver
before attempting to edit it.
Our goal is to alter the cdc_
string, which looks something like $cdc_lasutopfhvcZLmcfl
.
The methods below were tested on chromedriver version 2.41.578706
.
Using Vim
vim /path/to/chromedriver
After running the line above, you’ll probably see a bunch of gibberish. Do the following:
- Replace all instances of
cdc_
withdog_
by typing:%s/cdc_/dog_/g
.dog_
is just an example. You can choose anything as long as it has the same amount of characters as the search string (e.g.,cdc_
), otherwise thechromedriver
will fail.
- To save the changes and quit, type
:wq!
and pressreturn
.- If you need to quit without saving changes, type
:q!
and pressreturn
.
- If you need to quit without saving changes, type
Using Perl
The line below replaces all cdc_
occurrences with dog_
. Credit to Vic Seedoubleyew:
perl -pi -e 's/cdc_/dog_/g' /path/to/chromedriver
Make sure that the replacement string (e.g., dog_
) has the same number of characters as the search string (e.g., cdc_
), otherwise the chromedriver
will fail.
Wrapping Up
To verify that all occurrences of cdc_
were replaced:
grep "cdc_" /path/to/chromedriver
If no output was returned, the replacement was successful.
Go to the altered chromedriver
and double click on it. A terminal window should open up. If you don’t see killed
in the output, you’ve successfully altered the driver.
Make sure that the name of the altered chromedriver
binary is chromedriver
, and that the original binary is either moved from its original location or renamed.
My Experience With This Method
I was previously being detected on a website while trying to log in, but after replacing cdc_
with an equal sized string, I was able to log in. Like others have said though, if you’ve already been detected, you might get blocked for a plethora of other reasons even after using this method. So you may have to try accessing the site that was detecting you using a VPN, different network, etc.