Replacing PhantomJS with Headless Chrome in your Selenium testing stack

PhantomJS had one great advantage – it could burrow through authenticated proxies. That made it possible to combine with WonderProxy and see the screen rendering in Saudi Arabia, the Netherlands, Montreal, or 252 other cities in 87 other countries.

Strictly speaking PhantomJS is the browser-simulator, using WebDriver to drive the browser. Today that functionality exists in Headless Chrome, a faster, better maintained, and higher fidelity browser engine. The only advantage PhantomJS still has is its native support for authenticated proxies.

In this article, we'll replace PhantomJS with Headless Chrome. The examples will be in Node.js for how to install Selenium with Chrome and its dependencies. Projects in other languages that run WebDriver will need to Install WebDriver for Chrome in their own language, then pay close attention to the configuration sections, writing code in the local language. WonderProxy is developing Selenium HOWTOs by language; the Ruby-Selenium example is already up today.

Set up the Node.js project

The script below shows the version of Node.js. The selenium-webdriver npm package supports the stable and LTS releases of Node.js, so as of September 2021, that's versions 14.17 and 16.9. When npm init runs, select "Mocha" as the test runner, as the examples are in Mocha. If you're adding Selenium to an existing Node.js project, just continue testing in whatever you were using before, substituting out the driver name and adding the proxy code. In the code below, the npx gitignore command creates a .gitignore file for temporary files, while npm i -D installs WebDriver, the test runner, the assertion framework, and proxy-chain. proxy-chain is the package we'll use to support the authenticated proxies that WonderProxy provides, since Headless Chrome doesn't support authentication itself. (Do not cut and paste this code. Instead run each command separately, as npm init will ask for input and may take the call to npx as the input.)

node -v
npm init
npx gitignore node
npm i -D selenium-webdriver mocha chai proxy-chain

Configure WonderProxy credentials

Before getting started, you'll need a WonderProxy account, which is available as a free trial. Once you have a username and password, store those in environment variables.

export PROXY_USER=yourusername
export PROXY_PASS=yourpassword

Your tests will reference those environmental variables when they connect to WonderProxy servers. The easiest way to do this, of course, is to hard-code the username and password right into the source code. We do not recommend this, since it gives any person or process that can read your code access to your credentials. Most Continuous Integration tools, such as CircleCI, have methods to track environment variables and keep them secure.

Install Headless Chrome

ChromeDriver is a tool that accepts connections running over the W3C WebDriver Protocol, and runs a local browser that follows those commands. This is incredibly powerful, as it makes moving to the cloud or another server as easy as changing the connection string from localhost to a remote machine. Check your browser's version with Chrome → About, then download the appropriate ChromeDriver off the download page. Install it in a location known to your PATH environment variable, or put it in the same place that the test runner will run from.

Note that ChromeDriver is an executable file you downloaded from the internet. As of September 2021, the file is unsigned by MacOS and the operating system may refuse to let you run it. This support post describes how to approve the program on MacOS.

Now we're ready to change the code.

Replace WebDriver and configure proxy

WonderProxyTest.js is a complete test program written in JavaScript to run on Node.js using Selenium. The full project is on Github.

First, import the Chrome WebDriver, which is referenced on line 6.

const chrome = require('selenium-webdriver/chrome');

Second, configure Selenium to use Chrome with the forBrowser() method.

const {Builder, By} = require('selenium-webdriver');

const driver = new Builder()
   .forBrowser('chrome')
   .setChromeOptions(options)
   .build();

Third, provide the options to run through a proxy. The function runWithProxiedDriver() does this. The example below uses ChromeDriver's Options class, which is available in every implementation including Java, Python, and Ruby. Earlier we did an npm install for proxy-chain; that will be the technology used to push Chrome through the proxy.

const ProxyChain = require('proxy-chain');

async function runWithProxiedDriver(proxyServer, proxyPort, testFn) {
	const username = process.env.PROXY_USER;
	const password = process.env.PROXY_PASS;
    
	const wonderProxyUrl = `http://${username}:${password}@${proxyServer}:${proxyPort}`;
	const internalProxyUrl = await ProxyChain.anonymizeProxy(wonderProxyUrl);
	const options = new chrome.Options().addArguments(`--proxy-server=${internalProxyUrl}`);
	await runWithDriver(async function (driver) {
		await testFn(driver);
	}, options);
	await ProxyChain.closeAnonymizedProxy(internalProxyUrl, true);
}

Everything just works!

Headless Chrome is now the browser engine that the tests run against, instead of PhantomJS. This swap "just" replaces the browser with a newer, faster version with better support and upgrades. New versions of Headless will support changes in the definitions of CSS or the JavaScript compiler. If the tests passed with PhantomJS, they will probably pass now, with the possible exception of genuine changes in Chrome's behavior. If a link is "visible" in PhantomJS but not in the current version of Chrome, the new Selenium replacement will report a defect — and it should!

Here's the sample code running as a test from the command line.

mheusser@Matthews-MBP-2 sample2 % npm test                                  

> sample2@1.0.0 test /Users/mheusser/Desktop/sample2
> mocha



  Selenium WebDriver tests
    WonderProxy website - Puppeteer basics
      Home Page
        Title
          ✔ should be "Localization testing with confidence - WonderProxy" (2328ms)
      Status Page
        Title
          ✔ should be "Server Status - WonderProxy" (1492ms)
        Server status is up
          ✔ should show "up" status for lansing (1431ms)
          ✔ should show "up" status for nuremberg (1461ms)
          ✔ should show "up" status for koto (1426ms)
    Google.com world tour
      ✔ should have location specific text for Lansing (3738ms)
      ✔ should have location specific text for Amsterdam (4379ms)
      ✔ should have location specific text for Venice (5841ms)
      ✔ should have location specific text for Tokyo (7096ms)


  9 passing (29s)

This example worked through replacing PhantomJS/Selenium with Headless Chrome/Selenium, but it also provided a real working example in Node. Whether you are converting old code or starting new, WonderProxy, Selenium, and Headless Chrome have you covered.