1. BrowserMob Proxy
Browsermob proxy is a developers utility that helps to monitor network traffic of a web application. It is a technology that is developed under the Selenium open source project. It is originally called andBrowser Mob is now a part of Neustar.
A proxy is an (Apache 2.0 license) utility that can be used with Selenium or otherwise can be used independently as well.
Browsermob allows you to manipulate HTTP requests and response, capture the HTTP content and export the data as a HAR object, also known as HTTP archive.
2. Configuring BrowserMob proxy with selenium
Configuring BrowserMob proxy for the automation process will run your selenium scripts behind a proxy server. To configure browser-mob proxy you would need to do the following:
2.1 Browsermob Proxy library
Get the browser-mob proxy library from here. You are going to need this to create the proxy server. Download it and keep a copy of the browser-mob proxy in the project directory.
2.2 Python dependency for BrowserMob proxy
Use pip to install python dependency for the browser-mob proxy. To install the dependency just type in pip install browsermob-proxy.
2.3 Creating a server and instantiating a proxy instance
First of all, you need to create a server. Browsermob already has the configuration for Jetty server. You have to instantiate the server by giving the path for browser mob-proxy inside the browser mob-proxy/bin directory. By default, the Jetty server would be running on port 8080. The jetty server is a Java based Web Server with a servlet container. It helps in addressing requests and responses through the servlet container.
Instantiating a browser-mob server
# Configure proxy server and return server object
def configure_proxy_server(self):
server = Server("browsermob-proxy/bin/browsermob-proxy")
return server2.4 Instantiating a proxy from the server
Once the Jetty server is up and running, next a proxy needs to be created using the server instance of Jetty.
Creating a proxy
# Configure proxy using the server object and return proxy object def configure_proxy(self, server): server.start() proxy = server.create_proxy() return proxy
2.4.1 Configuring browser driver with proxy server configurations
Pass on the proxy configuration to the respective web driver object, so that it starts using the proxy of the jetty server of Browsermob rather than the proxy of the selenium server.
Configuring Chrome driver with proxy server configuration
# Configure selenium driver and return driver object
def configure_driver(self, proxy_data):
try:
co = webdriver.ChromeOptions()
co.add_argument('--proxy-server={host}:{port}'.format(host='localhost',
port=proxy_data.port))
driver = webdriver.Chrome(executable_path = "/usr/local/bin/chromedriver",
chrome_options=co)
return driver
except WebDriverException as e:
print(">>>WebDriver exception: {}".format(e))2.5 Storing the HTTP requests as HAR object
The outgoing HTTP requests from the browser can be stored in the HAR or Http Archive format. At the end of the execution, you can iterate the HAR object to get the HTTP requests. Also, you can also store the respective data in a .har file. Once the file.har is created you can use Google GSuite Analyzer to track the respective performance or monitor the site.
Assembling the HTTP requests and storing it as har object
# Track network traffic
def get_networkTraffic(self, server, proxy_data, driver):
try:
input_url = raw_input(">>>Enter the url to be checked for network traffic: ")
request = requests.get(input_url)
response_code = request.status_code
# Store the http requests as har object
if response_code == 200:
proxy_data.new_har("google")
driver.get(input_url)
proxy_data.har
for ent in proxy_data.har['log']['entries']:
print(ent['request']['url'])
else:
print(">>>Status code: "+str(response_code))
driver.quit()
server.stop()
except requests.exceptions.RequestException as e:
print(">>>Request Exception: {}".format(e))3. SSL Support
You have to explicitly set up the ca-certificate-rsa.cer for your respective browser. I have set up the certificate for Chrome browser. Without this the requests cannot be made over secure socket layer. To set up the certificate just open chrome browser goto settings > advanced > Privacy & Security > Manage Certificates. If you are working on OSX platform, you will get the KeyChain Access window. Just add the ca-certificate-rsa.cer. Once added, just click on ‘Trust’ and select the option ‘Always Trust’ and save the configuration by giving the root password. You will be able to find the ca-certificate-rsa.cer within the repository itself.
4. Download the Source Code
This was an example of Capturing HTTP requests using BrowserMobProxy.
You can download the full source code of this example here : Using BrowserMobProxy with Selenium
Why do you not use fiddler.?