Scraping Javascript page using Python

18 Oct 2014

Python library dryscape can be used to scrape javascript driven websites.

Code

To give an example, I created a sample page with following HTML code. (link):

<!DOCTYPE html>
<html>
<head>
  <meta charset="utf-8">
  <title>Javascript scraping test</title>
</head>
<body>
  <p id='intro-text'>No javascript support</p>
  <script>
     document.getElementById('intro-text').innerHTML = 'Yay! Supports javascript';
  </script> 
</body>
</html>

without javascript it says: No javascript support and with javascript: Yay! Supports javascript

Scraping without JS support:

>>> import requests
>>> from bs4 import BeautifulSoup
>>> response = requests.get(my_url)
>>> soup = BeautifulSoup(response.text)
>>> soup.find(id="intro-text")
<p id="intro-text">No javascript support</p>

Scraping with JS support:

>>> import dryscrape
>>> from bs4 import BeautifulSoup
>>> session = dryscrape.Session()
>>> session.visit(my_url)
>>> response = session.body()
>>> soup = BeautifulSoup(response)
>>> soup.find(id="intro-text")
<p id="intro-text">Yay! Supports javascript</p>