MowingBar
Posts: 13
Joined: Thu Dec 17, 2015 9:33 pm

Getting selenium python to work on Raspberry Pi Model B

Thu Dec 17, 2015 9:48 pm

I installed selenium on my headless raspberry pi model b using:

Code: Select all

pip install selenium
Since then I have tried running selenium using various different browsers including Firefox, Iceweasel, Chrome, Chromium and using phantomjs. I want to run the following code using Iceweasel:

Code: Select all

from pyvirtualdisplay import Display
import pyvirtualdisplay
from selenium.webdriver.firefox.firefox_binary import FirefoxBinary
from selenium import webdriver

display = Display(visible=0, size=(800, 600))
display.start()
fp = webdriver.FirefoxProfile("/usb1/uxfq8qiv.default")
#binary = FirefoxBinary("/usr/bin/iceweasel")
binary = FirefoxBinary("/usr/lib/iceweasel/iceweasel")
profile = webdriver.FirefoxProfile()
profile.native_events_enabled = False
driver = webdriver.Firefox(profile)

driver.get("http://www.python.org")
In all cases I run into the same error:

Code: Select all

("Can't load the profile. Profile "
selenium.common.exceptions.WebDriverException: Message: Can't load the profile. Profile Dir: %s If you specified a log_file in the FirefoxBinary constructor, check it for details.
I have searched for solutions to this however I cannot find anyone that actually runs selenium on a raspberry pi model b.

Is it possible to run selenium python scripts on a raspberry pi model b?
Last edited by MowingBar on Fri Jan 08, 2016 9:35 pm, edited 1 time in total.

ghans
Posts: 7438
Joined: Mon Dec 12, 2011 8:30 pm
Location: Germany

Re: Getting selenium python to work on Raspberry Pi Model B

Fri Dec 18, 2015 7:45 am

I can confirm that Selenium worked with X11 forwarding (XMing) and locally on the Pi (Raspbian default X) . A physical display was never connected to the Pi.

Code: Select all

import os

from selenium import webdriver
from selenium.webdriver.support.ui import Select
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

os.environ["DISPLAY"] = ":0.0"

fp = webdriver.FirefoxProfile()

#fp.set_preference("browser.download.folderList",2)
#fp.set_preference("browser.download.manager.showWhenStarting",False)
#fp.set_preference("browser.download.dir", os.getcwd())

browser = webdriver.Firefox(firefox_profile=fp)
browser.get("http://www.google.com")
Setting DISPLAY and then using the default X server might make the difference ?

ghans
• Don't like the board ? Missing features ? Change to the prosilver theme ! You can find it in your settings.
• Don't like to search the forum BEFORE posting 'cos it's useless ? Try googling : yoursearchtermshere site:raspberrypi.org

MowingBar
Posts: 13
Joined: Thu Dec 17, 2015 9:33 pm

Re: Getting selenium python to work on Raspberry Pi Model B

Mon Dec 21, 2015 12:10 pm

OK. After lots and lots of attempts I finally got it running. Not sure what finally did it but the last command I ran was:

Code: Select all

sudo apt-get install openssl gem ruby xvfb
Hopefully this helps anyone else trying to do this.

**********UPDATE***********
Please use the answer provided by glycerine to solve this issue.
Last edited by MowingBar on Fri Jan 08, 2016 9:36 pm, edited 1 time in total.

ipretender
Posts: 1
Joined: Wed Dec 23, 2015 2:49 am

Re: Getting selenium python to work on Raspberry Pi Model B

Wed Dec 23, 2015 3:08 am

I have got same problem. Can you post 100% working sample?
I installed xvfb, selenium, iceweasel.

MowingBar
Posts: 13
Joined: Thu Dec 17, 2015 9:33 pm

Re: Getting selenium python to work on Raspberry Pi Model B

Wed Dec 23, 2015 11:40 am

ipretender wrote:I have got same problem. Can you post 100% working sample?
I installed xvfb, selenium, iceweasel.
As I said in a previous post, make sure you have installed openssl, gem, ruby and xvfb. My working example is:

Code: Select all

from selenium.webdriver.firefox.webdriver import WebDriver
from selenium import webdriver
from pyvirtualdisplay import Display
import pyvirtualdisplay
import json

display = Display(visible=0, size=(800, 600))
display.start()

fp = webdriver.FirefoxProfile()
success = True
wd = WebDriver(fp)
wd.implicitly_wait(600)
wd.get("http://www.google.com")

display.stop()
Post again if you have any problems.
Last edited by MowingBar on Fri Jan 08, 2016 4:00 pm, edited 1 time in total.

tidaboy9
Posts: 7
Joined: Wed Dec 30, 2015 4:15 am

Re: Getting selenium python to work on Raspberry Pi Model B

Sat Jan 02, 2016 11:48 pm

Can anyone help, I've installed all the apt-get/pip/gem packages along with using your script on my B+ Rasbian Jessie but when I try to execute the script w/ sudo python test.py I get

("Can't load the profile. Profile "
selenium.common.exceptions.WebDriverException: Message: Can't load the profile. Profile Dir: %s If you specified a log_file in the FirefoxBinary constructor, check it for details.

I think it has something to do with the newest version of iceweasal,but I'm not sure. Is their any log I can post that would help diagnost this problem?

ghans
Posts: 7438
Joined: Mon Dec 12, 2011 8:30 pm
Location: Germany

Re: Getting selenium python to work on Raspberry Pi Model B

Sun Jan 03, 2016 9:15 am

Any reason for using sudo ?

ghans
• Don't like the board ? Missing features ? Change to the prosilver theme ! You can find it in your settings.
• Don't like to search the forum BEFORE posting 'cos it's useless ? Try googling : yoursearchtermshere site:raspberrypi.org

glycerine
Posts: 1
Joined: Fri Jan 08, 2016 2:59 pm

Re: Getting selenium python to work on Raspberry Pi Model B

Fri Jan 08, 2016 3:05 pm

Hi,

I was having the exact same issue, and found the answer here; https://github.com/SeleniumHQ/selenium/issues/1300

Essentially, the firefox instance wasn't starting before the selenium script timed out (in my case it was probably exacerbated by limiting all processes to 30% cpu usage max, but i'd expect it to start more slowly on a Pi regardless).

To fix, edit /usr/local/lib/python2.7/dist-packages/selenium/webdriver/firefox/firefox_binary.py and change the value '30' on line 101 to something higher. I went with 120, but it could probably be much lower if you're sensitive to the amount of time it'll take before failing if there is genuinely an issue.

Hope that helps!

MowingBar
Posts: 13
Joined: Thu Dec 17, 2015 9:33 pm

Re: Getting selenium python to work on Raspberry Pi Model B

Fri Jan 08, 2016 9:33 pm

glycerine wrote:Hi,

I was having the exact same issue, and found the answer here; https://github.com/SeleniumHQ/selenium/issues/1300

Essentially, the firefox instance wasn't starting before the selenium script timed out (in my case it was probably exacerbated by limiting all processes to 30% cpu usage max, but i'd expect it to start more slowly on a Pi regardless).

To fix, edit /usr/local/lib/python2.7/dist-packages/selenium/webdriver/firefox/firefox_binary.py and change the value '30' on line 101 to something higher. I went with 120, but it could probably be much lower if you're sensitive to the amount of time it'll take before failing if there is genuinely an issue.

Hope that helps!
This worked for me when I set the value to "240" as I was playing a video at the same time. This is a much better solution than my suggestion to install random packages.

honda4life
Posts: 74
Joined: Thu Mar 15, 2012 7:27 pm

Re: Getting selenium python to work on Raspberry Pi Model B

Sat May 14, 2016 7:25 am

This does not work anymore,
It works when you change the code, not the timeout value because it is a default value when no argument is given.

The real timeout comes from :
/usr/local/lib/python2.7/dist-packages/selenium/webdriver/firefox/webdriver.py

Code: Select all

class WebDriver(RemoteWebDriver):

    # There is no native event support on Mac
    NATIVE_EVENTS_ALLOWED = sys.platform != "darwin"

    def __init__(self, firefox_profile=None, firefox_binary=None, timeout=30,

All you need to install is from minimal clean image:

Code: Select all

sudo apt-get install python-pip iceweasel xvfb
sudo pip install pyvirtualdisplay selenium
(not sure if python-pip is needed)

To run:

Code: Select all

from selenium import webdriver
from pyvirtualdisplay import Display

display = Display(visible=0, size=(800, 600))
display.start()

driver = webdriver.Firefox()

Pi_n_Peas
Posts: 10
Joined: Sun Oct 21, 2012 11:34 am

Re: Getting selenium python to work on Raspberry Pi Model B

Sat Jun 24, 2017 1:03 pm

I had a fully working Python script using Selenium Webdriver,
pyvirtualdisplay, Xvfb wrapper and Firefox (Iceweasel/geckodriver) that scraped perfectly for me.
Only problem was the hardware this was working on was a Pi3. My Pi3 is now busy pretending to be an Amazon Alexa so I am trying to migrate the code to run on an old Pi Rev2 Model B. My Pi3 code created all sorts of errors (as you can imagine :o :o ) when I tried to run my 'working' Python script on the Pi Rev2 Model B so I decided to get something simpler working first and go back to basics and start with the simple code on this thread posted by @honda4life.

Code: Select all

from selenium import webdriver
from pyvirtualdisplay import Display

display = Display(visible=0, size=(800, 600))
display.start()

driver = webdriver.Firefox()
Error code I am getting is:

Code: Select all

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/firefox/webdriver.py", line 140, in __init__
    self.service.start()
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/common/service.py", line 96, in start
    self.assert_process_still_running()
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/common/service.py", line 109, in assert_process_still_running
    % (self.path, return_code)
selenium.common.exceptions.WebDriverException: Message: Service geckodriver unexpectedly exited. Status code was: -11
My H/W is a 512kB RPi rev2 Model B

OS on 8GB Card is:
PRETTY_NAME="Raspbian GNU/Linux 8 (jessie)"
NAME="Raspbian GNU/Linux"
VERSION_ID="8"
VERSION="8 (jessie)"
ID=raspbian
ID_LIKE=debian

Python 2.7.9
Name: selenium
Version: 3.0.2
Location: /usr/local/lib/python2.7/dist-packages

and Geckodriver.exe is in /home/pi/Downloads and this path has been added to $PATH

Pi_n_Peas
Posts: 10
Joined: Sun Oct 21, 2012 11:34 am

Re: Getting selenium python to work on Raspberry Pi Model B

Mon Jun 26, 2017 10:38 am

Issue appears to be that geckodriver is only compatible with ARM7 architecture.
https://github.com/mozilla/geckodriver/issues/560

Getting selenium to work on RPi Model B will require a new (retro) build of geckodriver that is compiled for ARM6 architecture.

Return to “General discussion”

Who is online

Users browsing this forum: andrum99, HermannSW, mamgeorge and 75 guests