PiTeR, OpenCV and MagPi issue 28

[guzunty]

Hi, I'm PiTeR,



I hope you found the MagPi article about my vision system interesting.

If you have any questions about using OpenCV image recognition on the Raspberry Pi, please post them here and I'll do my best to answer them.

best regards,

PiTeR

[ChrisLid]

Thanks for a really good article. I'd like to have a go at building the robot and implementing the OpenCV image recognition (demands of work not withstanding). Can you post a more detailed list of the parts you used for the robot?

[guzunty]

Hi Chris,

That's a great idea.

I have been meaning to put together a parts list. I'll build one tonight and post back here when it's done.

In the meantime, were there any parts you were particularly interested to know about?

best,

Derek

[windy54]

A great article, just what I have been looking for, got the robot, just started to work through your blog.

I will let you know how I get on,

Steve

[ChrisLid]

Hi, thanks for the reply. I haven't got a question about any one particular part. I'll have a go at creating my own wiring schematic when I've got all the parts ready. I've just ordered a guzunty so that's one part less on my shopping list.

[guzunty]

Hi again Chris,

Your Guzunty is on its way to you. I'm sure you'll find it very useful for many projects, not just robotics. There is a topic about all the things you can do with it here:

http://www.raspberrypi.org/forums/viewt ... 41&t=49352

[guzunty]

There is a detailed parts list for PiTeR here now.

https://github.com/Guzunty/Pi/wiki/PiTeRs-Parts

best,

Derek

[ChrisLid]

Hi Derek, thanks for the parts list. I'll have a look at it over the weekend and get back to you if I have any further questions.

The guzunty I ordered from you has turned up. Thanks

[windy54]

Hi Derek,

I have followed your instructions and can detect my traffic signs.

One question with opencv 2.4.9 , the image windows seem to double in size, they are created at the "correct" size but when the image is loaded they double in size. Did you notice this? Have not got the courage yet to build 2.4.10

I have also looked at your Pietar code for symbolFinder and symbol Identifier to see if I can reuse it. Just need to understand classes, threading, etc.

Cheers

Steve

[guzunty]

Hi Steve,

Good work getting the example code running.

Yes, I have noticed an issue with OpenCV 2.4.9. It looks like the integration with v4l2 got broken at that release. It fails to set the capture size, so what you're getting is the default. I also see some errors reported. Unfortunately, it impacts the performance significantly. It can take a few minutes to start showing images, so that it is easy to think that it has gone into the weeds.

It is in my to do list to Google the error and see if there is a solution or if it has been reported. If it hasn't, I'll take care of reporting it. If I find out any more I'll update the wiki.

I did build 2.4.10. The bug is there too, I'm afraid.

[guzunty]

Feel free to explore how PiTeR uses OpenCV. It's very similar, except that it does the heavy lifting on a separate thread so that his control loop remains responsive.

How all that works is the subject of a follow up MagPi article, so keep an eye out for that. (I'm hoping to hit the press deadlines for the December issue).

best,

Derek

[cswiger]

Really cool project - was looking thru the repo but cannot find the arduino sketch for the balancing robot?

Just wanted to compare yours with what I put together last year making a copy of Kas balanceBot - It sort of worked. Now I ordered a kit of parts from sainsmart to try again, and mounting an RPi Camera on it would be a challenge.

Anyway, always wanted to try OpenCV on the RPi and your code walkthru will make it easy!

[guzunty]

Hi there, glad you're enjoying the PiTeR project.

The sketch is there for sure, it's here:

https://github.com/Guzunty/Pi/tree/mast ... etch/PiTeR

I've promoted different things at different times, so maybe you just need to do a 'git pull' it you cloned at an earlier time.

You will find some similarities with the KasBot code. It was one of the projects I used as inspiration. One of the issues I found with the KasBot code was that it relies on exchanging a lot of telemetry with a host computer to help tune the robot. This data makes for some nice, heartbeat like, plots. However, exchanging all that data steals a lot of processing power away from the balancing task and affects the way the robot balances. Worse, that overhead can be lumpy which can cause the robot behaviour to be unpredictable. Since we are using the telemetry to tune the balancing, that lumpy loss of processing power starts working against you.

I decided that the plots don't tell you all that much about how to adjust the parameters and cut down massively how much is sent. Instead of relying on the plot data, I opted for a more procedural approach and implemented a method where I could adjust the balancing parameters directly from a wii-mote. The procedural approach I used is described here:

https://github.com/Guzunty/Pi/wiki/PiTeR-balance-tuning.

That said, I recognise that the telemetry data is useful. I have it in my ToDo list to implement a 'crash-dump' procedure, where the full telemetry data is stored locally on the microcontroller. If the robot should fall over (which is much rarer than it was, I'm glad to say), sending a dump command will retrieve the last few seconds worth of full telemetry. That way, we get the detailed data only when we really need it.

Hope that helps and good luck with your second try. I rebuilt PiTeR three times in different configurations before I was happy with his balancing behaviour, so you're not alone in having to have another go (and I plan to revisit it sometime because I believe it could still be even better).

Do please come back and let us know how you get on.

Finally, I'm curious about which Sainsmart kit you ordered, do please post a link if you can.

Derek

[windy54]

Hi,
I am ready to start testing the symbol recognition and am struggling with the command

export LD_LIBRARY_PATH=/usr/local/lib/opencv

To avoid you reading the rest of the post, do you set this automatically, if so, how, or do you always connect to your robot via a shell and manually set it?

I want to set this automatically when the PI boots up. Currently I run a script which detects if a bluetooth dongle is fitted, it then runs my Python program to control the robot and connects to a WII controller. I use the cursor keys to control the robot. The program has been modified so that if the 1 button is held down it runs the symbol detection algorithm and then moves automatically.

I have tried adding the command in .bashrc as suggested on several posts but it does not work.
Cheers
Steve

[guzunty]

Hi Steve,

LD_LIBRARY_PATH is not set in my default environment.

I don't need it to launch PiTeR.py. I'm not in a position to test the OpenCV functionality right now, but it appears to start just fine and successfully opens the camera without error.

I'm not sure why I recommended setting LD_LIBRARY_PATH in the Wiki, possibly because some of the OpenCV C++ test programs need it to be set. I'll test this hypothesis later today.

What makes you think you need it set? If you have error messages, please post them and I'll help interpret them.

FWIW, to launch the PiTeR.py script automatically at boot time, I call it from the startup script /etc/rc.local. If you find you really need LD_LIBRARY_PATH set it would make sense to run the export command just before launching the Python script in rc.local. I start it like this:

(python /home/pi/Projects/Pi/src/gz_piter/Python/PiTeR.py > home/pi/Projects/Pi/src/gz_piter/Python/piterlog 2>&1)&

Note the full path from the root of the filesystem and the pipe into a log file to help debug startup issues.

HTH, good luck and let us know how you get on,

Derek

[windy54]

Hi Steve,

LD_LIBRARY_PATH is not set in my default environment

What makes you think you need it set? If you have error messages, please post them and I'll help interpret them.

FWIW, to launch the PiTeR.py script automatically at boot time, I call it from the startup script /etc/rc.local. If you find you really need LD_LIBRARY_PATH set it would make sense to run the export command just before launching the Python script in rc.local. I start it like this:

(python /home/pi/Projects/Pi/src/gz_piter/Python/PiTeR.py > home/pi/Projects/Pi/src/gz_piter/Python/piterlog 2>&1)&


Derek

I started again with my scripts and got it to work by editing rc.local and adding the modprobe command, the export command and then starting my script using your example.
This now works.
Without the export command my scripts can't find opencv, the libraryfile...so ( sorry can't remember full name)
So, onto the next step, sending camera video back to my iPad, I have just managed to do this using silvan Melchior post on Rpi cam control.
Once I have got everything combined I will let you know how I get on.

Cheers

Steve

[guzunty]

Hi Steve,

Two comments:

1) Note that rc.local is run as root, so your script will be too. This will affect the environment, including the paths. This could well be the reason you need the explicit script calls where you do.

2) While as you have seen, putting a modprobe in the script works, it is not normal practice. Instead you should list the module you want started in the /etc/modules config file. Of course, raspbian uses modprobe on each of these entries anyway, but the idea is to keep them all in one well known place.

The only other explanation I can think of for your needing the explicit export is that we somehow installed openCV differently. As part of my research for the articles and wiki, I installed and re-installed OpenCV a number of times and at a number of different versions and builds. It is possible that during one of those procedures something got statically linked, or the LD_LIBRARY_PATH got implicitly set inside Python.

Anyway, you're running the way you want to now I guess, so all's well that ends well.

HTH,

Derek

[sjm]

Hi Derek,

First, thanks for a great article which I have found really useful.

I have a question about performance. I'm writing a script to detect red circles in an image. I used your colour.py script to get the high and low HSV values which worked a treat.

However when I use these values in a separate script to produce the mask, the following 1 line of code:

Code: Select all

imgThresholded = cv2.inRange(imgHSV, (lowH,lowS, lowV), (highH, highS, highV))

takes over 1 second to complete (even for a small sized image).

I note in your article you "achieved a processing rate for live images of about 2 to 3 frames per second" so I am clearly doing something wrong when just 1 line of code takes over 3 times longer than your entire program! Would you have any ideas why my script is so slow?

Just for reference how long does the above line take to execute on Piter?

Thanks in advance for any insights.

Jonathan

[guzunty]

Hi Johnathan,

Yes, performance is definitely a challenge on the Pi compared to running it on an i7 or something.

As you have said, image size is a big factor when tuning an OpenCV program. I will try to do some measurements on PiTeR and let you know what I get, but it's definitely quicker than 1s to do the inRange call.

Other things to look at are; OpenCV release level, they are working hard to improve library performance all the time, maybe you are running an older version? Overclocking, have you cranked your Pi up a bit, moderate overclocking is safe for your Pi. Also, note that there is probably significant OpenCV startup overhead, so measuring the inRange function performance in a standalone script may give misleading results. Better to run the call inside the script you actually need to use it in and insert some before and after system clock reports.

HTH, and I'll be back on when I've measured the runtime of PiTeR's inRange() call.

[sjm]

Hi,

Thanks for the quick reply. What size image does Piter use?

I'll check the version of openCV I have installed.

I only installed openCV a couple of weeks ago but I used:

Code: Select all

sudo apt-get install libopencv-dev python-opencv

instead of using

Code: Select all

sudo apt-get install python-opencv

as you suggested as I read this article http://trevorappleton.blogspot.co.uk/20 ... 5w36tW412j before I read yours. I'm too much of a noob to understand what the difference is. Could this be the issue?

I was measuring the time in the main script. I just had:

Code: Select all

other code  (loading image, converting to HSV format etc)

now=time.time()
imgThresholded = cv2.inRange(imgHSV, (lowH,l owS, lowV), (highH, hi ghS, highV))
print (time.time()-now)

more code...

I assume this is giving me an accurate timing.

Jonathan

[guzunty]

PiTeR captures at 320 x 240:

self.cap.set(cv2.cv.CV_CAP_PROP_FRAME_WIDTH, 320)
self.cap.set(cv2.cv.CV_CAP_PROP_FRAME_HEIGHT, 240)

(SymbolFinder.py around line 83)

Your measurement method looks fine.

[sjm]

OK thanks.

My Pi was overclocked to the "Medium" setting - I'll try at the max setting and make sure I'm using a 320 x 240 image and see if there is an improvement. I'd still be interested in hearing how long Piter takes to perform the inRange function if you get a chance to check.

The version of OpenCv I have installed is 2.4.1. I'll try installing a higher version if the above doesn't work...

[guzunty]

Sorry for the delay, caused by Raspbian distro upgrade fallout which gave PiTeR a big headache. It broke the python library I use to control the servos and LEDs.

So having got that going again, I finally got around to measuring the time taken to execute inRange. On the Model B Pi mounted in PiTeRs chassis time, as measured by the same code as you posted above, reports as taking between 7 and 32 ms. Here's a sample of the output:

0.0224049091339
0.00859785079956
0.00758099555969
0.00889682769775
0.00862789154053
0.0312020778656
0.0186188220978
0.0079448223114
0.0181300640106
0.0181319713593

So it does look like you have some kind of installation issue. These results were obtained with OpenCV 2.4.9, which is significantly faster than 2.4.1, I believe, but even that would not explain a two orders of magnitude difference in performance.
The only other difference I can see is that my code is passing numpy arrays and yours doesn't look like it is.

Do please let me know how you get on.

In the meantime, I'm planning on running some performance tests on a Pi 2 to see if it is worth giving PiTeR a brain transplant. I suspect the answer will be 'yes'.

[sjm]

Thanks for your reply. Shortly after posting I managed to brake the SD card slot on my Pi. Still it at least gave me an excuse to upgrade to Raspberry Pi 2!

Things are much faster now!

[guzunty]

Yes, I'm sure things will be much faster!

It is in my TODO list to characterise and quantify the performance of Raspberry Pi 2 platform with respect to OpneCV. I'll publish the results here when I get to it, but I am expecting very significant differences compared to a previous generation Pi.

For sure, the Pi 2 is going to be a very popular computer vision platform.

I'm expecting PiTeR to be able to home in on his navigation symbols much more quickly than he does today. I just need to make the time to upgrade his Pi board...