Arne Baeyens
Posts: 50
Joined: Mon Jul 14, 2014 8:38 pm
Location: Flanders

A Robot for RoboCup Junior with image processing - with code

Fri Jul 18, 2014 9:54 am

UPDATE 2014/24/10: code is now on GitHub!
Instructions here.

Hi all!

I would like to show my robot I built this year for the Robocup Junior Competition Flanders 2014, category ‘advanced rescue’. I got a third prize with my robot.
A video of the robot following the line: link.

The robot is –of course – based on a Raspberry Pi. I used a model B running Raspbian.


ImageImageImage

This is not my first robot for the RCJ competition. In 2013 I won the competition with a robot with the Dwengo board as control unit. It used reflectance and home-made colour sensors. The Dwengo board uses the popular pic18f4550 microcontroller and has amongst other functionalities a motor driver, a 2x16 char screen and a big, 40pin extension connector. The Dwengo board is, like the RPi, designed for educational purposes, with projects in Argentina and India.
As the Dwengo board is a good companion for the Raspberry Pi, I decided to combine both boards in my new robot. While the Pi does high-level image processing, the microcontroller controls the robot.


The Raspberry Pi was programmed in C++ using the openCV libraries, the wiringPi library (from Gordon Henderson) and the RaspiCam openCV interface library (from Pierre Raufast and improved by Emil Valkov ). I overclocked the Pi to 1GHz to get a frame rate of 12 to 14 fps.

Using a camera has some big advantages: first of all, you don’t have that bunch of sensors mounted close to the ground that are interfering with obstacles and deranged by irregularities. The second benefit is that you can see what is in front of the robot without having to build a swinging sensor arm. So, you have information about the actual position of the robot above the line but also on the position of the line in front, allowing calculation of curvature of the line.
In short, following the line is much more controllable.
By using edge detection rather than greyscale thresholding, the program is virtually immune for shadows and grey zones in the image.
If the line would have had less hairpin bends and I would have had a bit more time, I would have implemented a speed regulating algorithm on the base of the curvature of the line. This is surely something that would improve the performance of the robot.

I also used the camera to detect and track the green direction fields at a T-junction where the robot has to take the right direction. I used a simple colour blob tracking algorithm for this.

A short video of what the robot thinks.
Please note that in reality the robot goes a little bit slower following the line.

Different steps of the image processing:
Image acquired by the camera (with some lines and points already added):
Image

The RPi converts the colour image to a greyscale image.
Then the pixel values on a horizontal line in the image are extracted and put into an array. This array is visualized by putting the values in a graph (also with openCV):
Image

From the first array, a second is calculated by taking the difference from two successive values. In other words, we calculate the derivative.
Image

An iterating loop then searches for the highest and lowest value in the array. To have the horizontal relative position of the line in the array, the two position values - on the horizontal x axe in the graphed image – are averaged. The position is put in memory for the next horizontal scan with a new image. This makes that the scan line does not have to span the whole image but only about a third of it. The scan line moves horizontally with the centre about above the line.

But this is not enough for accurate tracking. From the calculated line position, circles following the line are constructed, each using the same method (but with much more trigonometry calculations as the scan lines are curved). For the second circle, not only the line position but also the line angle is used. Thanks to using functions, adding a circle is a matter of two short lines of code.

The colour tracking is done by colour conversion to HSV, thresholding and then blob tracking, like explained in this excellent video.
The colour tracking slows the line following down with a few fps but this is acceptable.
HSV image and thresholded image:
ImageImage

As seen in the video, after all the scan lines and some info points are plotted on the input image so we can see what the robot ‘thinks’.

My question: Has anyone made or seen projects using a similar line finding method?


And then?
After the Raspberry Pi has found the line, it sends the position data and commands at 115,2 kbps over the hardware serial port to the Dwengo microcontroller board. The Dwengo board does some additional calculations, like taking the square root of the proportional error and squaring the ‘integral error’ (curvature of the line). I also used a serial interrupt and made the serial port as bug-free as possible by receiving each character separetely. Thus, the program does not wait for the next character while in the serial interrupt.
The Dwengo board sends an answer character to control the data stream. The microcontroller also takes the analogue input of the SHARP IR long range sensor to detect the obstacles and scan for the container.
In short, the microcontroller is controlling the robot and the Raspberry Pi does an excellent job by running the CPU intensive line following program.


But let's get a bit more technical.
Electrical wiring
As a common power supply may give problems with ground loops, electrical noise, ... the robot has two 2S LiPo batteries of each 1300mAh to power the Rpi and Dwengo board separately.

Both devices are interconnected by two small boards – one attaches to the RPi and the other to the Dwengo board – that are joined by a right angle header connection. The first does with some resistors the logic level converting (the Dwengo board runs on 5V),
the latter board also has two DC jacks zit diodes in parallel for power input to the RPi. To regulate the power to the Pi, I used a Turnigy UBEC that delivers a stable 5.25V and feeds it into the Pi by the micro USB connector. This gives a bit more protection to the sensitive Pi. As the camera image was a bit distorted I added a 470uF capacitor to smooth things out. This helped. Even though the whole Pi got hot, the UBEC stayed cold. The power input was between 600 and 700mA at around 8.2 volts.

Technical things
In most linefollowers, the driving wheels are mounted behind the point of gravity (some info about linefollowers). This is to make that the sensors in front of the robot can be positioned above the line by making one wheel run slower than the other (or simply blocking it, like most other robots did). However, there were two reasons I wanted the driving wheels in front: Firstly, the robot has to take some bumps and with an undriven front wheel it is very likely the robot will be pushed off the line; secondly, as I am using a camera, I am totally not concerned about the front off the robot being exactly above the line. One of the spectator at the RCJ competition noticed: “your robot turns before the turn – it looks intelligent!”

I used a swivelling wheel to make the robot as free as possible in its movements. Last year I used a ball caster, and I found it didn’t work well on rougher terrain.
However, the drawback this year was that the robot had the tendency to keep turning for a short time after a turn. If I had had some more time I would have implemented a reflectance sensor on each wheel and control the motor speed. This would also have been useful to see if the robot was in front of a bump or hill.

The Sharp IR long range sensor in front of the robot is used for detecting the objects. At first, I mounted it horizontally. However, this turned out to be not a good arrangement for this model (I will look up the exact type). By mounting it vertically with the IR source at the top I had no problems anymore.

The standard horizontal FOV of the Pi camera is roughly 53 degrees. This is not enough to follow the line. With a wide angle lens a widened it to around 100 degrees.

With the extra battery, camera lens, RPi, ... the weight of the robot increased. Last year's robot weighted around 450g, this one weights about 800! However, as the motors were over calculated this gave not much problems. The robot handles bumps of 10mm without effort.

Last year, I almost missed the first place as the robot only just pushed the can out of the field. Not a second time! Having this in thought, I constructed two 14cm long arms that could be turned open by two 9g servos. With the two grippers opened, the robot spans almost 40 centimetres. Despite this, the robot managed – to everyone's annoyance – ‘to take its time before doing its job’, as can be seen in the video.


Building the robot platform

To build the robot platform I followed the same technology as the year before (link, in Dutch). I made a design in SketchUp, then converted it to a 2D vector drawing and finally lasercutted it at FabLab Leuven. However, the new robot platform is completely different in design. Last year, I made a ‘riding box’ by taking almost the maximum dimensions and mounting the electronics somewhere on or in it.

This time, I took a different approach. Instead of using an outer shell (like insects have), I made a design that supports and covers the parts only where necessary.
The result of this is that the robot not only looks much better, but also makes that the different components are much easier to mount and that there is more space for extensions and extra sensors.
The design files can be found here: Robot RoboCup Junior – FabLab Leuven.

3D renders in SketchUp:
ImageImage



On the day of the RCJ competition I had some bad luck as there wasn’t enough light in he competition room. The shutter time of the camera became much longer. As a consequence, the robot had much more difficulties in following sharp bends in the line. However, this problem did not affect the final outcome of the competition.
Maybe I should have mounted some LEDs to illuminate the line...


Arne Baeyens, 17 - Robotanicus
Last edited by Arne Baeyens on Mon Nov 03, 2014 9:54 am, edited 7 times in total.

Ravenous
Posts: 1956
Joined: Fri Feb 24, 2012 1:01 pm
Location: UK

Re: A Robot for RoboCup Junior with image processing

Mon Jul 21, 2014 1:15 pm

That is impressive. Bookmarked!

Hackertje
Posts: 29
Joined: Thu Feb 21, 2013 2:56 pm
Location: Hoofddorp

Re: A Robot for RoboCup Junior with image processing

Mon Jul 28, 2014 2:24 pm

Arne Baeyens wrote:My question: Has anyone made or seen projects using a similar line finding method?
Well, I know that they have such thing at the High School of Arnhem (HAN).
A year ago I was there at the open day, and I saw it, it was pretty cool.

By the way, you've done some nice work. Very interesting :ugeek: nice

Regards,
Ruben
#include <stdio.h>

int main()
{
. . . . . printf("If you wanna know, I'm Dutch\n");
}

abroun
Posts: 31
Joined: Sat Jun 29, 2013 10:13 am
Contact: Website

Re: A Robot for RoboCup Junior with image processing

Sat Aug 16, 2014 12:42 pm

Hi Arne,

First up, really fantastic work, and that's a great write up you've produced. With regards to line following with a camera, the RS4 self balancing robot, performs line following with a slightly different technique, which the guy who built the robot describes on his blog here. Also, if you're interested he describes a technique for getting the robot to recognise arbitrary black and white signs (shown in action here) which is really elegant.

I like your method of identifying the line and using edge detection to overcome the effects of varying illumination. What size image did you run the algorithm on, and what's the smallest size image you think it would work on to reduce computational load?

Looking forward to seeing your future robots. :)

Regards

Alan

Arne Baeyens
Posts: 50
Joined: Mon Jul 14, 2014 8:38 pm
Location: Flanders

Re: A Robot for RoboCup Junior with image processing

Sat Aug 16, 2014 8:57 pm

Hi Alan,

Thank you for your input.

Yes, Samuel Matos' projects were a source of inspiration! He has done very interesting work and his RS4 robot looks really nice.

Before writing my own algorithm I took a look at his method of line finding. One of the things I didn't like was that it used grey-value thresholding and I found that under the difficult light conditions the greyscale thresholding did not work well. By thresholding the image there's a loss of information as you are converting an 8-bit matrix (image) to a 1-bit matrix.
By the way, I found my robot even follows a shoelace on a tiled floor. It will follow a tape line if both are in the robots view.
Secondly, his way made it more difficult to calculate the curvature of the line.
I haven't done exact time measurements but I think his algorithm is more computationally extensive, especially if it's run multiple times. I found that finding the line slows down the program only a little bit. Grabbing and converting the images seems to take most of the time.
abroun wrote:
What size image did you run the algorithm on, and what's the smallest size image you think it would work on to reduce computational load?
The resolution is 320x240. I tried with 160x120 but the image was distorted and uncomplete for some reason.
If only a black line has to be followed I suppose it might even work with lower resolutions like 80x60.


Regards,
Arne

Arne Baeyens
Posts: 50
Joined: Mon Jul 14, 2014 8:38 pm
Location: Flanders

Re: A Robot for RoboCup Junior with image processing

Sun Sep 07, 2014 5:19 pm

Some have been asking for publishing the code...
I plan to release the full (improved and commented) C++ file in a few months. But don't hesitate to pose a question!

Regards,
Arne

kremor
Posts: 2
Joined: Mon Oct 13, 2014 4:26 pm

Re: A Robot for RoboCup Junior with image processing

Wed Oct 15, 2014 2:33 pm

Hi Arne,
Please post a code for lane tracking you used in competion. Github would be the best
Thanks you.

Arne Baeyens
Posts: 50
Joined: Mon Jul 14, 2014 8:38 pm
Location: Flanders

Re: A Robot for RoboCup Junior with image processing

Wed Oct 15, 2014 6:01 pm

Hi,
Thank you for your interest!
I've currently some hardware-related problems so I can't debug the code and it really needs cleaning up.
But it will be on github.
best,
Arne

kremor
Posts: 2
Joined: Mon Oct 13, 2014 4:26 pm

Re: A Robot for RoboCup Junior with image processing

Thu Oct 16, 2014 9:49 am

By the way what's your github profile name? Cannot find yet

Arne Baeyens
Posts: 50
Joined: Mon Jul 14, 2014 8:38 pm
Location: Flanders

Code now on GitHub

Fri Oct 24, 2014 12:38 pm

Hi all,

The code is now on GitHub!
https://github.com/abaeyens/image-proce ... r/RCJ_2014

Installation instructions:

1. Install the openCV library

Code: Select all

sudo apt-get install libopencv-dev
2. Install the raspicam library
Follow Emil Valkov's instructions in his README but replace in step 3 before running the 'make' command the RaspiCamCV.c file by that in my repository.

3. Compile the C++ program
Copy the C++ file 'RCJ_2014' to a directory somewhere on your Pi and compile it with

Code: Select all

g++ "%f" -lopencv_highgui -lopencv_core -lopencv_legacy -lopencv_video -lopencv_features2d -lopencv_calib3d -lopencv_imgproc -lpthread -lm -L/home/pi/git/raspberrypi/userland/build/lib -lmmal_core -lmmal -l mmal_util -lvcos -lbcm_host -lX11 -lXext -lrt -lstdc++ -L/home/pi/git/robidouille/raspicam_cv -lraspicamcv -I/usr/include/opencv -o "%e" -L/usr/local/lib -lwiringPi
You can use the above command with the geany editor, but if compiling in a terminal you have to change "%f" to the name of the output executable and "%e" to the name of the input c++ file.

4. Run the program
It is preferable to run the program in terminal as it gives verbose output during running. Navigate to the directory and run

Code: Select all

./RCJ_2014
To exit the program, select one of the windows and press 'q'.


Please report if something doesn't work and feel free tot share any thoughts.

Kind Regards,

Arne

rexry
Posts: 1
Joined: Thu Oct 27, 2016 3:16 am

Re: A Robot for RoboCup Junior with image processing - with

Thu Oct 27, 2016 3:27 am

Hi Arne

The step2 from the complete code was unavailable, the webpage was down.
(2. Install the raspicam library
Follow Emil Valkov's instructions in his README but replace in step 3 before running the 'make' command the RaspiCamCV.c file by that in my repository.)

Hope you can resolve this problem asap. Thanks a lot!
Your tutorial was interesting for me to learn.

Thanks & Regards
Rexry

Arne Baeyens
Posts: 50
Joined: Mon Jul 14, 2014 8:38 pm
Location: Flanders

Re: A Robot for RoboCup Junior with image processing - with

Thu Oct 27, 2016 8:10 pm

Hi Rexry,

thank you for posting about the issue!
I've added some new links in the README. It should work now.
I'm glad you enjoyed reading my article!

Regards,
Arne

gerardnorris549
Posts: 1
Joined: Wed Dec 07, 2016 9:37 am

Re: A Robot for RoboCup Junior with image processing - with

Wed Dec 07, 2016 9:45 am

Hii Arne! Congratulations for getting the third prize in RoboCup Junior Competition Flanders in the advance rescue category. I liked the features of your Robot. and video you shared with us. Thanks for sharing. All the best for your future projects.

Business Process Automation

Arne Baeyens
Posts: 50
Joined: Mon Jul 14, 2014 8:38 pm
Location: Flanders

Re: A Robot for RoboCup Junior with image processing - with

Tue Jan 03, 2017 4:21 pm

gerardnorris549 wrote:Hii Arne! Congratulations for getting the third prize in RoboCup Junior Competition Flanders in the advance rescue category. I liked the features of your Robot. and video you shared with us. Thanks for sharing. All the best for your future projects.

Business Process Automation
Thanks a lot for your kind message! I'm currently studying at university (engineering), so new projects will come!

kk123
Posts: 4
Joined: Tue Sep 12, 2017 11:34 pm

Re: A Robot for RoboCup Junior with image processing - with code

Wed Sep 13, 2017 3:06 pm

Hello Arne,

Great work !!!! Loved your presentation a lot ! I am new to using Raspberry-Pi. I am using the Pi 3 to drive a Rover and follow a line using the Pi-Camera. I need to develop a C++ application, which detects a curve ahead, when the Rover follows a line (using Pi-camera) and print "true" for a curve ahead and "false" for no curve.

Any help will be greatly appreciated!

Thank you for your help.

Regards,
Sudharshan

User avatar
OutoftheBOTS
Posts: 388
Joined: Tue Aug 01, 2017 10:06 am

Re: A Robot for RoboCup Junior with image processing - with code

Wed Sep 13, 2017 10:10 pm

cool robot :)


I made something similar for rescue myself using OpenCV and made half a dozens step by step videos to teach others to do the same thing.

see https://www.youtube.com/watch?v=tpwokAPiqfs

If you have already been doing this you may want to skip though some bits but I am sure there is ideas here to help you improve you bot

In the end I was getting processing up 50fps and the robot moving at high speed around the track without error.
Last edited by OutoftheBOTS on Thu Sep 14, 2017 7:30 am, edited 1 time in total.

Arne Baeyens
Posts: 50
Joined: Mon Jul 14, 2014 8:38 pm
Location: Flanders

Re: A Robot for RoboCup Junior with image processing - with code

Thu Sep 14, 2017 6:55 am

Hello Sudharshani,

Nice to hear you like my project :).

Probably the most difficult part of developing your program will be getting the code compiled and running. I know that my code runs on the Raspberry Pi 3.

I suggest you try to get my program up and running on your Raspberry Pi. Once compiled and launched, the program will just start working and analyzing the camera image but it doesn't do anything with that data, apart from showing it as a graphic overlay in the OpenCV camera preview.

The curvature of the line can be found in the variables already used in the program. You can add your code to my C++ program to read the variables and convert the data in them to booleans (using a threshold value) and to export/do something else with this data (I suggest you use the WiringPi C++ library if you need GPIO). There's some code (unused functions) for serial communication at the end of the file you can use. It sends the P and D values - P being the actual error and D how the error will change (D can be used for the 'curvature' of the line :) ) - in a small message to be used by a microcontroller (e.g. Arduino) to drive the motors. It thus can send sort of a first-order approximation of the line to the microcontroller.

Kind regards,

Arne

kk123
Posts: 4
Joined: Tue Sep 12, 2017 11:34 pm

Re: A Robot for RoboCup Junior with image processing - with code

Wed Sep 20, 2017 9:45 pm

Hello Arne,

Thank you for your reply! I have been trying to analyse the code and remove the piece of code that i do not require, but i have reached a road block. Your entire application detects and follows the line, furthermore does object tracking as well. I am just looking to detect if the path and i'm finding it difficult to understand which part of your code handles it and where. I see a variable called "angle" can i use that? It would have been great if you could assist me with the same!

Regards,
Sudharshan

User avatar
OutoftheBOTS
Posts: 388
Joined: Tue Aug 01, 2017 10:06 am

Re: A Robot for RoboCup Junior with image processing - with code

Thu Sep 21, 2017 6:18 am

If your interested here is a very simple way to detect angle of black lines in the rescue. see https://www.youtube.com/watch?v=o2ul4KrLT-s&t=114s

You will find a link in the description to the source code :)

Arne Baeyens
Posts: 50
Joined: Mon Jul 14, 2014 8:38 pm
Location: Flanders

Re: A Robot for RoboCup Junior with image processing - with code

Thu Sep 21, 2017 9:15 am

kk123 wrote:
Wed Sep 20, 2017 9:45 pm
Hello Arne,

Thank you for your reply! I have been trying to analyse the code and remove the piece of code that i do not require, but i have reached a road block. Your entire application detects and follows the line, furthermore does object tracking as well. I am just looking to detect if the path and i'm finding it difficult to understand which part of your code handles it and where. I see a variable called "angle" can i use that? It would have been great if you could assist me with the same!

Regards,
Sudharshan

Hi Sudharsan,

Nice to hear you got the program working!
I've written the code a long time ago and I'm not sure if I remember correctly how it works. I should have documented it better! I hope the following works:

In the file RCJ_2014.cpp, insert at line 243 the following lines:

Code: Select all

int my_angle = 0;
my_angle = lineangle();
std::cout << ""Angle of the line: " << my_angle << " °" << endl;
This should save the angle of the line to the variable my_angle in degrees and also print it to the standard output. You can initialise the variable my_angle in a better place if you want.

Really hope this works, if it doesn't, please let me know.

Regards,
Arne

kk123
Posts: 4
Joined: Tue Sep 12, 2017 11:34 pm

Re: A Robot for RoboCup Junior with image processing - with code

Sun Sep 24, 2017 7:10 pm

Hello Arne,
I was trying to install the Raspberry Pi camera Library, by seeing the tutorial you had suggested. I also replaced the RaspiCamCV.c file with the one provided by you. But when I executed the make command, there were few errors, for instance : #include <cv.h> and
#include <highgui.h> were not resolved. I tried removing them and compile again, but there were other errors. I suppose the removal lead to other errors.

Would you happen to know the issue?

Thank you for your help !!

Regards,
Sudharshan

Return to “Automation, sensing and robotics”

Who is online

Users browsing this forum: No registered users and 20 guests