Vlada
Posts: 10
Joined: Thu Jan 02, 2020 10:42 pm

Tennis ball collector

Thu Jan 02, 2020 11:14 pm

Hello everyone,

thinking to make a RPI tennis ball collector robot for my son (because collecting is boring :) )
For a V1 the idea is to use camera to detect tennis ball, pick it up then return to some position and drop the ball.

Problems to overcome:
1. move 4wd car around the tennis court and avoid objects (net, poles, fence)
2. detect tennis ball on the court
3. move car close to the detected tennis ball and pick it up
4. return car to home position and drop the ball
5. use phone to start/stop collecting activity

Hardware:
1. For camera I was thinking to use 5 megapixel OV5647 module.
2. Standard ultrasonic sensor to detect if car is close enough to collect the ball (or use camera for that as well)
3. 4 standard motors for wheels
4. few analog servo motors for collector arm

Links, ideas and advises are more than welcome, specially for that "return to home position" task because I still have no clue how to do it.

scotty101
Posts: 4050
Joined: Fri Jun 08, 2012 6:03 pm

Re: Tennis ball collector

Fri Jan 03, 2020 8:26 am

For the return to home feature, use the camera much like you do for detecting the tennis balls. At the "home" position have a large symbol which can be detected by the camera and computer vision software. Choose something that is high contrast.

Rather than having a complex pick up mechanism. Why not just have a plough on the front of the vehicle to guide the balls back to the home zone.
Electronic and Computer Engineer
Pi Interests: Home Automation, IOT, Python and Tkinter

Vlada
Posts: 10
Joined: Thu Jan 02, 2020 10:42 pm

Re: Tennis ball collector

Fri Jan 03, 2020 9:43 am

Thank you for your reply.

Using same camera for returning home, excellent idea! Thank you :)

In V2, plan is to put the basket on the car and store collected balls there so that is why I want to pickup balls instead of guide them back to home location.

JohnsUPS
Posts: 185
Joined: Fri Jul 06, 2018 2:13 am
Location: USA

Re: Tennis ball collector

Mon Jan 06, 2020 10:17 pm

For your V1, just gathering the balls, I built a robot that did just that. It was limited to four billiard balls, but the mechanism could easily be adapted to tennis balls.

Here is a suggestion:

The gathering mechanism was simply a rotating bar that was set to about 1/8" of an inch below the top of the ball, and constantly spun with a small DC motor. When the bar contacted the ball, it was pulled into a corral at the base of the robot. The ball remained on the ground and rolled around in the corral as the robot moved. This technique was used a couple of different times with great success. The rotating bar won't let the balls escape the "corral" unless the motor spinning it was reversed. Simple and effective.

Best of luck. Sounds like a fun project.....

Vlada
Posts: 10
Joined: Thu Jan 02, 2020 10:42 pm

Re: Tennis ball collector

Sat Jan 11, 2020 10:05 pm

Thank you for your reply.
Still not decided how to collect the ball, will see later.

So far I am able to detect balls using cv2.HoughCircles()
And now I am trying to calculate the distance to the ball based on ball size and circle diameter.
Will check in real world during son's next training session.

ElEscalador
Posts: 844
Joined: Tue Dec 15, 2015 4:55 pm
Location: Detroit, MI USA
Contact: Website

Re: Tennis ball collector

Sat Jan 11, 2020 11:36 pm

May I suggest (after doing the "I can write all the code myself way" myself) that you learn ROS sooner rather than later - yes, it's a little time commitment up front but the ability to so easily add just one feature (and turn just one feature on and off and edit it - all while everything else is already running) is really a time-saver. Not to mention all the message-handling, visualization tools, and software packages already written for you if you want them - even if all you want to use is sensor drivers it's a huge time-saver.
Robotics tips, hacks, book extras https://youtube.com/practicalrobotics

Vlada
Posts: 10
Joined: Thu Jan 02, 2020 10:42 pm

Re: Tennis ball collector

Wed Jan 29, 2020 8:03 pm

Hello everyone,

here is the quick update on my progress.
All main parts are printed but I will have to improve camera mount and add a tilt because ball gets out of sight when it is close.
Will start to work on electronics, board and battery mount.
tbv1.jpg
tbv1.jpg (126.54 KiB) Viewed 1903 times

User avatar
Gavinmc42
Posts: 4603
Joined: Wed Aug 28, 2013 3:31 am

Re: Tennis ball collector

Thu Jan 30, 2020 1:24 am

Replace those wheels with Mecanum wheels and you will be able to go sideways.
Rotocaster wheels at 45 degrees will do the same thing.

Latest software is still playing catch up for the Pi4.
OpenVX can now use OpenGLES 3.1 Compute Shaders.
So new I don't think anyone has done it yet on a Pi4.

ROS/OpenCV are big pieces of software, expect to use a lot of time learning them.
I played with Darknet/Yolo3 over Xmas, much easier.
Robocup- robotic soccer has some interesting tricks.
RoboDNN runs on x86 Nao bots but an Arm64 ported version would be ideal, faster too.

Tensorflow Lite would be another option now.

Those toy RS130 size low voltage motors will suck battery power.
You can get 12V wound 130 size motors for about $1-2 each.
Higher voltage, low current motors are more efficient = longer battery life.
Less power wasted in the motor drivers too.

Wide angle, clip on mobile phone lens?
5MP is plenty, most recog is done at about 300x300 pixels.
I'm dancing on Rainbows.
Raspberries are not Apples or Oranges

Vlada
Posts: 10
Joined: Thu Jan 02, 2020 10:42 pm

Re: Tennis ball collector

Fri Feb 07, 2020 12:38 am

Gavinmc42, than you for your comment.
Wheels, motors,... will be upgraded for sure.

So another quick update.
I've abandoned detecting circles and switched to TensorFlow Lite.
It took me some learning time but finally managed to create TFL model using Google Colab
Detection works fine even with models with small amount of training steps.
RPI 3 A+ struggles a bit but need to see if FPS can be improved. Now I have ~2FPS :)
Maybe Coral USB TPU or RPI 4 would be good to have.

User avatar
Gavinmc42
Posts: 4603
Joined: Wed Aug 28, 2013 3:31 am

Re: Tennis ball collector

Fri Feb 07, 2020 4:34 am

RPI 3 A+ struggles a bit
That's why a Pi4A should be a game changer.
Perhaps on Pi day?

I saw Elon talking abut the Tesla labeling speeds.
3 orders of magnitude improvement due to coding improvements.
The Tesla chip have 3 quad core A72 Arms, plus some NN stuff.
Deepspace and Squeezenet seem connected to this.

Time for me to try TFL on the Pi4.
Tensorflow seems to be able to use new stuff as it comes out.
https://blog.tensorflow.org/2019/01/ten ... obile.html
Pi 4 now has OpenGLES 3.1 :D
I still trying to get any OpenGLES 3.1 stuff to work, maybe look at TFL?

PI4 speeds for this?
https://blog.tensorflow.org/2020/01/pho ... -lite.html
Or I could just wait a bit longer and someone will do it eventually?
I'm dancing on Rainbows.
Raspberries are not Apples or Oranges

ElEscalador
Posts: 844
Joined: Tue Dec 15, 2015 4:55 pm
Location: Detroit, MI USA
Contact: Website

Re: Tennis ball collector

Fri Feb 07, 2020 12:33 pm

When it's time to move up from a Pi3 because you need more speed for image processing, it's time to skip the Pi4 and move right to the Nvidia Jetson Nano. A few $ more and many times faster. Make that "many, many, many times faster" if you run your processes on the 128 core GPU.
Robotics tips, hacks, book extras https://youtube.com/practicalrobotics

User avatar
Gavinmc42
Posts: 4603
Joined: Wed Aug 28, 2013 3:31 am

Re: Tennis ball collector

Sat Feb 08, 2020 12:25 am

Make that "many, many, many times faster" if you run your processes on the 128 core GPU.
But that makes it too easy, because it will just work and I won't learn anything.

The Nao bots used in Robocup have x86 cpu's that have less power than a Pi4.
Get it working on a Pi4 and then see if it can be tuned to run faster and port it down to Pi3 or a Zero.
Or get some k210 RISC-V boards and try them?

The Pi3 has 48 QPU cores and the Pi4 32 cores that run faster.
And I have lots of Pi's., 20+ ;)
I'm dancing on Rainbows.
Raspberries are not Apples or Oranges

ElEscalador
Posts: 844
Joined: Tue Dec 15, 2015 4:55 pm
Location: Detroit, MI USA
Contact: Website

Re: Tennis ball collector

Sat Feb 08, 2020 1:02 am

That's fair - I'm all about doing things for the challenge.
Robotics tips, hacks, book extras https://youtube.com/practicalrobotics

Vlada
Posts: 10
Joined: Thu Jan 02, 2020 10:42 pm

Re: Tennis ball collector

Sun Feb 09, 2020 11:55 am

For a comparison I've tried how single class tflite model works on android phones.
Aaaaand it works a bit faster :)
Got 4.5-5.5 FPS on Huawei p20 lite and was able to set tflite to use all cpu cores. I did not see this option in tflite for RPI. Anyone knows more about this?

Seems I will have to think some smart way of using 2 FPS on RPi :)

PiGraham
Posts: 4095
Joined: Fri Jun 07, 2013 12:37 pm
Location: Waterlooville

Re: Tennis ball collector

Sun Feb 09, 2020 1:40 pm

Vlada wrote:
Fri Feb 07, 2020 12:38 am
Gavinmc42, than you for your comment.
Wheels, motors,... will be upgraded for sure.

So another quick update.
I've abandoned detecting circles and switched to TensorFlow Lite.
It took me some learning time but finally managed to create TFL model using Google Colab
Detection works fine even with models with small amount of training steps.
RPI 3 A+ struggles a bit but need to see if FPS can be improved. Now I have ~2FPS :)
Maybe Coral USB TPU or RPI 4 would be good to have.
I can see Hough circles could be a problem. TFLite is overkill though. Try OpenCV blob detection working a Hue image. If you have bright yellow balls they should stand out well and blob detection should be robust. Check the area to perimeter ratio as a quick circularity measure.

You could get a much higher rate than TFLite.

https://www.pyimagesearch.com/2015/09/1 ... th-opencv/
.

Vlada
Posts: 10
Joined: Thu Jan 02, 2020 10:42 pm

Re: Tennis ball collector

Sun Feb 09, 2020 3:41 pm

I've tried that exact code and I get awful results out of the box.
It depends a lot on a ball color and I need robot to work in both indoor and outdoor fields with different light levels.
Probably I will go with TFLite and Coral USB TPU on RPI4. This combo should give me 20+ FPS.

PiGraham
Posts: 4095
Joined: Fri Jun 07, 2013 12:37 pm
Location: Waterlooville

Re: Tennis ball collector

Mon Feb 10, 2020 9:54 am

Vlada wrote:
Sun Feb 09, 2020 3:41 pm
I've tried that exact code and I get awful results out of the box.
It depends a lot on a ball color and I need robot to work in both indoor and outdoor fields with different light levels.
Probably I will go with TFLite and Coral USB TPU on RPI4. This combo should give me 20+ FPS.
You could do better than "out of the box". HSV splits RGB to a Hue value that is fairly independent of light levels. However, if saturation is low or intensity is low Hue is unreliable. So use Hue = yellow threshold but mask it with Sat>X and V>Y

Once you have the contours labelled you can check for width,height, area and also area/perimeter is a simple circularity measure
Last edited by PiGraham on Tue Feb 11, 2020 10:29 am, edited 1 time in total.

blimpyway
Posts: 416
Joined: Mon Mar 19, 2018 1:18 pm

Re: Tennis ball collector

Mon Feb 10, 2020 10:05 pm

Different light settings should be approachable in a simpler way than tensorflow..
e.g. see this tutorial which uses only color matching https://www.pyimagesearch.com/2015/09/1 ... th-opencv/

No need to detect circles or neural networks - anything matching colour, height/width aspect ratio and size (see below) should be sufficient.

Since both robot and balls are resting on the same flat surface, distance to the potential ball (matched by colour/light values) can be calculated from its position in frame.

Knowing distance, computing size of potential ball from image blob pixel size - to see if it reasonably matches size of a real ball -should also be easy.

---------------------------
Expect problems with several balls grouped together, bumping into any "object" matching colour but not size and/or aspect ratio should spread grouped balls and make them visible individually.

If that doesn't work I would even go as far as having a "training mode" in which you point a laser pointer at a ball, the robot follows the laser dot, when laser turns off the robot stores the remaining "coloured blob" as target.

----------------
PS I have pair of the same yellow motor-gear-wheels and I want to make a three wheel robot.
Also to have it play with a pi zero, camera and a laser pointer.
Can you please share details on how do they attach to your frame? You 3d printed it?

Thanks

User avatar
Gavinmc42
Posts: 4603
Joined: Wed Aug 28, 2013 3:31 am

Re: Tennis ball collector

Tue Feb 11, 2020 1:04 am

Been trying to understand this stuff off and on for years.
Break though came the other night watching a Lex Fridman Video.
A Perceptron is like single neuron in a neural network.
A google for "Python Perceptron" found code I could understand :idea:

A cut and paste and soon I had trained my first neuron :D
Still a long way to go, more videos to watch, more code to play with.
Now busy seeing if I can understand the py-videocore6 code.
It really needs more than just one example?
I'm dancing on Rainbows.
Raspberries are not Apples or Oranges

PiGraham
Posts: 4095
Joined: Fri Jun 07, 2013 12:37 pm
Location: Waterlooville

Re: Tennis ball collector

Tue Feb 11, 2020 10:35 am

Something I was thinking about on this is how does the ballbot find stray balls in the far corner of the court? Maybe a long focal length lens and a slow turn to scan the area?

But I like the idea of a camera on a pole with a view of the whole court. Make the ballbot visible and distinctive and spot balls and bot from the bird's eye view. Then navigate the bot to the balls. That way you don't need high frame rates. Just wouk out the path and the bot can drive itself autonomously. Probably just needs a closeup camera or sensor to ensure getting the ball into the catcher.

Not so practical if you are using public courts but it's your own court and you can fix the camera to a pole I think it would be good solution.

Vlada
Posts: 10
Joined: Thu Jan 02, 2020 10:42 pm

Re: Tennis ball collector

Thu Feb 13, 2020 12:50 am

Attached is the current chassis. Plan to improve wheel mounting points so they can be aligned and keep bot running straight as much as possible.

For sure bot will not be able to detect balls that are in far corners of the field because of camera lens, but my plan is to make algorithm that will do something like this:
1. Try ball detection with left-right movements.
2. If no balls are detected lower threshold level.
3. If no balls are detected do some movement around the field avoiding obstacles. Stop occasionally and scan for balls.
4. When possible ball object is detected, raise threshold level and move closer then verify detection.
...

So far:
- tried tflite for python and with just RPI I got 2FPS because only CPU core can be used.
- tried with green blob detection and got bad results in low light conditions (indoor fields)

Now:
- working on c++ version of tflite and will post results for comparison. (atm I have video from rpi camera and working on tflite libs)

Next:
- Coral TPU will come soon and will try tflite with that.
- check Python Perception so we can have multiple options compared.
Attachments
Tennis Bot Ch.zip
(42.47 KiB) Downloaded 26 times

User avatar
Gavinmc42
Posts: 4603
Joined: Wed Aug 28, 2013 3:31 am

Re: Tennis ball collector

Thu Feb 13, 2020 1:15 am

I have been studying jumping spider vision.
Spiders normally have 8 eyes.
On a jumping spider, 6 are used for motion detection and the two big ones are for stereo fine vision.

By reducing the field of view to less than 3 degrees, they rotate their bodies and can move the retinas inside their heads.
Moving the retina is a scanning function and can generate a 3d map in front of the spider.
They have tiny brains but can see nearly as well as humans and can plan attacks on non visible prey.
This rewrites what most people regard as intelligence in bugs.

Motion tracking moving tennis balls to see where they end up?
"Optical flow python" is my latest google exercise.
Simple motion tracking software to track balls then high res recog to zoom in on them?

Pi's should be able to do spider level intelligence?
I'm dancing on Rainbows.
Raspberries are not Apples or Oranges

User avatar
rpdom
Posts: 17542
Joined: Sun May 06, 2012 5:17 am
Location: Chelmsford, Essex, UK

Re: Tennis ball collector

Thu Feb 13, 2020 6:32 am

Gavinmc42 wrote:
Thu Feb 13, 2020 1:15 am
Spiders normally have 8 eyes.
On a jumping spider, 6 are used for motion detection and the two big ones are for stereo fine vision.
Jumping spiders are fun and cute (FCVO). They are different to most other spiders with that vision thing. They can recognise shapes and even pictures of other jumping spiders.

Most of the others can only recognise dark and light patches. They mostly rely on picking up vibrations through their feet.
Unreadable squiggle

Vlada
Posts: 10
Joined: Thu Jan 02, 2020 10:42 pm

Re: Tennis ball collector

Wed Feb 19, 2020 7:12 pm

I've converted same model to be used by Coral TPU, updated python code a bit and can confirm great speed-up on RPI 3 A+

With one tennis ball in video:
- without Coral: 0.45 seconds
- with Coral: 0.03 seconds

C++ installation is a nightmare but will spend more time on it to see if multi-thread gives better results without Coral.

Sergey_Marvelmind
Posts: 2
Joined: Tue Feb 25, 2020 10:18 am

Re: Tennis ball collector

Tue Feb 25, 2020 10:22 am

Hi,

Just for idea - for more accuracy tracking an navigating your robat you can use ultrasonic system (Marvelmind Robotics).

If need more info - you can ask directly me.

Return to “Automation, sensing and robotics”