"Raspivid" -o option behaving strange, affecting -x output

[cpunk]

I will make my best attempt to solve this problem myself, since it would likely take others a long while to even recreate the conditions exactly... but since it is such an interesting problem for me, I want to also document what I saw, hence I'm opening this thread here. :)

I'm working with my home-brew modified "raspivid" command that accepts the "!" parameter value to the -x argument, and if this is supplied, directs motion vectors to the standard error data stream.

I'm also using the -o argument to send raw video to various locations, for example serve it via TCP for a telemetry display program, or log it into a file named "/root/dump". Or to send it to "/dev/null" if I want nothing of it...

...and I find that motion vector content seems affected by where I'm sending the video output with -o. When I'm sending -o to TCP, my motion plotting program works fine, when I'm dumping -o to /dev/null, suddenly, the program "sees less motion". To make it even more interesting, when I try to record the video into "/root/dump", the file appears nonsensically small for multiple seconds of 30fps video.

The ways I'm calling Raspivid are as follows:

Code: Select all

raspivid --colfx 128:128 -awb off -awbg 0.0,0.0 -ex off -sh 0 -br 70 -co 10 -f -rot 90 -n -t 0 \
-w $1 -h $2 -b 400000 -fps 30 -pf high -fli 50hz -l -o /root/dump -x !

#raspivid --colfx 128:128 -awb off -awbg 0.0,0.0 -ex off -sh 0 -br 70 -co 10 -f -rot 90 -n -t 0 \
#-w $1 -h $2 -b 400000 -fps 30 -pf high -fli 50hz -l -o tcp://0.0.0.0:3333 -x !

Arguments 1 and 2 are resolution, and I'm passing 320 and 320 in them.

The first one produces problematic motion vectors and nonsensically small video output, the second one works beautifully. :)

Any ideas or suggestions are quite welcome, and of course I'll mention when I've found the bug. :)

[cpunk]

My first observation would be that I'm a bit silly. I'm passing the -f or "fullscreen" argument to a command that has no screen to draw on, and the -l or "listen" argument to a command that is outputting to a file...

...but removing those bits of my sillyness doesn't make the problem disappear... instead, removing these arguments *together* corrects the problem:

Code: Select all

-awb off -awbg 0.0,0.0 -ex off

So it's something that is corrected either by outputting via TCP or alternatively alleviated by disabling automatic white balance + exposure. It may be that I'm chasing two different ghosts here...

[cpunk]

Additional notes:

- passing all ("-ex off", "-awb off" and AWB gains) --> somehow visibly altered motion vectors, tiny unreadable video file
- omitting "-ex off", omitting "-awb off" and passing AWB gains --> correct behaviour
- omitting "-ex off", passing "-awb off", and omitting AWB gains --> no motion vectors, tiny unreadable video file
- omitting "-ex off", passing "-awb off" and passing AWB gains --> no motion vectors, tiny unreadable video file
- passing all and adding "-l -o tcp://0.0.0.0:3333" instead of "-o /root/dump" --> correct behaviour

[cpunk]

Update:

After upgrading all components and rewriting my homebrew "raspivid" command to totally exclude status messages polluting the "stderr" stream when it's being used for motion vectors, I was able to collect sample video. :) The better screenshot is from the TCP stream, the totally useless one from the file dump. :D

Conclusion? :D

I was expecting bleached out video. In fact, I was demanding it from the system, by requiring that automatic white balance be disabled. I need it too, and after a bit of adjustment, my robot works with that better than anything else - because it is totally consistent, even if horrible to watch. :)

But for some reason, depending on the output chosen (TCP or file), my system is now generating different quality levels of video.

Which explains why the vectors can be found in one case, and cannot be found in the other case. But why on earth would it do so, depending on the output chosen? :D I'm impressed by this bug. It is fancy. :D

[cpunk]

Sadly, for the time being, the bug defeated me. Since I'm preparing a robot for competition, I had to go around it - and my video pipeline script has transformed too, it now reads as below, and the problem doesn't happen:

Code: Select all

raspivid -sh 0 -sa -100 -br 70 -co 10 -fl -n -t 0 \
-w $1 -h $2 -b 400000 -fps 30 -pf high -fli 50hz -o udp://192.168.1.100:3333 -x ! -rf gray -r -

[cpunk]

It's getting more and more complicated. :D I hereby (almost) promise to figure it out and make sense of it later, after the contest (on Nov 25).

I was thinking that all was fine. Then I noticed instabilty in the robot's vision. I started investigating the fishy frames. I understood that either their top or bottom half was grayed out, and the likely cause was either:

a) buffering issues
b) buffering issues and something else, because the defective part of the frame was uniform grey, not black

Finally I realized that my previous white balance and exposure settings *would* produce featureless grey image, but something seems to be "magically" avoiding that. That "something" seems to run once in the beginning of the video, and then, maybe (unconfirmed) occasionally.

Understanding that the interactions were complicated, I made yet another bug-avoidance maneuver and changed my pipeline to:

Code: Select all

raspivid -n -t 0 -awb off -awbg 8.0,8.0 -ex off -ISO 1000 -ss 80000 -sa -100 -br 60 -co 15 -fl -fli 50hz \
-w $1 -h $2 -fps 30 -o udp://192.168.1.100:3333 -r ! -rf gray

...and nothing "magical" no longer happens. :) Rock solid and sturdy low-contrast grayscale, ideal for comparisons of pixel value to the darkest pixel and to the lightest pixel. :D Will come back to revisit this issue as soon as the robot is flying. :)

[6by9]

Sorry but I've been ignoring most of this thread as your command-line is such a mass of conflicting/non-sensical parameters.

-fli is going to do nothing as you have -ex off and you have specified the shutter speed.
-ISO is going to do nothing as you have -ex off. You can now set analogue and digital gain explicitly since https://github.com/raspberrypi/userland/pull/430 and the Oct 10th 2017 firmware release with -ag and -dg.
-awbg 0.0,0.0 means that you're just using the green pixels off the sensor. Red and blue still contribute to luma output so you're just skewing the colours that it will respond to and compromising the resolution in the process.
-sa -100 is going to be synonymous with --colfx 128:128 that you started with.
-br 70 is just shifting the black point, so black isn't represented by a luma value of 0 (or 16 if in BT601/709). That seems unnecessary.
-co 10 increases the gain slightly.

As to why the files come out small is probably because they've ended up with near 0 content and so compress incredibly well.
You'll have totally screwed up the statistics due to your awb gains, so at a guess you may have also therefore caused the dynamic lens shading algorithm misbehave (I'm assuming this is a V2.1 camera)

[cpunk]

Thanks for the really useful critique and explanation. :) I knew I had *some* contradictory arguments in that command line, but I didn't know I had so _many_. :)

I'll try to narrow them down to the bare minimum needed, and see if I still encounter oddities or if they disappear - and I'll make use of analog and digital gain values. :) I didn't know they were accessible and thus pointlessly messed around with AWB gains. :)

But yes, the core of my observation is that choosing a different output (TCP) inexplicably affected the motion vector stream (input to machine vision).

I'll try to see of I can recreate that in a manner that is simple, doesn't involve telling the software to do contradictory things, and can be recreated. :)

[cpunk]

I can no longer recreate the bug when I work with the methods you recommended. :)

But, I can show the advantages of manually adjusted settings over automatic settings, though. :)

Here are 3 videos:
- tiny video...
- the yellow bar is an artifact from a different task entirely, plz ignore...
- the blue boxes show where the code thinks interesting areas are...
- camera is looking at a line pattern on the floor
- there is a ceiling lamp reflection on the floor to confuse the code (working on it :) )
- AND there is a bright strobing lamp looking into the camera for half the scene length (simulating flash photography occurring nearby)...

...and here are the command lines that produced them:

Code: Select all

# No automatic adjustment, set everything manually... 
raspivid -fl -hf -vf -n -t 0 -awb off -ex off -ss 98765 -sa -100 -ag 0.0 -dg 8.0 \
-w $1 -h $2 -fps 30 -o udp://192.168.1.101:3333 -r ! -rf gray

# Use manual digital gain, other stuff automatic...
raspivid -fl -hf -vf -n -t 0 -fli 50hz -sa -100 -dg 4.0 \
-w $1 -h $2 -fps 30 -o udp://192.168.1.101:3333 -r ! -rf gray
 
# Fully automatic, does not work right...
raspivid -fl -hf -vf -n -t 0 -sa -100 \
-w $1 -h $2 -fps 30 -o udp://192.168.1.101:3333 -r ! -rf gray

...and the result for me is: never leave the settings automatically controlled - for machine vision, results will suffer. Always mess with the settings, (including the "make silly mistakes and issue contradictory commands" part, apparently). :D :)

[cpunk]

P.S. For entertainment only... a test run that recognizes distinct areas and color-codes them. :) No longer a bug report... just getting close to something functional. :)