MMALSharp - C# API for Raspberry Pi Camera

[techyian]

Hi all,

Over the last few years I've been working on MMALSharp which is a C# API to the Pi camera. Instead of wrapping Raspistill/Raspivid in a Process call, MMALSharp calls the native MMAL library giving you a lot of configurable settings for your app. I'm choosing to post it officially on the forums now as I've just released the latest version on NuGet (0.4.3) and I'm at a point where I'm really happy with how it's performing.

The library is over at GitHub https://github.com/techyian/MMALSharp and the currently implemented functionality can be found here https://github.com/techyian/MMALSharp/wiki.

I'm still learning a lot about image processing, but I hope the library as it currently stands can be a positive contribution to people wanting access to the Pi camera via C#. Both Mono and .NET Core runtimes are supported.

Thanks,
Ian

[techyian]

Thought I'd post this here to keep everything about MMALSharp in one place.
I'm having a little bit of trouble with application hangs at the minute and it's when calling mmal_port_disable. My understanding is that this function will block indefinitely until all buffers are released which is handled by the IN_TRANSIT_INCREMENT macro. I believe I've tracked down what's causing the hangs - within the mmal_port_buffer_header_callback function, the IN_TRANSIT_DECREMENT macro is not called until the buffer_header_callback is complete and this is causing an issue for me personally because my completion signalling is done within the buffer callback (as is done in raspistill/vid), this then appears to be causing a race condition because my code reaches mmal_port_disable before the IN_TRANSIT_DECREMENT macro is called.

To work around this at the minute, I'm having to wait for an arbitrary length of time (100ms) to reduce the risk of hanging, however it doesn't appear to work 100% of the time - removing the wait guarantees application hangs.

What would be nice is if there was a public API to return port->priv->core->transit_buffer_headers, this way I could check there were definitely no references left on any buffer headers. I've tried to mock up the MMAL_PORT_PRIVATE_CORE_T struct but it's being a bit of a pig so far to do so.

Thanks,
Ian

[techyian]

Any design suggestions/fixes are most welcome. I'd like the library to perform as efficiently as possible so forcing it to wait for an arbitrary length of time isn't great

[6by9]

What completion signalling?
RaspiVid releases the buffer in the callback, and if the port is then still enabled it pulls it from the pool and sends it back to the component. I'm not clear what you're considering as completion signalling from there.

AFAIK There is no issue with calling mmal_port_disable whilst there are buffers with the component. disable inherently calls flush, so they will all be returned before disable completes.

If you are calling mmal_port_disable from the callback context then that is the wrong thing to do because it will block and stop any further buffers being returned.

[techyian]

Yes, buffer releasing is done in the callback also. There is an asynchronous wait within the library which will be released when MMAL states any of the following or if the async wait is cancelled by the user:

MMAL_BUFFER_HEADER_FLAG_EOS
MMAL_BUFFER_HEADER_FLAG_FRAME_END
MMAL_BUFFER_HEADER_FLAG_TRANSMISSION_FAILED

The issue I believe I'm having is that once completion has been triggered, MMALSharp will begin to disable any active ports in use (this isn't done in the buffer callback BTW) and if the buffer_header_callback function https://github.com/raspberrypi/userland ... ort.c#L896 hasn't completed by decreasing the reference count, we will have a hang whilst MMAL waits for this to occur.

My understanding may be incorrect, and I may be barking up the wrong tree so apologies if this is the case, however at the minute that's the only conclusion I've come to.

[6by9]

The issue I believe I'm having is that once completion has been triggered, MMALSharp will begin to disable any active ports in use (this isn't done in the buffer callback BTW) and if the buffer_header_callback function https://github.com/raspberrypi/userland ... ort.c#L896 hasn't completed by decreasing the reference count, we will have a hang whilst MMAL waits for this to occur.

- Your callback is called by mmal_port_buffer_header_callback. Buffer is released back to the pool.
- Your callback signals a completion to a different thread.
- Callback is pre-empted by other thread, which calls mmal_port_disable.
- mmal_port_disable will call across to VideoCore to disable the component there. That will flush any buffers left and call your callback for them.
- mmal_port_disable blocks (doesn't hang) on the transit_sema counting semaphore in IN_TRANSIT_WAIT
- callback should be rescheduled and complete the IN_TRANSIT_DECREMENT, which posts transit_sema and thereby allows mmal_port_disable to complete.

raspivid always relies on this behaviour, although admittedly it does default to only having one output buffer. Bump it up to say 8 in https://github.com/raspberrypi/userland ... id.c#L2089, use

Code: Select all

export VC_LOGLEVEL="mmal:trace"

to bump up the logging, or add additional logging of your own.
Admittedly I'm seeing all buffer callbacks having been made before your mmal_port_disable_internal is called and checking for all buffers to be returned. Actually I think that is guaranteed via https://github.com/raspberrypi/userland ... api.c#L318

Are you sure you haven't claimed a mutex that is blocking the callback from running? Sorry, I'd need a nice simple test case to confirm this before digging further into this myself.

Your code in DisablePort does look a little backwards - https://github.com/techyian/MMALSharp/b ... se.cs#L348.
I would have expected you to disable the port and hence get all the callback BEFORE you then destroy all the buffers in the pool.

[techyian]

Thanks for taking the time to get back to me. My callback is definitely being called, and in this scenario I'm describing there is only 1 buffer in use. I appreciate your comment regarding the DisablePort method, it was paranoid programming on my part as I've been struggling with this blocking issue for a while now, although I agree it is unnecessary and I'll be removing that code in the near future.

As I mentioned, my thoughts on this currently are:

- Buffer callback is entered, completed status is found and wait is released.
- Within the time it takes for the thread handling the callback to complete the IN_TRANSIT_DECREMENT call, mmal_port_disable is called on my side causing the blocking.

Your comments re: raspivid are interesting, I will give that a test with multiple buffer headers and observe the behaviour there.

I'll continue digging on my end to see if I get anywhere. To reproduce this you'd need to build my code from source without the following line https://github.com/techyian/MMALSharp/b ... ra.cs#L367. There are lots of examples in the Wiki, the most basic of which is https://github.com/techyian/MMALSharp/w ... ge-Capture.

[techyian]

It does seem like this is the issue I'm having. Raspivid behaves correctly even with multiple buffer headers so it's going to be something I need to address with my design. I've added additional logging, and MMALSharp reaches the port disable code before the callback manages to finish. I'll keep plugging away and let you know how I get on.

Thanks again,
Ian

[techyian]

This has now been resolved I believe. 3rd party dependency I was relying on allowed main thread to reach disable port code before exiting callback method. Removed dependency and replaced with my own implementation of what I needed it to do. Seems to be pretty solid now with no blocking so far.

[Eric89]

I just started to look at MMALSharp and it looks really great. Do you know how I could send the video stream from PiCamera with MMALSharp to the accord net framework (http://accord-framework.net/). This framework is awesome with computer vision, deep learning capabilities, lego robotic handling, etc...

Thanks in advance.

Eric

[techyian]

Hi Eric,

The Accord framework does look good, I wasn't aware of it so thanks for the heads up. What operations are you looking to carry out on a video stream with Accord? There doesn't appear to be any documentation regarding Video streams in their Wiki - do you have a link to any documentation please?

Thanks,
Ian

[Eric89]

Thanks for the quick reply. Basically, as a first step, I am trying to implement the example of this url (http://accord-framework.net/docs/html/T ... tector.htm) on the live feed of the raspberry pi cam. It requires in this precise example to extract each video frame, to convert them to bmp data and then to let the accord framework do the magic of face detection .

These url will give you some insigts of the accord video namespace: http://accord-framework.net/docs/html/N ... _Video.htm and subnamespace http://accord-framework.net/docs/html/R ... rd_NET.htm

[techyian]

https://gist.github.com/techyian/fa3295 ... f422a4a5c7

I think this should hopefully get you on the right tracks. It's making use of the ICaptureHandler interface which MMALSharp calls back to with image frame data. You'll also need to download the ImageSharp Nuget package which I'm using to save the individual JPEG frames from the stream to BMP; I got the HaarObjectDetector code from here https://github.com/accord-net/framework ... torTest.cs. The results can be a bit hit-and-miss, and using a high resolution video stream will decrease performance significantly so the gist provided is at 640x480.

Let me know if there's anything else I can do to help.

Ian

[Eric89]

I thought the result would be faster, the time to save the bitmap is huge. I saw this trial with opencv which allows to get 4-5 fps on a pi (https://www.raspberrypi.org/blog/facial ... era-board/).

I have few questions:
- is it possible to decrease the resolution below 640x480 with MMLSharp (in the enum it seems to be the minimum)?
- on this line: vidEncoder.ConfigureOutputPort(0, MMALEncoding.MJPEG, MMALEncoding.RGBA, 90, 25000000); encodingType et Pixelformat are both MMALEncoding which make it very difficult to find a working pair of value when you are not an expert... do you have any doc showing all the working combination? Do you think one of them could speed up things?

[techyian]

Yes, I agree the performance seen here isn't brilliant unfortunately.

You can adjust the resolution for the video port as follows:

Code: Select all

MMALCameraConfig.VideoResolution = new Resolution(320, 240);

Reducing to the resolution above will give you around 1fps, but I appreciate it's still not fantastic.

The quality parameter on ConfigureOutputPort for MJPEG encoding does not affect anything. I'm not seeing any difference in performance when lowering the bitrate parameter either, I'll investigate whether that's an issue with MMALSharp or whether it's simply not honored for MJPEG.

I chose MJPEG for you as you're wanting to strip out the individual image frames from the video stream and convert them to BMP. There may be ways in which to improve the saving of BMP using ImageSharp - reach out to them on Github, or alternatively you could search for a different library to provide this support? There may also be a way of just sending the raw JPEG stream to Accord but I haven't had time to investigate this possibility yet.

To retrieve the supported encodings of a given port, you can call GetSupportedEncodings. In the case of the vidEncoder from the gist it would be:

Code: Select all

vidEncoder.Outputs[0].GetSupportedEncodings();

That will give you an array of FourCC codes, which you can parse with the int extension method ParseEncoding to get its human-friendly name.

Hope that's of use to you.

[rasp14]

Hi OP,

I've just found out this API of yours and it look great! Thanks for taking the effort to expose the native camera API for C#! A few questions for you hopefully you can help to answer.

1. Is there a way to capture still image in bytes instead of saving it into storage?
2. Is there a method similar to cam.TakePictureTimelapse() where i can control the timing of capturing image by using an external signal instead of specifying constant interval using Timelapse class? I'm looking for the feature similar to -s parameter in raspistill.

I'm working on a custom interface that handles multiple type of cameras, and i plan to support Raspberry Pi camera as well. My interface will handles the connection to Pi and transfer the captured images in bytes through TCP/IP to other destination. I'm looking to utilize your API to achieve that.

Looking forward to hear from you soon. Thanks in advance!

[techyian]

Hi rasp14,

To capture frame data to a byte array, the easiest thing to do would be to wire up your own ICaptureHandler which writes to a byte array. Something like the following should work using a list for expandable storage (not tested this but hopefully you get the jist):

Code: Select all

public class ByteArrayCaptureHandler : ICaptureHandler
{
        public List<byte> WorkingData { get; set; }

        public ByteArrayCaptureHandler()
        {
            this.WorkingData = new List<byte>();
        }
        
        public void Dispose()
        {            
        }

        public ProcessResult Process(uint allocSize)
        {
            throw new NotImplementedException();
        }

        public void Process(byte[] data)
        {
            this.WorkingData.AddRange(data);
        }

        public void PostProcess()
        {
            // Anything final after capture is complete.
        }
}

Then you can just pass that to the encoder component you're using, i.e:

Code: Select all

using (var imgCaptureHandler = new ByteArrayCaptureHandler())
using (var imgEncoder = new MMALImageEncoder(imgCaptureHandler))

Regarding externally signalling a timelapse capture, this is something you'll have to wire up yourself too as it's not something in the library by default. Maybe wiring up an event delegate would be the best way forward here that you can signal upon a button press or something similar?

Hope that helps.

[informed]

Hi

I am currently doing the same thing myself.

It is a great library and lots of hard/good work.

Did you actually get this to work? How do you incorporate the encoder to acquire the jpeg format?

[techyian]

Ohhh sorry I must have been on auto-pilot, just take the capture handler out of the using block and that should be fine, there's no disposable objects in the ByteArrayCaptureHandler class anyway. You can then access the WorkingData list from the capture handler.

For reference:

Code: Select all

var imgCaptureHandler = new ByteArrayCaptureHandler();
using (var imgEncoder = new MMALImageEncoder(imgCaptureHandler))
using (var renderer = new MMALVideoRenderer())
{
    cam.ConfigureCameraSettings();
        
    // Create our component pipeline.         
    imgEncoder.ConfigureOutputPort(0, MMALEncoding.JPEG, MMALEncoding.I420, 90);
                
    cam.Camera.StillPort.ConnectTo(imgEncoder);                    
    cam.Camera.PreviewPort.ConnectTo(renderer);
        
    // Camera warm up time
    await Task.Delay(2000);        
    await cam.ProcessAsync(cam.Camera.StillPort);
                
    // Do what you need with the byte array...let's see how big it is.
    Console.WriteLine(imgCaptureHandler.WorkingData.Count);
}

[techyian]

Happy New Year everyone,

I'm almost ready to release v0.5 of MMALSharp, however I've hit a bit of a stumbling block with implementing Video Encode/Decode directly from a file. The operations complete successfully, however the resulting video isn't correct and I was hoping someone might be able to tell me what's going on. I've had a look at an example similar to what I'm trying to achieve on @6by9's GitHub https://github.com/6by9/rpi_mmal_exampl ... e_encode.c however this didn't complete successfully and I haven't had time to investigate why. For clarity my tests are using the same Buck Bunny H.264 file provided in the repo.

The current test I'm trying to do is simply decode the Buck Bunny data to a raw I420 stream, then re-encode to H.264 again in the hope I'll see a similar result to the original file. I appreciate it'll be encoded to a containerless file so I've used the relevant demux flag in VLC to check the result.

It looks as though there is some kind of corruption in the data and continuously slides from left to right, so perhaps it's an issue with what's being fed to the Decoder and then the Encoder however the components are usually pretty good at firing back a transmission failed or corrupted MMAL error if it's being fed rubbish.

I've uploaded the result to https://techyian.github.io/downloads/re ... _bunny.avi

Any ideas/suggestions would be very helpful.

Ian

[6by9]

Your clip doesn't want to play at all for me in VLC. It does identify as coded size 1280x738, display size as 1280x720, so I suspect something is off in your port formats.
Have you managed to round the format.es.video.height up to multiple of 32 instead of 16 (although that would be 736, not 738 - 738 is a really odd number to be reported)?

The rpi_mmal_examples repo was a fork of a test app that someone else wrote and reported didn't work so I fixed up at one point. I'll try to have a look on Monday when I'm back in the office to see whether there is something funny going on there.

[techyian]

Did you use the demux=h264 flag when starting VLC as that's what I needed to use?

I hadn't padded the format.es.video.height to a multiple of 32, I've now made those changes thanks, but I'm still seeing the strange result.

Just to run some file sizes past you, bearing in mind the original Buck Bunny video is 3.5Mb:

Prior to padding fix

• Decoded I420 - 473Mb

• Re-encoded H.264 using I420 - 8.5Mb

After padding fix

• Decoded I420 - 483Mb

• Re-encoded H.264 using I420 - 8.7Mb

Do the decoded file sizes seem acceptable to you coming from a 3.5Mb encoded file? I really have no idea myself. If the decoded file size seems bizarre then that might explain the issues I'm having. To note, the re-encoded file is also much larger than the original.

I've uploaded the file with the fixed height of 736 to https://techyian.github.io/downloads/re ... ny_736.avi - it might give you a headache, sorry!

[techyian]

I've tried playing around with some of the other settings that can be passed to the encoder's output port for H.264 streams such as the video profile, intra period etc. to no avail. I also wanted to check with you whether it should be possible to decode MJPEG streams as the decoder doesn't seem to process any data when using that encoding type?

[6by9]

Did you use the demux=h264 flag when starting VLC as that's what I needed to use?

No, because the file had an avi extension so I (and my Android phone) were assuming an avi container. I see now that it is a raw h264 elementary stream and can view it as such (use the extension .h264 and VLC autodetects it correctly anyway).
Your stride is mismatched - format.es.video.width.

I hadn't padded the format.es.video.height to a multiple of 32, I've now made those changes thanks, but I'm still seeing the strange result.

Not what I had said. I was guessing what you might have done as I couldn't view the stream, and wondered if you had the height incorrectly set. format.es.video.height only needs to be a multiple of 16.

Just to run some file sizes past you, bearing in mind the original Buck Bunny video is 3.5Mb:

Prior to padding fix

• Decoded I420 - 473Mb

• Re-encoded H.264 using I420 - 8.5Mb

After padding fix

• Decoded I420 - 483Mb

• Re-encoded H.264 using I420 - 8.7Mb

Do the decoded file sizes seem acceptable to you coming from a 3.5Mb encoded file? I really have no idea myself. If the decoded file size seems bizarre then that might explain the issues I'm having. To note, the re-encoded file is also much larger than the original.

Input has almost no affect, solely the bitrate configured on the output.

I've uploaded the file with the fixed height of 736 to https://techyian.github.io/downloads/re ... ny_736.avi - it might give you a headache, sorry!

Now both stride and chroma alignment are wrong. I'm surprised it isn't scrolling vertically as well as the buffer sizes would be unlikely to match. Then again it would be working on one buffer in, one buffer out, so that may explain it.

[techyian]

use the extension .h264 and VLC autodetects it correctly anyway

Thanks, and apologies for the confusion.

Not what I had said. I was guessing what you might have done as I couldn't view the stream, and wondered if you had the height incorrectly set. format.es.video.height only needs to be a multiple of 16.

Have you managed to round the format.es.video.height up to multiple of 32 instead of 16

Crossed wires, sorry. In the first file I uploaded, the width is/should be padded to 32 and height padded to 16. I'll revert the changes and try and figure out why the width stride is wrong, thanks for the heads up. Can I ask how you're checking that the stride is incorrect so I don't need to keep nagging you?