MMALSharp - C# API for Raspberry Pi Camera

[techyian]

Hi all,

Over the last few years I've been working on MMALSharp which is a C# API to the Pi camera. Instead of wrapping Raspistill/Raspivid in a Process call, MMALSharp calls the native MMAL library giving you a lot of configurable settings for your app. I'm choosing to post it officially on the forums now as I've just released the latest version on NuGet (0.4.3) and I'm at a point where I'm really happy with how it's performing.

The library is over at GitHub https://github.com/techyian/MMALSharp and the currently implemented functionality can be found here https://github.com/techyian/MMALSharp/wiki.

I'm still learning a lot about image processing, but I hope the library as it currently stands can be a positive contribution to people wanting access to the Pi camera via C#. Both Mono and .NET Core runtimes are supported.

Thanks,
Ian

[techyian]

Thought I'd post this here to keep everything about MMALSharp in one place.
I'm having a little bit of trouble with application hangs at the minute and it's when calling mmal_port_disable. My understanding is that this function will block indefinitely until all buffers are released which is handled by the IN_TRANSIT_INCREMENT macro. I believe I've tracked down what's causing the hangs - within the mmal_port_buffer_header_callback function, the IN_TRANSIT_DECREMENT macro is not called until the buffer_header_callback is complete and this is causing an issue for me personally because my completion signalling is done within the buffer callback (as is done in raspistill/vid), this then appears to be causing a race condition because my code reaches mmal_port_disable before the IN_TRANSIT_DECREMENT macro is called.

To work around this at the minute, I'm having to wait for an arbitrary length of time (100ms) to reduce the risk of hanging, however it doesn't appear to work 100% of the time - removing the wait guarantees application hangs.

What would be nice is if there was a public API to return port->priv->core->transit_buffer_headers, this way I could check there were definitely no references left on any buffer headers. I've tried to mock up the MMAL_PORT_PRIVATE_CORE_T struct but it's being a bit of a pig so far to do so.

Thanks,
Ian

[techyian]

Any design suggestions/fixes are most welcome. I'd like the library to perform as efficiently as possible so forcing it to wait for an arbitrary length of time isn't great

[6by9]

What completion signalling?
RaspiVid releases the buffer in the callback, and if the port is then still enabled it pulls it from the pool and sends it back to the component. I'm not clear what you're considering as completion signalling from there.

AFAIK There is no issue with calling mmal_port_disable whilst there are buffers with the component. disable inherently calls flush, so they will all be returned before disable completes.

If you are calling mmal_port_disable from the callback context then that is the wrong thing to do because it will block and stop any further buffers being returned.

[techyian]

Yes, buffer releasing is done in the callback also. There is an asynchronous wait within the library which will be released when MMAL states any of the following or if the async wait is cancelled by the user:

MMAL_BUFFER_HEADER_FLAG_EOS
MMAL_BUFFER_HEADER_FLAG_FRAME_END
MMAL_BUFFER_HEADER_FLAG_TRANSMISSION_FAILED

The issue I believe I'm having is that once completion has been triggered, MMALSharp will begin to disable any active ports in use (this isn't done in the buffer callback BTW) and if the buffer_header_callback function https://github.com/raspberrypi/userland ... ort.c#L896 hasn't completed by decreasing the reference count, we will have a hang whilst MMAL waits for this to occur.

My understanding may be incorrect, and I may be barking up the wrong tree so apologies if this is the case, however at the minute that's the only conclusion I've come to.

[6by9]

The issue I believe I'm having is that once completion has been triggered, MMALSharp will begin to disable any active ports in use (this isn't done in the buffer callback BTW) and if the buffer_header_callback function https://github.com/raspberrypi/userland ... ort.c#L896 hasn't completed by decreasing the reference count, we will have a hang whilst MMAL waits for this to occur.

- Your callback is called by mmal_port_buffer_header_callback. Buffer is released back to the pool.
- Your callback signals a completion to a different thread.
- Callback is pre-empted by other thread, which calls mmal_port_disable.
- mmal_port_disable will call across to VideoCore to disable the component there. That will flush any buffers left and call your callback for them.
- mmal_port_disable blocks (doesn't hang) on the transit_sema counting semaphore in IN_TRANSIT_WAIT
- callback should be rescheduled and complete the IN_TRANSIT_DECREMENT, which posts transit_sema and thereby allows mmal_port_disable to complete.

raspivid always relies on this behaviour, although admittedly it does default to only having one output buffer. Bump it up to say 8 in https://github.com/raspberrypi/userland ... id.c#L2089, use

Code: Select all

export VC_LOGLEVEL="mmal:trace"

to bump up the logging, or add additional logging of your own.
Admittedly I'm seeing all buffer callbacks having been made before your mmal_port_disable_internal is called and checking for all buffers to be returned. Actually I think that is guaranteed via https://github.com/raspberrypi/userland ... api.c#L318

Are you sure you haven't claimed a mutex that is blocking the callback from running? Sorry, I'd need a nice simple test case to confirm this before digging further into this myself.

Your code in DisablePort does look a little backwards - https://github.com/techyian/MMALSharp/b ... se.cs#L348.
I would have expected you to disable the port and hence get all the callback BEFORE you then destroy all the buffers in the pool.

[techyian]

Thanks for taking the time to get back to me. My callback is definitely being called, and in this scenario I'm describing there is only 1 buffer in use. I appreciate your comment regarding the DisablePort method, it was paranoid programming on my part as I've been struggling with this blocking issue for a while now, although I agree it is unnecessary and I'll be removing that code in the near future.

As I mentioned, my thoughts on this currently are:

- Buffer callback is entered, completed status is found and wait is released.
- Within the time it takes for the thread handling the callback to complete the IN_TRANSIT_DECREMENT call, mmal_port_disable is called on my side causing the blocking.

Your comments re: raspivid are interesting, I will give that a test with multiple buffer headers and observe the behaviour there.

I'll continue digging on my end to see if I get anywhere. To reproduce this you'd need to build my code from source without the following line https://github.com/techyian/MMALSharp/b ... ra.cs#L367. There are lots of examples in the Wiki, the most basic of which is https://github.com/techyian/MMALSharp/w ... ge-Capture.

[techyian]

It does seem like this is the issue I'm having. Raspivid behaves correctly even with multiple buffer headers so it's going to be something I need to address with my design. I've added additional logging, and MMALSharp reaches the port disable code before the callback manages to finish. I'll keep plugging away and let you know how I get on.

Thanks again,
Ian

[techyian]

This has now been resolved I believe. 3rd party dependency I was relying on allowed main thread to reach disable port code before exiting callback method. Removed dependency and replaced with my own implementation of what I needed it to do. Seems to be pretty solid now with no blocking so far.

[Eric89]

I just started to look at MMALSharp and it looks really great. Do you know how I could send the video stream from PiCamera with MMALSharp to the accord net framework (http://accord-framework.net/). This framework is awesome with computer vision, deep learning capabilities, lego robotic handling, etc...

Thanks in advance.

Eric

[techyian]

Hi Eric,

The Accord framework does look good, I wasn't aware of it so thanks for the heads up. What operations are you looking to carry out on a video stream with Accord? There doesn't appear to be any documentation regarding Video streams in their Wiki - do you have a link to any documentation please?

Thanks,
Ian

[Eric89]

Thanks for the quick reply. Basically, as a first step, I am trying to implement the example of this url (http://accord-framework.net/docs/html/T ... tector.htm) on the live feed of the raspberry pi cam. It requires in this precise example to extract each video frame, to convert them to bmp data and then to let the accord framework do the magic of face detection .

These url will give you some insigts of the accord video namespace: http://accord-framework.net/docs/html/N ... _Video.htm and subnamespace http://accord-framework.net/docs/html/R ... rd_NET.htm

[techyian]

https://gist.github.com/techyian/fa3295 ... f422a4a5c7

I think this should hopefully get you on the right tracks. It's making use of the ICaptureHandler interface which MMALSharp calls back to with image frame data. You'll also need to download the ImageSharp Nuget package which I'm using to save the individual JPEG frames from the stream to BMP; I got the HaarObjectDetector code from here https://github.com/accord-net/framework ... torTest.cs. The results can be a bit hit-and-miss, and using a high resolution video stream will decrease performance significantly so the gist provided is at 640x480.

Let me know if there's anything else I can do to help.

Ian

[Eric89]

I thought the result would be faster, the time to save the bitmap is huge. I saw this trial with opencv which allows to get 4-5 fps on a pi (https://www.raspberrypi.org/blog/facial ... era-board/).

I have few questions:
- is it possible to decrease the resolution below 640x480 with MMLSharp (in the enum it seems to be the minimum)?
- on this line: vidEncoder.ConfigureOutputPort(0, MMALEncoding.MJPEG, MMALEncoding.RGBA, 90, 25000000); encodingType et Pixelformat are both MMALEncoding which make it very difficult to find a working pair of value when you are not an expert... do you have any doc showing all the working combination? Do you think one of them could speed up things?

[techyian]

Yes, I agree the performance seen here isn't brilliant unfortunately.

You can adjust the resolution for the video port as follows:

Code: Select all

MMALCameraConfig.VideoResolution = new Resolution(320, 240);

Reducing to the resolution above will give you around 1fps, but I appreciate it's still not fantastic.

The quality parameter on ConfigureOutputPort for MJPEG encoding does not affect anything. I'm not seeing any difference in performance when lowering the bitrate parameter either, I'll investigate whether that's an issue with MMALSharp or whether it's simply not honored for MJPEG.

I chose MJPEG for you as you're wanting to strip out the individual image frames from the video stream and convert them to BMP. There may be ways in which to improve the saving of BMP using ImageSharp - reach out to them on Github, or alternatively you could search for a different library to provide this support? There may also be a way of just sending the raw JPEG stream to Accord but I haven't had time to investigate this possibility yet.

To retrieve the supported encodings of a given port, you can call GetSupportedEncodings. In the case of the vidEncoder from the gist it would be:

Code: Select all

vidEncoder.Outputs[0].GetSupportedEncodings();

That will give you an array of FourCC codes, which you can parse with the int extension method ParseEncoding to get its human-friendly name.

Hope that's of use to you.

[rasp14]

Hi OP,

I've just found out this API of yours and it look great! Thanks for taking the effort to expose the native camera API for C#! A few questions for you hopefully you can help to answer.

1. Is there a way to capture still image in bytes instead of saving it into storage?
2. Is there a method similar to cam.TakePictureTimelapse() where i can control the timing of capturing image by using an external signal instead of specifying constant interval using Timelapse class? I'm looking for the feature similar to -s parameter in raspistill.

I'm working on a custom interface that handles multiple type of cameras, and i plan to support Raspberry Pi camera as well. My interface will handles the connection to Pi and transfer the captured images in bytes through TCP/IP to other destination. I'm looking to utilize your API to achieve that.

Looking forward to hear from you soon. Thanks in advance!

[techyian]

Hi rasp14,

To capture frame data to a byte array, the easiest thing to do would be to wire up your own ICaptureHandler which writes to a byte array. Something like the following should work using a list for expandable storage (not tested this but hopefully you get the jist):

Code: Select all

public class ByteArrayCaptureHandler : ICaptureHandler
{
        public List<byte> WorkingData { get; set; }

        public ByteArrayCaptureHandler()
        {
            this.WorkingData = new List<byte>();
        }
        
        public void Dispose()
        {            
        }

        public ProcessResult Process(uint allocSize)
        {
            throw new NotImplementedException();
        }

        public void Process(byte[] data)
        {
            this.WorkingData.AddRange(data);
        }

        public void PostProcess()
        {
            // Anything final after capture is complete.
        }
}

Then you can just pass that to the encoder component you're using, i.e:

Code: Select all

using (var imgCaptureHandler = new ByteArrayCaptureHandler())
using (var imgEncoder = new MMALImageEncoder(imgCaptureHandler))

Regarding externally signalling a timelapse capture, this is something you'll have to wire up yourself too as it's not something in the library by default. Maybe wiring up an event delegate would be the best way forward here that you can signal upon a button press or something similar?

Hope that helps.

[informed]

Hi

I am currently doing the same thing myself.

It is a great library and lots of hard/good work.

Did you actually get this to work? How do you incorporate the encoder to acquire the jpeg format?

[techyian]

Ohhh sorry I must have been on auto-pilot, just take the capture handler out of the using block and that should be fine, there's no disposable objects in the ByteArrayCaptureHandler class anyway. You can then access the WorkingData list from the capture handler.

For reference:

Code: Select all

var imgCaptureHandler = new ByteArrayCaptureHandler();
using (var imgEncoder = new MMALImageEncoder(imgCaptureHandler))
using (var renderer = new MMALVideoRenderer())
{
    cam.ConfigureCameraSettings();
        
    // Create our component pipeline.         
    imgEncoder.ConfigureOutputPort(0, MMALEncoding.JPEG, MMALEncoding.I420, 90);
                
    cam.Camera.StillPort.ConnectTo(imgEncoder);                    
    cam.Camera.PreviewPort.ConnectTo(renderer);
        
    // Camera warm up time
    await Task.Delay(2000);        
    await cam.ProcessAsync(cam.Camera.StillPort);
                
    // Do what you need with the byte array...let's see how big it is.
    Console.WriteLine(imgCaptureHandler.WorkingData.Count);
}