Raspberry Pi Forums

Hi,

I am trying to setup something to record and broadcast local conferences, and I was lent a Senheiser Freeport which has an XLR-3M socket audio output (blanaced) and a 1/4" jack socket (unbalanced).
Do I need a hat to input that kind of audio into the Pi, or will the jack be enough?
Can I hope to manage to synchronize the audio and the video on the fly?
Thanks.

gounthar wrote: ↑

Thu Oct 04, 2018 1:59 pm

Hi,

I am trying to setup something to record and broadcast local conferences, and I was lent a Senheiser Freeport which has an XLR-3M socket audio output (blanaced) and a 1/4" jack socket (unbalanced).
Do I need a hat to input that kind of audio into the Pi, or will the jack be enough?
Can I hope to manage to synchronize the audio and the video on the fly?
Thanks.

The RPi SBC has no Audio Input what-so-ever.


So yes you will need an either a USB Sound Card or a GPIO Sound Card, such as:

https://www.amazon.co.uk/Dynamode-USB-S ... B0010JP9CY

or

http://www.audioinjector.net/rpi-hat

https://www.amazon.co.uk/Sound-Card-Ras ... B01HCC0210

Hi there Gounthar

You have suggested that you want to record sound and video from conferances

Firstly if you are wanting to record video then sound will not be a problem. Video needs a lot more bandwidth than sound.

If you are intending on capturing video then the resolution, and bitrate of your video will be the primary factor.
if your video comes in over USB (webcam kinda thing) then pretty much everything will be fine
if you are wanting to capture analogue video then you will need a HAT (or similar capture board) for that - try and find one with both Audio and Video inputs.
if, however, you are wanting to capture HDMI or SDI video / audio streams then the Pi probably is not the thing for you. There just isn't enough processing power, memory or bandwidth available to transcode the AV to a stream with low enough bandwith requirements to save over the network or onto a USB disk. (Note playback is easy, real-time single pass encoding takes quite a lot more resources)

If you want to capture both sound and Video togeather I would strongly recomend that you feed your audio into your camera BEFORE you digitise it, otherwise you are going to suffer a lot of synchronisation problems. The act of digitising needs a small buffer when using a pre-emptive OS, but transcoding that data into your desired format requires quite a large buffer and adds significant latancy (I would recomend that you consider the VP9 video CODEC - it is one of the lowest overhead CODECs we have run).
Latancy is not your friend when you are trying to combine multiple compressed sources. It is much much simpler to add your microphone input to your camera (or for larger setups add the output of your sound desk).

If you want to mix various video sources togeather (live) then this takes considerably more resources and bandwidth than is available on a Pi sorry

Finally if you just want to record sound - then the Pi is plenty capable of doing this. You will need some form of audio capture - there are plenty of small hats that can do this (or you use a USB sound card).

I have been involved in recording and streaming conferances for a few years now, mainly the Debian Conferance 'Debconf' - whilst this may be a more complex setup than what you are intending to have, take a look at the team's documentation (very much a work in progress) to get an idea of the problems and how they can be solved...


https://debconf-video-team.pages.debian ... index.html

Hi RattusRattus,

thanks a bunch for your detailed answer.
I will read your documentation with great attention.
I was going for capturing video thanks to a modified Pi camera (ArduCam).
If we need several Pis to do the job, that's ok too.
I was thinking of using Voctomix on a Renegade Elite to mix everything.
My ramblings about all that are there: https://gist.github.com/gounthar/6e2073 ... 78af#sound

gounthar wrote: ↑

Fri Oct 05, 2018 9:45 am

thanks a bunch for your detailed answer.
I will read your documentation with great attention.
I was going for capturing video thanks to a modified Pi camera (ArduCam).
If we need several Pis to do the job, that's ok too.
I was thinking of using Voctomix on a Renegade Elite to mix everything.

You had read the hardware spec for the kit that they're using?

Voctomix requires a fair amount of CPU time to run in the default configuration of 1920×1080 at 25fps. Our production systems have these CPUs: Intel Core i7-3770 CPU 4x 3.40GHz but we're also experimenting with newer ones like these: Intel Core i7-6700K, 4x 4.00GHz

Got a couple of grand spare for your kit?
Whilst SBCs often do have hardware acceleration of various codec related stuff, it's the pure number crunching to do video mixing that is the killer. There's no sensible way to spread that over multiple machines.

Hardware accelerated VP9 encode support is limited. H264 encoding is common-place (and the only encoder available on the Pi).
You can get HDMI input on the Pi doing up to 1080P50 and with audio - look for the Auvidea B101, but use the kernel drivers and NOT raspivid.
The Pi H264 encoder is specified for 1080P30 only, but can be overclocked to manage 1080P50.

Timestamps are your friend. Both audio and video capture devices should be timestamping the incoming data, and that is the correct wya to pull the two together. Disregarding timestamps will cause issues.

6by9 wrote: ↑

Fri Oct 05, 2018 10:50 am

You had read the hardware spec for the kit that they're using?

Sir, yes, Sir!

6by9 wrote: ↑

Fri Oct 05, 2018 10:50 am

Voctomix requires a fair amount of CPU time to run in the default configuration of 1920×1080 at 25fps. Our production systems have these CPUs: Intel Core i7-3770 CPU 4x 3.40GHz but we're also experimenting with newer ones like these: Intel Core i7-6700K, 4x 4.00GHz

Got a couple of grand spare for your kit?

Nope, but we'll see what we can do with what I have on hand, and what I could buy (at a reasonable price) to complement.

6by9 wrote: ↑

Fri Oct 05, 2018 10:50 am

Whilst SBCs often do have hardware acceleration of various codec related stuff, it's the pure number crunching to do video mixing that is the killer. There's no sensible way to spread that over multiple machines.

I got that, and if my setup is not up to the task, we'll use an i7 PC. But I have to try (and to fail?) first. This way, I'll know for sure.

6by9 wrote: ↑

Fri Oct 05, 2018 10:50 am

Hardware accelerated VP9 encode support is limited. H264 encoding is common-place (and the only encoder available on the Pi).
You can get HDMI input on the Pi doing up to 1080P50 and with audio - look for the Auvidea B101, but use the kernel drivers and NOT raspivid.
The Pi H264 encoder is specified for 1080P30 only, but can be overclocked to manage 1080P50.

Timestamps are your friend. Both audio and video capture devices should be timestamping the incoming data, and that is the correct wya to pull the two together. Disregarding timestamps will cause issues.

Thanks a lot for your advice.

gounthar wrote: ↑

Fri Oct 05, 2018 9:45 am

I was going for capturing video thanks to a modified Pi camera (ArduCam).
If we need several Pis to do the job, that's ok too.

one Pi, one PiCamera will be fine. But you will still have an issue with getting sound onto the system as I can't see any mention of a mic input on the ArduCam (but then I only had a quick look).
Each Pi / PiCAM must be able to handle streaming your video across ethernet to your mixer

gounthar wrote: ↑

Fri Oct 05, 2018 9:45 am

I was thinking of using Voctomix on a Renegade Elite to mix everything.

I wouldn't be sure that will have enough hosepower. Whilst yes we use an i7 machine for this in each talk room You will probably get away with a smaller box (think i5 or perhaps an i3) a mobile phone class ARM is probably not going to be up to the task.

We also use that machine to Record each talk, and to stream as well so there is also some transcoding going on on the Debian boxes.

Real time Transcoding takes a lot of CPU and IO, be warned...

If you are simply recording to disk to transcode later then you can sacrifice horsepower for time (Who cares if it takes overnight to transcode a 1hour talk that isn't going to be uploaded for another week?)

Of cause that means that you will need to be able to record to disk...
Be sure that you have enough IO bandwidth on your editing station to
(1) receive the video from all your sources
(2) transmit any and all output streams
(3) record your mix to disk - either locally or across a network

Good luck

/Andy

Thanks a lot, Andy.
I want/need to broadcast and at the same time record the session.
I wanted to do it all only with SBCs (because we all have dreams, don't we?), but I can add an i7 laptop (which we have, unfortunately under Windows) to the equation if I have no other choice.

Arducam does not have a mic or line in input (I think it is more or less a standard Raspberry Pi camera with an added lens/CS or M12 mount).
I am doing tests with USB sound cards and Sennheiser microphones and we'll see if I ever can manage to mix them on the Pi (or on another SBC) before streaming them to the mixer (hopefully with something fancier than netcat).

We hope (yes, we're naïve) to be able to broadcast to twitch or youtube thanks to restream.io whilst recording the file.
Transcoding time is not a problem. If a machine has to work one whole week before delivering the final version, that's ok (and we'll (well, sort of) have the Youtube/Twitch recording anyway...).

Stay tuned to get more information about this upcoming failure. I'm sure I will have learned tons of things along the way.
Thanks a lot for the advice, I do appreciate.

Bruno

Hmm, you've got me curious now, at least with the currently stated limits of 720p (you don't say 30 or 60fps).
For output such as https://debconf16.debconf.org/talks/94/ (random debconf video link that Google brought up), then the Pi may be able to keep up, mainly as it has dedicated video composition hardware (the HVS - Hardware Video Scaler), and H264 encoder. I've just finished writing a MMAL component that wraps the HVS in memory to memory mode for overlaying subtitles and similar in VLC.

I'm just wondering if a Compute Module 3 with TC358743 based HDMI to CSI2 bridge chip (eg Auvidea B102), a Pi camera, and a sound input card (the AudioInjector range look good) would allow a very similar setup for a single camera.

The TC358743 will happily do 720P60 RGB into the Pi including HDMI audio. It can do 1080P60, but that will start stressing the Pi infrastructure.
The HVS will compose that RGB24 as fullscreen plus overlays into memory.
Feed that into the H264 encoder and it'll convert from RGB to YUV using the ISP hardware.
So far the CPU and GPU generic processors are doing very little, but memory bandwidth will be taking a fair beating.
Add in the audio coming in over I2S and the CPU needs to encode that, but that won't stress an A53. A bit of containerisation and send it out as a network stream. A single Pi will happily real-time transcode and resize the stream.

Shame we only have one I2S interface, as that means we can't bring in the HDMI audio and have the external sound input simultaneously.

I think the main stumbling point would be the UI as I'm rubbish at writing those and I haven't looked at how voctomix has implemented the raw video processing. It looks like it's almost all in GStreamer, so it then depends on exactly how the pipeline is assembled whether we can drop in a single component to do all the resizes and overlaying.
For two cameras then drop the Pi camera and use two TC358743s with HDMI out of the camera. Externally switch between the cameras. Admittedly you can't PIP the two cameras then (unless you also allow the PC feed to be switched to camera B), but that's about it.

indeed

The pi would make (and does make) good electonics of an IP camera
Debian doesn't use Pis for this because we want nice big lenses and sensors that we can run 24x7 with motor zooms etc etc. hence using semi pro camera gear

we then use SDI to take a RAW feed from the camer into the PC we mix on. That is the PC that does all the heavy work and transcoding from.

RattusRattus wrote: ↑

Mon Oct 08, 2018 1:41 pm

indeed

The pi would make (and does make) good electonics of an IP camera
Debian doesn't use Pis for this because we want nice big lenses and sensors that we can run 24x7 with motor zooms etc etc. hence using semi pro camera gear

we then use SDI to take a RAW feed from the camer into the PC we mix on. That is the PC that does all the heavy work and transcoding from.

I'm saying that I suspect that the Pi can do the heavy work of the mixing and initial encoding.

https://www.amazon.co.uk/eSynic-Convert ... B00H8JVMPS to convert SDI to HDMI to take it in via a TC358743. I can't find a direct SDI to CSI2 bridge other than an FPGA solution.

I will go for 30fps, that would be great.
For my first tests, I bought an Arducam camera module.
I will continue to investigate.

I made my first try today in a Hackaton. I had a few parts to try and lots of brilliant minds with me.
My goal was to demo the PiP with the Arducam and a desktop captured in a way or another to simulate the target (desktop shared by HDMI2Whatever, and video of the speaker inserted in the top right part of the desktop).
I used v4l2rtspserver to stream the camera with great success. The Pi was not doing much. As for the sound, we used several USB sound cards with an HF mic. The sound was synchronised with the image, which was quite a nice surprise. Unfortunately, the sound cards gave a terrible echo, as if I was talking from the grave. We had better results with the Jabra 510, but it's so pricey and heavy I would like to get rid of it for the "kit". We also tried an USB mic, which was perfect, except the cable was only 1m long...
Then the HDMI2Ethernet device did not keep its promises, so we decided to use ffmpeg to capture the desktop (PPT slides) and send it to the Pi through the network. Not so much PlugAndPlay afterall...
The Pi managed to do the PiP with the desktop stream and the camera stream. I had some troubles to use directly /dev/video0 and the desktop stream (I can't remember the error for the time being).
I made a try with v4l2compress_omx to have a compressed stream in the hope of making the PiP easier.
The tool didn't give me any error, but as a check (before doing the PiP with ffmpeg), I tried to launch v4l2rtspserver on this new video device, and never got any picture.
So we went back to the first solution, v4l2rtspserver to stream the camera, and we made the PiP on the presenter PC with ffmpeg. The resulting video was then seen as a camera by the browser, and we were then able to stream to the video to Circuit (which can then broadcast it to everybody and save it for later too).
We wanted to use the circuit live cam bot on another Pi, but we did not manage to.
For the demo at the end, we went fully wireless with a battery and Wifi and filmed the audience with the Pi while demoing. It was not that bad.
The experiment is not fully satisfactory, but we now have some proof, new ideas and hints to continue.

If I can get a little bit of budget for that, I'd love to get an audio hat, and the HDMI2CSI converter. That would mean two Raspberries, but why not... Or using that kind of adapter card to have a camera and an HDMI2CSI converter at the same time?
Would the performance degrade?
By the way, what would be the advantage of using a compute node instead of a Pi 3B+?
Thanks.