B101 Auvidea record audio + video + stream

[Florent42]

Hello

I have a B101 card with an RPI4 4GB and an updated Buster.

I try according to the different topics to make this card work.

For the moment my objective is to create a file which will contain the image and the sound.
Then the objective is to make the image and sound available in stream

With the reading of the different subjects:
- my card is functional, the sound can be recorded
- the image with the command
./yavta --capture = 1000 -n 3 --encode-to = file.h264 -f UYVY -m -T / dev / video0
is OK

But with command:
gst-launch-1.0 -vvv -e v4l2src ! video/x-raw,format=UYVY,framerate=60/1 ! v4l2h264enc ! h264parse ! queue ! matroskamux name=mux ! filesink location=foo.mkv alsasrc device=hw:1,0 ! audio/x-raw,rate=48000 ! queue ! voaacenc ! aacparse ! mux.

I have:

Code: Select all

Définition du pipeline à PAUSED...
Le pipeline est actif et n’a pas besoin de phase PREROLL…
Passage du pipeline à la phase PLAYING…
New clock: GstAudioSrcClock
/GstPipeline:pipeline0/GstAlsaSrc:alsasrc0: actual-buffer-time = 200000
/GstPipeline:pipeline0/GstAlsaSrc:alsasrc0: actual-latency-time = 10000
Redistribution de latence…
/GstPipeline:pipeline0/GstAlsaSrc:alsasrc0.GstPad:src: caps = audio/x-raw, format=(string)S16LE, layout=(string)interleaved, rate=(int)48000, channels=(int)2, channel-mask=(bitmask)0x0000000000000003
/GstPipeline:pipeline0/GstCapsFilter:capsfilter1.GstPad:src: caps = audio/x-raw, format=(string)S16LE, layout=(string)interleaved, rate=(int)48000, channels=(int)2, channel-mask=(bitmask)0x0000000000000003
/GstPipeline:pipeline0/GstQueue:queue1.GstPad:sink: caps = audio/x-raw, format=(string)S16LE, layout=(string)interleaved, rate=(int)48000, channels=(int)2, channel-mask=(bitmask)0x0000000000000003
/GstPipeline:pipeline0/GstCapsFilter:capsfilter1.GstPad:sink: caps = audio/x-raw, format=(string)S16LE, layout=(string)interleaved, rate=(int)48000, channels=(int)2, channel-mask=(bitmask)0x0000000000000003
/GstPipeline:pipeline0/GstQueue:queue1.GstPad:src: caps = audio/x-raw, format=(string)S16LE, layout=(string)interleaved, rate=(int)48000, channels=(int)2, channel-mask=(bitmask)0x0000000000000003
/GstPipeline:pipeline0/GstVoAacEnc:voaacenc0.GstPad:sink: caps = audio/x-raw, format=(string)S16LE, layout=(string)interleaved, rate=(int)48000, channels=(int)2, channel-mask=(bitmask)0x0000000000000003
/GstPipeline:pipeline0/GstVoAacEnc:voaacenc0.GstPad:src: caps = audio/mpeg, mpegversion=(int)4, channels=(int)2, rate=(int)48000, level=(string)2, base-profile=(string)lc, profile=(string)lc, stream-format=(string)raw, codec_data=(buffer)1190
/GstPipeline:pipeline0/GstAacParse:aacparse0.GstPad:src: caps = audio/mpeg, mpegversion=(int)4, channels=(int)2, rate=(int)48000, level=(string)2, base-profile=(string)lc, profile=(string)lc, stream-format=(string)raw, codec_data=(buffer)1190, framed=(boolean)true
/GstPipeline:pipeline0/GstMatroskaMux:mux.GstMatroskamuxPad:audio_0: caps = audio/mpeg, mpegversion=(int)4, channels=(int)2, rate=(int)48000, level=(string)2, base-profile=(string)lc, profile=(string)lc, stream-format=(string)raw, codec_data=(buffer)1190, framed=(boolean)true
/GstPipeline:pipeline0/GstAacParse:aacparse0.GstPad:sink: caps = audio/mpeg, mpegversion=(int)4, channels=(int)2, rate=(int)48000, level=(string)2, base-profile=(string)lc, profile=(string)lc, stream-format=(string)raw, codec_data=(buffer)1190
/GstPipeline:pipeline0/GstV4l2Src:v4l2src0.GstPad:src: caps = video/x-raw, format=(string)UYVY, framerate=(fraction)60/1, width=(int)1920, height=(int)1080, interlace-mode=(string)progressive, colorimetry=(string)bt601
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = video/x-raw, format=(string)UYVY, framerate=(fraction)60/1, width=(int)1920, height=(int)1080, interlace-mode=(string)progressive, colorimetry=(string)bt601
/GstPipeline:pipeline0/v4l2h264enc:v4l2h264enc0.GstPad:src: caps = video/x-h264, stream-format=(string)byte-stream, alignment=(string)au, profile=(string)high, level=(string)4, width=(int)1920, height=(int)1080, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)60/1, interlace-mode=(string)progressive, colorimetry=(string)bt601
/GstPipeline:pipeline0/GstH264Parse:h264parse0.GstPad:sink: caps = video/x-h264, stream-format=(string)byte-stream, alignment=(string)au, profile=(string)high, level=(string)4, width=(int)1920, height=(int)1080, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)60/1, interlace-mode=(string)progressive, colorimetry=(string)bt601
Redistribution de latence…
/GstPipeline:pipeline0/v4l2h264enc:v4l2h264enc0.GstPad:sink: caps = video/x-raw, format=(string)UYVY, framerate=(fraction)60/1, width=(int)1920, height=(int)1080, interlace-mode=(string)progressive, colorimetry=(string)bt601
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps = video/x-raw, format=(string)UYVY, framerate=(fraction)60/1, width=(int)1920, height=(int)1080, interlace-mode=(string)progressive, colorimetry=(string)bt601
ERREUR : de l’élément /GstPipeline:pipeline0/v4l2h264enc:v4l2h264enc0 : Échec du traitement de frame.
Information de débogage supplémentaire :
gstv4l2videoenc.c(803): gst_v4l2_video_enc_handle_frame (): /GstPipeline:pipeline0/v4l2h264enc:v4l2h264enc0:
Maybe be due to not enough memory or failing driver
EOS à la fermeture est activé -- attente d’EOS après erreur
Attente d’EOS.
ERREUR : de l’élément /GstPipeline:pipeline0/GstV4l2Src:v4l2src0 : Internal data stream error.
Information de débogage supplémentaire :
gstbasesrc.c(3055): gst_base_src_loop (): /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
streaming stopped, reason error (-5)

and I block it. Do you have an idea?


After is it that the following command will be able to transmit the flow and be recoverable by a VLC?
gst-launch-1.0 flvmux streamable=true name=mux ! rtmpsink location="rtmp://127.0.0.1/live" v4l2src io-mode=4 ! "video/x-raw,framerate=60/1,format=UYVY" ! v4l2h264enc output-io-mode=5 extra-controls="controls,video_bitrate=2000;" ! video/x-h264,profile=high ! h264parse ! queue ! mux. alsasrc device=hw:1,0 ! audio/x-raw,rate=48000,channels=2 ! audioconvert ! avenc_aac bitrate=128000 ! aacparse ! queue ! mux.

How can I add security type authentication and encryption?

Thanks for your help

[Florent42]

I try:

arecord -d 0 -r 48000 -c 2 -f S16_LE -t wav -D sysdefault:CARD=tc358743 | ffmpeg -f rawvideo -vcodec rawvideo -re -pix_fmt uyvy422 -s:v 1280x720 -i /dev/video0 -i pipe:0 -c:v libx264 -crf 23 -f flv -preset ultrafast -qp 0 -c:a aac test.flv


sound OK but image very very poor totally unreadable image

[Florent42]

I specify: I have this error with gstreamer if I launched before the command:
v4l2-ctl --set-dv-bt-timings query

On the other hand if I launch the command directly I have the same problem as with arecord and ffmeg: unusable image but sound OK

have you a idea?

I have a connected camcorder

thanks

[NKnuelle]

Please confirm that u followed all the steps mentioned by 6by9 in this thread (page 17):
https://www.raspberrypi.org/forums/view ... 0#p1339178

And please post the output of

Code: Select all

v4l2-ctl --log-status

as well as your kernel and firmware:

Code: Select all

uname -a

and

Code: Select all

vcgencmd version

Thanks!

[Florent42]

thanks for you answer


my kernel is 4.19.113, the last kernel.

Code: Select all

pi@raspberrypi:~ $ v4l2-ctl --log-status

Status Log:

   [ 1060.238212] unicam fe801000.csi: =================  START STATUS  =================
   [ 1060.239892] tc358743 0-000f: -----Chip status-----
   [ 1060.240515] tc358743 0-000f: Chip ID: 0x00
   [ 1060.241135] tc358743 0-000f: Chip revision: 0x00
   [ 1060.241147] tc358743 0-000f: Reset: IR: 1, CEC: 1, CSI TX: 0, HDMI: 0
   [ 1060.241157] tc358743 0-000f: Sleep mode: off
   [ 1060.241166] tc358743 0-000f: Cable detected (+5V power): yes
   [ 1060.241695] tc358743 0-000f: DDC lines enabled: yes
   [ 1060.242224] tc358743 0-000f: Hotplug enabled: yes
   [ 1060.242845] tc358743 0-000f: CEC enabled: no
   [ 1060.242854] tc358743 0-000f: -----Signal status-----
   [ 1060.242863] tc358743 0-000f: TMDS signal detected: yes
   [ 1060.242872] tc358743 0-000f: Stable sync signal: yes
   [ 1060.242881] tc358743 0-000f: PHY PLL locked: yes
   [ 1060.242890] tc358743 0-000f: PHY DE detected: yes
   [ 1060.250971] tc358743 0-000f: Detected format: 1920x1080p50.0 (2640x1125)
   [ 1060.250983] tc358743 0-000f: horizontal: fp = 0, -sync = 720, bp = 0
   [ 1060.250994] tc358743 0-000f: vertical: fp = 0, -sync = 45, bp = 0
   [ 1060.251003] tc358743 0-000f: pixelclock: 148500000
   [ 1060.251015] tc358743 0-000f: flags (0x0):
   [ 1060.251025] tc358743 0-000f: standards (0x0):
   [ 1060.251038] tc358743 0-000f: Configured format: 1920x1080p50.0 (2640x1125)
   [ 1060.251048] tc358743 0-000f: horizontal: fp = 0, -sync = 720, bp = 0
   [ 1060.251058] tc358743 0-000f: vertical: fp = 0, -sync = 45, bp = 0
   [ 1060.251068] tc358743 0-000f: pixelclock: 148500000
   [ 1060.251078] tc358743 0-000f: flags (0x0):
   [ 1060.251088] tc358743 0-000f: standards (0x0):
   [ 1060.251097] tc358743 0-000f: -----CSI-TX status-----
   [ 1060.251107] tc358743 0-000f: Lanes needed: 2
   [ 1060.251116] tc358743 0-000f: Lanes in use: 2
   [ 1060.251736] tc358743 0-000f: Waiting for particular sync signal: no
   [ 1060.252357] tc358743 0-000f: Transmit mode: no
   [ 1060.252977] tc358743 0-000f: Receive mode: no
   [ 1060.253596] tc358743 0-000f: Stopped: no
   [ 1060.253605] tc358743 0-000f: Color space: YCbCr 422 16-bit
   [ 1060.254136] tc358743 0-000f: -----HDMI status-----
   [ 1060.254145] tc358743 0-000f: HDCP encrypted content: no
   [ 1060.254155] tc358743 0-000f: Input color space: RGB limited range
   [ 1060.254684] tc358743 0-000f: AV Mute: off
   [ 1060.255215] tc358743 0-000f: Deep color mode: 8-bits per channel
   [ 1060.257718] tc358743 0-000f: HDMI infoframe: Auxiliary Video Information (AVI), version 2, length 13
   [ 1060.257730] tc358743 0-000f:     colorspace: RGB
   [ 1060.257741] tc358743 0-000f:     scan mode: Overscan
   [ 1060.257752] tc358743 0-000f:     colorimetry: ITU709
   [ 1060.257763] tc358743 0-000f:     picture aspect: 16:9
   [ 1060.257774] tc358743 0-000f:     active aspect: Same as Picture
   [ 1060.257784] tc358743 0-000f:     itc: No Data
   [ 1060.257796] tc358743 0-000f:     extended colorimetry: xvYCC 601
   [ 1060.257807] tc358743 0-000f:     quantization range: Default
   [ 1060.257818] tc358743 0-000f:     nups: Unknown Non-uniform Scaling
   [ 1060.257829] tc358743 0-000f:     video code: 31
   [ 1060.257839] tc358743 0-000f:     ycc quantization range: Limited
   [ 1060.257887] tc358743 0-000f:     hdmi content type: Graphics
   [ 1060.257898] tc358743 0-000f:     pixel repeat: 0
   [ 1060.257911] tc358743 0-000f:     bar top 0, bottom 0, left 0, right 0
   [ 1060.257921] unicam fe801000.csi: -----Receiver status-----
   [ 1060.257936] unicam fe801000.csi: V4L2 width/height:   1920x1080
   [ 1060.257949] unicam fe801000.csi: Mediabus format:     0000200f
   [ 1060.257962] unicam fe801000.csi: V4L2 format:         UYVY
   [ 1060.257974] unicam fe801000.csi: Unpacking/packing:   0 / 0
   [ 1060.257987] unicam fe801000.csi: ----Live data----
   [ 1060.257999] unicam fe801000.csi: Programmed stride:   3840
   [ 1060.258012] unicam fe801000.csi: Detected resolution: 0x0
   [ 1060.258024] unicam fe801000.csi: Write pointer:       e1f00000
   [ 1060.258035] unicam fe801000.csi: ==================  END STATUS  =========

but I think I found the problem. In fact I think it was just 60 frames per second in the command

I try:
gst-launch-1.0 -vvv -e v4l2src ! video/x-raw,format=UYVY,framerate=30/1 ! v4l2h264enc ! h264parse ! queue ! matroskamux name=mux ! filesink location=foo.mkv alsasrc device=hw:1,0 ! audio/x-raw,rate=48000 ! queue ! voaacenc ! aacparse ! mux.

and it works the file seems ok


so hopefully i come to my second question: how to make the video and sound stream accessible via VLC (secure) or via a web page?

thanks

[NKnuelle]

I am streaming using the module via Gstreamer to my local Nginx Server with rtmp enabled via this pipeline:

Code: Select all

GST_DEBUG=2 gst-launch-1.0 flvmux streamable=true name=mux ! rtmpsink location="rtmp://localhost:1935/live/stream" v4l2src io-mode=4 ! "video/x-raw,framerate=50/1,format=UYVY" ! v4l2h264enc output-io-mode=5  extra-controls="controls,video_bitrate=6000000;" ! video/x-h264,profile=high ! h264parse ! queue ! mux. alsasrc device=hw:1 ! audio/x-raw,rate=48000,channels=2 ! audioconvert ! voaacenc bitrate=48000 ! aacparse ! queue ! mux.

At the moment I am experiencing problems in long running streams (stream suddenly stops with a critical error). You can then use VLC to play that network stream.

How to configure your own web server or RTMP Service to be secure is up to you (blocking IP-Adresses etc.). However - As far as I know a network stream (e.g. via RTMP) could only be hided (using a secure URL) but is always accessible.

[6by9]

The specified framerate needs to match that which you are providing to the B101, ie in your case

Code: Select all

[ 1060.250971] tc358743 0-000f: Detected format: 1920x1080p50.0 (2640x1125)

50fps.

Note that you may need to increase the H264 level to allow encoding above 1080p30. Memory says the default is 4.0 which is limited at that, whilst level 4.2 should allow up to 1080p60.

[Florent42]

thank you for your help


@NKnuelle
in my I use an apache
you know if we can use rtmp in apache?

But otherwise yes in principle launching the rtmp in the web page and consulting it via the browser in SSL for example would be perfect.
If VLC also works it would be ideal

Can you tell me what you have configured at nginx level regarding the port: rtmp://localhost:1935/live/stream ?

@6by9
big thank you for all the topics where you gave informations

strangely if I configure 50fps it does not work
Where do we increase the H264?

[6by9]

strangely if I configure 50fps it does not work
Where do we increase the H264?

v4l2h264enc output-io-mode=5 extra-controls="controls,h264_level=13;"
("v4l2-ctl -d 11 --list-ctrls-menu" will list all menu options for enumerated/menu types)

[electronx]

dear 6by9, I have followed your explanations along with the posts, but I think I need further assistance. My name is Cristian and I have started last year a raspberry pi streaming project, and now I need help to finish it since the first order came already from the client.

I have searched for your contact, and since I was not able to find it, I am writing to you here to kindly ask you if you could teach me how to deal with HDMI audio-video input in a paid lesson, over a video call software like zoom, and anydesk, etc.

I would really appreciate it if you could consider my offer, I think IMHO that 2 hours of your time might be enough, this of course if you are not interested to go along with me onto a Kickstarter project, or a long term collaboration in this sense.

I have a raspberry pi 4, 4GB, B101 which does work on video, but I do not know how to get the sound out of it and stream it further.
Monday I will receive also a 4G modem to use GSM internet, in order to make the device portable.

Here is my email in case you can assist me on this project, as I mentioned, paid assistance, I don't want to get for free what u have spent dozen of time on it: electronrecord@gmail.com

Still, in case you are too busy and unable to help me by accepting my offer I totally understand that,
please briefly write the logical steps in order to achieve the getting the sound from B101 (what pins from the board to connect on what pins on the GPIO), and what you would suggest to use in order to sum the audio and video from the B101 and stream to a custom rtmp server.

warm greetings,
Cristian

[electronx]

--

[rpdom]

dear 6by9, my name is Cristian and I am developing a streaming project based on Rasp Pi 4, from about 1 year. I am now stuck and I already received an order from a client. So, please consider my offer to pay you to teach me how to sum the video from HDMI B101 and audio into a stream and send it to a custom rtmp server. I guess it will not take to long, I am a fast learner.

Also, if u will find the project attractive, maybe you will be interested to join in, as a supervisor, I would love to do the work and have a specialist guiding me.

In any case, if you are really tight with your time, please just briefly describe the steps (1,2,3...) to achieve my need.

many thanks for your knowledge sharing,
Cristian

Did you not read 6x9's signature block?

I'm not interested in doing contracts for bespoke functionality - please don't ask.

[electronx]

oh, I apologize, I have not seen his signature although I have checked his profile.

Thanks for the quick reply, I will start again reading the posts.

[6by9]

viewtopic.php?f=38&t=120702&p=1339178#p1339178 for configuration and wiring.

You should get a standard ALSA capture device. You can not change the sample frequency - whatever is reported via V4L2 as control "audio_sampling_rate" is the sampling rate, and your capture needs to ask for that rate. Any resampling needs to be done in software afterwards.

[electronx]

aha, so, 6by9, you are saying that there is no way to get the digital I2S audio stream from B101 module and send it to the Rpi GPIO and then mix it with the video from HDMI and then send it to a rtmp URL link?

You say that I need a digital capture device (ALSA) to collect the digital stream from B101 and then use it further to make the Audio-Video stream?

PS: I have a Fe-Pi soundcard which works great as analog audio input but it does not have digital I2S in...

many many thanks for your answers.

cheers,
Cristian

[6by9]

aha, so, 6by9, you are saying that there is no way to get the digital I2S audio stream from B101 module and send it to the Rpi GPIO and then mix it with the video from HDMI and then send it to a rtmp URL link?

You say that I need a digital capture device (ALSA) to collect the digital stream from B101 and then use it further to make the Audio-Video stream?

PS: I have a Fe-Pi soundcard which works great as analog audio input but it does not have digital I2S in...

You don't mix audio and video, you multiplex them (so that they can be extracted as independent streams again). That's a job for ffmpeg or GStreamer.

The Pi has I2S in, which your Fe-Pi soundcard will probably be currently connected to - you can't have both connected simultaneously.

From the B101 you wire out the 8 pin B101 connector to the Pi 40 pin GPIO header

LRCK/WFS to GPIO 19, BCK/SCK to GPIO 18, and DATA/SD to GPIO 20. (those are 7, 6 and 5 respectively on the B101/B102 boards).

Add "dtoverlay=tc358743-audio" to /boot/config.txt, and it will configure the I2S peripheral appropriately for the TC358743. You now have a suitably configured ALSA driver for receiving the audio.

[electronx]

Hi 6by9, I admire for your patience (I was reading all the posts related to B101 in my pursuit in trying to record an audio-video file from B101 HDMI in and I2S out)

So, after few days of "hunting" answers, I am a bit stuck by too many trials and error, and I am not sure I am following the right path

I am using raspbian 10 buster and Rpi 4 B+, kernel 4.19

Shortly, what I did:
- wired the I2S audio from B101 to GPIO as u detailed and also as I have found on other forums
- added dtoverlay=tc358743-audio to /boot/config.txt
- changed this param > dtparam=audio=off

When running raspivit -t 0 I get the video from my iPhone which spits out HDMI audio-video

Installed and tested yavta and I get the video recorded fine.

But, after typing "raspivit -t 0" and starting video onscreen, I have tried recording audio with
sudo arecord -Dhw:0,0 -d 10 -f cd -t wav test.wav

and I get
arecord: pcm_read:2031: read error: Input/output error

When typing alsamixer I get the tctc358743 just fine, of course, without any fader to set the input volume.

Btw, is there any way to have also main soundcard enabled while tc358743 IN, and listen the HDMI audio input on the audio jack?

So, my goal is: RECORD AUDIO to make sure the audio IN works

I have the feeling that I missed some steps (something with EDID), and also some extra config in /boot/config.txt

Some hints would be greatly appreciated, just in very short the steps to follow in order to capture and record audio.
Then I will move on to multiplex

many thanks for your time!

cheers,
Cristian

[6by9]

At the point anyone writes that they are still using raspivid with the tc358743, I walk away. IT IS UNSUPPORTED.

You can not mix and match the firmware drivers through raspivid with the audio drivers through the kernel.
viewtopic.php?f=38&t=120702&p=1339178#p1339178 lays out relatively step by step how to configure the chip using the kernel drivers.

viewtopic.php?f=38&t=120702&p=1394316#p1394316 gives a GStreamer pipeline that should record audio and video into an MKV container. I'm sure there have been updated pipelines posted which use v4l2h264enc instead of omxh264enc, and that would be preferable.

[electronx]

Great, got it, I will follow your leads.

as usual, many thanks for the clarifications.


cheers,
Cristian

[electronx]

Hi Florent42 and NKnuelle, do u think you could assist me a bit in finding the right way of streaming audio-video from B101 to a rtmp server?

Have u succeded in doing that?

thanks,
Cristian