6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5815
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

TC358743 and interlaced HDMI modes

Wed Jul 25, 2018 10:25 am

Branched off luiscgalo's discussion on viewtopic.php?f=43&t=109137&start=450#p1331592

Basic synopsis is trying to receive 1080i50 from a source as RGB, line interlace in the app, use the ISP to convert to YUV, and then image_fx to deinterlace and feed to the encoder.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5815
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: TC358743 and interlaced HDMI modes

Wed Jul 25, 2018 10:31 am

Copying the last posts
luiscgalo wrote:Hi all,
After some days of inactivity on my project, I finally had some time to continue investigating the issue with the deinterlacer.

I've performed multiple tests and I think that the problem is indeed related with the deinterlacer, misuse and/or incorrect configuration on my side...

Basically, if I set the "img_fx_param.effect_parameter[2] = 1" to half the output framerate (25fps instead of 50fps), the output video is nice and smooth, but at 25fps of course.
When I try to output 50fps, the output is almost ok... It seems that the deinterlacer/encoder chain struggles some times and cyclically the video is not smooth.

Better than explaining my problem using words is to provide you some example videos captured by my application.
I've created an WeTransfer link where you are able to download two H264 files, one for the 25fps test (nice and smooth) and a second one for the 50fps test (where you can see the issue that I'm describing)
https://we.tl/wArhB50GGs

The source code that I'm using to capture the files is the following (my latest version, excluding the rawcam and TC358743 parts since they are already working fine)

Code: Select all

#include "converter.h"

#include <stdio.h>
#include "interface/mmal/mmal.h"
#include "interface/mmal/mmal_buffer.h"
#include "interface/mmal/util/mmal_util.h"
#include "interface/mmal/util/mmal_util_params.h"
#include "interface/mmal/util/mmal_connection.h"
#include "interface/mmal/util/mmal_default_components.h"

/**
 * Flow of image processing: RawFrame -> ISP -> ImageFx -> Encoder
 */

// ISP variables (converter from BGR24 to I420)
MMAL_COMPONENT_T *pISP = NULL;
MMAL_PORT_T *isp_input = NULL;
MMAL_PORT_T *isp_output = NULL;
MMAL_POOL_T *isp_pool_in = NULL;

// Image Fx variables (deinterlacer)
MMAL_COMPONENT_T *pImageFx = NULL;
MMAL_PORT_T *deint_input = NULL;
MMAL_PORT_T *deint_output = NULL;

// H264 Encoder variables
MMAL_COMPONENT_T *pEncoder = NULL;
MMAL_PORT_T *enc_input = NULL;
MMAL_PORT_T *enc_output = NULL;
MMAL_POOL_T *enc_pool_out = NULL;

// Connections between components
MMAL_CONNECTION_T* conn_isp_deint = NULL;
MMAL_CONNECTION_T* conn_deint_enc = NULL;

// hard-coded constants for test purposes only
#define VIDEO_WIDTH 	1920
#define VIDEO_HEIGHT	1080

int64_t pts = 0;
FILE * pFile2 = NULL;

/**
 * Function called when a new raw (BGR24) interlaced frame is ready to be converted
 */
void ConvertFrame(uint8_t* punBuffer, const uint32_t unBufferSize) {
	MMAL_BUFFER_HEADER_T *buffer;

	// send data to the input port of ISP component
	buffer = mmal_queue_get(isp_pool_in->queue);
	if (buffer != NULL) {
		mmal_buffer_header_mem_lock(buffer);
		buffer->flags = MMAL_BUFFER_HEADER_VIDEO_FLAG_INTERLACED | MMAL_BUFFER_HEADER_VIDEO_FLAG_TOP_FIELD_FIRST;
		buffer->length = unBufferSize;

		//buffer->pts = buffer->dts = MMAL_TIME_UNKNOWN;
		buffer->pts = buffer->dts = pts;
		pts += 1000000 / 50;

		memcpy(buffer->data, punBuffer, unBufferSize);
		mmal_buffer_header_mem_unlock(buffer);

		//printf("BGR24 frame -> ISP input (%d bytes)\n", buffer->length);
		if (mmal_port_send_buffer(isp_input, buffer) != MMAL_SUCCESS) {
			printf("Error sending data to ISP input!\n");
		}
	} else {
		printf("\t\tISP input buffer not available!\n");
	}
}

/*
 * ISP input port callback function
 */
void isp_input_cb(MMAL_PORT_T *port, MMAL_BUFFER_HEADER_T *buffer) {
	mmal_buffer_header_release(buffer);
}

/*
 * H264 encoder output port callback function
 */
void enc_output_callback(MMAL_PORT_T *port, MMAL_BUFFER_HEADER_T *buffer) {
	printf("Encoder out data (%d bytes)!\n", buffer->length);

	if (pFile2 == NULL) {
		pFile2 = fopen("capture.h264", "wb");
	}

	// These are the header bytes, save them for final output
	mmal_buffer_header_mem_lock(buffer);
	fwrite(buffer->data, 1, buffer->length, pFile2);
	mmal_buffer_header_mem_unlock(buffer);

	// release buffer back to the pool
	mmal_buffer_header_release(buffer);

	// and send one back to the port (if still open)
	if (port->is_enabled) {
		MMAL_STATUS_T status;
		MMAL_BUFFER_HEADER_T* new_buffer = mmal_queue_get(enc_pool_out->queue);

		if (new_buffer)
			status = mmal_port_send_buffer(port, new_buffer);

		if (!new_buffer || status != MMAL_SUCCESS)
			printf("Unable to return a buffer to the encoder port!\n");
	}
}

/*
 * Converter/encoder initialization function
 */
void InitConverter() {
	MMAL_STATUS_T status;

	// ###################################### ISP converter ##################################################

	status = mmal_component_create("vc.ril.isp", &pISP);
	if (status != MMAL_SUCCESS) {
		printf("Failed to create ISP!\n");
		return;
	}

	isp_input = pISP->input[0];
	isp_output = pISP->output[0];

	isp_input->format->type = MMAL_ES_TYPE_VIDEO;
	isp_input->format->encoding = MMAL_ENCODING_BGR24;
	isp_input->format->es->video.width = VCOS_ALIGN_UP(VIDEO_WIDTH, 32);
	isp_input->format->es->video.height = VCOS_ALIGN_UP(VIDEO_HEIGHT, 16);
	isp_input->format->es->video.crop.x = 0;
	isp_input->format->es->video.crop.y = 0;
	isp_input->format->es->video.crop.width = VIDEO_WIDTH;
	isp_input->format->es->video.crop.height = VIDEO_HEIGHT;
	isp_input->format->es->video.frame_rate.num = 0;
	isp_input->format->es->video.frame_rate.den = 1;
	status = mmal_port_format_commit(isp_input);
	if (status != MMAL_SUCCESS) {
		printf("Failed to commit converter input format!\n");
		return;
	}

	isp_input->buffer_size = isp_input->buffer_size_recommended;
	isp_input->buffer_num = isp_input->buffer_num_recommended;
	//printf("ISP input buffer size %i bytes\n", isp_input->buffer_size);

	// create pool for input data
	isp_pool_in = mmal_port_pool_create(isp_input, isp_input->buffer_num, isp_input->buffer_size);
	if (isp_pool_in == NULL) {
		printf("Failed to create ISP input pool!\n");
	}

	// Setup ISP output (copy of input format, changing only the encoding)
	mmal_format_copy(isp_output->format, isp_input->format);
	isp_output->format->encoding = MMAL_ENCODING_I420;
	status = mmal_port_format_commit(isp_output);
	if (status != MMAL_SUCCESS) {
		printf("Failed to commit converter output format!\n");
		return;
	}

	isp_output->buffer_size = isp_output->buffer_size_recommended;
	isp_output->buffer_num = isp_output->buffer_num_recommended;
	//printf("ISP output buffer size %i bytes\n", isp_output->buffer_size);

	printf("ISP buffers In=%i | Out=%i\n", isp_input->buffer_num, isp_output->buffer_num);

	// Enable ports and ISP component
	status = mmal_port_enable(isp_input, isp_input_cb);
	if (status != MMAL_SUCCESS) {
		printf("Error enabling ISP input port!\n");
		return;
	}

	// ################################## deinterlacer ######################################################

	status = mmal_component_create("vc.ril.image_fx", &pImageFx);
	if (status != MMAL_SUCCESS) {
		printf("Failed to create image_fx!\n");
		return;
	}

	deint_input = pImageFx->input[0];
	deint_output = pImageFx->output[0];

	MMAL_PARAMETER_IMAGEFX_PARAMETERS_T img_fx_param;
	memset(&img_fx_param, 0, sizeof(MMAL_PARAMETER_IMAGEFX_PARAMETERS_T));
	img_fx_param.hdr.id = MMAL_PARAMETER_IMAGE_EFFECT_PARAMETERS;
	img_fx_param.hdr.size = sizeof(MMAL_PARAMETER_IMAGEFX_PARAMETERS_T);
	img_fx_param.effect = MMAL_PARAM_IMAGEFX_DEINTERLACE_FAST;
	img_fx_param.num_effect_params = 3;
	img_fx_param.effect_parameter[0] = 3; // interlaced input frame with both fields / top field first
	img_fx_param.effect_parameter[1] = 0; // frame period (1000000 * 1 / 25);
	img_fx_param.effect_parameter[2] = 1; // half framerate ?
	if (mmal_port_parameter_set(deint_output, &img_fx_param.hdr) != MMAL_SUCCESS) {
		printf("Failed to configure deinterlacer output port (mode)\n");
		return;
	}

	// Setup image_fx input format (equal to ISP output)
	mmal_format_copy(deint_input->format, isp_output->format);

	status = mmal_port_format_commit(deint_input);
	if (status != MMAL_SUCCESS) {
		printf("Failed to commit image_fx input format!\n");
		return;
	}

	deint_input->buffer_size = deint_input->buffer_size_recommended;
	deint_input->buffer_num = deint_input->buffer_num_recommended;

	// Setup image_fx output format (equal to input format)
	mmal_format_copy(deint_output->format, deint_input->format);

	status = mmal_port_format_commit(deint_output);
	if (status != MMAL_SUCCESS) {
		printf("Failed to commit image_fx output format!\n");
		return;
	}

	deint_output->buffer_size = deint_output->buffer_size_recommended;
	deint_output->buffer_num = deint_output->buffer_num_recommended;

	printf("Create connection ISP output to image_fx input...\n");
	status = mmal_connection_create(&conn_isp_deint, isp_output, deint_input,
	MMAL_CONNECTION_FLAG_TUNNELLING);
	if (status != MMAL_SUCCESS) {
		printf("Failed to create connection status %d: ISP->image_fx\n", status);
		return;
	}

	status = mmal_connection_enable(conn_isp_deint);
	if (status != MMAL_SUCCESS) {
		printf("Failed to enable connection ISP->image_fx\n");
		return;
	}

	// ################################## encoder ######################################################

	// create H264 video encoder component
	status = mmal_component_create(MMAL_COMPONENT_DEFAULT_VIDEO_ENCODER, &pEncoder);
	if (status != MMAL_SUCCESS) {
		printf("Unable to create video encoder component!\n");
		return;
	}

	enc_input = pEncoder->input[0];
	enc_output = pEncoder->output[0];

	enc_input->format->es->video.frame_rate.num = 0;
	enc_input->format->es->video.frame_rate.den = 1;
	enc_input->buffer_size = enc_input->buffer_size_recommended;
	enc_input->buffer_num = enc_input->buffer_num_recommended;

	printf("Create connection ISP output to image_fx input...\n");
	status = mmal_connection_create(&conn_deint_enc, deint_output, enc_input, MMAL_CONNECTION_FLAG_TUNNELLING);
	if (status != MMAL_SUCCESS) {
		printf("Failed to create connection status %d: Deint->encoder\n", status);
		return;
	}

	mmal_format_copy(enc_output->format, enc_input->format);
	enc_output->format->encoding = MMAL_ENCODING_H264;
	enc_output->format->bitrate = 1000 * 1000 * 4; // 4 Mbps output video
	enc_output->format->es->video.frame_rate.num = 0;
	enc_output->format->es->video.frame_rate.den = 1;
	status = mmal_port_format_commit(enc_output);
	if (status != MMAL_SUCCESS) {
		printf("Video encoder output format couldn't be set!\n");
		return;
	}

	enc_output->buffer_size = enc_output->buffer_size_recommended;
	enc_output->buffer_num = enc_output->buffer_num_recommended;

	printf("H264 encoder buffers In=%i | Out=%i\n", enc_input->buffer_num, enc_output->buffer_num);

	// set H264 profile and level
	MMAL_PARAMETER_VIDEO_PROFILE_T param2;
	param2.hdr.id = MMAL_PARAMETER_PROFILE;
	param2.hdr.size = sizeof(param2);
	param2.profile[0].profile = MMAL_VIDEO_PROFILE_H264_HIGH;
	param2.profile[0].level = MMAL_VIDEO_LEVEL_H264_4;
	status = mmal_port_parameter_set(enc_output, &param2.hdr);
	if (status != MMAL_SUCCESS) {
		printf("Unable to set H264 profile!\n");
		return;
	}

	// Set keyframe interval
	MMAL_PARAMETER_UINT32_T param = { { MMAL_PARAMETER_INTRAPERIOD, sizeof(param) }, 500 };
	status = mmal_port_parameter_set(enc_output, &param.hdr);
	if (status != MMAL_SUCCESS) {
		printf("Unable to set intraperiod!\n");
		return;
	}

	enc_pool_out = mmal_port_pool_create(enc_output, enc_output->buffer_num, enc_output->buffer_size);
	if (enc_pool_out == NULL) {
		printf("Failed to create encoder output pool!\n");
	}

	// Enable output port
	status = mmal_port_enable(enc_output, enc_output_callback);
	if (status != MMAL_SUCCESS) {
		printf("Error enabling deinterlacer output port!\n");
		return;
	}

	status = mmal_connection_enable(conn_deint_enc);
	if (status != MMAL_SUCCESS) {
		printf("Failed to enable connection Deint->encoder\n");
		return;
	}

	// ###################################### generic initialization ##################################################

	status = mmal_component_enable(pISP);
	if (status != MMAL_SUCCESS) {
		printf("Error enabling ISP!\n");
		return;
	}

	status = mmal_component_enable(pImageFx);
	if (status != MMAL_SUCCESS) {
		printf("Error enabling deinterlacer!\n");
		return;
	}

	status = mmal_component_enable(pEncoder);
	if (status != MMAL_SUCCESS) {
		printf("Error enabling encoder!\n");
		return;
	}

	// Send buffers for output pool
	for (uint8_t i = 0; i < enc_output->buffer_num; i++) {
		MMAL_BUFFER_HEADER_T *buffer = mmal_queue_get(enc_pool_out->queue);

		if (!buffer) {
			printf("Buffer is NULL!\n");
			exit(1);
		}
		status = mmal_port_send_buffer(enc_output, buffer);
		if (status != MMAL_SUCCESS) {
			printf("mmal_port_send_buffer failed deint on buffer %p, status %d\n", buffer, status);
			exit(1);
		}
	}

	printf("Image converter init OK!\n");
}

/*
 * Converter/encoder deinitialization function
 */
void CloseConverter() {
	MMAL_STATUS_T status;

	printf("Closing converter...\n");

	if (pFile2 != NULL) {
		fclose(pFile2);
	}

	// disable and destroy connection between ISP and deinterlacer
	mmal_connection_disable(conn_isp_deint);
	mmal_connection_destroy(conn_isp_deint);

	mmal_connection_disable(conn_deint_enc);
	mmal_connection_destroy(conn_deint_enc);

	// disable ports
	mmal_port_disable(isp_input);
	mmal_port_disable(enc_output);

	// disable ISP
	status = mmal_component_disable(pISP);
	if (status != MMAL_SUCCESS) {
		printf("Failed to disable ISP component!\n");
	}

	// disable deinterlacer
	status = mmal_component_disable(pImageFx);
	if (status != MMAL_SUCCESS) {
		printf("Failed to disable ISP component!\n");
	}

	// disable encoder
	status = mmal_component_disable(pEncoder);
	if (status != MMAL_SUCCESS) {
		printf("Failed to disable Encoder component!\n");
	}

	// destroy components
	mmal_component_destroy(pISP);
	mmal_component_destroy(pImageFx);
	mmal_component_destroy(pEncoder);
}

Maybe I'm missing any detail or not setting properly some parameters on the processing chain...
I didn't understood yet how I can solve this issue in order to achieve smooth 50fps...
Do you see anything wrong in my code 6by9?
Thanks again for your help.
luiscgalo wrote:One idea that comes to my mind (I don't know if it works) which may solve (or not!) the problem could be changing the way of using the image_fx deinterlacer component.

Currently I'm setting "img_fx_param.effect_parameter[0] = 3" which means that the image_fx is expecting the reception of a full HD interlaced input frame containing both top and bottom fields.

However, according to the info from a previous post, there are more options available for this parameter:

6by9 wrote:
All algorithms support a first int is an override for the buffer flags.
0 - The data is not interlaced, it is progressive scan
1 - The data is interlaced, fields sent separately in temporal order, with upper field first
2 - The data is interlaced, fields sent separately in temporal order, with lower field first
3 - The data is interlaced, two fields sent together line interleaved, with the upper field temporally earlier
4 - The data is interlaced, two fields sent together line interleaved, with the lower field temporally earlier
5 - The stream may contain a mixture of progressive and interlaced frames (all bets are off).

Since I'm already receiving separated top and bottom fields from the rawcam component, what if I set "img_fx_param.effect_parameter[0] = 1" in order to provide individual fields to the deinterlacer?
Using this approach I will be directly sending received fields to the ISP/image_fx blocks instead of having to merge each top/bottom field into a FullHD interlaced frame.
Another difference will be the fact that I will be feeding the deinterlacer with the final rate of 50fps (field rate) instead of sending 25fps interlaced that are latter on converted to 50fps.
Do you think that this could work?

The problem is that I cannot see a single example about how to use the image_fx deinterlacer with the "img_fx_param.effect_parameter[0]" parameter set to anything different from 3 (from my research, all available source code examples on the Internet are sending the two fields together).
In this scenario, what will be the input and output format of the image_fx block?
Will be the input resolution set to field resolution (1920x540)?
Will be the output resolution set to the target FullHD size (1920x1080)?
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

luiscgalo
Posts: 29
Joined: Fri Jun 22, 2018 9:09 am

Re: TC358743 and interlaced HDMI modes

Wed Jul 25, 2018 11:44 am

Hi 6by9,
Yes, thanks for creating a new thread! ;)
It is the best option otherwise multiple and different topics were being discussed in the same thread, at the same time...

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5815
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: TC358743 and interlaced HDMI modes

Wed Jul 25, 2018 12:05 pm

I don't have that much experience of the deinterlacing support.

I wonder if you're just hitting the limits of the hardware.
If you use "sudo vcdbg hist gnuplot" whilst your use case is running it'll create two gnuplot files of the loading on the VPU. "gnuplot task.gpt" will render those files ("sudo apt install gnuplot" first if necessary). There are 3 graphs - Core1/VPU0, Core2/VPU1, and a combined graph. If either of the first two are maxed out then you're a little stuck. Post the files to Dropbox or similar if you want me to have a look at them.
I'm assuming you are using the QPU deinterlace, as I'm fairly certain that is required for 1080i.

Looking at the code rather than the comments, modes 1 and 2 appear to not be handled. The line doubler algorithm only takes values 3 or 4, whilst the other two accept 0, 3, 4, or 5.
It also drops back to the puzzler I had before of how to deinterlace two fields that are vertically chroma-subsampled (ie YUV420).
So great idea, but sadly I don't think it flies.

I am mulling over deinterlace support for YUV422, and may be able to handle independent frames in. I'm thinking of the composite video to CSI2 chips which tend to spit out one of the 4 flavours of YUYV, but it might be an option for you too. It's a lowish priority though.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

luiscgalo
Posts: 29
Joined: Fri Jun 22, 2018 9:09 am

Re: TC358743 and interlaced HDMI modes

Wed Jul 25, 2018 12:21 pm

Thanks for the tip.
Tonight I will run the "sudo vcdbg hist gnuplot" command to trace the usage of the VPU. (I didn't know that one ;) )
6by9 wrote:I'm assuming you are using the QPU deinterlace, as I'm fairly certain that is required for 1080i.
Well, actually I'm not using the QPUs because when I've performed a test to enabling them, the "mmal_port_send_buffer(isp_input, buffer)" operation was always unsuccessful for some unknown reason (unable to send image data to the ISP input)...

For that test I've changed the following source code lines:

Code: Select all

...
img_fx_param.effect = MMAL_PARAM_IMAGEFX_DEINTERLACE_ADV;
img_fx_param.num_effect_params = 4;
img_fx_param.effect_parameter[0] = 3; // interlaced input frame with both fields / top field first
img_fx_param.effect_parameter[1] = 0; // default frame period
img_fx_param.effect_parameter[2] = 0; // half framerate ?
img_fx_param.effect_parameter[3] = 1; // enable QPU ?
...
You say that is better to enable the QPUs, but for which type of deinterlacing mode ("advanced" or "fast")?
There is any significant advantage in which regards to performance by enabling QPUs? (sorry if this is a very basic question)

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5815
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: TC358743 and interlaced HDMI modes

Wed Jul 25, 2018 1:47 pm

Within VideoCore you have multiple blocks, many are dedicated to particular tasks (eg H264 CABAC, JPEG encode/decode, video composition, etc), with the VPUs (Vector Processing Unit) being the general purpose processor. There are 2 cores, and they support a set of 16 way vector operations.
WIthin the 3D block there are the Quad Processing Units (QPUs). Whilst optimised for 3D operations they are still fairly general purpose. I'll cheat and quote Wikipedia - https://en.wikipedia.org/wiki/VideoCore#3D_engine
"Each processor has two vector floating-point ALUs which carry out multiply and non-multiply operations in parallel with single instruction cycle latency. Internally the QPU is a 4-way SIMD processor multiplexed 4× over four cycles, making it particularly suited to processing streams of quads of pixels," according to Broadcom's VideoCore® IV 3D Architecture Reference Guide. [2]

"QPUs are organized into groups of up to four, termed slices, which share certain common resources," the document continues. cf. Vertex and shader.
IIRC You've got 3 slices on BCM283x, so 48 ways of SIMD, and independent of most other processing. In theory that's 3 times the processing of the VPU (deinterlacing only ever runs on a single core as parts of the vector engine are shared between the 2 VPUs).

I've checked with Dom. di_adv is the only one to support the QPUs. It offloads the whole process to the QPUs, and is the only approach that can run 1080i50.
If you've hit problems with the QPU mode then not submitting to the ISP, please post a test case and I can give a 10 minute debug. I appreciate it can be a faff producing test cases, but reality is that support is not my primary job, so it has to be something I can do in a reasonable period of time. I have the hardware here so should be able to test out your situation (although using a Pi as the 1080i source).
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

luiscgalo
Posts: 29
Joined: Fri Jun 22, 2018 9:09 am

Re: TC358743 and interlaced HDMI modes

Wed Jul 25, 2018 10:15 pm

Hi again 6by9,
I've tried to follow your recommendation of using the ADV deinterlacer with QPUs.
I tested multiple combinations, always without success (unable to send image data to the ISP input).

Since you have a similar hardware setup, maybe it's easier if you try to compile and run my app on your RPi.
I've attached my working sources to this post (it's a CMake project).
It should be easy to run my app: you just need to compile and run the executable :)
When you run the application, an H264 elementary file will be saved with the deinterlaced video (producing video files identical to what I've shared in a previous post) until interrupting the application.

In order to test the deinterlacer options, please take a look to the "converter.c" file.
Between lines 184 and 210 you will find some blocks of commented code, representing my tests with the deinterlacer...

Please let me know if you have any problem trying to run this prototype app.

By the way, my overclock settings are the following (with very stable results since I have heatsinks and fan on the RPi):
arm_freq=1400
gpu_mem=200
over_voltage=7
core_freq=550
sdram_freq=550
sdram_schmoo=0x02000020
force_turbo=1
boot_delay=1
Attachments
camlive-source-25-7-2018.zip
Test App Sources
(17.42 KiB) Downloaded 17 times

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5815
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: TC358743 and interlaced HDMI modes

Thu Jul 26, 2018 2:19 pm

Some funnies running your app, and that's before I've been looking at the image_fx qpu stuff. I have got a debug firmware with various other tweaks, so things may be different.

It crashes within a few seconds. Looking at the VPU logs the same buffer appears to be returned twice.
This does appear to be something that has been changed in the firmware. I'll have to look in more detail at a later date, but there may be regressions in the very latest firmware. (The other person who works on this is away on holiday at the moment, and I'm off soon).

Something appears very wrong in the alignments as I'm getting big shifts in the images. https://drive.google.com/file/d/1YTaEWD ... sp=sharing is a 1920x1080 I420 image. That's not how the end credits of Big Buck Bunny should look.

A definite point to improve are how you are setting buffer_num on various ports. The defaults tend to be minimise memory usage, which does not necessarily account for keeping the pipeline busy. eg video_encode only needs one input buffer to work with, but that means that the thing supplying video_encode has to stop whilst encode is doing what it needs on the buffer. Assigning 2 buffers to those ports means that both source and sink can be busy at the same time.
- rawcam output really wants at least 6 (I think that may be the default on the current firmware).
- isp input will want at least 3 to keep it occupied and yet still allow you to have a buffer to be line interleaving into.
- isp_ouptut and deinterlace input want the same number. I'd say 3 was a minimum, preferably more.
- deinterlace output and encode input want at least 2.
- encode output will be happy with 1, but they're small so it's worth adding a couple of extra into the mix.

image_fx appears to only say it wants a single buffer on the input. I think that is probably why your pipe stalls as that isn't the case for the more advanced deinterlacing modes that want context.

I suspect I'm going to have to get to the bottom of the rawcam double release once I'm back. I'd generally seen it fairly infrequently, but your test seems to be triggering it fairly reliably. A reliable crash tends to be the key to finding issues.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

luiscgalo
Posts: 29
Joined: Fri Jun 22, 2018 9:09 am

Re: TC358743 and interlaced HDMI modes

Mon Jul 30, 2018 1:25 pm

Hi 6by9,
Sorry for my delayed response...
Thanks for testing my prototype application and for sharing valuable information and recommendations to improve performance of my video capture system.

Sometimes, I also have the same dramatic behavior of "crashing" something, triggering the kernel panic LED.
It is something that we have to take a look to understand what should be corrected/improved.
6by9 wrote: I suspect I'm going to have to get to the bottom of the rawcam double release once I'm back. I'd generally seen it fairly infrequently, but your test seems to be triggering it fairly reliably. A reliable crash tends to be the key to finding issues.
Well... In that case I'm already happy because, at least, my prototype code is useful to help detecting points of improvement on the RPi firmware. :D

In the meantime, I will try increasing the number of buffers on my data processing chain, evaluating the results.
We keep in touch... ;)

luiscgalo
Posts: 29
Joined: Fri Jun 22, 2018 9:09 am

Re: TC358743 and interlaced HDMI modes

Fri Aug 03, 2018 6:22 pm

Today I've made some more trial and error tests following 6by9 suggestions:
6by9 wrote: A definite point to improve are how you are setting buffer_num on various ports. The defaults tend to be minimise memory usage, which does not necessarily account for keeping the pipeline busy. eg video_encode only needs one input buffer to work with, but that means that the thing supplying video_encode has to stop whilst encode is doing what it needs on the buffer. Assigning 2 buffers to those ports means that both source and sink can be busy at the same time.
- rawcam output really wants at least 6 (I think that may be the default on the current firmware).
- isp input will want at least 3 to keep it occupied and yet still allow you to have a buffer to be line interleaving into.
- isp_ouptut and deinterlace input want the same number. I'd say 3 was a minimum, preferably more.
- deinterlace output and encode input want at least 2.
- encode output will be happy with 1, but they're small so it's worth adding a couple of extra into the mix.

image_fx appears to only say it wants a single buffer on the input. I think that is probably why your pipe stalls as that isn't the case for the more advanced deinterlacing modes that want context.
Basically, on my prototype code, I've set the buffer_num of input and output ports to 8 (it is a reasonable value, right?) on all components (ISP, image_fx and H264 encoder).
However, I'm still experiencing the same problems related with the deinterlacer:

1. When I use the MMAL_PARAM_IMAGEFX_DEINTERLACE_FAST deinterlacer, the application works perfectly with the img_fx_param.effect_parameter[2] option set to 1 (half framerate which means 25fps output).
When "half_framerate" is disabled (50fps output) the output video is choppy like described on a previous post.
It seems that the deinterlacer struggles with 1080p50 video.
From what 6by9 said on a previous post, it seems that I'm reaching the limits of the hardware...

2. Usage of the MMAL_PARAM_IMAGEFX_DEINTERLACE_ADV deinterlacer is unsuccessful (at least in my pipeline).
When the application starts, the ISP input port (first port on my pipeline) is never available to be written.
Honestly I don't know why this mode is not working.
Theoretically, this deinterlacer mode should work and support 1080p50 video...

Additionally, I also saw some sporadic crashes on the RPi (triggering kernel panic) when running the application.
This could be related to the double buffer release problem described by 6by9 on a previous post...

As side note, I've updated the firmware of my Raspberry Pi today for its latest version.

Again, I need some more help from expert guys like 6by9 in order to solve this issue related with the deinterlacer.

luiscgalo
Posts: 29
Joined: Fri Jun 22, 2018 9:09 am

Re: TC358743 and interlaced HDMI modes

Mon Aug 06, 2018 10:02 am

Hi again,
I'm still experimenting and trying to understand why the deinterlacer is struggling to produce 1080p50 from 1080i50 video... (without success yet)

By the way, another idea that comes to my mind (probably someone will say that it is quite stupid), will be to evaluate if the H264 encode block can support encoding of interlaced video directly.
If that's the case, I will be able to encode directly the 1080i50 video comming from the TC358743 chip, without using the deinterlacer component (image_fx).

6by9 (or someone more expert than me :) ) do you think that's possible?

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5815
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: TC358743 and interlaced HDMI modes

Mon Aug 06, 2018 2:23 pm

No, the encoder can't encode interlaced video. Or at least it can't set up the headers correctly to denote interlaced.

I'm back in the office so will have a look at what is going on with image_fx.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

luiscgalo
Posts: 29
Joined: Fri Jun 22, 2018 9:09 am

Re: TC358743 and interlaced HDMI modes

Wed Aug 08, 2018 9:52 am

Thanks again 6by9 for taking your time to check what's happening with the image_fx component (deinterlacer functionality) and the sporadic "crash" issue that we both have when running my prototype source code.

I will wait for your feedback to point out what should be the problem and if I have to change something on my code to use the "advanced" deinterlacer with QPU enabled.

luiscgalo
Posts: 29
Joined: Fri Jun 22, 2018 9:09 am

Re: TC358743 and interlaced HDMI modes

Fri Aug 10, 2018 10:54 am

Hi 6by9, probably you've been quite busy with other tasks.
Without trying to put any pressure, do you had some time to take a look on my issues related with the deinterlacer (struggling with 1080p50 output since I cannot active the QPU) or with sporadic crash of the pipeline (it seems related with a double buffer release problem)?
Thanks in advance for your help...

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5815
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: TC358743 and interlaced HDMI modes

Fri Aug 10, 2018 4:53 pm

No, I've had a further play with the code you'd posted and it always crashes in a heap very rapidly.
Please add "start_debug=1" to /boot/config.txt and take a "sudo vcdbg log msg" and "sudo vcdbg log assert" log when you're running.
For me I always get "vcos_abort: Halting" in the log msg output, which basically means bad things have happened and the VPU can't continue safely. That thread is stuck into a while(1) loop or equivalent.
In the assert log I'm getting
052112.107: assert( (subpool) && (subpool)->magic == VCOS_BLOCKPOOL_SUBPOOL_MAGIC && subpool->start >= subpool->mem ) failed; ../../../../../interface/vcos/threadx/../generic/vcos_generic_blockpool.c::vcos_generic_blockpool_free line 340 rev ae4ae72
which is a reuse of a freed buffer. That shouldn't happen. raspi_tc358743 seems vaguely happy with the input, so I'm not sure quite what is going on there.

Please could you put your code into something like git/github so that I can ensure I'm working from your latest hacking.

I need to reread the code of the QPU deinterlacer. I think it may be requiring an internal format called SAND or YUVUV that the video codec spits out, and if the input isn't correct it does a conversion on the VPU. That'll be hurting if it is the case, but I can't convince myself that it is. What I really need is a test case that I can easily run.

I had suggested you try "sudo vcdbg log hist" to graph the VPU usage whilst the deinterlacing was running. What did that reveal? Can you post the output to pastebin/dropbox or similar?
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

luiscgalo
Posts: 29
Joined: Fri Jun 22, 2018 9:09 am

Re: TC358743 and interlaced HDMI modes

Sat Aug 11, 2018 11:15 pm

Today I've tried to debug and test again my prototype code using the latest version of RPi firmware.
Now, I'm experiencing exactly the same problems as you and I cannot put the application running anymore... :?

Here are the scenarios which I've tried (all unsuccessful):
1. When using MMAL_PARAM_IMAGEFX_DEINTERLACE_FAST deinterlacer, the VPU crashes always with kernel panic led blink.
The message in the log is "vcos_abort: Halting" and "sudo vcdbg log assert" shows exactly the same text message you presented on the previous post)
In the past (with older RPi firmware versions) the crash was sporadic only...
Something is going "dramatically wrong" within the image_fx component code... :(
I'm almost sure that is related with the image_fx component because it's the only component triggering this catastrophic failure of the system.
I can have a setup with Rawcam, ISP and H264 video encoder and everything works fine.
However, as soon as I add the image_fx component to the pipeline this problem comes again and I don't know why.

2. When trying to use MMAL_PARAM_IMAGEFX_DEINTERLACE_ADV with QPU enabled the result is quite different.
There is no crash on the RPi system, but the ISP input port never accepts data (always fails to send interfaced BGR24 frame data to the ISP input).
Again, I don't know what is the problem here since I have a setup that is supposed to work, theoretically...

3. By curiosity, I've tried to convert BGR24 to SAND format instead of I420 (I know that recently you have referred that image_fx only supports I420 but it was just a test :D )
The result here was even more dramatic that the first one!
When I try to run the application (converting to SAND format), the RPi freezes completely (kernel panic LED is not even triggered)!
After this only a power cycle restores RPi operation.
Do you know why the system as this catastrophic failure when trying to work with SAND format?

Finally, I've followed your suggestion and I've created a Github repository for storing this prototype application.
The link for the repository is: https://github.com/luiscgalo/rpi-video-recorder/

From this repo you can checkout the code, make some tests and change what you want (I've invited you to provide "push" permissions).
Right now, in order to simplify our tests related with image_fx component, I've disabled (commented out) all code related with H264 encoder.
Basically, interlaced video frames only are sent to the ISP and then forwarded to the image_fx.
At the moment, image_fx output port is just printing some text to the stdout... (I'm discarding the data).

Let's see if we catch what's going on the image_fx component... :D

luiscgalo
Posts: 29
Joined: Fri Jun 22, 2018 9:09 am

Re: TC358743 and interlaced HDMI modes

Thu Aug 16, 2018 12:42 pm

Yesterday I've tried again some more attempts to enable functionality of my prototype application.
However, after uploading the RPi firmware to its latest version I'm not able to run the app anymore.
The deinterlacer crashes the RPi some moments after starting the app.
It's so annoying because with this catastrophic failure it is quite difficult to proceed with the development of my application... :?

6by9, do you know what's happened since last update performed or can you guess what's happening with the image_fx component?
By the way, I don't know if you've tried to access to my GitHub repository or if you are experiencing any difficulty.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5815
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: TC358743 and interlaced HDMI modes

Thu Aug 16, 2018 2:00 pm

There were a couple of patches to rawcam that were merged 24th July as part of https://github.com/raspberrypi/firmware ... c43a5b3699 to address race condition when swapping buffers.
I haven't tried it against TC358743 other than via your app and 1080i50.

Can you try "sudo SKIP_KERNEL=1 rpi-update d985893ae67" to sync to the version before and confirm if it behaves?
I'm afraid I really don't have the time to investigate too much at the moment even if we can pinpoint that as the regression. The overall behaviour is now correct, even if it crashes in your instance.
I did have a quick look at the Github repo - thanks, it just saves me messing around with having to set up my own to be able to track how my copy of the app differs to yours.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5815
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: TC358743 and interlaced HDMI modes

Wed Aug 22, 2018 2:46 pm

Firstly I think I've tracked down the regression in rawcam. I'll post a test firmware in a bit.

One thing I have discovered is that mmal_connection's have a flag MMAL_CONNECTION_FLAG_KEEP_BUFFER_REQUIREMENTS to make it keep the buffer sizes and number that you request, rather than what the components think may be best (the recommended values). I definitely needed that on the ISP to deinterlace connection, otherwise we end up with a single buffer on the link and the pipeline stalls.

Your app runs reasonably reliably with the new firmware. I am seeing a couple of odd frames where it appears not to be linear in time. Using Big Buck Bunny, where the acorn drops on the bird's head I get a frame jump backwards in time, and then back to normal. Also the chroma planes appear not to correspond to the luma planes, so we have a colour shadow. I don't know what is going on there.
I was suspecting something dubious going on over the field detection. In the event that we drop one buffer from rawcam due to not returning buffers fast enough, we could get a second bottom field without the corresponding top field and it'll still call convert.
Adding the patch

Code: Select all

--- a/rawcam.c
+++ b/rawcam.c
@@ -302,6 +302,12 @@ void rawcam_callback(MMAL_PORT_T *port, MMAL_BUFFER_HEADER_T *buffer) {
 
                if (g_bFirstTopFieldFound == true) {
                        video_field_cb(sField);
+                       if (sField.unField == FIELD_BOTTOM)
+                               g_bFirstTopFieldFound = false;
+               }
+               else
+               {
+                       printf("Dropping valid field as field order wrong\n");
                }
        }
stops and logs that, and I'm not seeing drops logged.

BTW I've switched my version of the code back to i2c-0. Add

Code: Select all

dtparam=i2c_vc=on
dtoverlay=i2c0-bcm2708,pins_44_45=on
to config.txt and it should sort out all the pinmuxing for you.
I've pushed my version back up to as a branch called 6by9 in your repo. It's based on the original code you posted here rather than your repo, but I've moved stuff to match your repo, and it gives you the reference.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5815
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: TC358743 and interlaced HDMI modes

Wed Aug 22, 2018 2:52 pm

Hang on, the chroma issues may well come back to the subsampling via the ISP.
We want to create an I420 image where chroma line 1 corresponds to the luma on lines 1&3, and chroma line 2 corresponds to the luma on lines 2&4. Currently you're getting chroma line 1 from luma lines 1&2, and chroma line 2 from luma lines 3&4.
We need the ISP to treat the frame as 1920*2 x 1080/2 in size, or 3840 x 540, in order to get the chroma subsampling correct. I think I'd mentioned this in passing on a different thread, but we haven't adopted it here and need to.

I'll see if I can sort that quickly.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5815
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: TC358743 and interlaced HDMI modes

Wed Aug 22, 2018 4:13 pm

Chroma subsampling sorted and pushed. It appears to sort out the odd chroma effects I was seeing.
It needs https://github.com/raspberrypi/userland/pull/483 to work.

edit Phil was on the ball and had merged the userland PR already. Clone the standard userland repo and ./buildme, and you should get the required functionality.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

luiscgalo
Posts: 29
Joined: Fri Jun 22, 2018 9:09 am

Re: TC358743 and interlaced HDMI modes

Wed Aug 22, 2018 10:43 pm

Hi again 6by9, thanks for your valuable investigation and help.

I've followed your suggestion (cloning your GIT branch and compiling/installing the latest version of userland libraries) and I was able to successfully compile your version of "rpi-video-recorder".

However, when trying to run the application, for a fraction of a second (about half a second) it seems to be working perfectly and valid video is being recorded until a point where the GPU crashes and the application stalls... :(

The log shows me an following error similar to what we've saw before:
assert( (subpool) && (subpool)->magic == VCOS_BLOCKPOOL_SUBPOOL_MAGIC && subpool->start >= subpool->mem ) failed; ../../../../../interface/vcos/threadx/../generic/vcos_generic_blockpool.c::vcos_generic_blockpool_free line 340 rev 31e0613
vcdbg_ctx_get_dump_stack: dump_stack failed

Probably this issue is still related with the MMAL buffer release issues recently detected.
I've run the "sudo rpi-update" but it seems that I'm already running the latest version of the firmware for RPi.

Are you using any specific/new version of RPi firmware?
If yes, that could explain why the application works on your Raspberry Pi but not on my environment.
Otherwise I'm doing something wrong trying to setup the things...

Once again, thanks for help and support on this. ;)

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5815
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: TC358743 and interlaced HDMI modes

Thu Aug 23, 2018 9:55 am

https://www.raspberrypi.org/forums/view ... 8#p1356219
Firstly I think I've tracked down the regression in rawcam. I'll post a test firmware in a bit.
That hasn't happened yet - too many things going on. I need to get a releasable firmware built, and I'll stick it on Google Drive for testing.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5815
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: TC358743 and interlaced HDMI modes

Thu Aug 23, 2018 10:45 am

New firmware at https://drive.google.com/file/d/1pfj5OY ... sp=sharing
"vcgencmd version" should report 96e9622cd164078fdc93a72706ea441efcda8b82, built today.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

luiscgalo
Posts: 29
Joined: Fri Jun 22, 2018 9:09 am

Re: TC358743 and interlaced HDMI modes

Fri Aug 24, 2018 1:23 pm

6by9 wrote: Firstly I think I've tracked down the regression in rawcam. I'll post a test firmware in a bit.
Sorry, I was so motivated with the fix that I did not even notice that statement. :D

Thanks for providing a custom firmware for testing purposes.
Now the application works without any crash on the VPU and the image_fx with QPUs enabled is working without any major problem.
The color conversion seems to be working perfectly also.

From my tests, when requesting 25fps as output framerate (with "half framerate" flag set to one in image_fx) the produced video is perfect and smooth all the time.
However, I'm still facing some problems related with smooth motion of the video when requesting 50fps (with "half framerate" flag set to zero in image_fx). In this scenario the video motion is choppy (back and forward movements)...

Do you also saw this problem during your tests?
Since the pipeline is working perfectly for 25fps, what could be causing this issue when requesting 50fps?

My suspicion is a problem between the image_fx output port and encoder input port. But I could be wrong...
Basically, when trying to deinterlace 25i into 25p, everything works perfectly since there is a direct "one to one" match between input and output frame rates on image_fx (25fps input [interlaced] and 25fps output [progressive]).
However, when converting from 25i to 50p we are doubling the output rate on the image_fx component. This means that we generating two output frames (progressive) for one input frame (interlaced).
Maybe there is an issue (or incorrect configuration on my side) on the connection between image_fx output and encoder input which ignores/inverts/corrupts the order of the frames produced by the deinterlacer... That could explain why we are getting choppy movements...
I don't know if my rational makes any sense to you 6by9. What do you think?

Return to “Camera board”