pigpio in uninterruptible sleep

[Mal_B]

Hi,
while working on C code linked to pigpio library, I'm now experiencing a weird issue : when trying to kill a correctly running process, it does not terminate, because it is stuck in DL state (uninterruptible sleep).

This was not the case until recently and I'm trying to find what is causing this in my recent code changes.
I'm working with wave calls in recent changes.

Does anyone ever experienced such a situation due to pigpio library, or is it something else ?

[joan]

Have you got some code you can share which shows the problem? Also do you know which version was okay and which version is faulty?

[Mal_B]

Hi Joan, thank you for caring.

Here below is the wave code I'm working on. I'm not sure at all it is related, to be honest.
The fact is that I was able to kill and relaunch until recently.

Is it possible that some invalid value (a mess with some negative value in uint_32) on delay in wave cause such a situation ?

FYI, this code is part of a regulator to command phase thyristor switch.

Code: Select all

#define DEL_F 20 //us délai de prise en compte front montant période, décalage supposé du trigger de phase
			int32_t del_pulse = (v_calc.Tm / 2); //demi-période courante pour ne pas dériver
			if (v_pers.Cy > del_pulse) v_pers.Cy = del_pulse;
			uint32_t tac = gpioWaveTxBusy() ? t_w : gpioTick();
 			int32_t delai = tick - DEL_F + del_pulse - tac; //delai avant Q1 (alternance négative)
			del_pulse -= v_pers.Cy;
			pulse[0].usDelay = delai + del_pulse ;
			pulse[1].usDelay = v_pers.Cy;	//durée de Q1
			pulse[2].usDelay = del_pulse;	//
			pulse[3].usDelay = v_pers.Cy;	//durée de Q2
			gpioWaveClear(); //vérifier si ça ne casse pas la wave Tn-Q2 en cours
			gpioWaveAddNew();
			gpioWaveAddGeneric(4, pulse);
			int wave_id = gpioWaveCreate();
			///activation séquence pour ce cycle
			if (wave_id >= 0) {
				gpioWaveTxSend(wave_id, PI_WAVE_MODE_ONE_SHOT_SYNC);
				int duree = gpioWaveGetMicros();
				if (duree < 20500) {
					t_w = duree + tac; //on mémorise le top temps de la fin de wave pour le calcul délai prochaine itération.
				}
				else v_calc.marche = 0;
				sprintf(debug, "%s t_w %f durée %d delai %d",debug,  0.000001 * t_w, duree, delai);
			}
			else {
				sprintf(debug, "%s t_w %u erreur wave create %d",debug, t_w, wave_id);
				v_calc.marche = 0;
			}

[joan]

I'll have a look at that, but tomorrow!

[joan]

The only thing I notice with the code is the call to gpioWaveClear, then creating a new wave, followed by a call to gpioWaveTxSend(wave_id, PI_WAVE_MODE_ONE_SHOT_SYNC).

The PI_WAVE_MODE_ONE_SHOT_SYNC implies you have an existing running wave. But the space used by that would have been reused by the new wave you create after the clear and if it was still running that could cause all sorts of problems (crashes, memory corruption).

[Mal_B]

This may be source of problem indeed.
If I don't call gpioWaveClear, The memory gets saturated.
What should be the right strategy to generate a four pulse wave each iteration and chain it after the current running one ?

[joan]

Do something like the following (syntax liberties taken).

Code: Select all

old_id = -1
while (1)
{
   gpioWaveAddGeneric();
   new_id = gpioWaveCreate();
   if (new_id >= 0)
   {
      gpioWaveTxSend(new_id, PI_WAVE_MODE_ONE_SHOT_SYNC);
      while (gpioWaveTxAt() != new_id) time_sleep(0.001);
      if (old_id >= 0) gpioWaveDelete(old_id);
      old_id = new_id;
   }
}

[Mal_B]

OK. I'll do it with 3 id variables to avoid the blocking loop, since I know that at a given iteration, I have a running wave (close to terminate), a new one to generate and an old one terminated.

I'll let you know.

[Mal_B]

The uninterruptible state has gone.
I'm still unsure if it was due to a memory corruption as you suggested or an invalid value on delay. The last option is unlikely since the time value turn around each 90 minutes and the block was more persistant (lasted one night in a test).

to avoid destroying current running wave i'm now doing, in the periodic callback:

Code: Select all

			if (wave_prec >= 0) gpioWaveDelete(wave_prec);
			wave_prec = wave_cour;
			gpioWaveAddGeneric(4, pulse);
			wave_cour = gpioWaveCreate();
			if (wave_cour >= 0) {
…

[Mal_B]

I'm bumping this thread even if this post is no longer related to the lockout in the OP.
Joan, tell me if you wish a new thread to discuss this issue.
My current problem is that I'm unable to chain waves periodically.

What I try to achieve:

1. Each 20ms a function is called on input change (gpioSetAlertFunc()). In this function some calculation is made and a control waveform is generated and added. Total duration of the generated wave is 20ms.
2. Since the function is called before the previous wave ends. The new generated one is sent with PI_WAVE_MODE_ONE_SHOT_SYNC mode, in order to be executed after the currently running one.
3. At any invocation of the function, we have three waves id involved. One is the currently running and about to finish. Other is the new generated an queued, and last is the "old" one, that is the one trailing from the previous iteration.
4. At the beginning of the invocation the "old" one is deleted to free space in wave buffer. The code I posted in my previous answer outlines the algorithm.

But after an variable number of iteration (8-20), something gets messy in the waves id/buffer. A call to gpioWaveTxAt() returns the wave id of the "old" one (it should be the current). The numbering is not logical in order.

It looks like the "garbage collection" in the waves buffer does not works well. Presumably it may be logical to reuse the same 3 "slot" in a round-robin scheme and I was expecting to see the same 0-1-2 wave ids. But this is not what happens. id can raise up to 7 or 8 before being "recycled" and moreover the numbers (and buffers I presume) get messed.

What would be the correct way to code this ?

[joan]

If you can provide a small program which shows the problem I'll run it and check for errors.

[Mal_B]

OK, here we are:

Code: Select all

//
//  test_wave.c
//  regul_pi
//

#include <stdio.h>
#include <stdlib.h>
#include <stdarg.h>
//#include <unistd.h>
//#include <signal.h>
#include <string.h>
//#include <fcntl.h>
//#include <byteswap.h>
#include <math.h>

#include <pigpio.h>

#define FREQ_IN		17	//frequency in signal
#define Q1			9	//command Q1
#define Q2			10	//Q2

static uint8_t run = 1;

void func_20ms(int user_gpio, int level, uint32_t tick) {
	static int wave_cour = -1, wave_prec = -1, wave_old = -1, n_iter = 0;
	static gpioPulse_t pulse[4] =  {
		//on, off, delay
		{1<<Q2,0, 5000},
		{0, 1<<Q1, 5000},
		{1<<Q1, 0, 5000},
		{0, 1<<Q2, 5000}
	};

	if (level == PI_LOW) return;

	if (wave_old >= 0) {
		int w_at = gpioWaveTxAt();
		if (w_at != wave_old) gpioWaveDelete(wave_old);
		else {
			fprintf(stderr, "mess in waves : at %d old %d prec %d cour %d nb pulse %d\n", w_at, wave_old, wave_prec, wave_cour, n_iter);
			run = 0;
			return;
		}
	}
	n_iter++; //to count waves generated
	wave_old = wave_prec;
	wave_prec = wave_cour;
	gpioWaveAddNew();
	gpioWaveAddGeneric(4, pulse);
	wave_cour = gpioWaveCreate();
	if (wave_cour >= 0) {
		gpioWaveTxSend(wave_cour, PI_WAVE_MODE_ONE_SHOT_SYNC);
	}
	else {
		fprintf(stderr, "invalid wave id %d @ nb pulse %d\n", wave_cour, n_iter);
		run = 0;
		return;
	}
}

void terminate() {
	gpioSetAlertFunc(FREQ_IN, NULL);
	gpioTerminate();
	fprintf(stderr, "pigpio terminated.\n");
}

int main(int argc, char *argv[]) {
	if(gpioInitialise() < 0) {
		/* init dans main pour retour code erreur */
		fprintf(stderr, "pigpio initialisation failed.\n");
		return 1;
	}
	else fprintf(stderr, "pigpio ok\n");
	
	gpioSetMode(Q1, PI_OUTPUT);
	gpioSetMode(Q2, PI_OUTPUT);

//	//this is to be used with a hadware signal on input
//	gpioSetMode(FREQ_IN, PI_INPUT);
//	gpioSetAlertFunc(FREQ_IN, func_20ms);
	
	//alternate simple timer invocation
	gpioSetTimerFunc(0, 20, func_20ms);
	
	while (run) ;
	terminate();
}

Compilation invocation:

Code: Select all

pi@moulin-turbine ~/pi_turbine/regul_pi $ gcc -o test_wave test_wave.c -lpigpio -lpthread -lrt
pi@moulin-turbine ~/pi_turbine/regul_pi $ sudo ./test_wave 
pigpio ok
mess in waves : at 0 old 0 prec 1 cour 2 nb pulse 1113
pigpio terminated.
pi@moulin-turbine ~/pi_turbine/regul_pi $ 

This sample code fails at some point, when run with a square 50hz signal at input.
In the source code there is a alternate initialization to activate the wave function on a timer basis. Surprisingly enough, when run from timer, the code runs ok (did not wait it to fail). I discovered it when setting up the sample code.

So now the question becomes "Is 20ms too short for the wave gestion to proceed ?" It may depend on other task around. In my real code I have to read an ADC (current intensity value) through i2C. I'm doing it on the low edge of the 50hz input. The whole processing may be too much.

[joan]

I'll have a look. Possibly tomorrow. If not probably Tuesday.

[joan]

It works okay with the timer invocation.

May I ask why you are creating a new wave each time through the loop? If nothing has changed you can just send the old wave again.

Remember that gpioWaveTxAt may return one of these values

• the id of the current wave being transmitted
• PI_WAVE_NOT_FOUND (9998) if the id of the current wave can't be determined
• PI_NO_TX_WAVE (9999) if there is no wave being transmitted (i.e. a wave has completed and nothing else was queued).

If you get PI_WAVE_NOT_FOUND you need to make another call to gpioWaveTxAt.

[Mal_B]

Hi Joan,
Thank you for following up, sorry for answering late, I was off the computer this afternoon.

The wave is re-caclulated at each invocation (not a loop, but a callback on edge), because it's the way a regulation works. This is to regulate a micro hydro turbine+alternator. The wave is to trigger thyristor on a ballast charge, in order to regulate the alternative current to a steady 50hz. The load on the main is changing and one need to watch the frequency to anticipate deviation.
Also, a thyristor trigger must switch off on the zero crossing of the mains. So the pulse is adjusted to start where it needs to and finish just on the mid period.

If pigpio can't cope with a 20ms chaining of waves, I can do it once on two, sending the wave in repeat_sync for two period and adjusting as needed. I have to test this.
But the calculation in the function is fast enough (80us) and the wave is sent quite early. The point is that some delay incurs in pigpio waves processing that cause it to be lag back at some point.

I understand the gpioWaveTxAt return value. And you can see from my log the test program does not end on a special value but rather on a value corresponding to a wave id which is wrong in the context.

I'll try tonight if a calculation/adjustment of the wave on a 40ms time schedule is ok.

Also, notice that the timer invocation is not failing at 20ms, while the callback on edge does on the same time base. This may be a clue that the callback processing add some delay in the wave processing ?

Edit: Just to clarify, the code I posted is not the real one, where the wave is adjusted. It's a simplified test of the problem I encounter.

Edit 2: 40ms seems better, but still not error free yet (on my real code). I have to understand what is causing the delay in waves processing.

Edit 3: I eventually found the problem, on my side. Sorry for the noise. Once again it was a problem of invalide value for delay. When working on a real mains phase, it happened that adjustment of the pulse edge sometimes gave a slightly negative value, which was of course invalid. Such a value was interpreted as a huge unsigned delay value. Then the wave did not finish in time and we had a still active wave when it should have been finished. Timer invocation did not generate error since the timing is fair and no adjustment is needed.

[Mal_B]

Redundant post to raise a forum alert to Joan and followers. Il found the problem on my side, see edit 3 of previous post.
Also that negative value bug might have been related to original post question, somehow.

[Mal_B]

For the record, even after tracking down any bugs in my code, I found that trying to concatenate waves using *_SYNC mode was not reliable. At some point the wave_ids get mixed somehow. I'm not able to elaborate on the reason, but it looks like any situation were a wave has a "follower" waiting is candidate for the problem. Like if some of the cleaning-garbage_collection is postponed in some cases.

I solved the problem, in my specific use case, by only using ONE_SHOT mode, ensuring that a given wave is (just) finished at the moment I send a new one. It turns out it is possible since the processing delay from a GPIO input transition to the callback is not more than 1ms, averaging at 500us. By tricking on the wave form, it is possible to either activate the effective output change in the new transmitted wave or else, if the edge must occurs before the callback activates, appending a "pulse" (and output edge) to the at the end of the previous wave.

Using this scheme, there is no problem at all to calculate and send a new wave on a 20ms time base. The I2C reading is also done on the same time base, on the other edge of the "clock" signal.

[joan]

For the record, even after tracking down any bugs in my code, I found that trying to concatenate waves using *_SYNC mode was not reliable. At some point the wave_ids get mixed somehow. I'm not able to elaborate on the reason, but it looks like any situation were a wave has a "follower" waiting is candidate for the problem. Like if some of the cleaning-garbage_collection is postponed in some cases.
...

If you have some code which I can run and demonstrates the fault I'll have a look.

[Mal_B]

Joan, here under is a slightly modified test code. It is close to the one posted already. It is worth to note :

1. When called by a timer the code runs ok.
2. it fails when called by a callback on edge. Only one edge is relevant, function returns on the.
3. on first invocation of the function an initial delay is set, in order to have the wave still running when next invocation occurs
4. I'l testing the failing callback case on a "real life" clock signal from the mains. It has slight variation in time, something like 10-100us variation in period. I'm unsure if this plays a role.
5. Also, as you know the callback is by itself uneven to the edge, being called on a 1ms time base, it lags a bit.

Code: Select all

//
//  test_wave.c
//  regul_pi
//

#include <stdio.h>
#include <stdlib.h>
#include <stdarg.h>
//#include <unistd.h>
//#include <signal.h>
#include <string.h>
//#include <fcntl.h>
//#include <byteswap.h>
#include <math.h>

#include <pigpio.h>

#define FREQ_IN		17	//frequency in signal
#define Q1			9	//command Q1
#define Q2			10	//Q2

static uint8_t run = 1;

void func_20ms(int user_gpio, int level, uint32_t tick) {
//void func_20ms() {
	static int wave_cour = -1, wave_prec = -1, n_iter = 0;
	static gpioPulse_t pulse[5] =  {
		//on, off, delay
		{0,0, 5000},	//initial delay on first pass
		{1<<Q2,0, 5000},
		{0, 1<<Q1, 5000},
		{1<<Q1, 0, 5000},
		{0, 1<<Q2, 5000}
	};

//	static uint8_t mes_f = 0;
	if (level == PI_LOW) return;
//	mes_f = !mes_f;
//	if (mes_f) return;

	int w_at = gpioWaveTxAt();
	if (w_at < PI_WAVE_NOT_FOUND && w_at != wave_cour) {
		fprintf(stderr, "mess in waves : at %d prec %d cour %d nb pulse %d\n", w_at, wave_prec, wave_cour, n_iter);
		run = 0;
		return;
	}
	if (wave_prec >= 0) gpioWaveDelete(wave_prec);
	
	n_iter++; //to count waves generated
	wave_prec = wave_cour;
	gpioWaveAddNew();
//	pulse[2].usDelay = pulse[2].usDelay == 8000 ? 2000 : 8000;
	if (wave_cour == -1) gpioWaveAddGeneric(5, pulse);
	else gpioWaveAddGeneric(4, &pulse[1]);
	wave_cour = gpioWaveCreate();
	if (wave_cour >= 0) {
		gpioWaveTxSend(wave_cour, PI_WAVE_MODE_ONE_SHOT_SYNC);
	}
	else {
		fprintf(stderr, "invalid wave id %d @ nb pulse %d\n", wave_cour, n_iter);
		run = 0;
		return;
	}
}

void terminate() {
//	gpioSetAlertFunc(FREQ_IN, NULL);
	gpioSetTimerFunc(0, 20, NULL);
	gpioTerminate();
	fprintf(stderr, "pigpio terminated.\n");
}

int main(int argc, char *argv[]) {
	if(gpioInitialise() < 0) {
		/* init dans main pour retour code erreur */
		fprintf(stderr, "pigpio initialisation failed.\n");
		return 1;
	}
	else fprintf(stderr, "pigpio ok\n");
	
	gpioSetMode(Q1, PI_OUTPUT);
	gpioSetMode(Q2, PI_OUTPUT);

	//this is to be used with a hadware signal on input
	gpioSetMode(FREQ_IN, PI_INPUT);
	gpioSetAlertFunc(FREQ_IN, func_20ms);
	
	//alternate simple timer invocation
//	gpioSetTimerFunc(0, 20, func_20ms);
	
	while (run) ;
	terminate();
}

[joan]

As an aside you might be able to use http://abyz.co.uk/rpi/pigpio/cif.html#gpioSetISRFunc rather than an alert callback for what you are doing. That is likely to have less latency than the 500 µs or so of an alert callback.

[Mal_B]

As an aside you might be able to use http://abyz.co.uk/rpi/pigpio/cif.html#gpioSetISRFunc rather than a callback for what you are doing. That is likely to have less latency than the 500 µs or so of an alert callback.

I tried it at some point already, but found it worse failing. I must say, nevertheless, that I don't remember if it was before or after I purged negative delays bug. I might try it again. I have now a working setup so it is no longer a showstopper.

[Mal_B]

I looked again at gpioSetISRFunc.
It turns out that the time latency is much less (125us) and make it a candidate, but there is a tick value problem. For some reason, probably system related, the tick received by the call back is sometimes way late. On a 20ms period it is current to see two succeeding ticks at, say, 25 and 15ms. The last is in sync again but then one have to filter out invalide value.This is not the case when using pigpio callback.

I'm running Wheezy 7 on 4.1.7 kernel if it makes a difference.