User avatar
Hove
Posts: 1205
Joined: Sun Oct 21, 2012 6:55 pm
Location: Cotswolds, UK
Contact: Website

SOLVED: Interacting between 5 processors problem

Thu Mar 01, 2018 4:43 pm

Code: Select all

+—————+
|Sweep|———>———+
+—————+       |
        +—————+———+     +——————+
        |Autopilot|——>——|Motion|
        +—————+———+     +———+——+
+———+         |             |
|GPS|————>————+             |
+———+                       |
                            |
                +—————+     |
                |Video|——>——+
                +—————+
I have 5 processes, all running Python, interconnected via shared memory streams. The problem is I can run GPS or Sweep, but not both. Neither are passing much to the autopilot process - a few bytes a second max. The autopilot receives from both and does its processing sending the motion what it needs to do next. The autopilot output on the stream sent to the motion processor is not picked up by the motion process. Video is never hampered when GPS &| Sweep are running. The motion processor is very busy as it's handling input from the IMU processing also. The motion processes is polling for both video and autopilot poll fds, and as I said, the poll always picks up the video, but never sees the autopilot but only when GPS _&_ Sweep processes are running into the autopilot process.

This is running on a B3; I've tried varienties of os.nice() and mlockall() to tweak priorities and prevent paging, but the result is unchanged.

I'm stumped, so I'm open to any suggestions, however weird they may sound!

TIA
Last edited by Hove on Wed Mar 07, 2018 1:27 pm, edited 1 time in total.
www.pistuffing.co.uk - Raspberry Pi and other stuffing!

User avatar
PeterO
Posts: 5820
Joined: Sun Jul 22, 2012 4:14 pm

Re: Interacting between 5 processors problem

Thu Mar 01, 2018 4:51 pm

On something as complicated as this, without seeing your code, it's next to impossible to make any suggestions.
PeterO
PS: Thread title says processors (hardware) not processes (software).
Discoverer of the PI2 XENON DEATH FLASH!
Interests: C,Python,PIC,Electronics,Ham Radio (G0DZB),1960s British Computers.
"The primary requirement (as we've always seen in your examples) is that the code is readable. " Dougie Lawson

User avatar
Hove
Posts: 1205
Joined: Sun Oct 21, 2012 6:55 pm
Location: Cotswolds, UK
Contact: Website

Re: Interacting between 5 processors problem

Thu Mar 01, 2018 5:21 pm

By processors, I meant CPU's - i.e. there's 4 CPUs to handle my 5 processes so I'd have thought it would just work (t.m.)

Code is 4k lines, but if you can face it, it's on github here: https://github.com/PiStuffing/Quadcopte ... dcopter.py

The main process is the Quadcopter class, the rest of the processes are (Video|Autopilot|GPS|Sweep) | (Manager|Processor).

The problem, as I say, is that all works fine with GPS or Sweep but not both.

TIA
www.pistuffing.co.uk - Raspberry Pi and other stuffing!

PiGraham
Posts: 3926
Joined: Fri Jun 07, 2013 12:37 pm
Location: Waterlooville

Re: Interacting between 5 processors problem

Thu Mar 01, 2018 5:36 pm

This is multiple instance of the python interpreter process, or multiple threads, or are you using the multiprocessing library?
I can't really help but AFAIK there are some oddities to multiple threads and multiple processes with Python.
Appologies if this is not relevant but have you read this:
https://docs.python.org/3/glossary.html ... reter-lock
global interpreter lock
The mechanism used by the CPython interpreter to assure that only one thread executes Python bytecode at a time. This simplifies the CPython implementation by making the object model (including critical built-in types such as dict) implicitly safe against concurrent access. Locking the entire interpreter makes it easier for the interpreter to be multi-threaded, at the expense of much of the parallelism afforded by multi-processor machines.
However, some extension modules, either standard or third-party, are designed so as to release the GIL when doing computationally-intensive tasks such as compression or hashing. Also, the GIL is always released when doing I/O.

User avatar
PeterO
Posts: 5820
Joined: Sun Jul 22, 2012 4:14 pm

Re: Interacting between 5 processors problem

Thu Mar 01, 2018 5:49 pm

Hove wrote:
Thu Mar 01, 2018 5:21 pm
By processors, I meant CPU's - i.e. there's 4 CPUs to handle my 5 processes so I'd have thought it would just work (t.m.)
Exactly.. It's a 5 processes problem, not a 5 processors problem.
Code is 4k lines, but if you can face it, it's on github here: https://github.com/PiStuffing/Quadcopte ... dcopter.py
The main process is the Quadcopter class, the rest of the processes are (Video|Autopilot|GPS|Sweep) | (Manager|Processor).
The problem, as I say, is that all works fine with GPS or Sweep but not both.
TIA
Can you produce a smaller example that displays the same symptoms. I doubt anyone is going to look over 4000 lines of someone else's code.
PeterO
Discoverer of the PI2 XENON DEATH FLASH!
Interests: C,Python,PIC,Electronics,Ham Radio (G0DZB),1960s British Computers.
"The primary requirement (as we've always seen in your examples) is that the code is readable. " Dougie Lawson

User avatar
Hove
Posts: 1205
Joined: Sun Oct 21, 2012 6:55 pm
Location: Cotswolds, UK
Contact: Website

Re: Interacting between 5 processors problem

Thu Mar 01, 2018 6:00 pm

I'll have a go next week after the B'day party in the Cotswolds. My hope was that it might ring a faint bell by someone; I'm fairly sure it's not in the details, but in the interaction of 5 processes over 4 CPUs over shared memory streams. I know, my wishful thinking may be too extreme!
www.pistuffing.co.uk - Raspberry Pi and other stuffing!

IanS
Posts: 248
Joined: Wed Jun 20, 2012 2:51 pm
Location: Southampton, England

Re: Interacting between 5 processors problem

Thu Mar 01, 2018 6:24 pm

Rather than shared memory, I use UDP packets for sending data between python processes so I can split the processes across multiple Pis when I want to. However, the same problem I have needed to overcome may be applicable here.
Are you blocking on input in the autopilot process? e.g. Does it follow the process:
  • Wait for input 1
  • Wait for input 2
  • Process inputs
  • Output
  • Repeat
If either input blocks, waiting for something to be fed to it, then you will never reach the output step, so motion will never see anything.
I had to insert a check for whether there was any input to read before I tried to read something. I am now considering whether to move to using an MQ broker and a publish/subscribe model, if the latency is low enough.

User avatar
Hove
Posts: 1205
Joined: Sun Oct 21, 2012 6:55 pm
Location: Cotswolds, UK
Contact: Website

Re: Interacting between 5 processors problem

Thu Mar 01, 2018 7:08 pm

Thanks for the idea. In my case, a single poll() is used by autopilot to track GPS and Sweep. Similarly, the motion process uses a single poll() to track video and autopilot.

Both GPS and Sweep sensors are watched on a single poll() call by the autopilot process, and processed. The autopilot and the video processes are then watch in a single poll() by the motion process. What's odd is that it's the feed into the autopilot (either but not both GPS & Sweep) seems to be the trigger whether the motion process picks up input from autopilot as well as video.
www.pistuffing.co.uk - Raspberry Pi and other stuffing!

User avatar
Hove
Posts: 1205
Joined: Sun Oct 21, 2012 6:55 pm
Location: Cotswolds, UK
Contact: Website

Re: Interacting between 5 processors problem

Thu Mar 01, 2018 7:14 pm

P.S. I should have said that my use of shared memory streams is the same reason you use UDP; I just chose the shared memory stream as a speculative faster option than UDP as it didn't need to go through the IP stack - all very much guesswork though!
www.pistuffing.co.uk - Raspberry Pi and other stuffing!

PiGraham
Posts: 3926
Joined: Fri Jun 07, 2013 12:37 pm
Location: Waterlooville

Re: Interacting between 5 processors problem

Thu Mar 01, 2018 8:40 pm

Do you have a DSO? It may help to have each process toggle a gpio and monitor the timing of the processes on the scope. That way you add negligible load and latency and can see the sequence and timing of all the processes. In particular you will see the parallelism, if one process is waiting on another and so on.

If you don't have a DSO Joan's PiScope is probably a good option, although it wil put a bit more loading on the system.

User avatar
Hove
Posts: 1205
Joined: Sun Oct 21, 2012 6:55 pm
Location: Cotswolds, UK
Contact: Website

Re: Interacting between 5 processors problem

Thu Mar 01, 2018 8:49 pm

I do have a DSO, I'll give it a go. Thanks for the suggestion.
www.pistuffing.co.uk - Raspberry Pi and other stuffing!

User avatar
Hove
Posts: 1205
Joined: Sun Oct 21, 2012 6:55 pm
Location: Cotswolds, UK
Contact: Website

Re: SOLVED: Interacting between 5 processors problem

Wed Mar 07, 2018 1:39 pm

It's one of those things where symptoms and cause were about of far from each other as possible. If both the GPS and Sweep processes were connected to the Autopilot process, for some reason, the Motion process flushing the shared-memory FIFO stream for the Video process hard-looped, never being empty. Why is a crass error on my part, with the Autopilot stream also being the same poll() used for the Video stream flushing.

Adding the Autopilot to the poll() after the Video flush has completed makes all well.

Ended up tracking it down with Top, finding the Motion process CPU ran at 100% with Sweep and GPS processes in use, but 43% when only one.
Some additional prints in the code made it clear which bit of code was hard-looping, and the trivial fix took seconds.
www.pistuffing.co.uk - Raspberry Pi and other stuffing!

Return to “Advanced users”