How to run concurrent long(not longer than a couple minutes) running tasks flask/raspberry pi/mqtt

[mesand]

Hi all,

Here is my dilemma. I have a flask webserver and I have a user input where they enter the number of seconds to turn a piece of hardware on for.

Currently,
I send an ajax request to a route in flask where i send a message to turn on the hardware via mqtt... then in flask I time.sleep(user input) and then send an mqtt message to turn off the hardware.

I don't want to have a while loop in my route because it can make it inefficient and cause problems.

So, now I am trying to implement a redis & rq task que system. My problem is one, I have roughly 7 pieces of hardware to turn off and on, the RQ worker only runs one job at a time. So if I try to turn multiple hardware devices at the same time they turn on and off in sequence rather then at the same time. So from my understanding...I must create more workers. One for each piece of hardware. BUT, now my main worry is what if multiple clients are sending requests to turn on/off the hardware at the same time but on different browsers. Does this mean their job will wait for the other persons job to finish before its executed in the que?

Another idea I have is to to subscribe/publish to some new topics on mqtt on the raspberry pi. I would make a topic for each piece of hardware I want to have a timer for.


In the flask route(inside the while loop i mentioned above)...I also poll for the status of the device... So if I turn it on I dont notify that the pump is on until it is confirmed by the raspberry pi (in the form of a callback message over mqtt that updates a dictionary back on the server with the current state). So if anyone has any idea on how to pool for the devices state without using a while loop that would be helpful!

If anyone has any advice on a simple/effective way to do this I would be very grateful. My main goal is to be able to run these pump timers at the same time, be able to pool for their on/off state after the mqtt message is sent without blocking the app with a while loop, and not let other users requests block eachother in the even I was using redis/rq queue system.

[Heater]

Don't over think it. What about this, use a thread to manage ON/OFF timing:

1) You have a list of 7 elements, one per device. Each element contains the amount of time a device should be on for. It is initialized with zeros.

2) You have a thread that contains an endless loop. On every iteration of the loop it scans the list.
For every non-zero element it sets the corresponding device ON, via MQTT message or whatever you have. It then decrements the value in the element by 1.
For every zero element it sets the corresponding device OFF, via MQTT message or whatever you have.
Having scanned the list the thread sleeps for 1 second.

3) In your Flask web server you have routes that accept ON and OFF AJAX requests.
On receiving an ON request it sets the appropriate element in the list to the desired amount of ON time.
On receiving an OFF request it sets the appropriate element in the list to zero.

You may need some kind of mutual exclusion (mutex) around the list as it is shared between threds. I have no idea how Python deals with this.

You could perhaps do this with ASYNC instead of a thread. Again I have no idea how ASYN works in Python.

In general I would say Python is the wrong language to do this kind of thing in. Makes life far more complex than it need be. In Javascript under node.js such asynchronous handling of events is naturally built into the language.

[mesand]

Very good, thanks for the input.

https://www.youtube.com/watch?v=6ApFx-0Fmvk&t=76s


This is APscheduler and it has a flask extension. It has the ability to pool in the background of the flask application to check some internal state or something. I may utilize your suggestions and use this as the solution to the thread. If you fastforward to 5:15 in the video he has the code written and runs it. It is simple code. He creates an instance of the flask_apscheduler with the app and then runs an interval every 5 seconds.

My only other thought is how this could scale to more than just me using the flask app.

Because lets say now two people are using the app and there are 14 pieces of hardware between the two of us. Would I have to assign each piece of hardware to a specific user in this scenario and now pool the state of 14 devices on the flask app?

Right now I have a service running on the pi that accepts mqtt messages for specific topics, all the topics include a serial number affiliated with a user(right now its just me), making the topic unique to user of each pi. Maybe I will do the timer code on the raspberry pi and use mqtt callback to notify of the state. I dont know tho, I 'll revisit this in the morning since Im pretty confused

[Heater]

The simple way to handle users is to ignore the problem. If there is only a few, trusted, users then just let anyone turn anything on and off. I don;t have locks and keys for the light switches in my house.

If you have lots of untrusted users and you want to restrict access to only what each user is allowed you have a much bigger problem.

You will likely need a data base of users their passwords and what devices what user is allowed to use. You will need some kind of login scheme for your web server. That will dictate using HTTPS, cookies and all the rest.

Meanwhile the actual ON/OFF mechanism remains much the same. Only each element of that timer array will now need to be a structure that contains a user ID along with the remaining ON time.

All depends what you want to do.

[mesand]

Ok perfect. Yes. I have cookies and a login authentication system in place. Where only logged in users (Just me for now) can visit the site to turn on the hardware

Thank you for the advice

[Heater]

Cool. Sounds like you are half way there already.

Let us know how this turns out.