User avatar
QuasiMojo
Posts: 5
Joined: Fri Apr 20, 2018 3:17 pm

PHP web interface to execute python script

Tue Jul 31, 2018 7:19 pm

I have been able to execute a simple Python script with PHP. test.py is a hello world script. However, I want to execute a much more complex Python script that requires arguments and a lot of dependencies (st.py imports both custom and native modules). All the files have been made executable and added to the www-data group. Both st.py and test.py have the shebang line.

Code: Select all

$command  = 'python /home/pi/st/staging/st1.6.1/st.py 1 1';
// $command = python ../testing/test.py; // this works and is normally commented out
$output = exec($command);
echo($output);
The st.py file is a very long running script that could take an hour to complete. I have updated the max_timeout in php.ini to allow unlimited time to run scripts. st.py has print statements that inform the operator the status of operations through a web page. I don't need any help with how to get web stuff to work, I need help to understand why test.py reports as expected and st.py doesn't.

User avatar
DougieLawson
Posts: 33616
Joined: Sun Jun 16, 2013 11:19 pm
Location: Basingstoke, UK
Contact: Website

Re: PHP web interface to execute python script

Tue Jul 31, 2018 9:26 pm

The idea behind a web CGI program is that you have a short lived "transaction" that grabs some input from a web browser, runs a short, one-shot process (no big loops) and sends some HTML, CSS, Javascript, XML and/or JSON back to the requester. The scripting stuff you send back processes the XML/JSON stuff on the requester's machine.

Running what's essentially a long running batch process is outside of the design spec for CGI.
Microprocessor, Raspberry Pi & Arduino Hacker
Mainframe database troubleshooter
MQTT Evangelist
Twitter: @DougieLawson

2012-18: 1B*5, 2B*2, B+, A+, Z, ZW, 3Bs*3, 3B+

Any DMs sent on Twitter will be answered next month.

User avatar
QuasiMojo
Posts: 5
Joined: Fri Apr 20, 2018 3:17 pm

Re: PHP web interface to execute python script

Wed Aug 01, 2018 2:07 pm

Thanks for the response Dougie. I have seen streaming text comments on live YouTube videos that suggest what I want to accomplish is possible.

Digging deeper into the subject, I learned that the CherryPy framework can facilitate HTTP chunked transfer encoding
https://en.wikipedia.org/wiki/Chunked_transfer_encoding
https://stackoverflow.com/questions/224 ... in-real-ti

If you have any caveats regarding the above process, please let me know.

User avatar
DougieLawson
Posts: 33616
Joined: Sun Jun 16, 2013 11:19 pm
Location: Basingstoke, UK
Contact: Website

Re: PHP web interface to execute python script

Wed Aug 01, 2018 5:12 pm

Anything in CGI that runs for more that a few microseconds is taking too long.

The other thing is don't wrap python inside PHP, that's a horrid prospect, python can run as a native CGI program (without too much effort - although it's exceedingly slow compared to C/C++).

The key to adding new text to an existing web page (without reloading the whole page) is AJAX. I use JQuery to do that stuff (because it was easy to learn when there was nothing else like it). Pick a web framework that you can learn easily and use that to dynamically build your output with small chunks of output from your CGI programs.
Microprocessor, Raspberry Pi & Arduino Hacker
Mainframe database troubleshooter
MQTT Evangelist
Twitter: @DougieLawson

2012-18: 1B*5, 2B*2, B+, A+, Z, ZW, 3Bs*3, 3B+

Any DMs sent on Twitter will be answered next month.

User avatar
QuasiMojo
Posts: 5
Joined: Fri Apr 20, 2018 3:17 pm

Re: PHP web interface to execute python script

Wed Aug 01, 2018 7:02 pm

I am using jQuery to display the result of the python script. I've been using jQuery for years and am quite familiar with it. Simple python scripts that finish quickly work fine ie; hello world. I have been tasked with developing a web interface that can send commands to the machine we are building and display status information while it is running. The machine we are attempting to control has a raspberry pi, several stepper motors, arduinos, database and a can-bus. When the machine goes to market, I will need to be able to troubleshoot any problems without having to get on a plane.

I have been in web app dev since '96 and am very familiar with it. However, I am relatively new to IoT and python. I am using PHP to execute a python script. PHP is best for me to display database information as I have already built a framework for that. If you have any framework that you can suggest that can accomplish what I need, I'm all ears.

Thanks again.

User avatar
DougieLawson
Posts: 33616
Joined: Sun Jun 16, 2013 11:19 pm
Location: Basingstoke, UK
Contact: Website

Re: PHP web interface to execute python script

Wed Aug 01, 2018 8:09 pm

If the database is MySQL or SQLite3 then doing stuff in python is trivial

SQLite3 version

Code: Select all

#!/usr/bin/env python
import sqlite3 as sql
import time
import cgi
import cgitb;
import datetime
cgitb.enable()  # for troubleshooting

sensorData = sql.connect('/home/pi/bmp180/sensordata.db')
bmp_data = sensorData.cursor()

endTime = datetime.datetime.now()
sqlEnd = endTime.strftime('%Y-%m-%d %H:%M:59')
startTime = datetime.datetime.now() - datetime.timedelta(minutes=60)
sqlStart = startTime.strftime('%Y-%m-%d %H:%M:00')

print "Content-type: text/html"
print

print """
<html>
<head><title>Current Temperature & Pressure</title></head>
<body>
<h3>Temp/Pressure for Last Hour</h3>
<h4>From %s to %s</h4>
<table>
""" % (cgi.escape(sqlStart), cgi.escape(sqlEnd),)

print """
<tr><th>Date</th><th>Temp</th><th>Pressure</th></tr>
"""

bmp_data.execute("select date_time, temp, pressure from bmp_data where date_time between ? and ?",(sqlStart, sqlEnd))
while True:
    row = bmp_data.fetchone()
    if row == None:
        break


    print """
<tr><td>%s</td><td>%f</td><td>%f</td></tr>
""" % (row[0], row[1], row[2],)

print """
</table>
</body>
</html>
"""
MySQL example (note the similarity apart from that slight syntax difference in parsing the start and end times into my SQL statement)

Code: Select all

#!/usr/bin/env python
import mysql.connector as sql
import time
import cgi
import cgitb;
import datetime
cgitb.enable()  # for troubleshooting

sensorData = sql.connect(user='dougie', password='secretpassword',
                              host='192.168.3.14',
                              database='sensordata')
bmp_data = sensorData.cursor()

endTime = datetime.datetime.now()
sqlEnd = endTime.strftime('%Y-%m-%d %H:%M:59')
startTime = datetime.datetime.now() - datetime.timedelta(minutes=60)
sqlStart = startTime.strftime('%Y-%m-%d %H:%M:00')

print "Content-type: text/html"
print

print """
<html>
<head><title>Current Temperature & Pressure</title></head>
<body>
<h3>Temp/Pressure for Last Hour</h3>
<h4>From %s to %s</h4>
<table>
print """
<tr><th>Date</th><th>Temp</th><th>Pressure</th></tr>
"""

bmp_data.execute("select date_time, temp, pressure from bmp_data where date_time between '%s' and '%s';" % (sqlStart, sqlEnd))
while True:
    row = bmp_data.fetchone()
    if row == None:
        break


    print """
<tr><td>%s</td><td>%f</td><td>%f</td></tr>
""" % (row[0], row[1], row[2],)

print """
</table>
</body>
</html>
"""
Microprocessor, Raspberry Pi & Arduino Hacker
Mainframe database troubleshooter
MQTT Evangelist
Twitter: @DougieLawson

2012-18: 1B*5, 2B*2, B+, A+, Z, ZW, 3Bs*3, 3B+

Any DMs sent on Twitter will be answered next month.

User avatar
QuasiMojo
Posts: 5
Joined: Fri Apr 20, 2018 3:17 pm

Re: PHP web interface to execute python script

Wed Aug 01, 2018 8:22 pm

I have MariaDB/MySQL installed on the Pi and have no problems accessing it with python. My background is PHP/MySQL so it was a walk in the park transitioning to python. Well, sort of. Python has its quirks but has been admirable in doing what we need out of the machine. Would Django meet my needs regarding the long running python script?

User avatar
DougieLawson
Posts: 33616
Joined: Sun Jun 16, 2013 11:19 pm
Location: Basingstoke, UK
Contact: Website

Re: PHP web interface to execute python script

Wed Aug 01, 2018 8:23 pm

I have no idea what Django is or what it does/doesn't do.
Microprocessor, Raspberry Pi & Arduino Hacker
Mainframe database troubleshooter
MQTT Evangelist
Twitter: @DougieLawson

2012-18: 1B*5, 2B*2, B+, A+, Z, ZW, 3Bs*3, 3B+

Any DMs sent on Twitter will be answered next month.

User avatar
QuasiMojo
Posts: 5
Joined: Fri Apr 20, 2018 3:17 pm

Re: PHP web interface to execute python script

Wed Aug 01, 2018 8:52 pm

Well that makes two of us lol. What I know is that it is a python based framework for building web apps. In the meantime I am giving cherrypy a go and will report back. CherryPy is supposed to allow me to change headers so I can maintain a persistent connection. Thanks for your help.

tpyo kingg
Posts: 204
Joined: Mon Apr 09, 2018 5:26 pm
Location: N. Finland

Re: PHP web interface to execute python script

Thu Aug 02, 2018 10:48 am

QuasiMojo wrote:
Tue Jul 31, 2018 7:19 pm
All the files have been made executable and added to the www-data group.
The executables should only be read only and not part of the www-group. So the normal group with 755 permissions is fine to run. The user and group www-data only exist to provide an unprivileged account from which the web server can read files from the document root (or other specifically designated areas). Neither users nor executable files should be in it. Think Write XOR eXecute aka w^x as another layer in secure design.

Also, you might look into using FastCGI instead of CGI. That keeps things more separate still, and can reduce overhead at a busy site.

About your actual question, how much data is being transfered from the server to the client once the task is complete? How important is it to return partial results instead of waiting for a completed run?

Keeping an HTTP connection open for longer than is needed to return a simple response is suboptimal. I usually prefer server-side solutions and thus mention that you can supply a token per job and have the initial page, plus subsequent queries using the same token, return an ETA until the task is complete. Once complete the results can be cached for that token until some time or space limit is reached. I've even seen sites which mail notifications when a long-running task has been completed after some hours or days, but mail is a whole other can of worms.

fbe
Posts: 325
Joined: Thu Aug 17, 2017 9:08 pm

Re: PHP web interface to execute python script

Wed Aug 22, 2018 10:23 pm

QuasiMojo wrote:
Tue Jul 31, 2018 7:19 pm
I have been able to execute a simple Python script with PHP. test.py is a hello world script. However, I want to execute a much more complex Python script that requires arguments and a lot of dependencies (st.py imports both custom and native modules). All the files have been made executable and added to the www-data group. Both st.py and test.py have the shebang line.

Code: Select all

$command  = 'python /home/pi/st/staging/st1.6.1/st.py 1 1';
// $command = python ../testing/test.py; // this works and is normally commented out
$output = exec($command);
echo($output);
The st.py file is a very long running script that could take an hour to complete. I have updated the max_timeout in php.ini to allow unlimited time to run scripts. st.py has print statements that inform the operator the status of operations through a web page. I don't need any help with how to get web stuff to work, I need help to understand why test.py reports as expected and st.py doesn't.
Simply avoid linebreaks in the output of your st.py script. According to the PHP documentation http://php.net/manual/en/function.exec.php exec returns the last line from the result (output) of the command, you execute. You could use the output parameter to get all lines.

Maybe I'm wong. Print two "Hello World!" lines in test.py to check that.

Of course the PHP processor procedes with the "echo($output);" only when the $output = exec($command); has finished. No output from this command will reach the webclient until this happens. You surely poll the intermediate status messages otherwise (e.g. by AJAX calls from your browser). So what is the output good for?

User avatar
topguy
Posts: 4750
Joined: Tue Oct 09, 2012 11:46 am
Location: Trondheim, Norway

Re: PHP web interface to execute python script

Thu Aug 23, 2018 9:29 am

I dont claim to be an expert, but this is probably how I would do it.
- PHP spawns off the pythons script in the background when requested ( no CGI ).
- The Pyhon script will continuously store its status and feedback/output in the database while its running. ( PHP will also check this to prevent running multiple instances of the script probably )
- Any request from the client for showing status of process will be read from database with PHP.

Return to “General programming discussion”

Who is online

Users browsing this forum: No registered users and 5 guests