py_pol perplexing problems persist (SOLVED)

[sonettguy]

pi-pol perplexing problems

BACKGROUND:
I seem to be marching steadily backwards on trying to use matplotlib and pi_pol with a new user. I can’t seem to get my pi 4 running Buster to find all the files I need. The following is some background, before I ask the question.

First of all, I try a simple matplotlib plot from my new user, which has super user privileges. When I run the simple python3 program, it fails with the following.

Code: Select all

newuser@pi08:~ $ python3 temp_plt02
No protocol specified
qy.qpa.screen: QXcbConnection: Could not connect to display :0.0
Cound not connect to any X display.

So, I exit back to the user “pi” and do the following.

Code: Select all

pi@pi08:~ $ export DISPLAY=:0.0
pi@pi08:~ $ Xhost +

When I go back to my new user and try the simple python3 program again, it works just fine, opening a new window with the plot, just as I specified it. (That is why it is not included in this posting.) But, I do get several warnings in the terminal window.

Code: Select all

LibEGL warning: DRI2: failed to create dri screen
LibEGL warning: DRI2: could not open /dev/dri/card1 (Permission denied)
etc

THE QUESTION:

So, now that I see matplotlib plots from my new user, I try the following program, temp_plt03, taken from https://py-pol.readthedocs.io/en/latest ... care.html.

Code: Select all

#!/usr/bin/env python3
# Poincare Sphere
import matplotlib
matplotlib.use('Qt5Cairo')
import matplotlib.pyplot as plt
import numpy as np
from py_pol import degrees
from py_pol.stokes import Stokes
from py_pol.mueller import Mueller
from py_pol.drawings import draw_poincare_sphere, draw_on_poincare
manifolds=1
inputs=2

s0=Stokes('s_0')
s0.linear_light(angle=0, intensity=2)
print(s0)
s0.draw_poincare(filename='poincare1.png')
plt.legend()

It fails with the following.

Code: Select all

newuser@pi08:~ $ python3 /home/pi/temp_plt03
s_0 = [ +2; +2; +0; +0]

libEGL warning: DRI2: failed to create dri screen
libEGL warning: DRI2: could not open /dev/dri/card1 (Permission denied)
QStandardPaths: wrong ownership on runtime directory /run/user/1000,
1000 instead of 1001 qt5ct: using qt5ct plugin
/usr/lib/python3/dist-packages/matplotlib/tight_layout.py:231:
UserWarning: tight_layout : falling back to Agg renderer
warnings.warn("tight_layout : falling back to Agg renderer") 

newuser@pi08:~$ exit

I try it from user “pi” now, which I think will work…but it doesn’t. It echoes the latter part of the previous error.

Code: Select all

pi@pi08:~ $ python3 temp_plt03
qt5ct: using qt5ct plugin
/usr/lib/python3/dist-packages/matplotlib/tight_layout.py:231:
UserWarning: tight_layout : falling back to Agg renderer
warnings.warn("tight_layout : falling back to Agg renderer") 

Well, what if I try it with the new user along with “sudo”? Nope.

Code: Select all

pi@pi08:~ $ su newuser
Password: 

newuser@pi08:/home/pi $ cd ~

newuser@pi08:~ $ sudo python3 /home/pi/temp_plt03
Traceback (most recent call last):
File "/home/pi/temp_plt03", line 7, in <module>
from py_pol import degrees
ModuleNotFoundError: No module named 'py_pol'

Now, it can’t even find py_pol.

So, I look through the py_pol install which I did previous to all of this, using pip3 install py_pol. I see at the bottom of the install, it has a series of warnings.

Code: Select all

Installing collected packages: scipy, mpmath, sympy, packaging,
sphinxcontrib-htmlhelp, sphinxcontrib-applehelp,
sphinxcontrib-serializinghtml, sphinxcontrib-qthelp, imagesize,
snowballstemmer, sphinxcontrib-devhelp, alabaster, pytz, babel,
sphinxcontrib-jsmath, sphinx, pandocfilters, defusedxml, mistune,
testpath, bleach, attrs, pyrsistent, jsonschema, nbformat, nbconvert,
nbsphinx, h5py, dill, hickle, py-pol [color=#BF0000]The script isympy is installed in
'/home/pi/.local/bin' which is not on PATH. Consider adding this
directory to PATH[/color] or, if you prefer to suppress this warning, use
--no-warn-script-location. The script pybabel is installed in
'/home/pi/.local/bin' which is not on PATH. Consider adding this
directory to PATH or, if you prefer to suppress this warning, use
--no-warn-script-location. The scripts sphinx-apidoc, sphinx-autogen,
sphinx-build and sphinx-quickstart are installed in
'/home/pi/.local/bin' which is not on PATH. Consider adding this
directory to PATH or, if you prefer to suppress this warning, use
--no-warn-script-location. The script jsonschema is installed in
'/home/pi/.local/bin' which is not on PATH. Consider adding this
directory to PATH or, if you prefer to suppress this warning, use
--no-warn-script-location. The script jupyter-trust is installed in
'/home/pi/.local/bin' which is not on PATH. Consider adding this
directory to PATH or, if you prefer to suppress this warning, use
--no-warn-script-location. The script jupyter-nbconvert is installed in
'/home/pi/.local/bin' which is not on PATH. Consider adding this
directory to PATH or, if you prefer to suppress this warning, use
--no-warn-script-location. The script py_pol is installed in
'/home/pi/.local/bin' which is not on PATH. Consider adding this
directory to PATH or, if you prefer to suppress this warning, use
--no-warn-script-location.

Hm. So, I look for instructions on adding this location to PATH. I find the following instructions by Gilles in StackExchange back in 2010.

The first place where PATH is set is /etc/login.defs. There's a setting for root and a setting for everyone else.

Another place where you can define environment variables is /etc/environment. These settings will apply to everyone (you can't write arbitrary shell code there).

A third place where you can define environment variables is /etc/profile. There you can write arbitrary shell code. If you want a user-specific setting, there is the corresponding per-user file ~www-data/.profile. But this will only apply to console interactive logins; in particular it won't apply to cron jobs unless they explicitly source /etc/profile.

If you only need that PATH setting in a user crontab, you can write it at the beginning of the crontab. Note that you need the full list (PATH=/usr/local/bin:/usr/bin:/bin:/usr/local/zend/bin), you can't use a variable substitution (PATH=$PATH:/usr/local/zend/bin won't work there).

So, I try the least intrusive first. I add /home/adi/.local/bin to crontab on my new user. I try it all again. Same result as above. I try adding it to crontab on “pi.” Try it all again; same result. I add it to /etc/profile in the new user. Same result. I try adding it to /etc/environment. There was no such file, so I added it. Tried it all again. Same result. I edited PATH in /etc/login.defs. Same result.

I’m at a loss. What is going on here where I install py_plot so neither my new user nor “pi” can find it?

[scruss]

Are you running another Python installation, or the stock one? Sometimes large packages (I've seen Jupyter do this) want their own whole installation, and they modify your PATH variable to point to their own python3 executable. If the command

Code: Select all

which python3

doesn't return /usr/bin/python3, then something's up with your installation.

There might also be something up with your new user setup. I don't think any of us can replicate that.

Even on a working (not Raspberry Pi, though) installation of PyPol, it gives warnings even when it works:

Code: Select all

 ./pypol.py 
s_0 = [ +2;  +2;  +0;  +0]

/usr/lib/python3/dist-packages/matplotlib/tight_layout.py:231: UserWarning: tight_layout : falling back to Agg renderer
  warnings.warn("tight_layout : falling back to Agg renderer")

This goes away if you comment out the matplotlib.use('Qt5Cairo') line.

[sonettguy]

Thanks for the quick response, Scruss.

Code: Select all

newuser@pi08:~ $ which python3
/usr/bin/python3

So, I'm good with python3. I use it frequently for some pretty complex programs employing numpy. I've never had a problem there. And, ignoring the warnings, matplotlib works just find with my new user ID. I set it up initially with the following.

Code: Select all

sudo adduser newuser
sudo usermod –aG sudo newuser
sudo usermod –aG gpio newuser
su newuser
<password>

Is that much correct?

[sonettguy]

Hmmm. I just noticed that if I change

Code: Select all

matplotlib.use('Qt5Cairo')

to

Code: Select all

matplotlib.use('Agg')

it works differently.

In that case, it just goes back to the prompt without a plot window opening and without an error or warning. I don't know what those represent. Are there different ones for different types of plots?

[sonettguy]

I think I'm being sucked in by a black hole. From https://matplotlib.org/faq/usage_faq.ht ... -a-backend, I see...

Use of an interactive backend (see What is a backend?) permits–but does not by itself require or ensure–plotting to the screen. Whether and when plotting to the screen occurs, and whether a script or shell session continues after a plot is drawn on the screen, depends on the functions and methods that are called, and on a state variable that determines whether matplotlib is in “interactive mode”.

If I follow the instructions for implementing a non-interactive mode drawing, which is supposed to show 3 consecutive plots, with a file as follows,

Code: Select all

import numpy as np
import matplotlib.pyplot as plt
plt.ioff()
for i in range(3):
    plt.plot(np.random.rand(10))
    plt.show()

Then, I get the first drawing only and the program hangs until I hit a <cntl c>. Why is this so hard?

[sonettguy]

The reason the code hangs is because plt.show() is a blocking function, meaning it is intended to be used only once and as the end statement in your code. It doesn't return control to the program until you physically close the pop-up window. I'm trying to sort through hundreds of posts on creating a matplotlib plot you can update. Some say to go interactive [plt.ion()], but that is really not a good solution. It is intended, as the name implies, for someone typing commands one at a time in an interactive terminal. I'm already sucked into the blackhole of endless forum threads. I'm trying to create a continuously updating plot created by reading a martrix data file with a program that runs in the background. Throw me a life-line if you have one.

[scruss]

…In that case, it just goes back to the prompt without a plot window opening and without an error or warning. I don't know what those represent. Are there different ones for different types of plots?

Your code is set to write a png image, which it does. You get poincare1.png in the current folder?

Code: Select all

s0.draw_poincare(filename='poincare1.png')

You're not going to get an interactive plot with that. Also, you import draw_on_poincare but use draw_poincare

I don't use this package, just saw some of the things that were causing errors in your commands.

[scruss]

If I follow the instructions for implementing a non-interactive mode drawing, which is supposed to show 3 consecutive plots, with a file as follows,

Code: Select all

import numpy as np
import matplotlib.pyplot as plt
plt.ioff()
for i in range(3):
    plt.plot(np.random.rand(10))
    plt.show()

Then, I get the first drawing only and the program hangs until I hit a <cntl c>. Why is this so hard?

Because Python's syntax makes it confusing. I think you want:

Code: Select all

import numpy as np
import matplotlib.pyplot as plt
plt.ioff()
for i in range(3):
    plt.plot(np.random.rand(10))

plt.show()

Whitespace makes all the difference in Python. The real language uses DO … END DO loops

[sonettguy]

Yes, I do indeed get the image file and it is correct when I view it from outside the code. Hm. I wonder if I could just keep re-writing the image file and use another program to read it and display it continuously. Seems clunky, but I haven't found a good alternative. And, yes, moving the plt.show() outside the loop seems to be the way matplotlib wants you to use the command. It moves control to the pop-up, so unless it's the last command in the code, you'll hang there until you manually close the pop-up image. Thanks for writing. Your observations are 100% correct. Still seeking a solution.

[scruss]

Still seeking a solution.

Please define clearly what you want to do. It's not just to help us, but it'll help you define when your project works or not. So far, all you've said is that it doesn't work, but none of us know what you mean by “work”.

[sonettguy]

Yes, of course. I am trying to display a continuously updating (however fast it can be updated by the pi) poincare sphere where the sphere (the coordinate map) stays the same, but the points on the sphere change according to data contained in a file containing a multidimensional matrix. The desired program would open the file, read data from it, close the file, display the data, check to see if the file is opened by some other process and, if not, open it again to repeat the process. Thus, the points on the sphere will be updated as often as practical. (Practical means that it doesn't need to be faster than the human eye can keep up with nor does it need to be faster than the pi can keep up with.) Step one is to use matplotlib and py_pol to display the data. That is done. Step two is to make the display update continuously. That is where I am stuck and in need of assistance. To make step two easier, I try to simply update the plot three times using the following loop.

Code: Select all

import numpy as np
import matplotlib.pyplot as plt
plt.ioff()
for i in range(3):
    plt.plot(np.random.rand(10))
    plt.show()

As you rightfully point out, that is not the intended use of the command plt.show(), as I found out the hard way. It is designed to be used only once in any code execution.

Code: Select all

import numpy as np
import matplotlib.pyplot as plt
plt.ioff()
for i in range(3):
    plt.plot(np.random.rand(10))
plt.show()

But, of course, this code only shows the last plot. At least, I think.... Am I wrong?

So, the fact that it is a poincare sphere is irrelevant to this problem, except that it places a need to use the combination of matplotlib and py_pol libraries. So, the solution should not exclude the use of those libraries.

More details: I desire to use python3 (which is what I'm used to). I'm using a pi 4B, 4Gb, running Buster, updated and upgraded. I'm using recent downloads of matplotlib and py_pol.

"To Work" in this case, means displaying a plot (any plot) using matplotlib and updating the curve (not the coordinate system) three times without human intervention.

[sonettguy]

I found an example that works to do what I'm trying to do. It updates (actually, I seem to have been using the wrong terminology...it animates) the plot with recurring new data. It is above my programming skill level so I can't read a single line of code. It looks nothing like the code that produces a sphere. Interestingly, if you combine both in one program, the last line, plt.show(), opens two separate windows and two separate plots. Maybe this has something to do what what matplotlib calls artists? Looks like i have a LOT of learning to do. Maybe I'll have this figured out by next year.

To summarize, I think the solution is somewhere in the following code. For anyone following this thread, look for "animation," not "updating." Specifically search for FuncAnimation().

I know I'm in trouble when I don't even know what three quotation marks mean on the first line....

Code: Select all

"""
================
The Bayes update
================

This animation displays the posterior estimate updates as it is refitted when
new data arrives.
The vertical line represents the theoretical value to which the plotted
distribution should converge.
"""

# update a distribution based on new data.
import numpy as np
import matplotlib.pyplot as plt
import scipy.stats as ss
from matplotlib.animation import FuncAnimation


class UpdateDist(object):
    def __init__(self, ax, prob=0.5):
        self.success = 0
        self.prob = prob
        self.line, = ax.plot([], [], 'k-')
        self.x = np.linspace(0, 1, 200)
        self.ax = ax

        # Set up plot parameters
        self.ax.set_xlim(0, 1)
        self.ax.set_ylim(0, 15)
        self.ax.grid(True)

        # This vertical line represents the theoretical value, to
        # which the plotted distribution should converge.
        self.ax.axvline(prob, linestyle='--', color='black')

    def init(self):
        self.success = 0
        self.line.set_data([], [])
        return self.line,

    def __call__(self, i):
        # This way the plot can continuously run and we just keep
        # watching new realizations of the process
        if i == 0:
            return self.init()

        # Choose success based on exceed a threshold with a uniform pick
        if np.random.rand(1,) < self.prob:
            self.success += 1
        y = ss.beta.pdf(self.x, self.success + 1, (i - self.success) + 1)
        self.line.set_data(self.x, y)
        return self.line,

fig, ax = plt.subplots()
ud = UpdateDist(ax, prob=0.7)
anim = FuncAnimation(fig, ud, frames=np.arange(100), init_func=ud.init,
                     interval=100, blit=True)
plt.show()

[sonettguy]

Update. So, although I thought I was on the brink of solving this problem, I am still unsuccessful. For others stumbling around to learn animation, I wanted to tie the best link I've found thus far. It is a set of 50 animations made with FuncAnimation in matplotlib.

https://www.programcreek.com/python/exa ... cAnimation

I am now wondering if FuncAnimation is written only for two dimensions. I, of course, am working in three. I've tried every combination of coding I can think of and it still does not work. Does anyone have experience with TimedAnimation()? I wonder if backing off the FuncAnimation() command would give me more flexibility.