User avatar
RPi_Mike
Posts: 191
Joined: Sat Dec 09, 2017 12:57 am
Location: United States

Re: STICKY: TUTORIAL: Build the #1 Image Viewer – Automatically in 3 Minutes!

Tue May 21, 2019 12:57 pm

geoffpeters wrote:
Mon May 20, 2019 7:34 pm
This may not be a question you can answer but here goes...

A few times the bottom half of the display changes to all gray. The top of the image is normal.

Any ideas?

NOTE TO OTHER READERS: This person is using a script to export the contents of a FileMaker database to a JPG image. The image file is continuously overwritten at regular intervals and displayed on a large screen to show employees the live status of items at a warehouse. The image display is controlled by my fehshow command. To understand the full context, please start here.

-----
Let me start by saying that no image viewer in the world can fully display an image that does not fully exist!

In the "few times" you saw an incomplete image, feh was almost certainly catching the image "in the act" of being written to the storage device by your script!

For example, if feh happens to reload the image at the exact moment only 70% of the JPG file has been written, then the top 70% of the image would appear normal and the bottom 30% would be gray (indicating "no data").

Feh is doing its job perfectly — it's showing you EXACTLY what the contents of the image file look like at that exact moment!

I of course have no clue what kind of "export script" you're using to generate the JPG file, but it could easily take a good chunk of a full second to complete the write. So if feh is reloading the same image every 10 seconds, for example, you might expect to see some amount of gray about 1% to 10% of the time.

Writing a file can take a surprisingly long time — especially when you consider that writing a file isn't necessarily just "writing" a file. Depending on what the software is doing and how it behaves, it could actually be a matter of "process 10%, write 10%; process the next 10%, write the next 10%".

As an extreme example, it could take 40 days to write a single JPG file if the processing component involved ray tracing a complex 3D model with photorealistic lighting. The "actual time spent writing" might only amount to 1/10 of a second, but the "total time to complete the write" would still be 40 days. If you opened the JPG file after a full 20 days, half the image would still be gray! That's not the fault of the image viewer — that's just the nature of the situation.

Fortunately, I have a solution for all of this.

One method would be to "auto detect" when 100% of the image file has been written — and only then trigger feh to display the completed image. That way, only completed images would ever be displayed. But the code to do that would first have to kill the current instance of feh before re-launching a fresh instance of feh to display the newly-updated image. Not only is "auto-detect / kill / re-launch" somewhat complicated, but it would introduce an annoying screen flicker every single time the entire feh application had to be closed and re-launched from scratch.

A better method is to DRAMATICALLY REDUCE the chance of feh catching an image file "in the act" of being written.

You should still use feh's INTERNAL "--reload" feature to refresh the image every 10 seconds or 5 seconds or whatever you want (as I explained in my previous post). This will not cause any screen flicker because it's NOT reloading feh itself — it's simply reloading the image within feh!

[Side Note: The longer the time between "reloads", the less likely that feh will ever catch the image "in the act" of being written. For example, if you set feh to reload the image every 600 seconds — and you didn't do anything else — that alone would make it VERY RARE to catch an incomplete image. But I realize you probably want a fairly rapid reload rate, such as 5 or 10 seconds, which is why I'm taking a much deeper dive on this subject.]

So we need to conceal the processing component of "writing" an image AND greatly minimize the actual time it takes to write the data itself. In other words, we need to "hide" as much file-writing activity from feh as we possibly can — so that feh rarely ever sees it!

We can accomplish both of these goals by creating 2 high-speed RAM disks — which I will refer to as "RAM folders". I call them folders because they "look and feel" just like regular folders on the Raspberry's internal SD card. But unlike regular folders, these are high-speed folders for temporary data — where all the data is kept inside the much faster RAM memory chip, instead of a slower storage device like the flash-based SD card.

In effect, we can use the 2 RAM folders to create a 2-stage concealment of the writing activity. The 1st folder will hide 100% of the "processing" part of writing the file — and the 2nd folder will hide about 90% of the "writing" part of writing the file!

However, be aware that since these special folders use "volatile" RAM, any reboot or loss of power will cause everything inside them to disappear forever. But that should not be a problem for your particular application. It's pure upside with no downside!

Beyond its faster speed, a RAM folder has the added benefit of reducing wear and tear on your storage device. The cells in flash storage, for example, can eventually break if repeatedly written to over and over again. RAM memory does not have that problem.

I'll explain how to create the RAM folders in just a moment — but here's the overall strategy:

First, modify your export script to write the JPG file to RAM_Folder_1.

Then, in the next line of your export script, make your code "sleep" for about 2 seconds. The idea is to make sure everything related to "processing + writing" the JPG file has fully settled down and finished before anything else is done. If you're using Bash, use "sleep 2" — or whatever time interval you believe to be sufficient to completely finish the JPG export/writing process.

Then, in your next line of code, simply copy the completed JPG file from RAM_Folder_1 to RAM_Folder_2. Assuming it's Bash, use the cp command.

Finally, simply point my fehshow command to RAM_Folder_2 in the "auto start" .desktop file you previously created — as described in my prior post. And then reboot your system.

With this strategy, the only opportunity feh will ever have to catch the JPG file "in the act" of being created is the TINY amount of time it takes to copy a single small file from one RAM folder to another RAM folder! This should make the display of incomplete images go from "a few times" to VERY RARELY!

Now that I've explained the concept, we can create the two RAM folders. I've assigned each of them a maximum capacity of 10 MiB — by setting "size=10m" in Step 3 below. That's plenty of room to handle a single 1080p JPG file. And thanks to the tmpfs facility, each folder will NOT gobble up 10 MiB of RAM. Instead, it will only use what it needs — up to the maximum limit of 10 MiB. So if your JPG file is only 1 MiB, it will only use 1 MiB of RAM!

Note: 10 MiB (mebibytes / binary) is almost 10.5 MB (megabytes / decimal).


STEP 1: CREATE THESE 2 FOLDERS:

mkdir /home/pi/RAM_Folder_1

mkdir /home/pi/RAM_Folder_2


STEP 2: OPEN THE FSTAB FILE IN LEAFPAD:

sudo leafpad /etc/fstab


STEP 3: CAREFULLY ADD THESE 2 LINES AND SAVE THE FILE:

tmpfs /home/pi/RAM_Folder_1 tmpfs rw,noatime,size=10m,mode=755,uid=pi,gid=pi
tmpfs /home/pi/RAM_Folder_2 tmpfs rw,noatime,size=10m,mode=755,uid=pi,gid=pi


STEP 4: REBOOT!


That's it!

From now on, those 2 regular-looking folders will be high-speed RAM folders! Change your "export script" to make use of them as I described — and, of course, point my fehshow command to RAM_Folder_2 in the "auto start" .desktop file.

Let me know how this technique works for you!

-----
PS: You seem to prefer my fehshow command — perhaps because it lets you point to an entire folder without having to worry about the exact file name of the single JPG image inside it. But if you do know the exact file name and it doesn't change, it would be more optimal to use my fehview command instead. In theory, by pointing to a specific file, fehview won't waste time assessing what's in the folder. I have not done a performance comparison, but it's certainly possible that it might shave several milliseconds off the reload. Shrinking the "time window" for the image reload would make it even more rare for it to overlap the "time window" for the file writing. For your single-image scenario, the outward behavior of both commands would be the same — both provide full-screen display, etc. If you do switch to my fehview command, be sure to update the auto-start .desktop file by changing the command to fehview while including the full path to the file — and be sure to add "--reload 10" (or whatever time interval) to the feh command line inside "Image Viewing Settings". My advice on using the high-speed RAM folders still applies, no matter what command you use.

retrkdr
Posts: 7
Joined: Sun Dec 10, 2017 10:04 pm

Re: STICKY: TUTORIAL: Build the #1 Image Viewer – Automatically in 3 Minutes!

Wed Jun 05, 2019 8:45 pm

I was having the devil's own time trying to get this to work. Luckily my new RPi3B+ arrived and I put it together with a corded keyboard and it worked perfectly. I switched to the wireless keyboard and discovered the right click key had issues. After full nuke and using rm -r Feh_Build 3 times, I felt pretty stupid.

Thank you. This is a really nice addition/replacement for watching a whole lot of pictures.

User avatar
RPi_Mike
Posts: 191
Joined: Sat Dec 09, 2017 12:57 am
Location: United States

Re: STICKY: TUTORIAL: Build the #1 Image Viewer – Automatically in 3 Minutes!

Thu Jun 06, 2019 9:08 pm

retrkdr wrote:
Wed Jun 05, 2019 8:45 pm
Thank you. This is a really nice addition/replacement for watching a whole lot of pictures.

I appreciate you saying that.

You're welcome!

geoffpeters
Posts: 10
Joined: Wed May 15, 2019 5:23 pm

Re: STICKY: TUTORIAL: Build the #1 Image Viewer – Automatically in 3 Minutes!

Tue Jul 02, 2019 5:23 pm

This is really not a reply but here goes...

I have had this up and running for a while, working well, and just recently I'm seeing a failure.

To refresh, there is only one .jpg in the folder that fehshow is pulling from.

What is happening is that the application quits and exits to the desktop.

What would you suggest I do to troubleshoot?

Thanks,
Geoff

User avatar
RPi_Mike
Posts: 191
Joined: Sat Dec 09, 2017 12:57 am
Location: United States

Re: STICKY: TUTORIAL: Build the #1 Image Viewer – Automatically in 3 Minutes!

Thu Jul 04, 2019 12:19 am

geoffpeters wrote:
Tue Jul 02, 2019 5:23 pm
I have had this up and running for a while, working well, and just recently I'm seeing a failure.

NOTE TO OTHER READERS: This person is using my fehshow command to display inventory data on a big screen at a warehouse. There's a whole backstory to this — so please check out the prior posts on this thread if you'd like to know more.

-----
To be honest, the information you provided is far too vague and generic for me to suggest anything.

I can't help but notice, however, that you've been using my command for two months without incident — but now, suddenly, you're having issues "just recently".

That clearly suggests an unknown change has occurred in your overall system that has nothing to do with my command.

In fact, just minutes ago, I performed the "gold standard" test of my code on a brand-new Raspberry Pi 4 with the equally brand-new Raspbian Buster operating system — in case that's what you meant by "just recently". Both products are only 9 days old.

To conduct my test, I pretended to be a total beginner and simply grabbed my own script from this website and pasted it into Terminal.

Everything still works perfectly!

When it comes to your warehouse, keep in mind that my command is being used inside the much larger context of an UNKNOWN script that you or someone connected with the warehouse has created. I obviously don't know anything about that code and how it functions — nor would it be appropriate to get into an off-topic discussion on this thread about a warehouse script that happens to use my command in a purely incidental manner.

I also took the time, several weeks ago, to write an in-depth explanation of how you can use a RAM drive to greatly speed up the file-writing time for the JPG image that your script is generating from the warehouse database — yet you never responded. Nor have you made any reference in your latest post to whether you're even using that technique.

So as you can probably tell, I'm justifiably reluctant to get pulled into the weeds on a topic like this.

I'll close with a philosophical statement I wrote some time ago — it does a good job at explaining my overall perspective. It appears, in red, at the bottom of the post.

geoffpeters
Posts: 10
Joined: Wed May 15, 2019 5:23 pm

Re: STICKY: TUTORIAL: Build the #1 Image Viewer – Automatically in 3 Minutes!

Thu Jul 04, 2019 7:17 pm

Hi Mike,

I do want to thank you for taking time to provide such a detailed explanation and solution to the image display issue. I actually did not see it until earlier this week. I had some other projects which have been higher priority and no one has been complaining about the "half screen" problem. I understand your solution and I will certainly give it a try. It makes sense.

As far as the new problem is concerned, all I was hoping for was to get an idea of what might cause the program to quit and revert to the desktop. I do understand that there may be numerous causes and I'm just looking for a clue.

Thanks,
Geoff

hydra3333
Posts: 107
Joined: Thu Jan 10, 2013 11:48 pm

Re: STICKY: TUTORIAL: Build the #1 Image Viewer – Automatically in 3 Minutes!

Sun Jul 07, 2019 10:42 am

Nice work.
I see feh is at version 3.1.3 ...
https://feh.finalrewind.org/
Is it worth considering updating the original script ?
Thank you.

User avatar
RPi_Mike
Posts: 191
Joined: Sat Dec 09, 2017 12:57 am
Location: United States

Re: STICKY: TUTORIAL: Build the #1 Image Viewer – Automatically in 3 Minutes!

Mon Jul 08, 2019 1:20 am

hydra3333 wrote:
Sun Jul 07, 2019 10:42 am
Nice work.
I see feh is at version 3.1.3..... Is it worth considering updating the original script?

As of this writing, the most recent version of feh — the one you reference — is only 3 months newer than the one my script builds: February 2019 vs. November 2018.

A review of feh's changelog reveals that only a few obscure changes were made in that short period of time, so I don't see a compelling reason to update it — at least for now.

In theory, I could easily change just a few characters in my script to build the latest version — by simply changing the source code URL and a few other things. But I hold myself to a high testing standard. So I'd have to start from freshly-nuked copies of both Raspbian Stretch and Buster — and thoroughly test everything, separately, on both the Raspberry 3 and 4.

Stretch won't run on the model 4, of course — but that still leaves 3 separate hardware / operating system combinations in the "modern Raspberry era". A quality testing process for all those combinations could easily take several hours. And all of that effort would be for what? Based on the changelog, it's clear that all that effort would not be warranted.

I've tested my existing script, however, on all possible combinations of Stretch and Buster — on both the Raspberry 3 and 4. I'm pleased to report that everything still works great!

In the future, if a significant enhancement of feh's source code emerges, I'll certainly update my script.

In the mean time, anyone is free to modify, use and update my script however they wish.

User avatar
RPi_Mike
Posts: 191
Joined: Sat Dec 09, 2017 12:57 am
Location: United States

Re: STICKY: TUTORIAL: Build the #1 Image Viewer – Automatically in 3 Minutes!

Mon Jul 08, 2019 2:00 am

geoffpeters wrote:
Thu Jul 04, 2019 7:17 pm
I do want to thank you for taking time to provide such a detailed explanation and solution to the image display issue..... I understand your solution and I will certainly give it a try. It makes sense.

As far as the new problem is concerned, all I was hoping for was to get an idea of what might cause the program to quit and revert to the desktop. I do understand that there may be numerous causes and I'm just looking for a clue.

NOTE TO OTHER READERS: This person is using my fehshow command INSIDE an unknown script at a warehouse. There's absolutely nothing wrong with my command — but if you'd like to know the full "warehouse story", please review this entire thread for the multiple exchanges I've had with this individual.

-----
Hypothetically, if you hired me as a consultant and I showed up at your warehouse to get to the bottom of things, my approach would be to "not accept the problem" as it was being presented to me.

In other words, I would not make the mistake of getting sucked into an unknown software environment — on an unknown device — with an unknown history — running unknown code.

This is especially true because of the giant elephant in the room: Everything was working well for two months — and now, suddenly, you're having strange problems!

For all I know, some dude at the warehouse snuck over to the Raspberry while you were on your lunch break and did a bizarre thing he calls "the freaky" — hacking the kernel with some weird A.I. code that deliberately creates inexplicable problems!

Or, as a much more mundane and likely example, perhaps your SD card is starting to wear out and do weird things from the thousands of writes and over-writes of the same JPG file that your unknown script is constantly burning to the same SD card — over and over again every 10 seconds — 12 hours a day — 6 days a week — or whatever the exact schedule of your warehouse is.

In theory, modern SD cards are supposed to do a good job at "wear leveling" — spreading around the areas being written to so that the same group of cells aren't getting repeatedly hammered. But maybe the algorithm in the tiny microcontroller chip hidden inside your SD card is not doing such a good job. Or maybe it's just a cheap SD card — or a name-brand card from a bad batch. Or maybe it was once a good card that's simply been "abused" a bit too much.

Or maybe at the end of the day, people are just pulling the plug on the Raspberry or flipping a wall switch without first doing a proper, software-based shutdown. That could easily corrupt files and create an unstable environment.

And then there's the code inside the mysterious warehouse script. Who knows what that's doing!

You get the idea — I could easily go on and on about all the possibilities.

So if I showed up at your warehouse, I'd follow this basic strategy:

BYOR — Bring Your Own Raspberry: No way would I automatically assume that your Raspberry is in perfect condition. So right off the bat, I'd eliminate the possibility of weird hardware issues by bringing my own Raspberry — a "known good" Raspberry!

BYOPSU — Bring Your Own Power Supply Unit: Maybe your warehouse got hit with a power surge that did some weird stuff to your power supply that causes it to generate an unsteady flow of electricity. The official Raspberry PSU is very small — about the size of a golf ball — so I'd have to be crazy not to bring that along too. And if I wanted to make this a reliable set-up for long-term industrial use, I'd also bring a high-end surge protector AND a UPS — Uninterruptible Power Supply. After all, many warehouses lose power for at least a brief second every year. What if the Raspberry was in the middle of writing data at that moment? That wouldn't be good!

BYOSDC — Bring Your Own SD Card: Who knows what the condition of your card is. I'd immediately kill that unknown variable by showing up with a high-quality, brand-new card!

BYOOS — Bring Your Own Operating System: I would never automatically assume that the existing Raspbian operating system at your warehouse was perfectly normal, undamaged and unmodified — and then just dive right in to my troubleshooting without questioning the entire foundation of EVERYTHING. NO WAY! So before I did anything, I'd first burn a brand-new copy of Raspbian on the brand-new SD card that I brought with me!

I'd also use the dd command to generate a validation image directly from the newly-burned card — and then perform a "binary compare" with the source image to confirm that every single bit of the 32 billion bits (4 GB) was mathematically identical! That may sound complicated, but it only takes about 5 minutes to achieve that level of extreme certainty. The Raspberry Pi Foundation describes this procedure at the bottom of this page.

BYOBTIETS — Bring Your Own Brain To Independently Evaluate The Script: I would never automatically assume that whoever wrote the warehouse script did everything perfectly. So I would carefully scrutinize all the code, line-by-line. I'd probably end up re-writing some of it as well.

So there ya go — that should give you a solid "30,000 foot" view of how to approach things.

At a minimum, you should get a brand-new, high-quality SD Card. I personally like the state-of-the-art SanDisk Extreme Pro A2 microSDXC card. But make sure you get it from a reputable source — because there's a big market out there for forgeries of name-brand cards.

When ordering on Amazon, for example, most people use whatever seller is automatically chosen by Amazon's algorithm. That can be a big mistake — especially on an easy-to-counterfeit item like an SD card!

[By easy-to-counterfeit, I don't mean electronically or physically reproducing the complicated stuff on the inside — I mean simply taking a cheap SD card and cosmetically changing the outer plastic shell. That, I suspect, is relatively easy to do and would fool almost anyone. To be clear, Amazon would have no involvement in any of this — but that doesn't mean that at least some of the numerous independent sellers might be doing some shady things.]

So on the product page, be sure to click the tiny link that says something like the following. Here's what it currently says for the 64 GB version of the SanDisk A2 card:

Used & new (19) from $18.99 & FREE shipping on orders over $25.00.

Once you do that, you'll see all the available "sellers". Carefully review each one. Several may offer free shipping with "Fulfillment by Amazon" — but you'll also see that the ratings of the individual sellers can vary dramatically. Some may have only 300 reviews in the last year with a 92% rating. But others may have 30,000 reviews with a 100% rating! That's a huge difference — in both "sample size" and rating.

Once you get a brand-new SD card, you should burn a completely fresh copy of Raspbian Buster Desktop — and then fully update it with sudo apt-get update / sudo apt-get dist-upgrade.

Only then should you proceed with copying over your mysterious warehouse script — and separately running my automated build and set-up script.

Finally, I can't stress enough that your warehouse script should not be constantly writing to the SD card in the first place — for reasons that go far beyond the wear issue. Instead, all of the writing should occur on two separate RAM drives, as I explained in great detail in my earlier post.

Good luck — let me know the outcome!

geoffpeters
Posts: 10
Joined: Wed May 15, 2019 5:23 pm

Re: STICKY: TUTORIAL: Build the #1 Image Viewer – Automatically in 3 Minutes!

Mon Jul 08, 2019 7:07 pm

Thanks again for the detailed response. What I was hoping for was that there might be a log file or files associated with FEH that would help in the debugging process.

I believe the problem was due to a change I made in my Filemaker script and not in your code. I'm testing that now. I guess I should have made my inquiry more specific in regards to any debugging aids. I was specifically looking for information regarding what conditions would cause the program to exit to the desktop. I still do not understand the mechanism.

For clarification, I am not running your program inside my script. Filemaker and its scripts are running on a Windows workstation and exporting images to the RPi.

Your code is doing what you intend it to do.

I will be setting up the RAM drives as you suggest, I had not considered the uSD card wear problem. Thanks for pointing it out.

Geoff

hydra3333
Posts: 107
Joined: Thu Jan 10, 2013 11:48 pm

Re: STICKY: TUTORIAL: Build the #1 Image Viewer – Automatically in 3 Minutes!

Tue Jul 09, 2019 4:35 am

I enjoy reading the detailed responses that RPi_Mike gives.
Nice work.

User avatar
RPi_Mike
Posts: 191
Joined: Sat Dec 09, 2017 12:57 am
Location: United States

Re: STICKY: TUTORIAL: Build the #1 Image Viewer – Automatically in 3 Minutes!

Tue Jul 09, 2019 7:43 pm

hydra3333 wrote:
Tue Jul 09, 2019 4:35 am
I enjoy reading the detailed responses that RPi_Mike gives.
Nice work.

I'm glad you like what I do.

Sometimes, I like to "go off" in great detail — especially if it has broad application to Linux or the Raspberry, rather than dealing with some narrow or obscure issue. A basic troubleshooting philosophy would certainly be a good example of general usefulness.

When I happen to be in just the right mood, my writing is effortless. It's like my fingers are on autopilot and the words just materialize.

At other times, however, it feels like a real grind. It all depends on my state of mind.

User avatar
RPi_Mike
Posts: 191
Joined: Sat Dec 09, 2017 12:57 am
Location: United States

Re: STICKY: TUTORIAL: Build the #1 Image Viewer – Automatically in 3 Minutes!

Tue Jul 09, 2019 8:14 pm

geoffpeters wrote:
Mon Jul 08, 2019 7:07 pm
Your code is doing what you intend it to do..... I believe the problem was due to a change I made in my Filemaker script and not in your code. I'm testing that now. I guess I should have made my inquiry more specific in regards to any debugging aids..... Filemaker and its scripts are running on a Windows workstation and exporting images to the RPi.

Microsoft Windows?

The "warehouse story" is definitely getting way off topic for a website about the Raspberry!

So not only is the warehouse script an unknown script, it's also a script that's running on an ALIEN operating system on an alien computer! And unless you happen to be running a third-party Bash emulator like Cygwin or the Windows Subsystem for Linux (WSL), your script would also be written in an alien language — like Microsoft PowerShell, for example.




geoffpeters wrote:
Mon Jul 08, 2019 7:07 pm
What I was hoping for was that there might be a log file or files associated with feh that would help in the debugging process.

Just to answer your question for purely academic purposes, you can build a special debug version of feh by (1) completely and properly uninstalling your existing copy of feh and then deleting the Feh_Build folder, as described in Warning #1 of my main tutorial; (2) adding "debug=1" to the make line of my build script and re-running it to automatically re-compile feh; (3) using the newly-added "--debug" option in the feh command line to do your "debugging".

Have I ever used or tested a debugging build of feh? No, I have not. And there's a good reason for that: Everything works perfectly on my end, so there's nothing to "debug" — nor is there anything to "debug" for the unknown hundreds or even thousands of people around the world that have used my image-viewing solution without reporting a single problem.

But let's step back for a moment and look at the big picture:

You sound fairly advanced in that you're even talking about "debugging" feh.

But ironically, it's the more advanced users that are also more prone to contemplate overly technical and unnecessarily complicated "solutions". If I were you, I would focus on the "change" you made to the alien system that caused everything to suddenly deteriorate after two months of successful operation.

I would also completely re-architect the script on the alien system so that it's writing exclusively to the Raspberry's RAM drives — not the SD card.

Remember, it's not just the SD card wear issue — which by itself is potentially huge. There's also a second, equally profound reason why you should use the RAM drives:

My two-stage RAM drive technique HIDES almost all of the file-writing "sausage making" from feh — thus making it much less likely that feh will ever get disturbed by whatever your alien system is doing in the first place! So I strongly encourage you to carefully re-read my lengthy explanation on how and why you should use RAM drives.

In your particular scenario, a two-stage RAM drive is a "win win win win" — it wins on speed; it wins on eliminating wear; it wins on concealing file-writing activity from feh; and it wins on reliability!

geoffpeters
Posts: 10
Joined: Wed May 15, 2019 5:23 pm

Re: STICKY: TUTORIAL: Build the #1 Image Viewer – Automatically in 3 Minutes!

Wed Jul 10, 2019 5:50 pm

Well, I have no options as Filemaker will not run on a Raspberry Pi system. The script that exports the image file is a Filemaker script.

User avatar
RPi_Mike
Posts: 191
Joined: Sat Dec 09, 2017 12:57 am
Location: United States

Re: STICKY: TUTORIAL: Build the #1 Image Viewer – Automatically in 3 Minutes!

Thu Jul 11, 2019 1:35 am

geoffpeters wrote:
Wed Jul 10, 2019 5:50 pm
Well, I have no options as Filemaker will not run on a Raspberry Pi system. The script that exports the image file is a Filemaker script.

I realize "it is what it is" — the other system at your warehouse is not a Raspberry.

For good or for bad, that's what you're dealing with.

I assume, however, that the FileMaker scripting language at least gives you the option of changing the output path for the JPG file that's being written. Or is that not even true?

For example, let's say the JPG is currently being written to the Raspberry's internal SD card at /home/pi/Pictures. Are you at least able to change the export path so that it's written to the Raspberry's more robust RAM chip instead? The idea is that after following my previous RAM drive instructions, you would simply change the output path to /home/pi/RAM_Folder_1.

If you can at least do that, it would be a major improvement in reliability. It would at least accomplish "stage 1" of my two-stage RAM drive technique.

This begs a critical question, however:

How, exactly, are you "transmitting" the JPG file from the Microsoft Windows system to the Raspberry? Is the Windows system writing to the Raspberry via a shared, hard-wire Ethernet network at the warehouse? For all I know, the file is being wirelessly FTP'd via ad-hoc WiFi — or some other method I'm not even contemplating. Please be as detailed as possible in your answer.

In fact, the details are so limited on what you're actually doing that it just occurred to me that the JPG file may not even physically end up on the Raspberry! In other words, for all I know, there may not even be any file writing taking place — at least not on the Raspberry. Is it possible that you're pointing my fehshow command to a "shared network folder" on a NAS server on the warehouse network? Is that where the JPG is being exported to?

Please let me know exactly what's going on — because I just came up with a solution that could make feh extremely happy by COMPLETELY hiding the file-writing "sausage making". But there's no reason for me to develop that solution if your overall set-up is totally different from what I think it is!

One last, critical thing:

Can you change the exported image format from JPG to BMP? That, potentially, could be a huge breakthrough — because the exact size of a JPG file will constantly change. Even though it's an image of the same database screen over and over again, the JPG compression algorithm will cause the file size to slightly fluctuate — because the contents of the database itself are also fluctuating.

A standard BMP, however, typically has no compression — so when the exporting and writing process is 100.000% complete, the file will always be the EXACT SAME SIZE no matter what. Even if the content of the image suddenly changed from a database to a herd of elephants, the completed file size would still be identical!

Do you see where I'm going with this?

If we can rely on the completed file size always being the same, it would be very easy to write a tiny script on the Raspberry that continuously "probes" the image file to make absolutely certain the writing process has totally finished before feh is even allowed to look at it!

I'd be happy to write the script — it would use a regular expression to continuously grep the output of the stat command and extract the file's exact byte size. It would then use that value to conditionally determine whether the file is truly complete — and therefore whether it gets copied to RAM folder #2. That way, all the images that end up in RAM folder #2 would be TOTALLY COMPLETE — not just 99.9% of the time, but 100.000% of the time!

[Technically, there would have to be a "two-stage confirmation" that the file is truly complete before it finally moves on to its final folder. That's because there would always be a tiny gap in time of X microseconds between the assessment of the file size and when the file is copied to the next folder. If an overwrite happened to occur during that tiny window of time, the assessment of the file size would no longer be valid and therefore an incomplete file would slip through my filter. But a two-stage confirmation completely solves that issue. So the final, confirmed file would actually have to end up in a RAM folder #3. This more advanced method would therefore be called my three-stage RAM drive technique! But all the technicalities are no big deal — I would take care of everything in the automated script I'm proposing.]

But I need to know the exact details of your set-up before I can determine whether any of this is even possible!

By completely HIDING the file-writing "sausage making" from feh, your warehouse would have a very robust database display system.

geoffpeters
Posts: 10
Joined: Wed May 15, 2019 5:23 pm

Re: STICKY: TUTORIAL: Build the #1 Image Viewer – Automatically in 3 Minutes!

Thu Jul 11, 2019 3:29 pm

Once again thanks for the detailed information.

I set up a Samba server on the Rpi and I'm sharing a folder. I will need to see if I need to configure the share differently if I'm using a RAM drive. I want to try that as a next step.

The Filemaker script allows me to export to any local or network location. I have the script exporting to the Samba share.

Occasionally we will add one or more other images to the shared folder on the RPi. For example, if the parking lot will be closed for some reason (paving?) we will post that information as part of the fehshow rotation. These are temporary and are removed after the event is completed.

I'll report back here with an update after I set up the share as a RAM drive.

Geoff

User avatar
RPi_Mike
Posts: 191
Joined: Sat Dec 09, 2017 12:57 am
Location: United States

Re: STICKY: TUTORIAL: Build the #1 Image Viewer – Automatically in 3 Minutes!

Fri Jul 12, 2019 12:15 am

geoffpeters wrote:
Thu Jul 11, 2019 3:29 pm
I set up a Samba server on the Rpi and I'm sharing a folder. I will need to see if I need to configure the share differently if I'm using a RAM drive. I want to try that as a next step.

The Filemaker script allows me to export to any local or network location. I have the script exporting to the Samba share.

Occasionally we will add one or more other images to the shared folder on the RPi. For example, if the parking lot will be closed for some reason (paving?) we will post that information as part of the fehshow rotation. These are temporary and are removed after the event is completed.

I'll report back here with an update after I set up the share as a RAM drive.

So a Samba file-sharing system is in the mix too! And it turns out that images unrelated to the database are also involved!

I wasn't too far off when I mentioned a picture of a herd of elephants in my previous post. Except in this case, it's an image about a parking lot instead of elephants!

When you first started asking your seemingly-uncomplicated questions about a slideshow two months ago, I realize you probably felt these numerous other details were not relevant — so it may have seemed perfectly reasonable to not bother mentioning them.

But you have to understand that from my equally reasonable perspective, this has been a steady "drip drip" of critical details that have slowly leaked out in the course of our conversation.

Nonetheless, your warehouse project has managed to intrigue me — which is why I've been pressing on with this.

Let me point out two things:

First: I have no personal experience with a Samba set-up — so I can't give you any competent advice on that. The reason is simple — I've never had a need for Samba! It's definitely a subject that falls far outside the topic of image viewing.

Second: I'm still fascinated by the potential of my three-stage RAM drive technique — whether or not it's applicable in your particular case. So I wanted to elaborate a bit on the third and final stage. The only requirement is that it has to be a standard ext4 Linux file system.

Believe it or not, it's actually possible to make the image file appear INSTANTANEOUSLY in RAM folder #3!

It's the Linux equivalent of a "quantum leap". In other words, there would not be the tiniest window of time where feh could catch the file transitioning from one state to another — not even a single nanosecond! There would simply be one moment where it was NOT a recognizable image file — and then the next moment, it would suddenly appear as a fully complete image file!

How is that even possible? How can a file pop into existence not just super fast — but INFINITELY FAST from the standpoint of feh?

It can easily be done with an automated Bash script. It would copy the file from RAM folder #2 to RAM folder #3 with the cp command — but with a fake extension that's unrelated to image files, like Database.txt instead of Database.bmp.

Then, in the final step, the file would be renamed back to Database.bmp with the mv command.

In Linux, file renaming is an "atomic" operation — it changes a fundamental building block of the file's metadata.

So there is never a "transition state" between old and new where the file can get caught with its pants down. One moment it's the old name — and the next moment it's the new name! There is no in-between or partial state. The file name either exists or it doesn't.

This is very similar to the atomic concept in physics.

In physics, when it comes to the thing we know as "gold", there is nothing smaller or more fundamental than a single atom of gold — or hydrogen or carbon or any other element. Sure, the atom itself is made of quarks and electrons — which in turn MIGHT be made of even smaller things called strings. Although these things are smaller and more fundamental than gold, they are NOT more fundamental when it comes to the thing we know as "gold" — because anything below the level of an atom of gold is no longer gold!

As I understand it, the mv command's core behavior conforms with the rename() function. Here's an excerpt on atomic behavior from section 14.7 of the official GNU libc manual:

One useful feature of rename is that the meaning of newname changes “atomically” from any previously existing file by that name to its new meaning (i.e., the file that was called oldname). There is no instant at which newname is non-existent “in between” the old meaning and the new meaning. If there is a system crash during the operation, it is possible for both names to still exist; but newname will always be intact if it exists at all.

innocent_bystander
Posts: 45
Joined: Mon Oct 15, 2018 12:15 am
Location: Florida, USA

Re: STICKY: TUTORIAL: Build the #1 Image Viewer – Automatically in 3 Minutes!

Thu Jul 25, 2019 8:29 pm

Not sure if you are aware, but feh 3.1.3 is available in Debian Buster repositories. To install, simply run:

Code: Select all

wget http://ftp.debian.org/debian/pool/main/f/feh/feh_3.1.3-1_armhf.deb
sudo apt install ./feh_3.1.3-1_armhf.deb

No need to compile anything or pollute your system with dev packages or unmanaged binaries.

Just my 2 cents. :)
The Sirius Cybernetics Corporation is the primary manufacturer and supplier of androids, robots and autonomic assistants for the known universe. ... The company motto is "Share and Enjoy."

User avatar
RPi_Mike
Posts: 191
Joined: Sat Dec 09, 2017 12:57 am
Location: United States

Re: STICKY: TUTORIAL: Build the #1 Image Viewer – Automatically in 3 Minutes!

Fri Jul 26, 2019 7:56 pm

innocent_bystander wrote:
Thu Jul 25, 2019 8:29 pm
Not sure if you are aware, but feh 3.1.3 is available in Debian Buster repositories. To install, simply run:

Code: Select all

wget http://ftp.debian.org/debian/pool/main/f/feh/feh_3.1.3-1_armhf.deb
sudo apt install ./feh_3.1.3-1_armhf.deb

No need to compile anything or pollute your system with dev packages or unmanaged binaries.

Just my 2 cents. :)

Unfortunately, your "2 cents" don't work on the newest Raspberry platform of only 5 weeks ago – the Raspberry 3 with the Raspbian Stretch operating system.

On these numerous deployed systems – likely to be in the millions – any user that follows your advice will get nothing but error messages due to "unmet dependencies". That's because the Buster version of feh that you're recommending that people download – version 3.1.3 – requires at least version 7.16.2 of libcurl4.

But it gets worse than that – because no such package exists in the world of Stretch! Why? Because the libcurl4 package was newly introduced with Buster.

But even for those who do have Buster, your advice is still flawed:

If someone already has Buster, it makes no sense to advise them to download an external package from a non-Raspberry site when that package is already conveniently available from the official Raspbian Buster repository.

In other words, for those who already have Buster, they can run a single command line to install feh:

sudo apt-get install feh

That's it. Simple as that! And it arrives the "proper" way – directly from the Raspberry Pi Foundation's official repository.

If someone already has Buster and they want to do that, they can simply remove the section of my script that builds feh – and replace it with the single installation line that I just referenced. That's perfectly fine if someone wants to do that.

However, they still need to use my overall script – with that one modification – if they want the convenient integration with File Manager – and the convenient control and customization my set-up provides.

Or, since no significant feature of feh has been introduced since I wrote my script, Buster users can simply ignore this entire discussion and run my script "as is". I've tested it on Buster and everything still works great! I even described a few weeks ago why I don't immediately jump on the latest version – especially if there's no compelling reason to do so.

Newer doesn't automatically mean "better" in the world of computing.

It's also important to realize that as time marches on, the Buster version of feh will no longer be the current version – and there may indeed be a useful new feature that people want in the future.

The good news is that my existing script has a kind of "timeless value" – in that it provides an easily-modifiable framework to acquire the latest version of feh at any time in the future.

Remember, many people on here are former Windows and Mac users who are being exposed to "real" computing for the very first time. That's one of the main purposes of the Raspberry Pi Foundation – to expand the number of people that get a "genuine" exposure to computing – an exposure that isn't limited to shiny, locked-down apps where everything that happens is completely hidden under the hood.

Because the prior experience of many Raspberry users is exclusively limited to locked-down GUI programs, many of them have never even heard about "building" software – nor could they tell you what "source code" even means.

Sure, a certain minority of people on here have computer science degrees and other advanced backgrounds that make all the nerdy stuff about building software seem so obvious. But ironically, it's that very knowledge that can blind them and make them ignorant about the true nature of the computing marketplace. There's even a term for this – it's a cognitive bias called the curse of knowledge. When I write my tutorials, I try my best to avoid any mental contamination from that curse.

There's also a good practical reason why many on here don't know what "building software" even means. In the Windows world, for example – thanks to a user base that utterly dwarfs the Raspberry's – the most recent versions of thousands of programs are always available. Why? Because the giant user base compels developers to pre-build all their programs and make them easily available to the common user.

That's certainly not the case with an "ARM-Linux desktop" like the Raspberry.

So, let's look for a moment at the "ARM-Linux desktop" world – and the value my tutorial provides in explicitly showing how software can be built and customized for an extremely common activity like image viewing.

Exact figures are impossible to know, but let's be generous and say that 2% of desktop users in the entire world use ANY flavor of Linux on ANY CPU architecture. The dismal numbers don't stop there. The majority of that tiny 2% are using Intel / AMD-based systems. So the fraction of people using ARM-based Linux as a "real computer" is what? Maybe 5 or 10 times smaller than 2%? This is just my very rough math of course, but the current number of people using ARM-based Linux as a "real computer" is perhaps a shockingly low 1 in 500. And that's not even taking into account the heavy fragmentation of "Linux" – that there are tons of different distributions that further dilute the influence of any single platform. This of course pushes any single platform in this category to levels even further below 1 in 500 – or whatever the exact tiny number is.

People love to trot out the billion+ Android users that use ARM-based Linux systems – but although these systems are full-blown computers in technical terms, in practical terms they are of course an entirely different product category known as the "smartphone". There aren't too many Android users out there Googling around for how to build software – or know what "building software" even means. They're just regular people – lawyers, doctors, bakers, housewives and candlestick makers. Even most MIT engineering professors are not using Android systems as desktop computers – or any ARM-based Linux desktop computer for that matter! So when it comes to incentivizing developers to build easy-to-install software for ARM-based Linux desktops like the Raspberry, the overwhelming majority of users aren't even in the game.

So that's a major part of my tutorial's value – at least for those who got a Raspberry for the first time and might be curious about these things. It exposes "regular people" to the largely hidden world of computing. It shows them the computing equivalent of how to bake a cake. It shows them how to build and customize software for one of the world's most popular activities on a computer – viewing images!

So there ya go – that's my 2 cents.

Finally, for those who like technical detail, this is exactly what will happen if anyone on Stretch follows the poster's advice:

sudo apt install ./feh_3.1.3-1_armhf.deb

Reading package lists... Done
Building dependency tree
Reading state information... Done
Note, selecting 'feh' instead of './feh_3.1.3-1_armhf.deb'
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
feh : Depends: libcurl4 (>= 7.16.2) but it is not installable
Recommends: libjpeg-progs
E: Unable to correct problems, you have held broken packages.

innocent_bystander
Posts: 45
Joined: Mon Oct 15, 2018 12:15 am
Location: Florida, USA

Re: STICKY: TUTORIAL: Build the #1 Image Viewer – Automatically in 3 Minutes!

Fri Jul 26, 2019 9:08 pm

So perhaps you could make your own .deb package with all the scripts included and spare the users from installing dev packages and unmanaged binaries?
The Sirius Cybernetics Corporation is the primary manufacturer and supplier of androids, robots and autonomic assistants for the known universe. ... The company motto is "Share and Enjoy."

User avatar
RPi_Mike
Posts: 191
Joined: Sat Dec 09, 2017 12:57 am
Location: United States

Re: STICKY: TUTORIAL: Build the #1 Image Viewer – Automatically in 3 Minutes!

Fri Jul 26, 2019 9:39 pm

innocent_bystander wrote:
Fri Jul 26, 2019 9:08 pm
So perhaps you could make your own .deb package with all the scripts included and spare the users

Perhaps I will be the one to decide what I do with my own tutorial.

To borrow your provocative phrasing, it is you that might wish to "spare the users" of flawed technical advice in the future.

innocent_bystander
Posts: 45
Joined: Mon Oct 15, 2018 12:15 am
Location: Florida, USA

Re: STICKY: TUTORIAL: Build the #1 Image Viewer – Automatically in 3 Minutes!

Mon Jul 29, 2019 7:14 pm

RPi_Mike wrote:
Fri Jul 26, 2019 9:39 pm
Perhaps I will be the one to decide what I do with my own tutorial.

To borrow your provocative phrasing, it is you that might wish to "spare the users" of flawed technical advice in the future.

@RPi_Mike wasn't trying to provoke you. A lot of people, myself included, appreciate the work you and other people in the RPi community do to help others. This is what makes this community so great.

However, it is my very humble opinion (which I believe is also the opinion of many OS maintainers) that installing files that are outside of package manager's control is not for the fainthearted.

You've done a great job in making newer version of feh available for Debian stretch users. But your script, no matter how foolproof, still has a chance of causing issues for end-users. That's the only reason I suggested for you to make a .deb package out of it.

Anyway, I understand your position and as a compromise, I've created a .deb package that includes your fehview and fehshow scripts. I've also added fehview-normal (no zoom), fehshow-normal (no zoom) and fehshow-filename (shows filename) versions, so that users don't have to modify anything by hand.

The package can be downloaded here: feh_3.2.1-1_armhf.deb

Installation is as simple as:

Code: Select all

sudo apt install ./feh_3.2.1-1_armhf.deb

I did not include my name in it, since you did most of the work. Feel free to use it as your own and make any further improvements to it.

Share and enjoy.
The Sirius Cybernetics Corporation is the primary manufacturer and supplier of androids, robots and autonomic assistants for the known universe. ... The company motto is "Share and Enjoy."

geoffpeters
Posts: 10
Joined: Wed May 15, 2019 5:23 pm

Re: STICKY: TUTORIAL: Build the #1 Image Viewer – Automatically in 3 Minutes!

Thu Aug 15, 2019 6:27 pm

OK, I can report that all is working as expected. About two weeks ago I added a RAM disk as the destination folder on the RPi. I only created one.

For sake of complete disclosure I believe that the "partial image" displays that I was seeing were due to my export scheme. There were two different screenshots, of different layouts, that I was exporting.These exports were scheduled to run at 30 second intervals alternating between the two screenshots. I was, however, exporting the screenshots to the same file name on the Rpi.

I changed the export scheme so the two screenshots are now exported to two different files. I have had no "partial image" problems since the change. I changed the export interval to 5 minutes, sufficient to accommodate database updates. I changed Fehshow to update every 30 seconds.

Again, thank you for your work and your suggestions.

Return to “Graphics, sound and multimedia”