ClassicMaN
Posts: 15
Joined: Tue Jun 12, 2018 10:33 am

Raspberry Pi 3 Model B+ clusters

Tue Jun 12, 2018 10:48 am

:) Hello

I'm kinda confused about this idea - i just wanna be sure before i buy those stuff

Q1: Does this means if i get 5* of Raspberry Pi 3 Model B+, the cluster's performance will be increase from 1.4GHz to 7GHz ?

Q2: if Q1 is yes - does this means that i could speed it up image resize as a batch (using python or PHP coding) to speed thing up much faster then regular desktop computer?

Image Resize - basically any image TIFF full resolution from DSLR camera - convert into any aspect ratio - 4x5, 7x10 or 8x10 save as TIFFs or JPEGs - i can do this on a mac but i have to wait for hours to get this batch done before i could do something else!.

Looking forward to hear from you guys...

ClassicMaN :)

Heater
Posts: 9841
Joined: Tue Jul 17, 2012 3:02 pm

Re: Raspberry Pi 3 Model B+ clusters

Tue Jun 12, 2018 12:09 pm

Clustering computers, like the Pi, does not magically make any software go many times faster.

But, if you have many jobs to do, like thousands of images to process, you could run the same program on many Pi and send it the images to work on. Then collect the processed images when it is done.

This is going to take some software effort to implement the distribution of work to the Pi, set the processing running on each Pi and collect the results.

Of course software has already been developed to do such work distribution. I can't think what any of them are called just now. I'd be inclined to roll my sleeves up and create my own job dispatcher/runner in Python or Javascript.

HermannSW
Posts: 767
Joined: Fri Jul 22, 2016 9:09 pm

Re: Raspberry Pi 3 Model B+ clusters

Tue Jun 12, 2018 12:15 pm

> Of course software has already been developed to do such work distribution. I can't think what any of them are called just now.
>
Many hits for MPI and Raspberry:
https://www.google.de/search?q=parallel ... +raspberry

E.g. "Raspberry Pi Supercomputer with MPI":
https://www.hackster.io/darthbison/rasp ... mpi-4602cb
bookmark list: https://stamm-wilbrandt.de/en/Raspberry_camera.html

https://github.com/Hermann-SW/fork-raspiraw      https://github.com/Hermann-SW/userland
https://github.com/Hermann-SW/wireless-control-Eachine-E52-drone      https://twitter.com/HermannSW

drgeoff
Posts: 8520
Joined: Wed Jan 25, 2012 6:39 pm

Re: Raspberry Pi 3 Model B+ clusters

Tue Jun 12, 2018 12:38 pm

It takes a woman 9 months to produce a baby.

9 women can produce 9 babies in 9 months.

9 women cannot produce 1 baby in 1 month.

Heater
Posts: 9841
Joined: Tue Jul 17, 2012 3:02 pm

Re: Raspberry Pi 3 Model B+ clusters

Tue Jun 12, 2018 12:44 pm

Indeed. However we are talking about computers and batch jobs.

Our OP used the word "batch" which I take to mean that there are many TIFF images to be processed not just one. That is to say many "babies". So there is some chance of getting multiple Pi to process the batch in less time than one.

ClassicMaN
Posts: 15
Joined: Tue Jun 12, 2018 10:33 am

Re: Raspberry Pi 3 Model B+ clusters

Tue Jun 12, 2018 5:39 pm

Heater wrote:
Tue Jun 12, 2018 12:09 pm
Clustering computers, like the Pi, does not magically make any software go many times faster.

But, if you have many jobs to do, like thousands of images to process, you could run the same program on many Pi and send it the images to work on. Then collect the processed images when it is done.

This is going to take some software effort to implement the distribution of work to the Pi, set the processing running on each Pi and collect the results.

Of course software has already been developed to do such work distribution. I can't think what any of them are called just now. I'd be inclined to roll my sleeves up and create my own job dispatcher/runner in Python or Javascript.
That''s a good idea my friend,

can you test for me? - anyone is welcome

In that case: how long does it read and resize image just using one Raspberry Pi b+

TIFF: average file size is 100MB-/+ (3280 × 4928) to 8x10 (2400x3000) - don't worry about aspect ratio that's another problems :mrgreen:

I will work it out in math how long it will take to process over 5,000 images !

ClassicMaN
Last edited by ClassicMaN on Tue Jun 12, 2018 6:11 pm, edited 3 times in total.

ClassicMaN
Posts: 15
Joined: Tue Jun 12, 2018 10:33 am

Re: Raspberry Pi 3 Model B+ clusters

Tue Jun 12, 2018 5:41 pm

HermannSW wrote:
Tue Jun 12, 2018 12:15 pm
> Of course software has already been developed to do such work distribution. I can't think what any of them are called just now.
>
Many hits for MPI and Raspberry:
https://www.google.de/search?q=parallel ... +raspberry

E.g. "Raspberry Pi Supercomputer with MPI":
https://www.hackster.io/darthbison/rasp ... mpi-4602cb
thank you my friend - I will look up some more information regarding "Work distribution"

ClassicMaN

ClassicMaN
Posts: 15
Joined: Tue Jun 12, 2018 10:33 am

Re: Raspberry Pi 3 Model B+ clusters

Tue Jun 12, 2018 5:45 pm

drgeoff wrote:
Tue Jun 12, 2018 12:38 pm
It takes a woman 9 months to produce a baby.

9 women can produce 9 babies in 9 months.

9 women cannot produce 1 baby in 1 month.
:mrgreen: it would be nice to have 1 baby in 1 month, 9 months is a long time to wait. :lol:

ClassicMaN

ClassicMaN
Posts: 15
Joined: Tue Jun 12, 2018 10:33 am

Re: Raspberry Pi 3 Model B+ clusters

Tue Jun 12, 2018 5:46 pm

Heater wrote:
Tue Jun 12, 2018 12:44 pm
Indeed. However we are talking about computers and batch jobs.

Our OP used the word "batch" which I take to mean that there are many TIFF images to be processed not just one. That is to say many "babies". So there is some chance of getting multiple Pi to process the batch in less time than one.
Thank you my friends !

ClassicMaN

ejolson
Posts: 1899
Joined: Tue Mar 18, 2014 11:47 am

Re: Raspberry Pi 3 Model B+ clusters

Wed Jun 13, 2018 12:33 am

ClassicMaN wrote:
Tue Jun 12, 2018 10:48 am
i can do this on a mac but i have to wait for hours to get this batch done before i could do something else!
I'm not sure why you would have to wait before doing anything else. Most computers, even Macintoshes (but maybe not the Mac Classic), have the ability to multitask so that more than one program can run at a time.

The task of resizing and compressing hundreds of images definitely has significant opportunities for parallel processing. The advantage of a cluster is that it can be expanded by adding more nodes as needed; the advantage of the Pi is that it's cheap and already includes four cores. My recommendation would be to buy one Pi and see how fast its four cores can process a batch of images. If one Pi is sufficient, you are done. If not, then you at least know one Pi can process the images in time T. Now choose N large enough until the time T/N is small enough. You now have an estimate which says you will need at least N Pi computers.

If N is larger than the number of ports on your network switch you will also need to buy a suitable gigabit switch--WiFi is unlikely to work well for this.

pootle
Posts: 270
Joined: Wed Sep 04, 2013 10:20 am
Location: Staffordshire
Contact: Website

Re: Raspberry Pi 3 Model B+ clusters

Wed Jun 13, 2018 7:32 am

A Raspberry pi is going to be a LOT slower than a mac at processing tiff files - unless it is a very old mac. You would probably need 5 Raspberry pi's to match the performance of your MAC, so to go faster you will need a lot of pi's.

You need to understand why your current processing is slow to see if a multiple machine approach will help.

Unless of course you really just want an excuse to play with a bunch of pi's!

You should first check out ways of multi-processing on the MAC and how much this can improve the time.

You should also confirm that you are doing the conversion in an efficient way to start with.

Use a performance monitor to check how much load is placed on CPU, memory and disc I/O.

With your tiff files being 100Mb it sounds like your existing tools may not be using TIFF compression. Your conversion may well be disc i/o limited because of this. Can you use tools that enable (lossless) tiff compression? - Standard options are LZW and zip style.

Raspberry Pi's do not have very fast network connections. Even with a Gigabit switch, they can only do about 300Mb(its) per second. It will take ~3 seconds to transfer each file to the pi, and another 3 to get it back (unless you can compress the files - see above). Your mac will only be able to transfer 3 in parallel before network congestion becomes the limiting factor.

Have you looked at any of the many tools available for fast batch processing tiff files?

http://www.batchphoto.com/blog/top-5-ba ... e-for-mac/

ClassicMaN
Posts: 15
Joined: Tue Jun 12, 2018 10:33 am

Re: Raspberry Pi 3 Model B+ clusters

Wed Jun 13, 2018 1:45 pm

ejolson wrote:
Wed Jun 13, 2018 12:33 am
ClassicMaN wrote:
Tue Jun 12, 2018 10:48 am
i can do this on a mac but i have to wait for hours to get this batch done before i could do something else!
I'm not sure why you would have to wait before doing anything else. Most computers, even Macintoshes (but maybe not the Mac Classic), have the ability to multitask so that more than one program can run at a time.

The task of resizing and compressing hundreds of images definitely has significant opportunities for parallel processing. The advantage of a cluster is that it can be expanded by adding more nodes as needed; the advantage of the Pi is that it's cheap and already includes four cores. My recommendation would be to buy one Pi and see how fast its four cores can process a batch of images. If one Pi is sufficient, you are done. If not, then you at least know one Pi can process the images in time T. Now choose N large enough until the time T/N is small enough. You now have an estimate which says you will need at least N Pi computers.

If N is larger than the number of ports on your network switch you will also need to buy a suitable gigabit switch--WiFi is unlikely to work well for this.

That is a good thinking my friend, why i didn't think of it :lol: i guess i should buy 1 of B+ for now and buy later 5 of Raspberry pi 4 only if they release soon, hopefully it should come with USB3 - 5gbp data transfer).

ClassicMaN
Posts: 15
Joined: Tue Jun 12, 2018 10:33 am

Re: Raspberry Pi 3 Model B+ clusters

Wed Jun 13, 2018 2:07 pm

pootle wrote:
Wed Jun 13, 2018 7:32 am
A Raspberry pi is going to be a LOT slower than a mac at processing tiff files - unless it is a very old mac. You would probably need 5 Raspberry pi's to match the performance of your MAC, so to go faster you will need a lot of pi's.

You need to understand why your current processing is slow to see if a multiple machine approach will help.

Unless of course you really just want an excuse to play with a bunch of pi's!

You should first check out ways of multi-processing on the MAC and how much this can improve the time.

You should also confirm that you are doing the conversion in an efficient way to start with.

Use a performance monitor to check how much load is placed on CPU, memory and disc I/O.

With your tiff files being 100Mb it sounds like your existing tools may not be using TIFF compression. Your conversion may well be disc i/o limited because of this. Can you use tools that enable (lossless) tiff compression? - Standard options are LZW and zip style.

Raspberry Pi's do not have very fast network connections. Even with a Gigabit switch, they can only do about 300Mb(its) per second. It will take ~3 seconds to transfer each file to the pi, and another 3 to get it back (unless you can compress the files - see above). Your mac will only be able to transfer 3 in parallel before network congestion becomes the limiting factor.

Have you looked at any of the many tools available for fast batch processing tiff files?

http://www.batchphoto.com/blog/top-5-ba ... e-for-mac/

Today i wrote a quick script to find out whats my result would be like on a mac pro with regular HD 7200rpm, currently using VMware fusion windows 10 pro.

My script run as classic asp/asp.net localhost as IIS

here is the result i get:

FileName: _D4M8455.tif
Resize from: 4,928 x 3280 to 3000 x 2400 (8x10)
Total Size:[/b ]123.3 MB
StopWatch: 3 seconds
Per MB: 41.1 MB per second

This script took 3 seconds to load

3 seconds for each file...

2 Minutes to complete 50 images
5 Minutes to complete 100 images
12 Minutes to complete 250 images
25 Minutes to complete 500 images
0 hours and 50 minutes to complete 1,000 images
4 hours and 10 minutes to complete 5,000 images
8 hours and 20 minutes to complete 10,000 images
16 hours and 40 minutes to complete 20,000 images

4 hours is a long time to wait and leave the computer alone to avoid any software conflict.... thats why i need to find the ways to speed it up.

somehow i need to push from 3 seconds to 1 second per image that would work out roughly 1 hr 38 min for 5,000 images

Ernst
Posts: 686
Joined: Sat Feb 04, 2017 9:39 am
Location: Germany

Re: Raspberry Pi 3 Model B+ clusters

Wed Jun 13, 2018 2:41 pm

I would not bother trying to solve this problem with one or more Raspberry Pi. The better, meaning better performing, solution would be to have a dedicated Intel based computer to do the job.

The reason is very simple, most of the work is moving data, from disk to RAM, from RAM to processor, from processor to RAM, from RAM to disk. According to your test you "measured" the time to process as 41.1 Mbyte per second, what this means that internally the transfer speed will be much higher. The general available PC should offer a better memory transfer speed and a better IO transfer speed compared to a Raspberry, and most likely a higher processor performance.

It is an interesting problem to see how such a problem can be solved with a bunch of networked pocket calculators but it is not the answer to getting the job done faster.
My first computer was an ICT1500, my first "personal" computer was the Science of Cambridge Mk14, followed by a TRS-80 Model I later

jerrm
Posts: 59
Joined: Wed May 02, 2018 7:35 pm

Re: Raspberry Pi 3 Model B+ clusters

Wed Jun 13, 2018 3:55 pm

A Pi would definitely be I/O bound. A Pi won't come close to 41MB/s with a straight "cp" command

I doubt it would be an exaggeration that it would take 5+ Pis all working in parallel just to average the output you'd get from a single x86 sub-$100 refurb PC.

ejolson
Posts: 1899
Joined: Tue Mar 18, 2014 11:47 am

Re: Raspberry Pi 3 Model B+ clusters

Thu Jun 14, 2018 3:21 am

ClassicMaN wrote:
Wed Jun 13, 2018 2:07 pm
here is the result i get:

FileName: _D4M8455.tif
Resize from: 4,928 x 3280 to 3000 x 2400 (8x10)
Total Size:123.3 MB
StopWatch: 3 seconds
Per MB: 41.1 MB per second
A single gigabit Ethernet port can move data at a rate of 100 MB/sec. A Mac Pro (the round one) has two gigabit Ethernet ports. If it were used as a file server for the Pi computers, you could expect at most a 4.6 times speed up as constrained by the dual gigabit networking. However, if you are generating multiple resized images from a single image, the speedup could be much more as the input image only needs to be transmitted across the network once.

Maximizing the use of a single gigabit Ethernet port takes at least 5 Raspberry Pi 3B+ computers assuming that each one is consuming data at 200 Mbit. More likely they will consume data at less than half that. In this case you would need around 20 Pi 3B+ computers maybe more to maximize the use of both gigabit Ethernet ports on the Mac Pro. Speeds beyond that could be achieved by using a big-data style distributed storage or a purpose-built file server. If the raw images are coming from SD cards, it would also be possible to attach SD card readers to each Pi, read the cards in parallel and only transmit the resized and compressed images back to the Mac Pro.

Sometimes the point of a cluster is not faster speeds, but a reliable place to perform long-running calculations that don't tie up your desktop computer.

fruitoftheloom
Posts: 17487
Joined: Tue Mar 25, 2014 12:40 pm

Re: Raspberry Pi 3 Model B+ clusters

Thu Jun 14, 2018 6:04 am

ClassicMaN wrote:
Tue Jun 12, 2018 10:48 am
:) Hello

I'm kinda confused about this idea - i just wanna be sure before i buy those stuff

Q1: Does this means if i get 5* of Raspberry Pi 3 Model B+, the cluster's performance will be increase from 1.4GHz to 7GHz ?

Q2: if Q1 is yes - does this means that i could speed it up image resize as a batch (using python or PHP coding) to speed thing up much faster then regular desktop computer?

Image Resize - basically any image TIFF full resolution from DSLR camera - convert into any aspect ratio - 4x5, 7x10 or 8x10 save as TIFFs or JPEGs - i can do this on a mac but i have to wait for hours to get this batch done before i could do something else!.

Looking forward to hear from you guys...

ClassicMaN :)

Brilliant idea, but like all similar posts only good as a Learning Exercise, and 5 RPis does not increase power by 5x.....


A cluster of RPis is good for certain tasks like Drupal and similar:

https://www.pidramble.com
Adieu

pootle
Posts: 270
Joined: Wed Sep 04, 2013 10:20 am
Location: Staffordshire
Contact: Website

Re: Raspberry Pi 3 Model B+ clusters

Thu Jun 14, 2018 8:40 am

ClassicMaN wrote:
Wed Jun 13, 2018 2:07 pm
Today i wrote a quick script to find out whats my result would be like on a mac pro with regular HD 7200rpm, currently using VMware fusion windows 10 pro.

My script run as classic asp/asp.net localhost as IIS

here is the result i get:

FileName: _D4M8455.tif
Resize from: 4,928 x 3280 to 3000 x 2400 (8x10)
Total Size:[/b ]123.3 MB
StopWatch: 3 seconds
Per MB: 41.1 MB per second

This script took 3 seconds to load

3 seconds for each file...

2 Minutes to complete 50 images
5 Minutes to complete 100 images

4 hours is a long time to wait and leave the computer alone to avoid any software conflict.... thats why i need to find the ways to speed it up.

somehow i need to push from 3 seconds to 1 second per image that would work out roughly 1 hr 38 min for 5,000 images


It seems you are going through many layers - VMWare, IIS asp.net - which will not be helping.

What is the actual tool you are using for the resize?

Can you run it native on the MAC? This would probably give you a speed boost.

Also from the figures you give it looks like this is only running in 1 core of your multi-core / multi-threading CPU. By using more processes you should get a big speed boost - although you may reach disc i/o limits before you reach CPU limits.

Is it at all possible that you can start processing the first of the 500 images before the last is taken? I assume that you are not taking 500 images in a couple of seconds!

How do you access the images from their original source? By that I mean are you copying them from an SD card onto the MAC before you start, or are they accessed over the network, or are they on some other external media? Ideally you want to read the images (once) from the original source and write them (once) to the final destination without any intermediate files being used.

If the conversion tool can compress the output files using ZIP or LZW compression, it will reduce the file size and probably reduce the overall time (there is an extra cpu cost, but I suspect you will be disc limited once you are running multiple processes.

ClassicMaN
Posts: 15
Joined: Tue Jun 12, 2018 10:33 am

Re: Raspberry Pi 3 Model B+ clusters

Fri Jun 15, 2018 8:10 am

Ernst wrote:
Wed Jun 13, 2018 2:41 pm
I would not bother trying to solve this problem with one or more Raspberry Pi. The better, meaning better performing, solution would be to have a dedicated Intel based computer to do the job.

The reason is very simple, most of the work is moving data, from disk to RAM, from RAM to processor, from processor to RAM, from RAM to disk. According to your test you "measured" the time to process as 41.1 Mbyte per second, what this means that internally the transfer speed will be much higher. The general available PC should offer a better memory transfer speed and a better IO transfer speed compared to a Raspberry, and most likely a higher processor performance.

It is an interesting problem to see how such a problem can be solved with a bunch of networked pocket calculators but it is not the answer to getting the job done faster.
Yes you are right my friend, however i thought i could get away with cheaper option but i guess not due hardware limitation via USB 2.0 + Micro SD.

Hopefully raspberry pi 4 will offer better hardware such as USB 3.1 and boot from SSD.

ClassicMaN
Posts: 15
Joined: Tue Jun 12, 2018 10:33 am

Re: Raspberry Pi 3 Model B+ clusters

Fri Jun 15, 2018 8:13 am

jerrm wrote:
Wed Jun 13, 2018 3:55 pm
A Pi would definitely be I/O bound. A Pi won't come close to 41MB/s with a straight "cp" command

I doubt it would be an exaggeration that it would take 5+ Pis all working in parallel just to average the output you'd get from a single x86 sub-$100 refurb PC.
Yes that's a good idea to get refurb PC to start off with, i was hoping i could get away with raspberry pi :lol:

ejolson
Posts: 1899
Joined: Tue Mar 18, 2014 11:47 am

Re: Raspberry Pi 3 Model B+ clusters

Fri Jun 15, 2018 8:23 am

ClassicMaN wrote:
Fri Jun 15, 2018 8:10 am
Yes you are right my friend, however i thought i could get away with cheaper option but i guess not due hardware limitation via USB 2.0 + Micro SD.

Hopefully raspberry pi 4 will offer better hardware such as USB 3.1 and boot from SSD.
There are two considerations: how fast the data can be read by the Pi and how fast it can be provided by the file server.

In this case you are limited by the file server, presumably a Mac Pro, because gigabit Ethernet can only transfer data at about 100 MB/sec. It doesn't matter how fast the Pi can read data, because if the file server has bandwidth left over you can always buy more Pi computers. In particular, the bottleneck for parallel speedup is how fast you can get the data off the Mac Pro.

ClassicMaN
Posts: 15
Joined: Tue Jun 12, 2018 10:33 am

Re: Raspberry Pi 3 Model B+ clusters

Fri Jun 15, 2018 8:41 am

ejolson wrote:
Thu Jun 14, 2018 3:21 am
ClassicMaN wrote:
Wed Jun 13, 2018 2:07 pm
here is the result i get:

FileName: _D4M8455.tif
Resize from: 4,928 x 3280 to 3000 x 2400 (8x10)
Total Size:123.3 MB
StopWatch: 3 seconds
Per MB: 41.1 MB per second
A single gigabit Ethernet port can move data at a rate of 100 MB/sec. A Mac Pro (the round one) has two gigabit Ethernet ports. If it were used as a file server for the Pi computers, you could expect at most a 4.6 times speed up as constrained by the dual gigabit networking. However, if you are generating multiple resized images from a single image, the speedup could be much more as the input image only needs to be transmitted across the network once.

Maximizing the use of a single gigabit Ethernet port takes at least 5 Raspberry Pi 3B+ computers assuming that each one is consuming data at 200 Mbit. More likely they will consume data at less than half that. In this case you would need around 20 Pi 3B+ computers maybe more to maximize the use of both gigabit Ethernet ports on the Mac Pro. Speeds beyond that could be achieved by using a big-data style distributed storage or a purpose-built file server. If the raw images are coming from SD cards, it would also be possible to attach SD card readers to each Pi, read the cards in parallel and only transmit the resized and compressed images back to the Mac Pro.

Sometimes the point of a cluster is not faster speeds, but a reliable place to perform long-running calculations that don't tie up your desktop computer.
if you have the time to test it out - can you let me know how long does it take you to transfer from 100MB data from a USB HD to each pi and 35MB data send back to the USB HD?

ClassicMaN
Posts: 15
Joined: Tue Jun 12, 2018 10:33 am

Re: Raspberry Pi 3 Model B+ clusters

Fri Jun 15, 2018 8:50 am

fruitoftheloom wrote:
Thu Jun 14, 2018 6:04 am
ClassicMaN wrote:
Tue Jun 12, 2018 10:48 am
:) Hello

I'm kinda confused about this idea - i just wanna be sure before i buy those stuff

Q1: Does this means if i get 5* of Raspberry Pi 3 Model B+, the cluster's performance will be increase from 1.4GHz to 7GHz ?

Q2: if Q1 is yes - does this means that i could speed it up image resize as a batch (using python or PHP coding) to speed thing up much faster then regular desktop computer?

Image Resize - basically any image TIFF full resolution from DSLR camera - convert into any aspect ratio - 4x5, 7x10 or 8x10 save as TIFFs or JPEGs - i can do this on a mac but i have to wait for hours to get this batch done before i could do something else!.

Looking forward to hear from you guys...

ClassicMaN :)

Brilliant idea, but like all similar posts only good as a Learning Exercise, and 5 RPis does not increase power by 5x.....


A cluster of RPis is good for certain tasks like Drupal and similar:

https://www.pidramble.com
Thank you my friends, the link you provided is very very interesting informations....

Does this web server allow you to run PHP & MySQL database? does it store all pi or just one pi?

ClassicMaN
Posts: 15
Joined: Tue Jun 12, 2018 10:33 am

Re: Raspberry Pi 3 Model B+ clusters

Fri Jun 15, 2018 1:27 pm

pootle wrote:
Thu Jun 14, 2018 8:40 am
ClassicMaN wrote:
Wed Jun 13, 2018 2:07 pm
Today i wrote a quick script to find out whats my result would be like on a mac pro with regular HD 7200rpm, currently using VMware fusion windows 10 pro.

My script run as classic asp/asp.net localhost as IIS

here is the result i get:

FileName: _D4M8455.tif
Resize from: 4,928 x 3280 to 3000 x 2400 (8x10)
Total Size:[/b ]123.3 MB
StopWatch: 3 seconds
Per MB: 41.1 MB per second

This script took 3 seconds to load

3 seconds for each file...

2 Minutes to complete 50 images
5 Minutes to complete 100 images

4 hours is a long time to wait and leave the computer alone to avoid any software conflict.... thats why i need to find the ways to speed it up.

somehow i need to push from 3 seconds to 1 second per image that would work out roughly 1 hr 38 min for 5,000 images


It seems you are going through many layers - VMWare, IIS asp.net - which will not be helping.

What is the actual tool you are using for the resize?

Can you run it native on the MAC? This would probably give you a speed boost.

Also from the figures you give it looks like this is only running in 1 core of your multi-core / multi-threading CPU. By using more processes you should get a big speed boost - although you may reach disc i/o limits before you reach CPU limits.

Is it at all possible that you can start processing the first of the 500 images before the last is taken? I assume that you are not taking 500 images in a couple of seconds!

How do you access the images from their original source? By that I mean are you copying them from an SD card onto the MAC before you start, or are they accessed over the network, or are they on some other external media? Ideally you want to read the images (once) from the original source and write them (once) to the final destination without any intermediate files being used.

If the conversion tool can compress the output files using ZIP or LZW compression, it will reduce the file size and probably reduce the overall time (there is an extra cpu cost, but I suspect you will be disc limited once you are running multiple processes.




It seems you are going through many layers - VMWare, IIS asp.net - which will not be helping.

Yes you are right my friend, but i'm just using VMWare just for the web development purpose...

What is the actual tool you are using for the resize?

i used .dll which was made by 3rd party that allow me to create, edit, convert and resize images.

Can you run it native on the MAC? This would probably give you a speed boost.

yes that's why im looking at pi b+ or something better as long it is cheap things to do....

How do you access the images from their original source? By that I mean are you copying them from an SD card onto the MAC before you start, or are they accessed over the network, or are they on some other external media? Ideally you want to read the images (once) from the original source and write them (once) to the final destination without any intermediate files being used.

Well, i used DSLR to send the Raw files to the laptop via USB but the will not stored in Laptop only external SSD HD. Photoshop wll be edit and save within same SSD HD.

If the conversion tool can compress the output files using ZIP or LZW compression, it will reduce the file size and probably reduce the overall time (there is an extra cpu cost, but I suspect you will be disc limited once you are running multiple processes.

yes you are right, i could've use those image compression however i rather to stay out of any compression, i like to preserve the highest quality as much i can do.

Heater
Posts: 9841
Joined: Tue Jul 17, 2012 3:02 pm

Re: Raspberry Pi 3 Model B+ clusters

Fri Jun 15, 2018 1:40 pm

zip and lzw compression are lossless compressors. They do not degrade files. What you have before compression is exactly what you get back after decompression.

Return to “General discussion”

Who is online

Users browsing this forum: Douglas6 and 51 guests