User avatar
yv1hx
Posts: 360
Joined: Sat Jul 21, 2012 10:09 pm
Location: Zulia, Venezuela
Contact: Website

WRF & NetCDF on Raspberry-Pi

Thu Oct 04, 2012 11:01 pm

Hi R-Pi fanatics!

I´m trying to build the Weather forecasting model (WRF) on my R-Pi, towards to build a two nodes cluster when my second R-Pi arrives at some point, but I cannot build the NetCDF package because i´m always getting the message:

Code: Select all

<snip>
checking for ptrdiff_t... yes
checking for stdlib.h... (cached) yes
checking for GNU libc compatible malloc... yes
checking for special C compiler options needed for large files... no
checking for _FILE_OFFSET_BITS value needed for large files... 64
checking netcdf.h usability... no
checking netcdf.h presence... no
checking for netcdf.h... no
configure: error: netcdf.h could not be found. Please set CPPFLAGS.
pi@raspberrypi:~/netcdf-cxx4-4.2$ 
Has anyone tried to build & install those packages in a R-Pi?

EDIT: I goggled about the ´configure: error: netcfd ....' but the proposed solutions where not successful

P.S.:
This is a purely academic exercise, I'm more a "hardware guy" than a "linux guy", so don't blame me please... and i´m aware the fact the R-Pi can be not enough powerful to run those tools :mrgreen:

Thank you in advance...
Marco-Luis
Telecom Specialist (Now Available for Hire!)

http://www.meteoven.org
http://yv1hx.ddns.net
http://twitter.com/yv1hx

BlackJack
Posts: 288
Joined: Sat Aug 04, 2012 8:28 am
Contact: Website

Re: WRF & NetCDF on Raspberry-Pi

Fri Oct 05, 2012 6:34 pm

@yv1hx: It seems you have not installed the netcdf library or at least not the development headers for it. The package name for the headers should be `libnetcdf-dev`.

Code: Select all

while not self.asleep():
    sheep += 1

User avatar
yv1hx
Posts: 360
Joined: Sat Jul 21, 2012 10:09 pm
Location: Zulia, Venezuela
Contact: Website

Re: WRF & NetCDF on Raspberry-Pi

Sat Nov 10, 2012 5:51 pm

BlackJack wrote:@yv1hx: It seems you have not installed the netcdf library or at least not the development headers for it. The package name for the headers should be `libnetcdf-dev`.
Dear BlackJack, I have this project in the freezer for now, a bunch of bad things have occurred to my and my parents in the last month :? so I was forced to stop all of my research.

The good new is: I have my second R-Pi, and two 16GB SD cards for continuing my experiments :mrgreen:

Thank you for point me the way....
Marco-Luis
Telecom Specialist (Now Available for Hire!)

http://www.meteoven.org
http://yv1hx.ddns.net
http://twitter.com/yv1hx

User avatar
yv1hx
Posts: 360
Joined: Sat Jul 21, 2012 10:09 pm
Location: Zulia, Venezuela
Contact: Website

Re: WRF & NetCDF on Raspberry-Pi

Sun Dec 30, 2012 7:19 pm

Well, back to playing with my two RP-i's:

A colleague, well trained with the WRF package, has been guiding me about this topic, just for the records:

0) the netcdf sources are here: http://www.unidata.ucar.edu/downloads/netcdf/index.jsp
1) The netcdf MUST be compiled using the same compiler than the rest of the packages to be compiled later (HDF5 , and others); at this point I'm very unsure what compiler is using the automated scripts :oops:

After trying the builds 3 or 4 times, ]I have a serious doubt about the c compiler that is using the automated scripts, sorry for asking this silly question, but would be possible have 2 or more c compilers installed at the same time ?
Marco-Luis
Telecom Specialist (Now Available for Hire!)

http://www.meteoven.org
http://yv1hx.ddns.net
http://twitter.com/yv1hx

everdaniel
Posts: 8
Joined: Thu Aug 30, 2012 4:00 pm
Location: Encarnacion, Paraguay
Contact: Website AOL

Re: WRF & NetCDF on Raspberry-Pi

Sat Oct 29, 2016 7:31 pm

Hi yv1hx,

did you managed to compile WRF on a Raspberry Pi? Interesting to know how you managed to do it if you did.

Thanks!

User avatar
yv1hx
Posts: 360
Joined: Sat Jul 21, 2012 10:09 pm
Location: Zulia, Venezuela
Contact: Website

Re: WRF & NetCDF on Raspberry-Pi

Sun Oct 30, 2016 5:55 pm

everdaniel wrote:Hi yv1hx,

did you managed to compile WRF on a Raspberry Pi? Interesting to know how you managed to do it if you did.

Thanks!
Nop, I cannot compile the WRF on the Raspberry Pi.

As far as I can remember, the package needs an Intel processor, also in the case I can manage to compile, I suspect that would be need a lot more of computing power that can deliver an Rpi.

Good luck.
Marco-Luis
Telecom Specialist (Now Available for Hire!)

http://www.meteoven.org
http://yv1hx.ddns.net
http://twitter.com/yv1hx

everdaniel
Posts: 8
Joined: Thu Aug 30, 2012 4:00 pm
Location: Encarnacion, Paraguay
Contact: Website AOL

Re: WRF & NetCDF on Raspberry-Pi

Sun Oct 30, 2016 8:04 pm

Hi yv1hx and everyone,

I'm very happy to report that I was finally able to compile WRF 3.8.1 and WPS 3.8.1 on the Raspberry Pi. I've been working on this the past two months occasionally and the last two weeks more intensively, and was finally able to pull it off.

Basically, this is the stack and library versions I used:
- most recent version of Raspbian, currently Raspbian Jessy Lite
- MPICH 3.0.4
- NETCDF 4.1.3
- Jasper 1.900.1
- libpng 1.2.50
- ZLib 1.2.7
- WRF 3.8.1
- WPS 3.8.1

Based on what I was able to find, it's possible in theory to compile all versions of WRF all the way down WRF 3.2 although I haven't tried this yet.

I focused on 3.8.1 since that was the most recent version, I was able to compile it in both serial and dmpar modes.

I'm going to organise my notes and then publish a post explaining how I did it since I was not able to find a lot of info online (only a few notes here and there), I'm sure there will be other people who will try to do the same thing in the future.

Please see screenshots:
Screen Shot 2016-10-30 at 4.57.23 PM.png
WRF binaries compiled on the Raspberry Pi
Screen Shot 2016-10-30 at 4.57.23 PM.png (44.16 KiB) Viewed 3298 times
Screen Shot 2016-10-30 at 4.58.11 PM.png
WPS binaries compiled on the Raspberry Pi
Screen Shot 2016-10-30 at 4.58.11 PM.png (27.39 KiB) Viewed 3298 times
Going to work on running a few of the test cases available on the WRF site now! :D

everdaniel
Posts: 8
Joined: Thu Aug 30, 2012 4:00 pm
Location: Encarnacion, Paraguay
Contact: Website AOL

Re: WRF & NetCDF on Raspberry-Pi

Sun Oct 30, 2016 8:09 pm

In my excitement, I forgot to mention one other thing, our intention (I'm working on this with a friend) is to run the WRF on a cluster of Pi's. Our idea is to get between 24 or 32 Pi's to build the cluster and then run WRF on it (it depends on our budget).

We already have WRF running on a non-arm cluster, now that we have the compilations step solved on the Pi, we will now start working on building the cluster and then running WRF on it.

Will report back here once we have that up and running as well.

User avatar
yv1hx
Posts: 360
Joined: Sat Jul 21, 2012 10:09 pm
Location: Zulia, Venezuela
Contact: Website

Re: WRF & NetCDF on Raspberry-Pi

Mon Oct 31, 2016 3:17 am

everdaniel wrote:In my excitement, I forgot to mention one other thing, our intention (I'm working on this with a friend) is to run the WRF on a cluster of Pi's. Our idea is to get between 24 or 32 Pi's to build the cluster and then run WRF on it (it depends on our budget).

We already have WRF running on a non-arm cluster, now that we have the compilations step solved on the Pi, we will now start working on building the cluster and then running WRF on it.

Will report back here once we have that up and running as well.
Huge congrats to you and your friend. :D :D

At some point I was very pessimistic about the success of this experiment, in fact, since the forced closing of my former Meteorology Research unit I lost totally the interest.

Well done, please keep us posted!
Marco-Luis
Telecom Specialist (Now Available for Hire!)

http://www.meteoven.org
http://yv1hx.ddns.net
http://twitter.com/yv1hx

wefoust
Posts: 1
Joined: Mon May 29, 2017 6:46 pm

Re: WRF & NetCDF on Raspberry-Pi

Mon May 29, 2017 6:54 pm

Hello everdaniel,
I am trying install WRF 3.8.1 on a Raspberry Pi, but I am having trouble compiling it. My first issue was that there were no build options when I I ran ./configure. When I did so, I received the following prompt.

"Please select from among the following supported platforms.


Enter selection [1-0]"

I think I fixed this by going to the WRFV3/arch directory and editing the file "configure_new.defaults". Here I replaced all references of ‘x86_64’ with ‘armv7l’, and this forced the script to bring up my build options. Now, I am can't seem to compile the model with any of the test cases. It makes an attempt at compiling, but I just get a log.compile file with many fatal errors.

Did you have similar trouble when you compiled the model? If so, how did you solve the problem. As a reference, I am using a Raspberry Pi 3 Model B running Raspbian GNU/Linux 8 (jessie). Any help you can provide will be appreciated.

Thanks,
Eliott

uupadhyaya
Posts: 1
Joined: Thu Aug 17, 2017 10:51 am

Re: WRF & NetCDF on Raspberry-Pi

Thu Aug 17, 2017 11:02 am

I could configure WRF 3.8.1 in my Raspberry Pi 2 Model B+ . However, configuring WPS is not providing me full options.

./configure
Will use NETCDF in dir: /home/rroy/Develops/Dependencies/NetCDF/NetCDF-4.4.4.1[4.4.4]
Found Jasper environment variables for GRIB2 support...
$JASPERLIB = /home/rroy/Develops/Dependencies/Grib2/Jasper/Jasper-2.0.12/lib
$JASPERINC = /home/rroy/Develops/Dependencies/Grib2/Jasper/Jasper-2.0.12/include
------------------------------------------------------------------------
Please select from among the following supported platforms.


Enter selection [1-0] :

Can anyone suggest which files needs to be edited?
Thank You.
Best,
Umesh.

everdaniel
Posts: 8
Joined: Thu Aug 30, 2012 4:00 pm
Location: Encarnacion, Paraguay
Contact: Website AOL

Re: WRF & NetCDF on Raspberry-Pi

Sun Aug 27, 2017 3:42 pm

Hi Eliott, really sorry! I saw your message just now.

Were you able to move forward with this? I did exactly what you did as well, I edited the configure_new.defaults and copy one of the GFortran/x86_64 block at the bottom of the file and replace it with armv7 (I think), I still have the notes handy so let me know and I'll share them with you.

Let me know!

wefoust wrote:
Mon May 29, 2017 6:54 pm
Hello everdaniel,
I am trying install WRF 3.8.1 on a Raspberry Pi, but I am having trouble compiling it. My first issue was that there were no build options when I I ran ./configure. When I did so, I received the following prompt.

"Please select from among the following supported platforms.


Enter selection [1-0]"

I think I fixed this by going to the WRFV3/arch directory and editing the file "configure_new.defaults". Here I replaced all references of ‘x86_64’ with ‘armv7l’, and this forced the script to bring up my build options. Now, I am can't seem to compile the model with any of the test cases. It makes an attempt at compiling, but I just get a log.compile file with many fatal errors.

Did you have similar trouble when you compiled the model? If so, how did you solve the problem. As a reference, I am using a Raspberry Pi 3 Model B running Raspbian GNU/Linux 8 (jessie). Any help you can provide will be appreciated.

Thanks,
Eliott

everdaniel
Posts: 8
Joined: Thu Aug 30, 2012 4:00 pm
Location: Encarnacion, Paraguay
Contact: Website AOL

Re: WRF & NetCDF on Raspberry-Pi

Sun Aug 27, 2017 3:44 pm

Hi Umesh,

you don't get any compilation options becase the default configure.defaults doesn't have compilation options for the arm architecture.

I know this is a late reply so let me know if you still have this problem, if that's the case, I'll share my notes with you.
uupadhyaya wrote:
Thu Aug 17, 2017 11:02 am
I could configure WRF 3.8.1 in my Raspberry Pi 2 Model B+ . However, configuring WPS is not providing me full options.

./configure
Will use NETCDF in dir: /home/rroy/Develops/Dependencies/NetCDF/NetCDF-4.4.4.1[4.4.4]
Found Jasper environment variables for GRIB2 support...
$JASPERLIB = /home/rroy/Develops/Dependencies/Grib2/Jasper/Jasper-2.0.12/lib
$JASPERINC = /home/rroy/Develops/Dependencies/Grib2/Jasper/Jasper-2.0.12/include
------------------------------------------------------------------------
Please select from among the following supported platforms.


Enter selection [1-0] :

Can anyone suggest which files needs to be edited?
Thank You.
Best,
Umesh.

mp_pi3
Posts: 2
Joined: Thu Oct 05, 2017 7:02 pm

Re: WRF & NetCDF on Raspberry-Pi

Thu Oct 05, 2017 7:16 pm

everdaniel wrote:
Sun Oct 30, 2016 8:09 pm
In my excitement, I forgot to mention one other thing, our intention (I'm working on this with a friend) is to run the WRF on a cluster of Pi's. Our idea is to get between 24 or 32 Pi's to build the cluster and then run WRF on it (it depends on our budget).

We already have WRF running on a non-arm cluster, now that we have the compilations step solved on the Pi, we will now start working on building the cluster and then running WRF on it.

Will report back here once we have that up and running as well.
Hi everdaniel,

Very interesting project! I run the WRF on my desktop (old and slow...) and I'm wondering how fast it would run on a Pi's cluster or how fast it run in a single Pi. Do you have some stats of run times on a single Pi for some test case?

Thanks!

everdaniel
Posts: 8
Joined: Thu Aug 30, 2012 4:00 pm
Location: Encarnacion, Paraguay
Contact: Website AOL

Re: WRF & NetCDF on Raspberry-Pi

Thu Oct 05, 2017 10:36 pm

mp_pi3 wrote:
Thu Oct 05, 2017 7:16 pm
Hi everdaniel,

Very interesting project! I run the WRF on my desktop (old and slow...) and I'm wondering how fast it would run on a Pi's cluster or how fast it run in a single Pi. Do you have some stats of run times on a single Pi for some test case?

Thanks!
Hi there!

the run time depends a lot, in my particular case, on how big the domain is in the WRF Model and the parametrization used for the simulation, but, to give you an idea, these are the numbers we got on our research:
- Domain Grid: 100 cells by 100 cells
- Resolution: 5km (each cell is a 5km square)
- Total hours simulated: 6
- Compute nodes: 20 Raspberry Pi's
- Login node: 1 Raspberry Pi (not used for compute)

Based on the above, we run a few tests using a different number of cores and the average run time for each number of cores configuration:
64 cores: 53 minutes
72 cores: 1 hour, 2 minutes
80 cores: 51 minutes

A full WRF run, simulating 72 hours, takes 8 hours 52 minutes to run on this Raspberry Pi cluster. For comparison, we ran the same domain (same setup, same WRF, same libraries, same domain, same parametrization, same everything) on a Dell PowerEdge R530 with 2 x Intel Xeon CPU E5-2640 @ 2.40GHz (10 cores per socket, 2 threads per core, 40 cores total) and it took an average of 35 minutes.

Now you can definitely setup a much smaller domain and reduce the total compute time but I still don't think you might be able to run the model for each of the 4 daily cycles available from global models (GFS in this case which was what we used). However, I do think it's perfect to run 2 cycles, for example, 00UTC and 12UTC.

Happy to answer any other questions you might have! :D

mp_pi3
Posts: 2
Joined: Thu Oct 05, 2017 7:02 pm

Re: WRF & NetCDF on Raspberry-Pi

Sat Oct 07, 2017 12:56 pm

Well that's impressive! And how is the system in terms of power consumption?
everdaniel wrote:
Thu Oct 05, 2017 10:36 pm
Hi there!

the run time depends a lot, in my particular case, on how big the domain is in the WRF Model and the parametrization used for the simulation, but, to give you an idea, these are the numbers we got on our research:
- Domain Grid: 100 cells by 100 cells
- Resolution: 5km (each cell is a 5km square)
- Total hours simulated: 6
- Compute nodes: 20 Raspberry Pi's
- Login node: 1 Raspberry Pi (not used for compute)

Based on the above, we run a few tests using a different number of cores and the average run time for each number of cores configuration:
64 cores: 53 minutes
72 cores: 1 hour, 2 minutes
80 cores: 51 minutes

A full WRF run, simulating 72 hours, takes 8 hours 52 minutes to run on this Raspberry Pi cluster. For comparison, we ran the same domain (same setup, same WRF, same libraries, same domain, same parametrization, same everything) on a Dell PowerEdge R530 with 2 x Intel Xeon CPU E5-2640 @ 2.40GHz (10 cores per socket, 2 threads per core, 40 cores total) and it took an average of 35 minutes.

Now you can definitely setup a much smaller domain and reduce the total compute time but I still don't think you might be able to run the model for each of the 4 daily cycles available from global models (GFS in this case which was what we used). However, I do think it's perfect to run 2 cycles, for example, 00UTC and 12UTC.

Happy to answer any other questions you might have! :D

everdaniel
Posts: 8
Joined: Thu Aug 30, 2012 4:00 pm
Location: Encarnacion, Paraguay
Contact: Website AOL

Re: WRF & NetCDF on Raspberry-Pi

Sat Oct 07, 2017 1:02 pm

mp_pi3 wrote:
Sat Oct 07, 2017 12:56 pm
Well that's impressive! And how is the system in terms of power consumption?
That's a really good question, however, we did not measure power consumption, how we would go about doing that?

Return to “Other languages”

Who is online

Users browsing this forum: No registered users and 7 guests