User avatar
yv1hx
Posts: 368
Joined: Sat Jul 21, 2012 10:09 pm
Location: Zulia, Venezuela
Contact: Website

WRF & NetCDF on Raspberry-Pi

Thu Oct 04, 2012 11:01 pm

Hi R-Pi fanatics!

I´m trying to build the Weather forecasting model (WRF) on my R-Pi, towards to build a two nodes cluster when my second R-Pi arrives at some point, but I cannot build the NetCDF package because i´m always getting the message:

Code: Select all

<snip>
checking for ptrdiff_t... yes
checking for stdlib.h... (cached) yes
checking for GNU libc compatible malloc... yes
checking for special C compiler options needed for large files... no
checking for _FILE_OFFSET_BITS value needed for large files... 64
checking netcdf.h usability... no
checking netcdf.h presence... no
checking for netcdf.h... no
configure: error: netcdf.h could not be found. Please set CPPFLAGS.
pi@raspberrypi:~/netcdf-cxx4-4.2$ 
Has anyone tried to build & install those packages in a R-Pi?

EDIT: I goggled about the ´configure: error: netcfd ....' but the proposed solutions where not successful

P.S.:
This is a purely academic exercise, I'm more a "hardware guy" than a "linux guy", so don't blame me please... and i´m aware the fact the R-Pi can be not enough powerful to run those tools :mrgreen:

Thank you in advance...
Marco-Luis
Telecom Specialist (Now Available for Hire!)

http://www.meteoven.org
http://yv1hx.ddns.net
http://twitter.com/yv1hx

BlackJack
Posts: 288
Joined: Sat Aug 04, 2012 8:28 am
Contact: Website

Re: WRF & NetCDF on Raspberry-Pi

Fri Oct 05, 2012 6:34 pm

@yv1hx: It seems you have not installed the netcdf library or at least not the development headers for it. The package name for the headers should be `libnetcdf-dev`.

Code: Select all

while not self.asleep():
    sheep += 1

User avatar
yv1hx
Posts: 368
Joined: Sat Jul 21, 2012 10:09 pm
Location: Zulia, Venezuela
Contact: Website

Re: WRF & NetCDF on Raspberry-Pi

Sat Nov 10, 2012 5:51 pm

BlackJack wrote:@yv1hx: It seems you have not installed the netcdf library or at least not the development headers for it. The package name for the headers should be `libnetcdf-dev`.
Dear BlackJack, I have this project in the freezer for now, a bunch of bad things have occurred to my and my parents in the last month :? so I was forced to stop all of my research.

The good new is: I have my second R-Pi, and two 16GB SD cards for continuing my experiments :mrgreen:

Thank you for point me the way....
Marco-Luis
Telecom Specialist (Now Available for Hire!)

http://www.meteoven.org
http://yv1hx.ddns.net
http://twitter.com/yv1hx

User avatar
yv1hx
Posts: 368
Joined: Sat Jul 21, 2012 10:09 pm
Location: Zulia, Venezuela
Contact: Website

Re: WRF & NetCDF on Raspberry-Pi

Sun Dec 30, 2012 7:19 pm

Well, back to playing with my two RP-i's:

A colleague, well trained with the WRF package, has been guiding me about this topic, just for the records:

0) the netcdf sources are here: http://www.unidata.ucar.edu/downloads/netcdf/index.jsp
1) The netcdf MUST be compiled using the same compiler than the rest of the packages to be compiled later (HDF5 , and others); at this point I'm very unsure what compiler is using the automated scripts :oops:

After trying the builds 3 or 4 times, ]I have a serious doubt about the c compiler that is using the automated scripts, sorry for asking this silly question, but would be possible have 2 or more c compilers installed at the same time ?
Marco-Luis
Telecom Specialist (Now Available for Hire!)

http://www.meteoven.org
http://yv1hx.ddns.net
http://twitter.com/yv1hx

everdaniel
Posts: 11
Joined: Thu Aug 30, 2012 4:00 pm
Location: Encarnacion, Paraguay
Contact: Website AOL

Re: WRF & NetCDF on Raspberry-Pi

Sat Oct 29, 2016 7:31 pm

Hi yv1hx,

did you managed to compile WRF on a Raspberry Pi? Interesting to know how you managed to do it if you did.

Thanks!

User avatar
yv1hx
Posts: 368
Joined: Sat Jul 21, 2012 10:09 pm
Location: Zulia, Venezuela
Contact: Website

Re: WRF & NetCDF on Raspberry-Pi

Sun Oct 30, 2016 5:55 pm

everdaniel wrote:Hi yv1hx,

did you managed to compile WRF on a Raspberry Pi? Interesting to know how you managed to do it if you did.

Thanks!
Nop, I cannot compile the WRF on the Raspberry Pi.

As far as I can remember, the package needs an Intel processor, also in the case I can manage to compile, I suspect that would be need a lot more of computing power that can deliver an Rpi.

Good luck.
Marco-Luis
Telecom Specialist (Now Available for Hire!)

http://www.meteoven.org
http://yv1hx.ddns.net
http://twitter.com/yv1hx

everdaniel
Posts: 11
Joined: Thu Aug 30, 2012 4:00 pm
Location: Encarnacion, Paraguay
Contact: Website AOL

Re: WRF & NetCDF on Raspberry-Pi

Sun Oct 30, 2016 8:04 pm

Hi yv1hx and everyone,

I'm very happy to report that I was finally able to compile WRF 3.8.1 and WPS 3.8.1 on the Raspberry Pi. I've been working on this the past two months occasionally and the last two weeks more intensively, and was finally able to pull it off.

Basically, this is the stack and library versions I used:
- most recent version of Raspbian, currently Raspbian Jessy Lite
- MPICH 3.0.4
- NETCDF 4.1.3
- Jasper 1.900.1
- libpng 1.2.50
- ZLib 1.2.7
- WRF 3.8.1
- WPS 3.8.1

Based on what I was able to find, it's possible in theory to compile all versions of WRF all the way down WRF 3.2 although I haven't tried this yet.

I focused on 3.8.1 since that was the most recent version, I was able to compile it in both serial and dmpar modes.

I'm going to organise my notes and then publish a post explaining how I did it since I was not able to find a lot of info online (only a few notes here and there), I'm sure there will be other people who will try to do the same thing in the future.

Please see screenshots:
Screen Shot 2016-10-30 at 4.57.23 PM.png
WRF binaries compiled on the Raspberry Pi
Screen Shot 2016-10-30 at 4.57.23 PM.png (44.16 KiB) Viewed 5107 times
Screen Shot 2016-10-30 at 4.58.11 PM.png
WPS binaries compiled on the Raspberry Pi
Screen Shot 2016-10-30 at 4.58.11 PM.png (27.39 KiB) Viewed 5107 times
Going to work on running a few of the test cases available on the WRF site now! :D

everdaniel
Posts: 11
Joined: Thu Aug 30, 2012 4:00 pm
Location: Encarnacion, Paraguay
Contact: Website AOL

Re: WRF & NetCDF on Raspberry-Pi

Sun Oct 30, 2016 8:09 pm

In my excitement, I forgot to mention one other thing, our intention (I'm working on this with a friend) is to run the WRF on a cluster of Pi's. Our idea is to get between 24 or 32 Pi's to build the cluster and then run WRF on it (it depends on our budget).

We already have WRF running on a non-arm cluster, now that we have the compilations step solved on the Pi, we will now start working on building the cluster and then running WRF on it.

Will report back here once we have that up and running as well.

User avatar
yv1hx
Posts: 368
Joined: Sat Jul 21, 2012 10:09 pm
Location: Zulia, Venezuela
Contact: Website

Re: WRF & NetCDF on Raspberry-Pi

Mon Oct 31, 2016 3:17 am

everdaniel wrote:In my excitement, I forgot to mention one other thing, our intention (I'm working on this with a friend) is to run the WRF on a cluster of Pi's. Our idea is to get between 24 or 32 Pi's to build the cluster and then run WRF on it (it depends on our budget).

We already have WRF running on a non-arm cluster, now that we have the compilations step solved on the Pi, we will now start working on building the cluster and then running WRF on it.

Will report back here once we have that up and running as well.
Huge congrats to you and your friend. :D :D

At some point I was very pessimistic about the success of this experiment, in fact, since the forced closing of my former Meteorology Research unit I lost totally the interest.

Well done, please keep us posted!
Marco-Luis
Telecom Specialist (Now Available for Hire!)

http://www.meteoven.org
http://yv1hx.ddns.net
http://twitter.com/yv1hx

wefoust
Posts: 1
Joined: Mon May 29, 2017 6:46 pm

Re: WRF & NetCDF on Raspberry-Pi

Mon May 29, 2017 6:54 pm

Hello everdaniel,
I am trying install WRF 3.8.1 on a Raspberry Pi, but I am having trouble compiling it. My first issue was that there were no build options when I I ran ./configure. When I did so, I received the following prompt.

"Please select from among the following supported platforms.


Enter selection [1-0]"

I think I fixed this by going to the WRFV3/arch directory and editing the file "configure_new.defaults". Here I replaced all references of ‘x86_64’ with ‘armv7l’, and this forced the script to bring up my build options. Now, I am can't seem to compile the model with any of the test cases. It makes an attempt at compiling, but I just get a log.compile file with many fatal errors.

Did you have similar trouble when you compiled the model? If so, how did you solve the problem. As a reference, I am using a Raspberry Pi 3 Model B running Raspbian GNU/Linux 8 (jessie). Any help you can provide will be appreciated.

Thanks,
Eliott

uupadhyaya
Posts: 1
Joined: Thu Aug 17, 2017 10:51 am

Re: WRF & NetCDF on Raspberry-Pi

Thu Aug 17, 2017 11:02 am

I could configure WRF 3.8.1 in my Raspberry Pi 2 Model B+ . However, configuring WPS is not providing me full options.

./configure
Will use NETCDF in dir: /home/rroy/Develops/Dependencies/NetCDF/NetCDF-4.4.4.1[4.4.4]
Found Jasper environment variables for GRIB2 support...
$JASPERLIB = /home/rroy/Develops/Dependencies/Grib2/Jasper/Jasper-2.0.12/lib
$JASPERINC = /home/rroy/Develops/Dependencies/Grib2/Jasper/Jasper-2.0.12/include
------------------------------------------------------------------------
Please select from among the following supported platforms.


Enter selection [1-0] :

Can anyone suggest which files needs to be edited?
Thank You.
Best,
Umesh.

everdaniel
Posts: 11
Joined: Thu Aug 30, 2012 4:00 pm
Location: Encarnacion, Paraguay
Contact: Website AOL

Re: WRF & NetCDF on Raspberry-Pi

Sun Aug 27, 2017 3:42 pm

Hi Eliott, really sorry! I saw your message just now.

Were you able to move forward with this? I did exactly what you did as well, I edited the configure_new.defaults and copy one of the GFortran/x86_64 block at the bottom of the file and replace it with armv7 (I think), I still have the notes handy so let me know and I'll share them with you.

Let me know!

wefoust wrote:
Mon May 29, 2017 6:54 pm
Hello everdaniel,
I am trying install WRF 3.8.1 on a Raspberry Pi, but I am having trouble compiling it. My first issue was that there were no build options when I I ran ./configure. When I did so, I received the following prompt.

"Please select from among the following supported platforms.


Enter selection [1-0]"

I think I fixed this by going to the WRFV3/arch directory and editing the file "configure_new.defaults". Here I replaced all references of ‘x86_64’ with ‘armv7l’, and this forced the script to bring up my build options. Now, I am can't seem to compile the model with any of the test cases. It makes an attempt at compiling, but I just get a log.compile file with many fatal errors.

Did you have similar trouble when you compiled the model? If so, how did you solve the problem. As a reference, I am using a Raspberry Pi 3 Model B running Raspbian GNU/Linux 8 (jessie). Any help you can provide will be appreciated.

Thanks,
Eliott

everdaniel
Posts: 11
Joined: Thu Aug 30, 2012 4:00 pm
Location: Encarnacion, Paraguay
Contact: Website AOL

Re: WRF & NetCDF on Raspberry-Pi

Sun Aug 27, 2017 3:44 pm

Hi Umesh,

you don't get any compilation options becase the default configure.defaults doesn't have compilation options for the arm architecture.

I know this is a late reply so let me know if you still have this problem, if that's the case, I'll share my notes with you.
uupadhyaya wrote:
Thu Aug 17, 2017 11:02 am
I could configure WRF 3.8.1 in my Raspberry Pi 2 Model B+ . However, configuring WPS is not providing me full options.

./configure
Will use NETCDF in dir: /home/rroy/Develops/Dependencies/NetCDF/NetCDF-4.4.4.1[4.4.4]
Found Jasper environment variables for GRIB2 support...
$JASPERLIB = /home/rroy/Develops/Dependencies/Grib2/Jasper/Jasper-2.0.12/lib
$JASPERINC = /home/rroy/Develops/Dependencies/Grib2/Jasper/Jasper-2.0.12/include
------------------------------------------------------------------------
Please select from among the following supported platforms.


Enter selection [1-0] :

Can anyone suggest which files needs to be edited?
Thank You.
Best,
Umesh.

mp_pi3
Posts: 2
Joined: Thu Oct 05, 2017 7:02 pm

Re: WRF & NetCDF on Raspberry-Pi

Thu Oct 05, 2017 7:16 pm

everdaniel wrote:
Sun Oct 30, 2016 8:09 pm
In my excitement, I forgot to mention one other thing, our intention (I'm working on this with a friend) is to run the WRF on a cluster of Pi's. Our idea is to get between 24 or 32 Pi's to build the cluster and then run WRF on it (it depends on our budget).

We already have WRF running on a non-arm cluster, now that we have the compilations step solved on the Pi, we will now start working on building the cluster and then running WRF on it.

Will report back here once we have that up and running as well.
Hi everdaniel,

Very interesting project! I run the WRF on my desktop (old and slow...) and I'm wondering how fast it would run on a Pi's cluster or how fast it run in a single Pi. Do you have some stats of run times on a single Pi for some test case?

Thanks!

everdaniel
Posts: 11
Joined: Thu Aug 30, 2012 4:00 pm
Location: Encarnacion, Paraguay
Contact: Website AOL

Re: WRF & NetCDF on Raspberry-Pi

Thu Oct 05, 2017 10:36 pm

mp_pi3 wrote:
Thu Oct 05, 2017 7:16 pm
Hi everdaniel,

Very interesting project! I run the WRF on my desktop (old and slow...) and I'm wondering how fast it would run on a Pi's cluster or how fast it run in a single Pi. Do you have some stats of run times on a single Pi for some test case?

Thanks!
Hi there!

the run time depends a lot, in my particular case, on how big the domain is in the WRF Model and the parametrization used for the simulation, but, to give you an idea, these are the numbers we got on our research:
- Domain Grid: 100 cells by 100 cells
- Resolution: 5km (each cell is a 5km square)
- Total hours simulated: 6
- Compute nodes: 20 Raspberry Pi's
- Login node: 1 Raspberry Pi (not used for compute)

Based on the above, we run a few tests using a different number of cores and the average run time for each number of cores configuration:
64 cores: 53 minutes
72 cores: 1 hour, 2 minutes
80 cores: 51 minutes

A full WRF run, simulating 72 hours, takes 8 hours 52 minutes to run on this Raspberry Pi cluster. For comparison, we ran the same domain (same setup, same WRF, same libraries, same domain, same parametrization, same everything) on a Dell PowerEdge R530 with 2 x Intel Xeon CPU E5-2640 @ 2.40GHz (10 cores per socket, 2 threads per core, 40 cores total) and it took an average of 35 minutes.

Now you can definitely setup a much smaller domain and reduce the total compute time but I still don't think you might be able to run the model for each of the 4 daily cycles available from global models (GFS in this case which was what we used). However, I do think it's perfect to run 2 cycles, for example, 00UTC and 12UTC.

Happy to answer any other questions you might have! :D

mp_pi3
Posts: 2
Joined: Thu Oct 05, 2017 7:02 pm

Re: WRF & NetCDF on Raspberry-Pi

Sat Oct 07, 2017 12:56 pm

Well that's impressive! And how is the system in terms of power consumption?
everdaniel wrote:
Thu Oct 05, 2017 10:36 pm
Hi there!

the run time depends a lot, in my particular case, on how big the domain is in the WRF Model and the parametrization used for the simulation, but, to give you an idea, these are the numbers we got on our research:
- Domain Grid: 100 cells by 100 cells
- Resolution: 5km (each cell is a 5km square)
- Total hours simulated: 6
- Compute nodes: 20 Raspberry Pi's
- Login node: 1 Raspberry Pi (not used for compute)

Based on the above, we run a few tests using a different number of cores and the average run time for each number of cores configuration:
64 cores: 53 minutes
72 cores: 1 hour, 2 minutes
80 cores: 51 minutes

A full WRF run, simulating 72 hours, takes 8 hours 52 minutes to run on this Raspberry Pi cluster. For comparison, we ran the same domain (same setup, same WRF, same libraries, same domain, same parametrization, same everything) on a Dell PowerEdge R530 with 2 x Intel Xeon CPU E5-2640 @ 2.40GHz (10 cores per socket, 2 threads per core, 40 cores total) and it took an average of 35 minutes.

Now you can definitely setup a much smaller domain and reduce the total compute time but I still don't think you might be able to run the model for each of the 4 daily cycles available from global models (GFS in this case which was what we used). However, I do think it's perfect to run 2 cycles, for example, 00UTC and 12UTC.

Happy to answer any other questions you might have! :D

everdaniel
Posts: 11
Joined: Thu Aug 30, 2012 4:00 pm
Location: Encarnacion, Paraguay
Contact: Website AOL

Re: WRF & NetCDF on Raspberry-Pi

Sat Oct 07, 2017 1:02 pm

mp_pi3 wrote:
Sat Oct 07, 2017 12:56 pm
Well that's impressive! And how is the system in terms of power consumption?
That's a really good question, however, we did not measure power consumption, how we would go about doing that?

alex_ant
Posts: 3
Joined: Thu May 10, 2018 3:27 pm

Re: WRF & NetCDF on Raspberry-Pi

Thu May 10, 2018 4:00 pm

Hi,

I know that it has been while since this has been posted but is there a chance you can provide some more info on how you accomplished installing WRF and WPS?

I tried wrf 3.8.1 and 3.9.1.1 without any luck when following this http://www2.mmm.ucar.edu/wrf/OnLineTuto ... torial.php

My hardware is pi 3 b+

Alex

User avatar
mahjongg
Forum Moderator
Forum Moderator
Posts: 10583
Joined: Sun Mar 11, 2012 12:19 am
Location: South Holland, The Netherlands

Re: WRF & NetCDF on Raspberry-Pi

Thu May 10, 2018 10:42 pm

slim chance as that was his last post on this forum.

well it seems I was a bit pessimistic. :oops: :mrgreen:

everdaniel
Posts: 11
Joined: Thu Aug 30, 2012 4:00 pm
Location: Encarnacion, Paraguay
Contact: Website AOL

Re: WRF & NetCDF on Raspberry-Pi

Fri May 11, 2018 9:42 pm

Hi Alex!

I've been planning on writing a blog post explaining what I did but I never find the time...

ANYWAYS!

here is what you need to do: increate the Raspbian swap size, from the default 100 to 2048 (you can probably do it with less but I didn't tested it), you can do this doing the following:

edit /etc/dphys-swapfile

change
CONF_SWAPSIZE=100
to
CONF_SWAPSIZE=2048
(2GB)

then reload

sudo /etc/init.d/dphys-swapfile stop
sudo /etc/init.d/dphys-swapfile start

and just in case, I would reboot.

Once this change is done, you can then safely follow the WRF compilation tutorial and you should be able to have the binaries on your system.

I stumble into quite a few more issues (although not compilation related) later on so if you need help, just shoot me an email to everdaniel at gmail dot com, will be more than happy to help!

Best!
alex_ant wrote:
Thu May 10, 2018 4:00 pm
Hi,

I know that it has been while since this has been posted but is there a chance you can provide some more info on how you accomplished installing WRF and WPS?

I tried wrf 3.8.1 and 3.9.1.1 without any luck when following this http://www2.mmm.ucar.edu/wrf/OnLineTuto ... torial.php

My hardware is pi 3 b+

Alex

everdaniel
Posts: 11
Joined: Thu Aug 30, 2012 4:00 pm
Location: Encarnacion, Paraguay
Contact: Website AOL

Re: WRF & NetCDF on Raspberry-Pi

Fri May 11, 2018 9:44 pm

For any person in the future who came to this post looking to know how to compile the WRF on the Raspberry Pi, please see comment above, that's the gist of it (increasing the swap size on the Raspberry Pi), for a more "complete" explanation you can visit my blog at www.everdaniel.com - I'll be posting more details there soon!

Best!

alex_ant
Posts: 3
Joined: Thu May 10, 2018 3:27 pm

Re: WRF & NetCDF on Raspberry-Pi

Sat May 12, 2018 7:41 am

Hi!

Thanks for your reply!
I had indeed added swap space and now WRF and WPS have compiled.
While geogrid,ungrid and metgrid finish ok, real.ex and wrf.exe exit with this message (WRF 3.6.1):

taskid: 0 hostname: node01
------ ERROR while reading namelist physics ------
------ ERROR while reading namelist dynamics ------
-------------- FATAL CALLED ---------------
FATAL CALLED FROM FILE: <stdin> LINE: 9965
ERRORS while reading one or more namelists from namelist.input.
-------------------------------------------
application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0


My version is compile for gfortran and dmpar.

Maybe someone has seen this before? I am still trying to understand, thinking it might be memory related the real_em case i am trying to run has a very small grid and 24 hours of forecast (will try even a shorter forecast window)

everdaniel
Posts: 11
Joined: Thu Aug 30, 2012 4:00 pm
Location: Encarnacion, Paraguay
Contact: Website AOL

Re: WRF & NetCDF on Raspberry-Pi

Sat May 12, 2018 2:39 pm

Hi Alex,

that error sounds like an issue with your namelist.input, more specifically, with parameterization in the physics and dynamics section?

I also ran into a number of similar issues while trying to use some options outlined in the documentation, just to find out later that they either don't work at all, or they are options for specific HPC clusters.

So what I would do is focus instead on getting some of test cases running on your cluster, and slowly move from there, try changing/adding a new option, try to run, if it works then try the next option, and so forth
alex_ant wrote:
Sat May 12, 2018 7:41 am
Hi!

Thanks for your reply!
I had indeed added swap space and now WRF and WPS have compiled.
While geogrid,ungrid and metgrid finish ok, real.ex and wrf.exe exit with this message (WRF 3.6.1):

taskid: 0 hostname: node01
------ ERROR while reading namelist physics ------
------ ERROR while reading namelist dynamics ------
-------------- FATAL CALLED ---------------
FATAL CALLED FROM FILE: <stdin> LINE: 9965
ERRORS while reading one or more namelists from namelist.input.
-------------------------------------------
application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0


My version is compile for gfortran and dmpar.

Maybe someone has seen this before? I am still trying to understand, thinking it might be memory related the real_em case i am trying to run has a very small grid and 24 hours of forecast (will try even a shorter forecast window)

alex_ant
Posts: 3
Joined: Thu May 10, 2018 3:27 pm

Re: WRF & NetCDF on Raspberry-Pi

Sun May 13, 2018 7:00 pm

it turns out that WRF 3.6.1 would not work at all for me. I was using namelist files from another system that I have WRF running in, so I was sure that there was no problem there.

I managed to compile WRF 3.9.1.1 and WPS 3.8.1 and I can confirm that they run fine on Raspberry Pi 3B+ (at least for real_em case of compilation) .

In the tutorial here http://www2.mmm.ucar.edu/wrf/OnLineTuto ... torial.php I had to change only a couple of things to make this work, more specifically the architecture in a couple of environmental variables in BASH

export FCFLAGS=-march=armv6
export FFLAGS=-march=armv6

Also I have added swap space as Everdaniel mentions above (for compiling it was ok if the swap is or example in /var/swap but in general to move it out of the filesystem I believe)

Return to “Other programming languages”

Who is online

Users browsing this forum: No registered users and 5 guests