ejolson
Posts: 6040
Joined: Tue Mar 18, 2014 11:47 am

Re: 2 node raspberry pi supercomputer

Mon Jan 04, 2016 6:03 pm

fruitoftheloom wrote: I ran iPerf using a USBNET Cable and I had similar results to mikerr, unfortunately do not have a A+ for comparison, see:

viewtopic.php?f=36&t=131042
Aha, that suggests the built-in hub of the 2B is not the bottleneck. Note that running iPerf seems just as good if not better than running netperf. Knowing whether Zero to Zero communication with the Pi 2B serving as a router between them results in any additional slowdown would be very interesting.

fruitoftheloom
Posts: 24578
Joined: Tue Mar 25, 2014 12:40 pm
Location: Delightful Dorset

Re: 2 node raspberry pi supercomputer

Mon Jan 04, 2016 6:17 pm

ejolson wrote:
fruitoftheloom wrote: I ran iPerf using a USBNET Cable and I had similar results to mikerr, unfortunately do not have a A+ for comparison, see:

viewtopic.php?f=36&t=131042
Aha, that suggests the built-in hub of the 2B is not the bottleneck. Note that running iPerf seems just as good if not better than running netperf. Knowing whether Zero to Zero communication with the Pi 2B serving as a router between them results in any additional slowdown would be very interesting.
It does seem that the LAN9512/4 (which is not a hub par-se) used in the B / B+ / 2B is a fairly sophisticated chipset:

http://www.microchip.com/wwwproducts/De ... ct=LAN9514
Thinking outside the box is better than burying your head in the sand...

mikerr
Posts: 2826
Joined: Thu Jan 12, 2012 12:46 pm
Location: UK
Contact: Website

Re: 2 node raspberry pi supercomputer

Mon Jan 04, 2016 7:51 pm

B+ to Zero:

Code: Select all

pi@bplus:~ $ iperf -c 169.254.64.64
------------------------------------------------------------
Client connecting to 169.254.64.64, TCP port 5001
TCP window size: 43.8 KByte (default)
------------------------------------------------------------
[  3] local 169.254.136.56 port 38622 connected with 169.254.64.64 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  45.0 MBytes  37.6 Mbits/sec


pi@bplus:~ $ iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[  4] local 169.254.136.56 port 5001 connected with 169.254.64.64 port 57334
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.1 sec  85.2 MBytes  71.0 Mbits/sec
Android app - Raspi Card Imager - download and image SD cards - No PC required !

ejolson
Posts: 6040
Joined: Tue Mar 18, 2014 11:47 am

Re: 2 node raspberry pi supercomputer

Mon Jan 04, 2016 8:16 pm

mikerr wrote:B+ to Zero
I'm I right to interpret this as 71 mbps upstream from Zero to B+ but only 37.6 mbps downstream from the B+ to the Zero? What about Zero to Zero speeds?

fruitoftheloom
Posts: 24578
Joined: Tue Mar 25, 2014 12:40 pm
Location: Delightful Dorset

Re: 2 node raspberry pi supercomputer

Mon Jan 04, 2016 8:48 pm

ejolson wrote:
mikerr wrote:B+ to Zero
I'm I right to interpret this as 71 mbps upstream from Zero to B+ but only 37.6 mbps downstream from the B+ to the Zero? What about Zero to Zero speeds?
I have tested the RPi Zero iPerf Server to my Lubuntu PC iPerf Client with the USBNET Cable:

Code: Select all

iperf -c 10.42.0.95
------------------------------------------------------------
Client connecting to 10.42.0.95, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  3] local 10.42.0.1 port 42384 connected with 10.42.0.95 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   156 MBytes   131 Mbits/sec
Thinking outside the box is better than burying your head in the sand...

mikerr
Posts: 2826
Joined: Thu Jan 12, 2012 12:46 pm
Location: UK
Contact: Website

Re: 2 node raspberry pi supercomputer

Mon Jan 04, 2016 9:53 pm

Hmm - my skills are failing me with multiple pizeros :?

Central Pi B+ ends up with the following network interfaces:

eth0: 192.168.0.100 + rest of network
usb0: 192.168.1.1
usb1: 192.168.2.1
usb2: 192.168.3.1

where usb pi zeroes are:

192.168.1.2
192.168.2.2
192.168.3.2

I can get one of the zeroes to have the internet via iptables, but my skills fail me on more than 2 interfaces.
At present B+ can talk to all zeroes, but zeroes can't talk directly to each other....

Any iptables gurus help?
Android app - Raspi Card Imager - download and image SD cards - No PC required !

User avatar
DougieLawson
Posts: 40246
Joined: Sun Jun 16, 2013 11:19 pm
Location: A small cave in deepest darkest Basingstoke, UK
Contact: Website Twitter

Re: 2 node raspberry pi supercomputer

Mon Jan 04, 2016 11:01 pm

mikerr wrote:
192.168.1.2
192.168.2.2
192.168.3.2
I think you'd find it easier to put all of the zeros in the same subnet. 192.168.1.xxx then a single iptables definition should route all three to the eth0 interface and out to the public internet.
Criticising any questions is banned on this forum.

Any DMs sent on Twitter will be answered next month.
All fake doctors are on my foes list.

Note: Any requirement to use a crystal ball or mind reading will result in me ignoring your question.

ejolson
Posts: 6040
Joined: Tue Mar 18, 2014 11:47 am

Re: 2 node raspberry pi supercomputer

Mon Jan 04, 2016 11:16 pm

DougieLawson wrote:
mikerr wrote:
192.168.1.2
192.168.2.2
192.168.3.2
I think you'd find it easier to put all of the zeros in the same subnet. 192.168.1.xxx then a single iptables definition should route all three to the eth0 interface and out to the public internet.
As each Pi Zero is actually connected to a different "physical" network, putting them in separate subnets seems reasonable to me. Otherwise, you would have to use netmasks that look like 255.255.255.254 in your routing table. Have you enabled IP forwarding in the kernel and made ACCEPT the default policy? Try something like

Code: Select all

# echo 1 > /proc/sys/net/ipv4/ip_forward
# iptables -P FORWARD ACCEPT
Also, what do the routing tables look like on the Pi B+ and Pi Zeros?

Code: Select all

# route -n

User avatar
Jim Manley
Posts: 1600
Joined: Thu Feb 23, 2012 8:41 pm
Location: SillyCon Valley, California, and Powell, Wyoming, USA, plus The Universe
Contact: Website

Re: 2 node raspberry pi supercomputer

Tue Jan 05, 2016 5:51 am

mikerr wrote:I can't seem to get >100Mbps out though, despite the virtual ethernet reporting 425Mbps,
similar to your usbnet tests.
As I've posted elsewhere, you're never going to see anywhere near the full _theoretical_ maximum _instantaneous_ 480 Mbps of USB 2.0 on any system, let alone on a Pi with its constrained USB port on the SoC (even without the Ethernet/USB controller) and other things going on with the CPU, and especially not with two or more devices on the bus. Noise, impedance-matching imperfections that cause reflected signals (including even minor connector dirt/corrosion, cable loops/kinks, etc.), and the asynchronous nature of such serial interfaces all degrade maximum actual goodput of bits successfully delivered intact.

A virtual interface is just that - it doesn't physically exist and is a software-driven artifact. Even loopback interfaces don't perform at the maximum theoretical instantaneous bit rate - there is a readily-measurable propagation delay within the interface hardware itself.
The best things in life aren't things ... but, a Pi comes pretty darned close! :D
"Education is not the filling of a pail, but the lighting of a fire." -- W.B. Yeats
In theory, theory & practice are the same - in practice, they aren't!!!

fruitoftheloom
Posts: 24578
Joined: Tue Mar 25, 2014 12:40 pm
Location: Delightful Dorset

Re: 2 node raspberry pi supercomputer

Tue Jan 05, 2016 7:37 am

Jim Manley wrote:
mikerr wrote:I can't seem to get >100Mbps out though, despite the virtual ethernet reporting 425Mbps,
similar to your usbnet tests.
As I've posted elsewhere, you're never going to see anywhere near the full _theoretical_ maximum _instantaneous_ 480 Mbps of USB 2.0 on any system, let alone on a Pi with its constrained USB port on the SoC (even without the Ethernet/USB controller) and other things going on with the CPU, and especially not with two or more devices on the bus. Noise, impedance-matching imperfections that cause reflected signals (including even minor connector dirt/corrosion, cable loops/kinks, etc.), and the asynchronous nature of such serial interfaces all degrade maximum actual goodput of bits successfully delivered intact.

A virtual interface is just that - it doesn't physically exist and is a software-driven artifact. Even loopback interfaces don't perform at the maximum theoretical instantaneous bit rate - there is a readily-measurable propagation delay within the interface hardware itself.
JM lost in what you are trying to impart but using different 3 different "Ethernet over USB" solutions get differing results with iPerf, see following for the 3rd:

viewtopic.php?f=63&t=131379


Though for the original topic of this thread creating a 2 Node Cluster, it would seem a RPi 1B+ or 2B as Master and 2 RPi Zero in Slave "Gadget" Mode would be the best solution....
Last edited by fruitoftheloom on Tue Jan 05, 2016 7:50 am, edited 1 time in total.
Thinking outside the box is better than burying your head in the sand...

User avatar
rpdom
Posts: 17736
Joined: Sun May 06, 2012 5:17 am
Location: Chelmsford, Essex, UK

Re: 2 node raspberry pi supercomputer

Tue Jan 05, 2016 7:45 am

DougieLawson wrote:
mikerr wrote:
192.168.1.2
192.168.2.2
192.168.3.2
I think you'd find it easier to put all of the zeros in the same subnet. 192.168.1.xxx then a single iptables definition should route all three to the eth0 interface and out to the public internet.
I thin I'd look into setting the three USB interfaces on the B+ as a bridge interface, so they share a common IP address and that will automatically route between the zeroes.

The bridge could also include eth0 to have everything on the same subnet, or use ipables if your want to filter traffic between the cluster and main lan.

Ben_is_best
Posts: 89
Joined: Fri Dec 25, 2015 11:01 pm
Location: USA

Re: 2 node raspberry pi supercomputer

Tue Jan 05, 2016 1:15 pm

fruitoftheloom wrote:Though for the original topic of this thread creating a 2 Node Cluster, it would seem a RPi 1B+ or 2B as Master and 2 RPi Zero in Slave "Gadget" Mode would be the best solution....
I know. Would it just work right when the Pi Zero got plugged in to the 2B or would you have to set up the software on the Pi 2?

echomancer
Posts: 13
Joined: Fri Feb 27, 2015 4:29 am

Re: 2 node raspberry pi supercomputer

Fri Jan 08, 2016 12:44 pm

Not to hijack the conversation, but adding some color commentary. Earlier this year I built a chassis w/ power and networking to power up to 5 PI 2s hanging off the back of the LCD to work on Einstein@Home work units. This was quite successful, and will be even more so when I upgrade everything to Jessie (no more compiling BOINC, etc). Of course when the Zero came out, I had the same desire to build a "cluster" for the same purpose - even if it's not as powerful, it would still look awesome sitting on a desk.

At first, this seemed pointless given all the extra peripherals you had to acquire, but when I came across the USB Gadget guides, I thought the problems were solved! To start, I created a Jessie Lite card, installed boinc-client, input my E@H Key, and away the little guy went. Initial impressions were good - <45C temp at full CPU utilization, and an estimated time to completion about 10% greater than what a single Pi2 core can do. Unfortunately, this has not been a smooth journey.

I'm currently attached to the Zero using a USB/TTL adapter and using the official RPi WiFi dongle with an OTG cable. It is powered with a 1A USB Power supply showing 5.02V. The Zero at full CPU utilization and WiFi powered is drawing between .2 and .25 A. Within the first three hours of operation at full CPU utilization, the machine locked up w/ no logs. After a hard reboot, the machine continued its efforts but it eventually ran in to an issue where syslogd was fighting BOINC for CPU (taking up 50% of the CPU). Another hard reset, I left it running overnight. I woke up this morning to find a kernel panic in the terminal (saved for later investigation).

There are many variables at play - power (shorter USB cable, higher PSU Voltage), trying a different Zero board, building BOINC from scratch, some other unknown configuration item. I will likely start a new thread to follow along in my adventures. I have my first Zero sized powered 4-port USB hub arriving today, and I have been designing a chassis to 3d print for the whole shebang.

ejolson
Posts: 6040
Joined: Tue Mar 18, 2014 11:47 am

Re: 2 node raspberry pi supercomputer

Fri Jan 08, 2016 5:38 pm

mikerr wrote:Hmm - my skills are failing me with multiple pizeros :?
Any progress with zero to zero communication?

mikerr
Posts: 2826
Joined: Thu Jan 12, 2012 12:46 pm
Location: UK
Contact: Website

Re: 2 node raspberry pi supercomputer

Sat Jan 09, 2016 12:06 am

zero to zero:

Code: Select all

pi@pizero2:~ $ iperf -c 192.168.1.2
------------------------------------------------------------
Client connecting to 192.168.1.2, TCP port 5001
TCP window size: 43.8 KByte (default)
------------------------------------------------------------
[  3] local 192.168.2.2 port 43780 connected with 192.168.1.2 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.1 sec  52.9 MBytes  44.0 Mbits/sec
pi@pizero2:~ $ iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[  4] local 192.168.2.2 port 5001 connected with 192.168.1.2 port 52116
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.2 sec  52.8 MBytes  43.3 Mbits/sec
Android app - Raspi Card Imager - download and image SD cards - No PC required !

fruitoftheloom
Posts: 24578
Joined: Tue Mar 25, 2014 12:40 pm
Location: Delightful Dorset

Re: 2 node raspberry pi supercomputer

Sat Jan 09, 2016 4:00 am

mikerr wrote:zero to zero:

Code: Select all

pi@pizero2:~ $ iperf -c 192.168.1.2
------------------------------------------------------------
Client connecting to 192.168.1.2, TCP port 5001
TCP window size: 43.8 KByte (default)
------------------------------------------------------------
[  3] local 192.168.2.2 port 43780 connected with 192.168.1.2 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.1 sec  52.9 MBytes  44.0 Mbits/sec
pi@pizero2:~ $ iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[  4] local 192.168.2.2 port 5001 connected with 192.168.1.2 port 52116
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.2 sec  52.8 MBytes  43.3 Mbits/sec
Does not seem particularly speedy, but at least it works :D
Thinking outside the box is better than burying your head in the sand...

ejolson
Posts: 6040
Joined: Tue Mar 18, 2014 11:47 am

Re: 2 node raspberry pi supercomputer

Sat Jan 09, 2016 10:33 pm

fruitoftheloom wrote:
mikerr wrote:zero to zero:
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.2 sec 52.8 MBytes 43.3 Mbits/sec
Does not seem particularly speedy, but at least it works :D
From my point of view creating a cluster using only a Pi B+ and three Pi zeros without any additional networking hardware is an amazing success. While one might have hoped for a faster than 200 mbps interconnect, 44 mbps is definitely sufficient for leaning about cluster computing and much faster than 4 mbps using ppp over a serial port. This opens up many learning opportunities with a very small investment in cost, space and electricity.

After bandwidth, the next question is network latency. This could be measured, for example, by an MPI ping pong test. Maybe iperf has latency test as well.

echomancer
Posts: 13
Joined: Fri Feb 27, 2015 4:29 am

Re: 2 node raspberry pi supercomputer

Sun Jan 10, 2016 12:13 am

I believe I have fixed my Oops 17 issues. I made two changes yesterday and my Pi Zero has now been chugging away with zero issues. From a thread about the PI Oopsing during an apt-get update, I went in and assigned only 16 megabytes for display memory - I was surprised to find that by default it was set to 0. Second, I wasn't aware that I had made an error by "overclocking" the Zero to 1 Ghz. I returned it to the default 700Mhz setting and verified that it is infact running at 1Ghz. All seem to be well now. I received my hub and my 1ft USB Micro Cables. I have 4 Pi Zeros here and hope to get it cobbled together over the next week.

wolf.z
Posts: 135
Joined: Thu Dec 24, 2015 8:34 pm

Re: 2 node raspberry pi supercomputer

Sun Jan 10, 2016 12:32 am

I am wondering, why nobody is speaking of networking Zeros with an enc28j60 SPI2ETH chip, making use of GPIO (SPI) pins as demanded by the thread starter Ben_is_best? He asked for networking a Pi2 and a zero only. As this thread viewtopic.php?f=91&t=131145&p=874936&hi ... 60#p874936 supposes, networking with enc28j60 is not that difficult any more. Just a straight small config job by adding a single line to config.txt. So hardware setup might end up with
on-board eth Pi2 <-> Pi2 <-> enc28j60 <-> enc28j60 <-> Zero.
In case Zeros will become available, I am planning to do exactly that. Am I missing something? In case not, could someone please give advice on how to do the iptable/routing thing for the Pi2?

Thanks
Wolfram

User avatar
mikronauts
Posts: 2817
Joined: Sat Jan 05, 2013 7:28 pm
Contact: Website

Re: 2 node raspberry pi supercomputer

Sun Jan 10, 2016 3:06 am

It will work fine, up to a theoretical maximum of 10Mbps.

You may also have limits on the number of simultaneous TCP connections (not sure, it has been a while since I read the data sheet)
wolf.z wrote:I am wondering, why nobody is speaking of networking Zeros with an enc28j60 SPI2ETH chip, making use of GPIO (SPI) pins as demanded by the thread starter Ben_is_best? He asked for networking a Pi2 and a zero only. As this thread viewtopic.php?f=91&t=131145&p=874936&hi ... 60#p874936 supposes, networking with enc28j60 is not that difficult any more. Just a straight small config job by adding a single line to config.txt. So hardware setup might end up with
on-board eth Pi2 <-> Pi2 <-> enc28j60 <-> enc28j60 <-> Zero.
In case Zeros will become available, I am planning to do exactly that. Am I missing something? In case not, could someone please give advice on how to do the iptable/routing thing for the Pi2?

Thanks
Wolfram
http://Mikronauts.com - home of EZasPi, RoboPi, Pi Rtc Dio and Pi Jumper @Mikronauts on Twitter
Advanced Robotics, I/O expansion and prototyping boards for the Raspberry Pi

User avatar
Jim Manley
Posts: 1600
Joined: Thu Feb 23, 2012 8:41 pm
Location: SillyCon Valley, California, and Powell, Wyoming, USA, plus The Universe
Contact: Website

Re: 2 node raspberry pi supercomputer

Sun Jan 10, 2016 6:39 am

ejolson wrote:While one might have hoped for a faster than 200 mbps interconnect, 44 mbps is definitely sufficient for leaning about cluster computing and much faster than 4 mbps using ppp over a serial port.
Hi ejolson - how do you expect to get 200 Mbps through an Ethernet interface that only has a 100 Mbps maximum theoretical instantaneous bit rate capatility? The 44 Mbps observed is right in line with what I've been trying to tell people to expect in real-world operation, and this is a two-node configuration AIUI, so it would be worse with more nodes on the same subnet, depending on activity levels of the various nodes.
The best things in life aren't things ... but, a Pi comes pretty darned close! :D
"Education is not the filling of a pail, but the lighting of a fire." -- W.B. Yeats
In theory, theory & practice are the same - in practice, they aren't!!!

ejolson
Posts: 6040
Joined: Tue Mar 18, 2014 11:47 am

Re: 2 node raspberry pi supercomputer

Sun Jan 10, 2016 8:20 am

mikronauts wrote:It will work fine, up to a theoretical maximum of 10Mbps.

You may also have limits on the number of simultaneous TCP connections (not sure, it has been a while since I read the data sheet)
wolf.z wrote:I am wondering, why nobody is speaking of networking Zeros with an enc28j60 SPI2ETH chip, making use of GPIO (SPI) pins as demanded by the thread starter Ben_is_best?
I don't think the enc28j60 implements a full TCP/IP stack but only hardware assisted IP checksumming and general packet filtering. The data sheet is

http://www.microchip.com/wwwproducts/De ... e=en022889

The enc28j60 connected through SPI would leave the USB port of each Pi Zero available to attach an external HD, which could be useful when building a hadoop cluster for data mining. When mimicking the architecture of a typical supercomputer, where user data is stored on a central disk array, it makes sense to use USB as the communications fabric, as there is no other need for the Pi Zero's USB port.

fruitoftheloom
Posts: 24578
Joined: Tue Mar 25, 2014 12:40 pm
Location: Delightful Dorset

Re: 2 node raspberry pi supercomputer

Sun Jan 10, 2016 10:11 am

Jim Manley wrote:
ejolson wrote:While one might have hoped for a faster than 200 mbps interconnect, 44 mbps is definitely sufficient for leaning about cluster computing and much faster than 4 mbps using ppp over a serial port.
Hi ejolson - how do you expect to get 200 Mbps through an Ethernet interface that only has a 100 Mbps maximum theoretical instantaneous bit rate capatility? The 44 Mbps observed is right in line with what I've been trying to tell people to expect in real-world operation, and this is a two-node configuration AIUI, so it would be worse with more nodes on the same subnet, depending on activity levels of the various nodes.
Unfortunately there is not much information regards USB Networking Speeds on a A / A+ / ZERO, so most people compare to a B / B+ which has the LAN9512/14 Chipset on the USB BUS

RPi Zero iPerf Server to my Lubuntu PC iPerf Client with the USBNET Cable:

Code: Select all

iperf -c 10.42.0.95
------------------------------------------------------------
Client connecting to 10.42.0.95, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  3] local 10.42.0.1 port 42384 connected with 10.42.0.95 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   156 MBytes   131 Mbits/sec
Lubuntu PC iPerf Server to my RPi Zero iPerf Client with the USBNET Cable:

Code: Select all

iperf -c 10.42.0.1 -t 180
------------------------------------------------------------
Client connecting to 10.42.0.1, TCP port 5001
TCP window size: 43.8 KByte (default)
------------------------------------------------------------
[  3] local 10.42.0.27 port 42722 connected with 10.42.0.1 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-180.0 sec   913 MBytes  42.5 Mbits/sec
Thinking outside the box is better than burying your head in the sand...

User avatar
jedi5200
Posts: 12
Joined: Sat Jan 09, 2016 8:27 pm

Re: 2 node raspberry pi supercomputer

Sun Jan 10, 2016 1:50 pm

has anyone actually gotten the rpi2+ zero to work? and how?
Would it be ironic if you died choking on a lifesaver?

fruitoftheloom
Posts: 24578
Joined: Tue Mar 25, 2014 12:40 pm
Location: Delightful Dorset

Re: 2 node raspberry pi supercomputer

Sun Jan 10, 2016 3:38 pm

jedi5200 wrote:has anyone actually gotten the rpi2+ zero to work? and how?
mikerr showed a setup earlier in this thread, if you want to direct connect a RPi 2B to a RPi ZERO running in Gadget Mode:

viewtopic.php?f=63&t=130815#p874890
Thinking outside the box is better than burying your head in the sand...

Return to “General discussion”