ejolson
Posts: 6349
Joined: Tue Mar 18, 2014 11:47 am

Re: iSCSI Root Like a Data Center

Sun May 31, 2020 12:40 am

sdwilsh wrote:
Sat May 30, 2020 11:51 pm
I was working on getting the Pi 4 to network boot as well for the past week and a half. I actually got it working without an SD card at all (after the initial setup), which is a little different from what you did.

I just finished writing it all up on https://shawnwilsher.com/2020/05/networ ... a-freenas/ and figured I'd share it here, since I came across this thread when I was debugging an issue (turned out to be a typo on my part).
Welcome to the forum!

Thanks for posting. I've been wondering whether to continue all the way to a full network boot or not. Getting rid of the SD card entirely makes the setup more like a data center. On the other hand the title of this thread intentionally did not mention booting. I've got the updated firmware installed and should be ready to network boot. I look forward to reading how you configured everything.

incognitum
Posts: 600
Joined: Tue Oct 30, 2018 3:34 pm

Re: iSCSI Root Like a Data Center

Sun May 31, 2020 1:24 pm

ejolson wrote:
Sun May 31, 2020 12:36 am
Thanks for the link. It looks like nolink is the only addition to the dhcpcd.conf file. I noticed that adding that option to the standard Raspbian image prevents dhcpd from starting the interface and acquiring a lease. If the network is already up, as happens with the iSCSI root initial RAM filesystem, does dhcpcd at least continue to maintain and renew the existing lease?
If network link state is up prior to dhcpcd being started (which should be the case in all network boot scenarios), it should indeed operate as normal.

Can check log/journal to verify.

ejolson
Posts: 6349
Joined: Tue Mar 18, 2014 11:47 am

Re: iSCSI Root Like a Data Center

Sun Jun 14, 2020 9:38 pm

So having spent the last week fitting cardboard replicas of the Raspberry Pi 4B inside a 2U server chassis in preparation for booting a box of many single board computers with iSCSI root

viewtopic.php?f=36&t=272660

I decided to do some additional performance testing before using encryption to make iSCSI root secure from a MAC flooding attack and ARP cache poisoning. The post

viewtopic.php?f=63&t=277210#p1679035

see also

https://www.raspberrypi.org/blog/sd-card-speed-test/

Putting the religious pun aside, I typed

Code: Select all

$ sudo apt-get install agnostics
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following additional packages will be installed:
  adwaita-icon-theme at-spi2-core dbus-user-session dconf-gsettings-backend
  dconf-service fio fontconfig fontconfig-config fonts-dejavu-core
  glib-networking glib-networking-common glib-networking-services
  gsettings-desktop-schemas gtk-update-icon-cache hicolor-icon-theme
  ibverbs-providers libaio1 libatk-bridge2.0-0 libatk1.0-0 libatk1.0-data
  libatspi2.0-0 libavahi-client3 libboost-atomic1.67.0
  libboost-iostreams1.67.0 libboost-regex1.67.0 libboost-system1.67.0
  libboost-thread1.67.0 libcairo-gobject2 libcairo2 libcolord2 libcroco3
  libcups2 libdatrie1 libdconf1 libepoxy0 libfontconfig1 libgdk-pixbuf2.0-0
  libgdk-pixbuf2.0-bin libgdk-pixbuf2.0-common libgfapi0 libgfrpc0 libgfxdr0
  libglusterfs0 libgraphite2-3 libgtk-3-0 libgtk-3-bin libgtk-3-common
  libgtksourceview-3.0-1 libgtksourceview-3.0-common libharfbuzz0b libibverbs1
  libjbig0 libjson-glib-1.0-0 libjson-glib-1.0-common liblcms2-2 libnspr4
  libnss3 libnuma1 libpango-1.0-0 libpangocairo-1.0-0 libpangoft2-1.0-0
  libpixman-1-0 libproxy1v5 librados2 librbd1 librdmacm1 librest-0.7-0
  librsvg2-2 librsvg2-common libsoup-gnome2.4-1 libsoup2.4-1 libthai-data
  libthai0 libtiff5 libwayland-client0 libwayland-cursor0 libwayland-egl1
  libwebp6 libxcb-render0 libxcb-shm0 libxcomposite1 libxcursor1 libxdamage1
  libxfixes3 libxi6 libxinerama1 libxkbcommon0 libxrandr2 libxrender1 libxtst6
  mousepad x11-common
Suggested packages:
  gnuplot gfio python-scipy colord cups-common gvfs liblcms2-utils
  librsvg2-bin
The following NEW packages will be installed:
  adwaita-icon-theme agnostics at-spi2-core dbus-user-session
  dconf-gsettings-backend dconf-service fio fontconfig fontconfig-config
  fonts-dejavu-core glib-networking glib-networking-common
  glib-networking-services gsettings-desktop-schemas gtk-update-icon-cache
  hicolor-icon-theme ibverbs-providers libaio1 libatk-bridge2.0-0 libatk1.0-0
  libatk1.0-data libatspi2.0-0 libavahi-client3 libboost-atomic1.67.0
  libboost-iostreams1.67.0 libboost-regex1.67.0 libboost-system1.67.0
  libboost-thread1.67.0 libcairo-gobject2 libcairo2 libcolord2 libcroco3
  libcups2 libdatrie1 libdconf1 libepoxy0 libfontconfig1 libgdk-pixbuf2.0-0
  libgdk-pixbuf2.0-bin libgdk-pixbuf2.0-common libgfapi0 libgfrpc0 libgfxdr0
  libglusterfs0 libgraphite2-3 libgtk-3-0 libgtk-3-bin libgtk-3-common
  libgtksourceview-3.0-1 libgtksourceview-3.0-common libharfbuzz0b libibverbs1
  libjbig0 libjson-glib-1.0-0 libjson-glib-1.0-common liblcms2-2 libnspr4
  libnss3 libnuma1 libpango-1.0-0 libpangocairo-1.0-0 libpangoft2-1.0-0
  libpixman-1-0 libproxy1v5 librados2 librbd1 librdmacm1 librest-0.7-0
  librsvg2-2 librsvg2-common libsoup-gnome2.4-1 libsoup2.4-1 libthai-data
  libthai0 libtiff5 libwayland-client0 libwayland-cursor0 libwayland-egl1
  libwebp6 libxcb-render0 libxcb-shm0 libxcomposite1 libxcursor1 libxdamage1
  libxfixes3 libxi6 libxinerama1 libxkbcommon0 libxrandr2 libxrender1 libxtst6
  mousepad x11-common
0 upgraded, 93 newly installed, 0 to remove and 0 not upgraded.
Need to get 45.4 MB of archives.
After this operation, 140 MB of additional disk space will be used.
Do you want to continue? [Y/n] n
Abort.
Although not agnostic, it appears humanly impossible to know whether the disk speed test really needs libboost, dbus, mousepad, the adwaita-icon-theme, wayland and 140 MB of additional fonts and libraries. I decided to meditate on the metaphysics of that question while installing the benchmark on the sacrificial Pi 3B+ currently solving the coronavirus problem at a rate of 190 points per day.

viewtopic.php?f=63&t=271456&start=175#p1678821

Upon inspecting the shell script, it appeared to be a wrapper for the fio program, so I installed fio on the 4B and copied the script over. The boost library was still needed but fortunately not mousepad. I also switched the SD card back to 50 MB/s mode to be fair.

The results for the SD card were

Code: Select all

$ bash ./sdtest.sh 
Run 1
prepare-file;0;0;9953;19
seq-write;0;0;7591;14
rand-4k-write;0;0;2529;632
rand-4k-read;11040;2760;0;0
Sequential write speed 7591 KB/sec (target 10000) - FAIL
Note that sequential write speed declines over time as a card is used - your card may require reformatting
Random write speed 632 IOPS (target 500) - PASS
Random read speed 2760 IOPS (target 1500) - PASS
Run 2
prepare-file;0;0;10163;19
seq-write;0;0;9863;19
rand-4k-write;0;0;2588;647
rand-4k-read;10871;2717;0;0
Sequential write speed 9863 KB/sec (target 10000) - FAIL
Note that sequential write speed declines over time as a card is used - your card may require reformatting
Random write speed 647 IOPS (target 500) - PASS
Random read speed 2717 IOPS (target 1500) - PASS
Run 3
prepare-file;0;0;10536;20
seq-write;0;0;9807;19
rand-4k-write;0;0;2549;637
rand-4k-read;10818;2704;0;0
Sequential write speed 9807 KB/sec (target 10000) - FAIL
Note that sequential write speed declines over time as a card is used - your card may require reformatting
Random write speed 637 IOPS (target 500) - PASS
Random read speed 2704 IOPS (target 1500) - PASS
./sdtest.sh: line 37: return: can only `return' from a function or sourced script
while the results for the iSCSI root were

Code: Select all

$ bash ./sdtest.sh
Run 1
prepare-file;0;0;77010;150
seq-write;0;0;76116;148
rand-4k-write;0;0;31859;7964
rand-4k-read;54795;13698;0;0
Sequential write speed 76116 KB/sec (target 10000) - PASS
Random write speed 7964 IOPS (target 500) - PASS
Random read speed 13698 IOPS (target 1500) - PASS
sdtest.sh: line 34: return: can only `return' from a function or sourced script
Run 2
prepare-file;0;0;76650;149
seq-write;0;0;71389;139
rand-4k-write;0;0;32395;8098
rand-4k-read;33505;8376;0;0
Sequential write speed 71389 KB/sec (target 10000) - PASS
Random write speed 8098 IOPS (target 500) - PASS
Random read speed 8376 IOPS (target 1500) - PASS
sdtest.sh: line 34: return: can only `return' from a function or sourced script
Run 3
prepare-file;0;0;65997;128
seq-write;0;0;55586;108
rand-4k-write;0;0;35617;8904
rand-4k-read;33868;8467;0;0
Sequential write speed 55586 KB/sec (target 10000) - PASS
Random write speed 8904 IOPS (target 500) - PASS
Random read speed 8467 IOPS (target 1500) - PASS
sdtest.sh: line 34: return: can only `return' from a function or sourced script
sdtest.sh: line 37: return: can only `return' from a function or sourced script
While it's interesting the script, as written, specifies bash in the #! at the beginning but is only compatible with dash, the main point was to obtain a performance information about iSCSI on the Raspberry Pi. Referring back to the post

viewtopic.php?f=63&t=277210#p1679035

we have the following table

Code: Select all

              KB/sec    IOPS Write   IOPS Read
SD Card        7591          632         2760
iSCSI         76116         7964        13698
SSD          330989        21933        15723
This suggests that iSCSI is significantly faster than the SD card but not as fast as a locally attached USB3 SSD. Of course SD cards vary in speed as do networks and SSD's. At the same time, I'm starting to understand why data centers generally do not mount root from SD cards.

ejolson
Posts: 6349
Joined: Tue Mar 18, 2014 11:47 am

Re: iSCSI Root Like a Data Center

Mon Jun 15, 2020 7:51 pm

There are two obvious ways to implement security for an iSCSI device mounted over an unsecured network:
  • Encrypt the data in motion.
  • Encrypt the data at rest.
Encrypting the data in motion can be achieved with a VPN such as WireGuard, IPSec or OpenVPN. The advantage is that the data stored on the file server is not encrypted so losing the password is no big deal. The disadvantage is that any security breach of the iSCSI target server compromises the data. The server also needs to spend a significant number of CPU cycles running the VPN.

Encrypting the data at rest can be achieved using dm-crypt for full disk encryption or a directory-based approach with ZFS or encfs. The advantage is that the file server does not need to run any decryption algorithms on the incoming data. The data also remains secure even if there is a security breach of the server. The disadvantage is that the encryption keys can not be easily changed nor is there perfect forward secrecy. Thus, an observer snooping the network for a long period will obtain more information about the secret key than with a VPN.

I have decided to use dm-crypt when mounting an iSCSI root like a data center. This allows one to use any commercial NAS to serve the iSCSI targets and keeps CPU requirements of that server to a minimum so it can scale better.

Since the default recommendation is based on the assumption of the existence of hardware AES instructions, the first thing to do is choose a cipher. I tested the available algorithms using

Code: Select all

# apt-get install cryptsetup-bin ; # Raspberry Pi 4B 32-bit
# cryptsetup benchmark
# Tests are approximate using memory only (no storage IO).
PBKDF2-sha1       358610 iterations per second for 256-bit key
PBKDF2-sha256     553046 iterations per second for 256-bit key
PBKDF2-sha512     262406 iterations per second for 256-bit key
PBKDF2-ripemd160  268590 iterations per second for 256-bit key
PBKDF2-whirlpool   48401 iterations per second for 256-bit key
argon2i       4 iterations, 248625 memory, 4 parallel threads (CPUs) for 256-bit key (requested 2000 ms time)
argon2id      4 iterations, 259391 memory, 4 parallel threads (CPUs) for 256-bit key (requested 2000 ms time)
#     Algorithm |       Key |      Encryption |      Decryption
        aes-cbc        128b        44.7 MiB/s        77.5 MiB/s
    serpent-cbc        128b               N/A               N/A
    twofish-cbc        128b               N/A               N/A
        aes-cbc        256b        36.8 MiB/s        58.5 MiB/s
    serpent-cbc        256b               N/A               N/A
    twofish-cbc        256b               N/A               N/A
        aes-xts        256b        84.9 MiB/s        74.7 MiB/s
    serpent-xts        256b               N/A               N/A
    twofish-xts        256b               N/A               N/A
        aes-xts        512b        66.3 MiB/s        57.0 MiB/s
    serpent-xts        512b               N/A               N/A
    twofish-xts        512b               N/A               N/A
and discovered the only cipher available is AES anyway. Undeterred, I decided to push forward by substituting a 20-year-old Athlon 1400 PC for the Pi 4B that also doesn't have cryptographic acceleration and obtained

Code: Select all

# cryptsetup benchmark ; # Athlon 1400 Thunderbird 32-bit
# Tests are approximate using memory only (no storage IO).
PBKDF2-sha1       202584 iterations per second for 256-bit key
PBKDF2-sha256     254015 iterations per second for 256-bit key
PBKDF2-sha512      55538 iterations per second for 256-bit key
PBKDF2-ripemd160  172236 iterations per second for 256-bit key
PBKDF2-whirlpool   37194 iterations per second for 256-bit key
#  Algorithm | Key |  Encryption |  Decryption
     aes-cbc   128b    31.7 MiB/s    43.4 MiB/s
 serpent-cbc   128b    25.3 MiB/s    26.7 MiB/s
 twofish-cbc   128b    44.4 MiB/s    49.4 MiB/s
     aes-cbc   256b    31.3 MiB/s    34.3 MiB/s
 serpent-cbc   256b    27.4 MiB/s    26.6 MiB/s
 twofish-cbc   256b    45.3 MiB/s    49.5 MiB/s
     aes-xts   256b    40.4 MiB/s    42.8 MiB/s
 serpent-xts   256b    28.1 MiB/s    26.5 MiB/s
 twofish-xts   256b    48.1 MiB/s    49.3 MiB/s
     aes-xts   512b    32.6 MiB/s    34.0 MiB/s
 serpent-xts   512b    28.4 MiB/s    26.6 MiB/s
 twofish-xts   512b    48.6 MiB/s    49.2 MiB/s
From this output one can see that twofish is faster than AES. Since the same is likely to hold true on the 4B, this, along with a fondness for the work of Dr Seuss

Image

as well as the work of Dr Schneier, makes me ask, does anyone know how to enable twofish in the Raspberry Pi OS?

Along different lines, why can't dm-crypt perform the cha cha or salsa? Are those dance-themed encryption algorithms only good for data in motion?

ejolson
Posts: 6349
Joined: Tue Mar 18, 2014 11:47 am

Re: iSCSI Root Like a Data Center

Tue Jun 16, 2020 6:04 pm

Before lowering performance by enabling AES encryption, I decided to check whether the use of jumbo packets could improve performance. The result

Code: Select all

# ip link set eth0 mtu 1501
Error: mtu greater than device maximum.
convinced me to give up on jumbo packets almost immediately. Are these difficulties with jumbo packet support on the 4B caused by the hardware or the Linux kernel?

On a Pi B+ running Raspbian Stretch with kernel 4.9.59+ I can do

Code: Select all

# ip link set eth0 mtu 7418
# ifconfig eth0
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 7418
        inet 192.168.174.154  netmask 255.255.255.128  broadcast 192.168.174.255
        inet6 fe80::6fdf:e7e8:a7cc:c6ba  prefixlen 64  scopeid 0x20<link>
        ether b8:27:eb:0b:0d:c2  txqueuelen 1000  (Ethernet)
        RX packets 363772  bytes 48626516 (46.3 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 324169  bytes 51619584 (49.2 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

# ip link set eth0 mtu 1500
If anyone reading this has a B+ running a recent release of Raspberry Pi OS 32-bit, could they try the above sequence of commands?

ejolson
Posts: 6349
Joined: Tue Mar 18, 2014 11:47 am

Re: iSCSI Root Like a Data Center

Sun Jun 21, 2020 5:30 pm

As a start for filling the case described in

viewtopic.php?t=272660#p1678218

with Pi computers, I installed a single Pi Zero next to the micro ATX motherboard. The Zero is connected to a tiny 4-port USB2 hub which is in turn connected to one of the motherboard headers. The hub is further powered by 5V from the ATX power supply in preparation for adding some Pi 4B computers later.

To understand why it's practical for a Pi 4B to mount iSCSI root like a data center, it is interesting to compare performance of doing the same with a Pi Zero. Note that the following tests are not fully comparable to the previous ones, because the iSCSI target server changed as well. In particular, the tests for the Zero were done using a disk image on a 1TB spinning disk served by an older AMD APU rather than a disk image stored on a 240GB SSD served by an ODROID-N2. I'll test the 4B with the AMD-based server in the near future.

The Zero was setup with the latest Raspberry Pi OS using an 8GB SD card purchased about 3 years ago from Amazon. The card is branded as Techkey, manufactured by Shenzhen Denos Trade Company and as stated on the packaging includes a 30-year warranty. Though the product is no longer listed on Amazon, I have no reason to believe the card was a fake. For reference the card looks like
.
techkey8GB.png
techkey8GB.png (55.68 KiB) Viewed 1626 times
.
The results for the Pi Zero with the SD card are

Code: Select all

$ sh sdtest.sh ; # Pi Zero with Techkey SD card
Run 1
prepare-file;0;0;12974;25
seq-write;0;0;15269;29
rand-4k-write;0;0;694;173
rand-4k-read;5603;1400;0;0
Sequential write speed 15269 KB/sec (target 10000) - PASS
Random write speed 173 IOPS (target 500) - FAIL
Random read speed 1400 IOPS (target 1500) - FAIL
Run 2
prepare-file;0;0;14209;27
seq-write;0;0;10701;20
rand-4k-write;0;0;666;166
rand-4k-read;5577;1394;0;0
Sequential write speed 10701 KB/sec (target 10000) - PASS
Random write speed 166 IOPS (target 500) - FAIL
Random read speed 1394 IOPS (target 1500) - FAIL
Run 3
prepare-file;0;0;15034;29
seq-write;0;0;10312;20
rand-4k-write;0;0;388;97
rand-4k-read;4993;1248;0;0
Sequential write speed 10312 KB/sec (target 10000) - PASS
Random write speed 97 IOPS (target 500) - FAIL
Random read speed 1248 IOPS (target 1500) - FAIL
The sequential write speed of the card seems to meet the 10 MB/s rating that a class 10 card should have. The slow random read and write speeds are as expected for a card designed for video recording and continuous shooting.

Testing the iSCSI network block device results in

Code: Select all

$ sh sdtest.sh ; # Pi Zero iSCSI MTU 1500
Run 1
prepare-file;0;0;9282;18
seq-write;0;0;8627;16
rand-4k-write;0;0;3354;838
rand-4k-read;3972;993;0;0
Sequential write speed 8627 KB/sec (target 10000) - FAIL
Note that sequential write speed declines over time as a card is used - your card may require reformatting
Random write speed 838 IOPS (target 500) - PASS
Random read speed 993 IOPS (target 1500) - FAIL
Run 2
prepare-file;0;0;8956;17
seq-write;0;0;8626;16
rand-4k-write;0;0;3418;854
rand-4k-read;4004;1001;0;0
Sequential write speed 8626 KB/sec (target 10000) - FAIL
Note that sequential write speed declines over time as a card is used - your card may require reformatting
Random write speed 854 IOPS (target 500) - PASS
Random read speed 1001 IOPS (target 1500) - FAIL
Run 3
prepare-file;0;0;8712;17
seq-write;0;0;9684;18
rand-4k-write;0;0;3384;846
rand-4k-read;3983;995;0;0
Sequential write speed 9684 KB/sec (target 10000) - FAIL
Note that sequential write speed declines over time as a card is used - your card may require reformatting
Random write speed 846 IOPS (target 500) - PASS
Random read speed 995 IOPS (target 1500) - FAIL
The random write speed now falls within recommendations. The others are lower than expected, but still good enough to expect overall better performance for the Zero when using the network block device for root. Setting this up is only slightly more difficult than the iSCSI root setup for the 4B. The details will be discussed in a future post.

Since the Linux Ethernet gadget supports jumbo packets (it didn't for a while but it's fixed again), we end with one more test. Setting the packet size to 7418 on both sides, restarting the target server, re-attaching the network block device on the initiator and remounting led to

Code: Select all

$ sh ./sdtest.sh ; Pi Zero iSCSI MTU 7418
Run 1
prepare-file;0;0;17683;34
seq-write;0;0;16318;31
rand-4k-write;0;0;3703;925
rand-4k-read;5365;1341;0;0
Sequential write speed 16318 KB/sec (target 10000) - PASS
Random write speed 925 IOPS (target 500) - PASS
Random read speed 1341 IOPS (target 1500) - FAIL
Run 2
prepare-file;0;0;16351;31
seq-write;0;0;16237;31
rand-4k-write;0;0;3689;922
rand-4k-read;5432;1358;0;0
Sequential write speed 16237 KB/sec (target 10000) - PASS
Random write speed 922 IOPS (target 500) - PASS
Random read speed 1358 IOPS (target 1500) - FAIL
Run 3
prepare-file;0;0;16516;32
seq-write;0;0;16532;32
rand-4k-write;0;0;3665;916
rand-4k-read;5330;1332;0;0
Sequential write speed 16532 KB/sec (target 10000) - PASS
Random write speed 916 IOPS (target 500) - PASS
Random read speed 1332 IOPS (target 1500) - FAIL
Now, both the sequential and random write speed pass while the random read speed is only 10 percent slower than the recommendations. On the other hand,

((76117/16318)*(7964/924)*(13698/1341))^(1/3) = 7.43

implies iSCSI root on the 4B is on average 7.43 times faster.

ejolson
Posts: 6349
Joined: Tue Mar 18, 2014 11:47 am

Re: iSCSI Root Like a Data Center

Mon Jun 22, 2020 6:35 am

It would be nice to boot the Zero over USB using rpiboot as is done on the super-cheap cluster

viewtopic.php?f=49&t=199994

The first obstruction in doing this is the fact the USB hubs that I'm using seem not to reliably recognize the Zero when first powered up. It is possible to work around this with the SD card by repeatedly removing and then installing the g_ether gadget until a functioning network connection is made to the server. Hopefully, toggling the run pin will have the same effect for rpiboot. If not, then pulling the USB cord out and reinserting it always brings the Zero online, so some sort of workaround for these hubs should be possible. That seems to be the joy of building a prototype.

Before any of that, however, I need a working version of rpiboot installed on that free PC which is functioning as the server. Thinking this should be easy, I downloaded the source from GitHub

https://github.com/raspberrypi/usbboot

and typed

Code: Select all

$ make
cc -Wall -Wextra -g -o bin2c bin2c.c
./bin2c msd/bootcode.bin msd/bootcode.h
./bin2c msd/start.elf msd/start.h
cc -Wall -Wextra -g -o rpiboot main.c -lusb-1.0
In file included from main.c:13:
fmemopen.c: In function 'seekfn':
fmemopen.c:62:18: error: invalid operands to binary >= (have 'fpos_t' {aka 'union _G_fpos64_t'} and 'int')
   62 |       if (offset >= 0) {
      |                  ^~
fmemopen.c:63:9: error: aggregate value used where an integer was expected
   63 |         pos = (size_t)offset;
      |         ^~~
fmemopen.c:70:18: error: invalid operands to binary >= (have 'fpos_t' {aka 'union _G_fpos64_t'} and 'int')
   70 |       if (offset >= 0 || (size_t)(-offset) <= mem->pos) {
      |                  ^~
fmemopen.c:70:35: error: wrong type argument to unary minus
   70 |       if (offset >= 0 || (size_t)(-offset) <= mem->pos) {
      |                                   ^
fmemopen.c:71:9: error: aggregate value used where an integer was expected
   71 |         pos = mem->pos + (size_t)offset;
      |         ^~~
fmemopen.c:77:5: error: aggregate value used where an integer was expected
   77 |     case SEEK_END: pos = mem->size + (size_t)offset; break;
      |     ^~~~
fmemopen.c:78:21: error: incompatible types when returning type 'int' but 'fpos_t' {aka 'union _G_fpos64_t'} was expected
   78 |     default: return -1;
      |                     ^
fmemopen.c:82:12: error: incompatible types when returning type 'int' but 'fpos_t' {aka 'union _G_fpos64_t'} was expected
   82 |     return -1;
      |            ^
fmemopen.c:86:10: error: cast to union type from type not present in union
   86 |   return (fpos_t)pos;
      |          ^
fmemopen.c: In function 'fmemopen':
fmemopen.c:95: warning: ignoring #pragma unused  [-Wunknown-pragmas]
   95 |   #pragma unused(mode)
      | 
fmemopen.c:106:10: warning: implicit declaration of function 'funopen'; did you mean 'fdopen'? [-Wimplicit-function-declaration]
  106 |   return funopen(mem, readfn, writefn, seekfn, closefn);
      |          ^~~~~~~
      |          fdopen
fmemopen.c:106:10: warning: returning 'int' from a function with return type 'FILE *' {aka 'struct _IO_FILE *'} makes pointer from integer without a cast [-Wint-conversion]
  106 |   return funopen(mem, readfn, writefn, seekfn, closefn);
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
fmemopen.c:94:52: warning: unused parameter 'mode' [-Wunused-parameter]
   94 | FILE *fmemopen(void *buf, size_t size, const char *mode) {
      |                                        ~~~~~~~~~~~~^~~~
fmemopen.c: In function 'seekfn':
fmemopen.c:87:1: warning: control reaches end of non-void function [-Wreturn-type]
   87 | }
      | ^
make: *** [Makefile:2: rpiboot] Error 1
The astonishing deluge of errors became even more astonishing when I examined the first and found the opaque data type fpos_t represented by some sort of union was being treated as a signed integer which was then tested for whether it was nonnegative. Naturally man fpos_t and apropos fpos_t returned nothing.

Searching the /usr/include directory encouraged me to try man fgetpos. This documentation included the statement
man fgetpos wrote: On some non-UNIX systems, an fpos_t object may be a complex object and these routines may be the only way to portably reposition a text stream.
which left me wondering whether Linux was, in fact, a non-UNIX system.

A little more grep magic found

Code: Select all

typedef union _G_fpos64_t {
    char __opaque[16];
    long long __lldata;
    double __align;
} fpos_t;
in stdio.h. Therefore, I decided to replace offset with offset.__lldata in the code to see if it would at least compile. There were 6 places this change was made. In addition there were 2 places where

Code: Select all

return -1;
needed to be replaced by

Code: Select all

{
   fpos_t minusone; minusone.__lldata=-1;
   return minusone;
}
and one place where

Code: Select all

  return (fpos_t)pos;
needed to be replaced by

Code: Select all

  {
    fpos_t returnpos; returnpos.__lldata=pos;
    return returnpos;
  }
At this point I obtained the link loader error undefined reference to funopen. I should have known, it seems all those casts might actually work with macOS but not with the musl C library. Could it be that funopen is actually an euphemism for miseryopen or sorrowopen? Maybe it should be called wearinessopen.

ejolson
Posts: 6349
Joined: Tue Mar 18, 2014 11:47 am

Re: iSCSI Root Like a Data Center

Mon Jun 22, 2020 9:44 pm

Since compiling rpiboot had turned into frustration, for more of the same I set up a socially-distanced Zoom appointment with the lead developer of FidoBASIC to contract a version of funopen that runs without misery, sorrow or weariness on Linux. The canine expert in n-level metaprogramming refused to consider my fpos_t changes and insisted on starting with the pristine sources from the GitHub repository. After some time Fido growled, the problem is the same as with most C programs--there are not enough #define, #include, #ifdef and #if preprocessing directives. The rest sounded like barking.

I decided to take a break from rpiboot and swapped the SD card from the Zero into a 4B recently connected via a USB-C cable to the same hub to see what would happen. I was amazed with the reverse and forward compatibility of the 4B and that the SD card just booted up.

Again enabling the USB Ethernet gadget I measured the iSCSI disk performance as

Code: Select all

$ bash sdtest.sh ; # 4B Ethernet gadget MTU 1500
Run 1
rand-4k-write;0;0;24956;6239
rand-4k-read;23753;5938;0;0
Sequential write speed 24481 KB/sec (target 10000) - PASS
Random write speed 6239 IOPS (target 500) - PASS
Random read speed 5938 IOPS (target 1500) - PASS
./sdtest.sh: line 34: return: can only `return' from a function or sourced script
Run 2
prepare-file;0;0;26575;51
seq-write;0;0;24317;47
rand-4k-write;0;0;23865;5966
rand-4k-read;23857;5964;0;0
Sequential write speed 24317 KB/sec (target 10000) - PASS
Random write speed 5966 IOPS (target 500) - PASS
Random read speed 5964 IOPS (target 1500) - PASS
./sdtest.sh: line 34: return: can only `return' from a function or sourced script
Run 3
prepare-file;0;0;25996;50
seq-write;0;0;24545;47
rand-4k-write;0;0;22215;5553
rand-4k-read;24014;6003;0;0
Sequential write speed 24545 KB/sec (target 10000) - PASS
Random write speed 5553 IOPS (target 500) - PASS
Random read speed 6003 IOPS (target 1500) - PASS
./sdtest.sh: line 34: return: can only `return' from a function or sourced script
./sdtest.sh: line 37: return: can only `return' from a function or sourced script
Then I set jumbo packets with MTU 7418 and obtained

Code: Select all

$ bash sdtest.sh ; # 4B Ethernet gadget MTU 7418
Run 1
prepare-file;0;0;29802;58
seq-write;0;0;29244;57
rand-4k-write;0;0;26892;6723
rand-4k-read;31417;7854;0;0
Sequential write speed 29244 KB/sec (target 10000) - PASS
Random write speed 6723 IOPS (target 500) - PASS
Random read speed 7854 IOPS (target 1500) - PASS
sdtest.sh: line 34: return: can only `return' from a function or sourced script
Run 2
prepare-file;0;0;30453;59
seq-write;0;0;29979;58
rand-4k-write;0;0;26298;6574
rand-4k-read;31432;7858;0;0
Sequential write speed 29979 KB/sec (target 10000) - PASS
Random write speed 6574 IOPS (target 500) - PASS
Random read speed 7858 IOPS (target 1500) - PASS
sdtest.sh: line 34: return: can only `return' from a function or sourced script
Run 3
prepare-file;0;0;30595;59
seq-write;0;0;28580;55
rand-4k-write;0;0;26565;6641
rand-4k-read;31356;7839;0;0
Sequential write speed 28580 KB/sec (target 10000) - PASS
Random write speed 6641 IOPS (target 500) - PASS
Random read speed 7839 IOPS (target 1500) - PASS
sdtest.sh: line 34: return: can only `return' from a function or sourced script
sdtest.sh: line 37: return: can only `return' from a function or sourced script
While the USB Ethernet gadget is slower than gigabit Ethernet, it is clear that the 4B can drive the gadget much faster than the Zero. In fact, the 4B is fast enough to obtain passing results in all of the tests.

ejolson
Posts: 6349
Joined: Tue Mar 18, 2014 11:47 am

Re: iSCSI Root Like a Data Center

Wed Jun 24, 2020 9:16 pm

Woohoo! I now have two Raspberry Pi Zero computers inside the server chassis. The first has an SD card and mounts root from the card. A GPIO output from the first is wired to the run pin of the second which can now be reset as desired. To make things even better, I followed Fido's advice and added an #ifdef to main.c in rpiboot as follows.

Code: Select all

/* Assume BSD without native fmemopen() if doesn't seem to be glibc */
#ifdef FIDO
#if defined(__APPLE__) || (!defined(_GNU_SOURCE) && (!defined(_POSIX_C_SOURCE) || _POSIX_C_SOURCE < 200809L))
#include "fmemopen.c" // BSD fmemopen() compat in terms of funopen()
#endif
#endif
This avoids the misery, sorrow and weariness that comes from trying to link the funopen routine referenced in the fmemopen.c file. Note that the comment (already present in the code) explains what the problem was: Musl in place of glibc does not turn Linux to BSD.

Stay tuned for iSCSI root on the Pi zero like a data center.

ejolson
Posts: 6349
Joined: Tue Mar 18, 2014 11:47 am

Re: iSCSI Root Like a Data Center

Thu Jun 25, 2020 1:40 am

Following the same procedure for the Zero as previously done for the Pi 4B

viewtopic.php?f=36&t=274553#p1664035

I mounted the iSCSI network block device on /iroot and typed

Code: Select all

# time rsync -glopqrtxDH --numeric-ids / /iroot

real    5m50.833s
user    0m52.955s
sys 1m41.928s
The calculation

350.833/50.495=6.95

indicates that it took almost 7 times longer for the Zero to copy the root file system to the iSCSI network block device than for the 4B to do the same. Coincidentally, the Zero is also exactly 7 times cheaper than the 4B. On the other hand, speeds like this likely have zero relevance in a data center.

swampdog
Posts: 474
Joined: Fri Dec 04, 2015 11:22 am

Re: iSCSI Root Like a Data Center

Thu Jun 25, 2020 3:12 pm

Have you noticed any dhcp duplication? Years ago when I experimented with iscsi I found it would randomly fail (nothing being unplugged). The rpi initrd would obtain a lease and when raspbian loaded, another lease would be obtained. I probably messed up the initrd. Too long ago to recall those details. Initially I thought I had fixed it by assigning a static ip in dhcpd.conf of my dns server. Nope. At some point the raspbian lease would expire leaving that ip waiting to be allocated to another box. As soon as it was, failure.

ejolson
Posts: 6349
Joined: Tue Mar 18, 2014 11:47 am

Re: iSCSI Root Like a Data Center

Fri Jun 26, 2020 4:01 am

swampdog wrote:
Thu Jun 25, 2020 3:12 pm
Have you noticed any dhcp duplication? Years ago when I experimented with iscsi I found it would randomly fail (nothing being unplugged). The rpi initrd would obtain a lease and when raspbian loaded, another lease would be obtained. I probably messed up the initrd. Too long ago to recall those details. Initially I thought I had fixed it by assigning a static ip in dhcpd.conf of my dns server. Nope. At some point the raspbian lease would expire leaving that ip waiting to be allocated to another box. As soon as it was, failure.
You may be right that the target doesn't like the initiator to change its IP number.

Currently I'm specifying the initiator-address explicitly in targets.conf and giving each Pi a fixed private IP number. I noticed some hiccups when switching IP numbers around and had to restart tgtd a couple times when experimenting. My plan is to translate the private IP numbers to public numbers using SNAT and DNAT (no masquerade) while performing some traffic shaping.

I now have the second Pi Zero booting without an SD card and mounting iSCSI root through the Ethernet gadget when I toggle the GPIO on the first Pi. It is a little slow, but the remote administration works just like a virtual machine in a data center. Now I need a clunky web interface so I can manage the Zero with my mouse and provision it with different operating-system images and configurations using one click.

ejolson
Posts: 6349
Joined: Tue Mar 18, 2014 11:47 am

Re: iSCSI Root Like a Data Center

Fri Jun 26, 2020 8:05 pm

This post describes setting up a network block device encrypted with dm-crypt for security in situations where either the local area network or the iSCSI server may be compromised.

As before create a new iSCSI target on the server. If you are using a commercial NAS this can be done with the mouse. Otherwise, it can be done following the same procedure as outlined before. For the rest of this discussion let's assume the Pi 4B initiator is running from a root filesystem on the SD card which has an extra boot partition in /dev/mmcblk0p3 and an attached iSCSI network block device as /dev/sda.

In order to make it difficult to identify the encrypted data, the the first thing to do is initialize the network block device with random data. Note that it might be more efficient to initialize the random data directly on the server; however, given the exponential growth of the epidemic, it's safer to assume the NAS has already been infected.

After checking /dev/sda is the network block device, randomly initialize it.

Code: Select all

# time dd if=/dev/urandom of=/dev/sda bs=4M
dd: error writing '/dev/sda': No space left on device
8193+0 records in
8192+0 records out
34359738368 bytes (34 GB, 32 GiB) copied, 1305.89 s, 26.3 MB/s

real    21m45.896s
user    0m0.020s
sys 21m44.175s
Note almost all the time was spent in system mode generating pseudo random numbers for this operation.

At this point install dm-crypt on the Pi 4B, create a key, mount the encrypted device and copy the root file system.

Code: Select all

# apt-get install cryptsetup-bin cryptsetup-initramfs
# cd /root
# dd if=/dev/random bs=80 count=1 | base64 | head -c72 >i2root.key
# cryptsetup open --type plain --key-file i2root.key /dev/sda iroot
# cd /
# mkdir /iroot
# mke2fs -t ext4 -L IROOT /dev/mapper/iroot
# mount /dev/mapper/iroot /iroot
# time rsync -glopqrtxDH --numeric-ids / /iroot

real    1m25.980s
user    0m12.714s
sys 0m23.694s
In comparison to the time of 50.495s posted in

viewtopic.php?f=36&t=274553#p1664035

the encrypted volume took 70 percent longer to copy. Before drawing too many conclusions, it's worth noting not only did the use of encryption change between the runs, but the server changed as well: The original iSCSI target was hosted by an ODROID-N2 single-board computer using an SSD while the encrypted target tested in this post was hosted by an AMD A6-5400K APU using mirrored 1TB HDs. This seems to be the same kind of science that led some cities to open the beaches and the bars as places for people to practice social distancing.

ejolson
Posts: 6349
Joined: Tue Mar 18, 2014 11:47 am

Re: iSCSI Root Like a Data Center

Sat Jun 27, 2020 4:25 am

Creating a working initial RAM filesystem that mounts the encrypted network block device proved difficult due to things labeled FIXME in the cryptroot hook. To make progress I edited the file cryptroot in /usr/share/initramfs-tools/hooks and commented out the return after the skipping message as

Code: Select all

     cryptsetup_message "WARNING: Skipping root target $CRYPTTAB_NAME: uses a key file"
#    return 1
A few lines later I removed the FIXME and wrote

Code: Select all

    if [ ! -e "$CRYPTTAB_KEY" ]; then
        cryptsetup_message "WARNING: Target $CRYPTTAB_NAME has a non-existing key file $CRYPTTAB_KEY"
    else
#       _CRYPTTAB_KEY="/FIXME-initramfs-rootmnt$_CRYPTTAB_KEY" # preserve mangled name
        cp $_CRYPTTAB_KEY $DESTDIR/cryptroot
        _CRYPTTAB_KEY="/cryptroot/${_CRYPTTAB_KEY##*/}"
    fi
Note that for clarity I have not reproduced the amazing levels of indentation that appear in the original script.

After this change, chroot to the encrypted network block device currently mounted under /iroot with the commands

Code: Select all

# mount /dev/mmcblk0p3 /boot
# mount --rbind /sys /iroot/sys
# mount --make-rslave /iroot/sys
# mount --rbind /dev /iroot/dev
# mount --make-rslave /iroot/dev
# mount --rbind /proc /iroot/proc
# mount --make-rslave /iroot/proc
# chroot /iroot
Edit /etc/fstab to read

Code: Select all

proc              /proc proc defaults                 0 0
/dev/mmcblk0p3    /boot vfat defaults                 0 2
/dev/mapper/iroot /     ext4 defaults,noatime,_netdev 0 1
Edit /etc/crypttab to read

Code: Select all

iroot /dev/sda /root/i2root.key plain,cipher=aes-cbc-essiv:sha256,size=256
Edit /etc/cryptsetup-initramfs/conf-hook to read

Code: Select all

CRYPTSETUP=y
and edit /boot/cmdline.txt to read

Code: Select all

ip=192.168.47.130::192.168.47.254:255.255.255.128:t2:eth0:off iscsi_username=t2 iscsi_initiator=iqn.1993-08.org.debian:01:6f8597faca9c iscsi_target_name=iqn.1999-01.wulf.silver:void:i2root iscsi_target_ip=192.168.47.254 console=serial0,115200 console=tty1 root=/dev/mapper/iroot rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait sdhci.debug_quirks2=4
Most of the options in cmdline.txt have been discussed earlier. Note the static IP number given in the ip= option has been added in this case, only because the subnet on which the Pi 4B is now connected does not have an active DHCP server.

Finally, add two lines to /boot/config.txt so the relevant part reads

Code: Select all

[pi4]
# Enable DRM VC4 V3D driver on top of the dispmanx display stack
dtoverlay=vc4-fkms-v3d
max_framebuffers=2
kernel=mykernel.img
initramfs myinitrd.img
Generate the initial RAM filesystem with

Code: Select all

# cd /boot
# update-initramfs -c -k `uname -r`
cryptsetup: WARNING: Skipping root target iroot: uses a key file
# mv initrd.img* myinitrd.img
Don't worry that it says it's skipping. Since we commented out the return, it didn't really do that. Other than exercise, a good reason for skipping appears to be the many complaints that key files were appearing unencrypted in the initial RAM filesystem. While that would be bad in the encrypt-your-laptop-for-when-it-gets-stolen scenario, placing the key file in the initial RAM filesystem is exactly what we want to do here.

Exit the chroot environment and unmount everything with

Code: Select all

# exit
# umount /iroot/boot
# umount -R /iroot/sys
# umount -R /iroot/dev
# umount -R /iroot/proc
# umount /iroot
# cryptsetup close iroot
# iscsiadm -m node -U all
If all goes well you should now be able to boot with

Code: Select all

# systemctl reboot 3
into an encrypted iSCSI root like a data center.
Last edited by ejolson on Sat Jul 11, 2020 12:37 am, edited 1 time in total.

ejolson
Posts: 6349
Joined: Tue Mar 18, 2014 11:47 am

Re: iSCSI Root Like a Data Center

Sat Jun 27, 2020 4:45 am

To compare the encrypted iSCSI root to the unencrypted one in a more scientific way while wearing a mandatory face mask, I ran

Code: Select all

$ bash sdtest.sh ; # Pi 4B encrypted iSCSI root
Run 1
prepare-file;0;0;38732;75
seq-write;0;0;38687;75
rand-4k-write;0;0;19574;4893
rand-4k-read;35310;8827;0;0
Sequential write speed 38687 KB/sec (target 10000) - PASS
Random write speed 4893 IOPS (target 500) - PASS
Random read speed 8827 IOPS (target 1500) - PASS
sdtest.sh: line 34: return: can only `return' from a function or sourced script
Run 2
prepare-file;0;0;36510;71
seq-write;0;0;35578;69
rand-4k-write;0;0;17332;4333
rand-4k-read;34240;8560;0;0
Sequential write speed 35578 KB/sec (target 10000) - PASS
Random write speed 4333 IOPS (target 500) - PASS
Random read speed 8560 IOPS (target 1500) - PASS
sdtest.sh: line 34: return: can only `return' from a function or sourced script
Run 3
prepare-file;0;0;38778;75
seq-write;0;0;33334;65
rand-4k-write;0;0;18419;4604
rand-4k-read;36368;9092;0;0
Sequential write speed 33334 KB/sec (target 10000) - PASS
Random write speed 4604 IOPS (target 500) - PASS
Random read speed 9092 IOPS (target 1500) - PASS
sdtest.sh: line 34: return: can only `return' from a function or sourced script
sdtest.sh: line 37: return: can only `return' from a function or sourced script
Compared to the unencrypted results from

viewtopic.php?f=36&t=274553&p=1686319#p1679209

we have

Code: Select all

              KB/sec    IOPS Write   IOPS Read
unencrypted   76116         7964        13698
encrypted     38687         4893         8827
percent loss    49%          39%          36%
While the encrypted results are still much better than the recommendations, as remarked earlier, the speeds would be faster if the twofish cipher were used instead of AES.

swampdog
Posts: 474
Joined: Fri Dec 04, 2015 11:22 am

Re: iSCSI Root Like a Data Center

Sat Jun 27, 2020 10:07 am

ejolson wrote:
Fri Jun 26, 2020 4:01 am
swampdog wrote:
Thu Jun 25, 2020 3:12 pm
Have you noticed any dhcp duplication? Years ago when I experimented with iscsi I found it would randomly fail (nothing being unplugged). The rpi initrd would obtain a lease and when raspbian loaded, another lease would be obtained. I probably messed up the initrd. Too long ago to recall those details. Initially I thought I had fixed it by assigning a static ip in dhcpd.conf of my dns server. Nope. At some point the raspbian lease would expire leaving that ip waiting to be allocated to another box. As soon as it was, failure.
You may be right that the target doesn't like the initiator to change its IP number.

Currently I'm specifying the initiator-address explicitly in targets.conf and giving each Pi a fixed private IP number. I noticed some hiccups when switching IP numbers around and had to restart tgtd a couple times when experimenting. My plan is to translate the private IP numbers to public numbers using SNAT and DNAT (no masquerade) while performing some traffic shaping.

I now have the second Pi Zero booting without an SD card and mounting iSCSI root through the Ethernet gadget when I toggle the GPIO on the first Pi. It is a little slow, but the remote administration works just like a virtual machine in a data center. Now I need a clunky web interface so I can manage the Zero with my mouse and provision it with different operating-system images and configurations using one click.
I couldn't progress with the ip problem back then because I was unable to reboot the iscsi target on a whim (production box). There was also the initramfs problem when an rpi update changed the rpi kernel. An ideal sandbox would be three rpi's on their own subnet. A target rpi. A DHCP/DNS rpi. An initiator rpi. With lots of reading and some tcpdumps I might have got somewhere figuring out "who" was responsible for the duplicate ip as it shouldn't have happened - once DHCP was tied to the mac address.

It's on my todo list. Sadly, I have a much larger mandatory todo list. I never get to have any fun! :-|

ejolson
Posts: 6349
Joined: Tue Mar 18, 2014 11:47 am

Re: iSCSI Root Like a Data Center

Sun Jun 28, 2020 8:37 pm

After simultaneously scrambling my EEPROM and /boot partition, I was able to recover the EEPROM, wonder why Ethernet didn't work and then recover the /boot partition.

The next time I was more careful and successfully configured the firmware as

Code: Select all

BOOT_UART=0
WAKE_ON_GPIO=1
POWER_OFF_ON_HALT=0
BOOT_ORDER=0xf31
FREEZE_VERSION=0
following the directions at

https://www.raspberrypi.org/documentati ... _config.md

At this point I was able to pull the SD card from the Pi 4B, copy the contents of the extra boot partition used for iSCSI root to the file server where rpiboot would find it and boot the 4B without an SD card by loading the kernel and related drivers over the USB-C power port.

Unfortunately systemd complained /dev/mmcblk0p3 was not available, so I typed the root password to enter maintenance mode and edited /etc/fstab to read as

Code: Select all

proc              /proc proc defaults                 0 0
/dev/mapper/iroot /     ext4 defaults,noatime,_netdev 0 1
Note, this is the copy of /etc/fstab on the encrypted iSCSI network block device.

Woohoo! Even though it appears not officially supported, I can securely load the initial RAM filesystem containing the dm-crypt password using rpiboot over the USB-C power port and then automatically mount the encrypted iSCSI root over gigabit Ethernet. Since the 4B experiences a 20 second delay when booting while it checks and then rechecks for the missing SD card, it's even more like a data center than expected.

To operate at scale, it would be nice to create a customized recovery SD card that programs the EEPROM with the needed boot options. Does anyone know how to do that?

ejolson
Posts: 6349
Joined: Tue Mar 18, 2014 11:47 am

Re: iSCSI Root Like a Data Center

Fri Jul 10, 2020 2:13 am

ejolson wrote:
Sat Jun 27, 2020 4:45 am

Code: Select all

Pi 4B         KB/sec    IOPS Write   IOPS Read
unencrypted   76116         7964        13698
encrypted     38687         4893         8827
percent loss    49%          39%          36%
I tested an encrypted iSCSI root on the Pi Zero and obtained

Code: Select all

Pi Zero       KB/sec    IOPS Write   IOPS Read
unencrypted   19733          948         1176
encrypted      4015          498          590
percent loss    80%          47%          50%
Clearly encryption results in a much bigger performance loss for the Zero than the 4B. If it weren't for that parabolic antenna mounted topside the dog house, it would be tempting to turn encryption off. However, good security trumps performance in the data center the same way Trump secures the coronavirus in the United States.

ejolson
Posts: 6349
Joined: Tue Mar 18, 2014 11:47 am

Re: iSCSI Root Like a Data Center

Fri Jul 10, 2020 9:59 pm

For purposes of comparison, I decided to see what difference the presence of ARM-based hardware acceleration for AES encryption makes on a dm-crypt secured iSCSI network block device. The ODROID-N2 is based on a 12nm big.LITTLE cluster comprised of four Cortex-A73 cores and two Cortex-A53 cores. In 64-bit mode /proc/cpuinfo reports the feature set

Code: Select all

Features    : fp asimd evtstrm aes pmull sha1 sha2 crc32 cpuid
For reference, the 64-bit Pi 4B reports

Code: Select all

Features    : fp asimd evtstrm crc32 cpuid
The results of sdtest shows that enabling encryption results in little or no loss of performance when the ARM cores include cryptographic acceleration. In particular,

Code: Select all

ODROID-N2     KB/sec    IOPS Write   IOPS Read
unencrypted   60794         8402        13287
encrypted     61248         8533        13429
percent loss    -1%          -2%          -1%
When I showed this result to the head of hardware development, the furry head of the developer tilted sideways. Why are the percent-loss figures negative? I replied that the performance actually seems a little bit faster with encryption turned on, possibly due to the use of larger buffers. At this the canine's tail stopped wagging, the A73 relies on out-of-order execution. What's the point of encryption given the side-channel security leaks that might be lurking inside that chip? I objected that the 4B is also out-of-order.

With a shake of the head Fido explained, ever since IBM open sourced the in-order Power-EN core with cryptographic and compression accelerators, it makes little sense to use anything else for security in the data center. Unconvinced, I pointed out that no sensible person wants parts salvaged from an old supercomputer and there is no new hardware in production based on the EN.

At this the developer's floppy ears perked up. I'm designing a new processor that combines the best features of the Power-EN with the low-power PET on a chip. The PPOC will be compatible with the Super PET and run fully twice as fast with encryption turned on compared to when it's off. The rest sounded like barking.

https://www.nextplatform.com/2020/06/30 ... computers/

See also

viewtopic.php?p=1668781#p1668781

and

viewtopic.php?p=1678991#p1678991

ejolson
Posts: 6349
Joined: Tue Mar 18, 2014 11:47 am

Re: iSCSI Root Like a Data Center

Tue Jul 21, 2020 2:59 am

I decided to compare NFS to the speed of the iSCSI root and obtained

Code: Select all

Pi 4B         KB/sec    IOPS Write   IOPS Read
iSCSI          76116         7964        13698
NFS           106045        11314        12564

Pi Zero       KB/sec    IOPS Write   IOPS Read
iSCSI          19733          948         1176
NFS            36048         1279         1367
It would appear that NFS is faster than iSCSI. This is unexpected because NFS bypasses the Linux buffer cache and spends extra effort maintaining the consistency of files simultaneously shared to multiple systems across the network. If anyone knows of a realistic test comparing iSCSI to NFS, that would be appreciated. Even if iSCSI turns out to be slower, since the network block device can be treated as a physical disk with a real ext4 file system on it, iSCSI root might still be more like a data center.

The output from sdtest for the NFS mount is

Code: Select all

$ bash sdtest.sh ; # Pi 4B with NFS over gigabit Ethernet
Run 1
prepare-file;0;0;109409;213
seq-write;0;0;106045;207
rand-4k-write;0;0;45259;11314
rand-4k-read;50257;12564;0;0
Sequential write speed 106045 KB/sec (target 10000) - PASS
Random write speed 11314 IOPS (target 500) - PASS
Random read speed 12564 IOPS (target 1500) - PASS
sdtest.sh: line 34: return: can only `return' from a function or sourced script
Run 2
prepare-file;0;0;107260;209
seq-write;0;0;104356;203
rand-4k-write;0;0;45103;11275
rand-4k-read;49349;12337;0;0
Sequential write speed 104356 KB/sec (target 10000) - PASS
Random write speed 11275 IOPS (target 500) - PASS
Random read speed 12337 IOPS (target 1500) - PASS
sdtest.sh: line 34: return: can only `return' from a function or sourced script
Run 3
prepare-file;0;0;108683;212
seq-write;0;0;104025;203
rand-4k-write;0;0;45637;11409
rand-4k-read;52809;13202;0;0
Sequential write speed 104025 KB/sec (target 10000) - PASS
Random write speed 11409 IOPS (target 500) - PASS
Random read speed 13202 IOPS (target 1500) - PASS
sdtest.sh: line 34: return: can only `return' from a function or sourced script
sdtest.sh: line 37: return: can only `return' from a function or sourced script
and

Code: Select all

$ bash sdtest.sh ; # Pi Zero with NFS over Ethernet gadget
Run 1
prepare-file;0;0;36068;70
seq-write;0;0;36048;70
rand-4k-write;0;0;5116;1279
rand-4k-read;5471;1367;0;0
Sequential write speed 36048 KB/sec (target 10000) - PASS
Random write speed 1279 IOPS (target 500) - PASS
Random read speed 1367 IOPS (target 1500) - FAIL
Run 2
prepare-file;0;0;36107;70
seq-write;0;0;36147;70
rand-4k-write;0;0;5141;1285
rand-4k-read;5472;1368;0;0
Sequential write speed 36147 KB/sec (target 10000) - PASS
Random write speed 1285 IOPS (target 500) - PASS
Random read speed 1368 IOPS (target 1500) - FAIL
Run 3
prepare-file;0;0;36107;70
seq-write;0;0;36127;70
rand-4k-write;0;0;5108;1277
rand-4k-read;5449;1362;0;0
Sequential write speed 36127 KB/sec (target 10000) - PASS
Random write speed 1277 IOPS (target 500) - PASS
Random read speed 1362 IOPS (target 1500) - FAIL
sdtest.sh: line 37: return: can only `return' from a function or sourced script

ejolson
Posts: 6349
Joined: Tue Mar 18, 2014 11:47 am

Re: iSCSI Root Like a Data Center

Thu Jul 23, 2020 8:08 pm

I showed the results of the previous test to the dog dentist who growled, why don't the patients follow safety guidelines while in the waiting room? We looked with N95-masked faces expressionless at each other for some time. Then Fido remarked, antivirus and synthetic benchmarks are easy to game. Have you tried something useful like unpacking the gcc-10.1.0.tar.xz source code archive?

On the NFS-mounted directory I typed

Code: Select all

$ time tar Jxf gcc-10.1.0.tar.xz ; # NFS

real    2m8.922s
user    0m15.561s
sys 0m29.719s
and then on the iSCSI root typed

Code: Select all

$ time tar Jxf gcc-10.1.0.tar.xz ; # iSCSI

real    0m19.340s
user    0m13.629s
sys 0m12.114s
Being 6.67 times faster compared to NFS, the iSCSI root was more like a data center. This also explains why those Sun workstations always seemed so slow.
    sunlab.jpg
    sunlab.jpg (232.31 KiB) Viewed 770 times
    https://web.njit.edu/~peterp/otherunix.html

    Return to “Networking and servers”