Network booting with isc-dhcp and tftpd not working

[Solskogen]

Hi!

I have an already existing infrastructure for network booting, so instead of using dns-masq i would rather like to use my existing setup. But I am having a hard time getting my RPi3 to network boot. It seems like it gets some kind of contact with my dhcp server, but it don't even as much as look at the tftp server.
What am I missing?

[FM81]

You know about bugs and limitations? See here: https://www.raspberrypi.org/blog/pi-3-b ... e-awesome/

It seems like it gets some kind of contact with my dhcp server, but it don't even as much as look at the tftp server.

Can we see some log files? Have you tested option 66 for example? Do you transmit the correct string via option 43 on the other side? Have you tested with an nearly empty SD-card with only a new 'bootcode.bin' on it? (Last point to exclude network-hardware-related problems only.)

Greetings, FM_81

[Solskogen]

Well, there you probably have it.
WTF is "option 66" and "option 43"? I've booted the Rpi3 with USB, and that works very fine.

[FM81]

It's from the DHCP-protocol.
Option 66 specifies the bootserver. (This one, which is running TFTP.)
Option 43 is vendor specific. In case of a RPi the string Raspberry Pi Boot is needed, may be padded with three zero-bytes at the end.

Complete list from here: http://www.iana.org/assignments/bootp-d ... ters.xhtml

Greetings, FM_81

[Solskogen]

That got me a bit further...

option option-43 code 43 = text;
option option-66 code 66 = text;

and this in the scope
option option-43 "192.168.0.1";
option option-66 "Raspberry Pi Boot ";

[FM81]

option option-43 "192.168.0.1";
option option-66 "Raspberry Pi Boot ";

Swap it! 66 is the server, 43 is the string! I'm also not familiar with the isc-dhcp-syntax, so may be there is an issue too?

MfG, FM_81

[Solskogen]

I've network boot to work with those options.

[marcosd]

Have you managed to make the network boot work with ISC DHCP server?
I've included both options 43 and 66 as explained above with no success. I've also included three spaces at the end of the vendor string as explained in: https://www.raspberrypi.org/documentati ... des/net.md but no luck. I'm still stuck with the RPi 3 sending a discover packet to the DHCP server a couple of times, which are replied with an offer that seems to be ignored by the Rpi.

Here is the tcpdump pcap with the interraction between the Pi and the server:

Code: Select all

12:10:16.790443 IP (tos 0x0, ttl 128, id 0, offset 0, flags [none], proto UDP (17), length 348)
    0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from b8:27:eb:ad:76:1e, length 320, xid 0x26f30339, Flags [none]
	  Client-Ethernet-Address b8:27:eb:ad:76:1e
	  Vendor-rfc1048 Extensions
	    Magic Cookie 0x63825363
	    DHCP-Message Option 53, length 1: Discover
	    Parameter-Request Option 55, length 12: 
	      Vendor-Option, Vendor-Class, BF, Option 128
	      Option 129, Option 130, Option 131, Option 132
	      Option 133, Option 134, Option 135, TFTP
	    ARCH Option 93, length 2: 0
	    NDI Option 94, length 3: 1.2.1
	    GUID Option 97, length 17: 0.68.68.68.68.68.68.68.68.68.68.68.68.68.68.68.68
	    Vendor-Class Option 60, length 32: "PXEClient:Arch:00000:UNDI:002001"
12:10:16.790792 IP (tos 0x10, ttl 128, id 0, offset 0, flags [none], proto UDP (17), length 328)
    192.168.2.1.67 > 192.168.2.12.68: BOOTP/DHCP, Reply, length 300, xid 0x26f30339, Flags [none]
	  Your-IP 192.168.2.12
	  Server-IP 192.168.2.1
	  Client-Ethernet-Address b8:27:eb:ad:76:1e
	  file "pxelinux.0"
	  Vendor-rfc1048 Extensions
	    Magic Cookie 0x63825363
	    DHCP-Message Option 53, length 1: Offer
	    Server-ID Option 54, length 4: 192.168.2.1
	    Lease-Time Option 51, length 4: 300
	    Vendor-Option Option 43, length 20: 82.97.115.112.98.101.114.114.121.32.80.105.32.66.111.111.116.32.32.32
	    Subnet-Mask Option 1, length 4: 255.255.255.0

I am just wondering whether I am missing something here.

[Solskogen]

Yes, I got it to work. Are you sure that you have tftpd going? Can you fetch the files manually from another machine?

This is how my dhcpd.conf looks like

Code: Select all

option option-43 code 43 = text;
option option-66 code 66 = text;

subnet 192.168.0.0 netmask 255.255.255.0 {
        option domain-name "antarctica.no";
        option domain-name-servers 192.168.0.4, 192.168.0.1;
        option routers 192.168.0.1;
        option netbios-name-servers 192.168.0.4;
        option tftp-server-name "192.168.0.1";
        option option-43 "192.168.0.1";
        option option-66 "Raspberry Pi Boot   ";

        filename "pxelinux.0";
        next-server 192.168.0.1;
        pool {
                range 192.168.0.100 192.168.0.200;
        }
}

[marcosd]

Solskogen, my dhcpd.conf was similar to yours, except that I swapped options 66 and 43 as described above. But the Rpi will still send a couple of DHCP Discover packets before it stops and it does not seem to send any ARP or TFTP request:

Code: Select all

16:05:55.898188 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from b8:27:eb:ad:76:1e (oui Unknown), length 320
16:05:55.898479 IP 192.168.2.1.bootps > 192.168.2.12.bootpc: BOOTP/DHCP, Reply, length 300
16:06:25.898433 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from b8:27:eb:ad:76:1e (oui Unknown), length 320
16:06:25.898636 IP 192.168.2.1.bootps > 192.168.2.12.bootpc: BOOTP/DHCP, Reply, length 300
16:06:55.898630 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from b8:27:eb:ad:76:1e (oui Unknown), length 320
16:06:55.898820 IP 192.168.2.1.bootps > 192.168.2.12.bootpc: BOOTP/DHCP, Reply, length 300

I've checked the firewall and the TFTP server, and yes, I can download files from other machines that are in the same network. Here is what my dhcpd.conf looks like:

Code: Select all

option option-66 code 66 = text;
option option-43 code 43 = text;

subnet 192.168.2.0 netmask 255.255.255.0 {
     pool {
        range 192.168.2.11 192.168.2.254;
     }
    option domain-name-servers 192.168.2.1;
    option routers 192.168.2.1;
    option domain-search "testbed";
    option tftp-server-name "192.168.2.1";
    option option-66 "192.168.2.1";
    option option-43 "Raspberry Pi Boot   ";
    filename "pxelinux.0";
    next-server 192.168.2.1;

    host rpi-1 {
        hardware ethernet b8:27:eb:ad:76:1e;
        fixed-address 192.168.2.12;
        option host-name "rpi-2";
    }
}

[marcosd]

The issue seems to be the fixed-address assignment in the host scope. If I remove the host declaration and let the DHCP server to pick an address from the pool (even if it happens to pick the same I was previously specifying as fixed-address), the boot process continues with the subsequent ARP/TFTP requests.

[Solskogen]

Have you tried swapping option-43 and option-66?

[marcosd]

I did, I swapped them. But the issue seems to be the fixed-address assignment. If I remove:

Code: Select all

    host rpi-1 {
        hardware ethernet b8:27:eb:ad:76:1e;
        fixed-address 192.168.2.12;
        option host-name "rpi-2";
    }

from the dhcpd.conf file and let the server pick an address from the pool, it works :-/

[marcosd]

I think I found the source of the problem, but I do not know how to solve it. If I do a packet capture when forcing a fixed-address I see something like this:

Code: Select all

07:28:00.324637 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from b8:27:eb:ad:76:1e (oui Unknown), length 320
07:28:00.324898 IP 192.168.2.1.bootps > 192.168.2.10.bootpc: BOOTP/DHCP, Reply, length 300

If I leave the server choose an address from the address pool, I see a slightly different pattern:

Code: Select all

07:29:21.325328 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from b8:27:eb:ad:76:1e (oui Unknown), length 320
07:29:21.325726 ARP, Request who-has 192.168.2.10 tell 192.168.2.1, length 28
07:29:22.324837 ARP, Request who-has 192.168.2.10 tell 192.168.2.1, length 28
07:29:22.326978 IP 192.168.2.1.bootps > 192.168.2.10.bootpc: BOOTP/DHCP, Reply, length 300

By reading the dhcpd.conf documentation I see that when assigning an address from a pool, the server first does a test by sending an ICMP echo to check whether the address in use in the network:

The DHCP server checks IP addresses to see if they are in use before allocating them to clients. It does this by sending an ICMP Echo request message to the IP address being allocated. If no ICMP Echo reply is received within a second, the address is assumed to be free. This is only done for leases that have been specified in range statements, and only when the lease is thought by the DHCP server to be free - i.e., the DHCP server or its failover peer has not listed the lease as in use.

This delay is apparently the key and has already been mentioned in other threads as a limitation in the current implementation of the network boot. I am trying to see whether I can add an artificial delay when assigning a fixed-address, but so far I have not found anything that is useful.

[marcosd]

To elaborate a little further on the topic, I still have not found a solution, but if I set

Code: Select all

ping-check false;

to remove the ICMP test for ip addresses assigned from the pool, the boot does not work. It does seem that the delay introduced by the ICMP test does the trick for network booting a Rpi 3.

[marcosd]

Here is a hack that worked for the time being. I leave the DHCP server to pick an address from the pool, hence performing an ICMP echo, and then forced the Rpi 3 to do a mac address spoofing once it boots to get a fixed IP address. My dhcpd.conf looks like this:

Code: Select all

subnet 192.168.2.0 netmask 255.255.255.0 {
   option broadcast-address 192.168.2.255;
   option domain-name-servers 192.168.2.1;
   option routers 192.168.2.1;
   option domain-search "testbed";
   option tftp-server-name "192.168.2.1";
   option option-66 "192.168.2.1";
   option option-43 "Raspberry Pi Boot   ";
   default-lease-time 60;
   max-lease-time 3600;
   next-server 192.168.2.1;

   pool {
        range 192.168.2.10 192.168.2.254;
   }

   host rpi-1 {
       hardware ethernet b8:27:eb:1e:1e:1e;
       fixed-address 192.168.2.5;
   }
}

And I added:

Code: Select all

smsc95xx.macaddr=b8:27:eb:1e:1e:1e

to the cmdline.txt file that the Rpi 3 downloads from the tftp server during boot, thus forcing it to change its mac address and get a new IP address from the DHCP server.

[dhenry]

I re-open this thread, after debugging with tcpdump, some informations is missing in your fixed-address block. Isc doesn't send informations included in the subnet block, otherwise informations is sended if you comment the line fixed-address..... why ? i dont know.

So if you copy some informations like :
option broadcast-address 192.168.2.255;
option domain-name-servers 192.168.2.1;
option routers 192.168.2.1;
option domain-search "testbed";
option tftp-server-name "192.168.2.1";
next-server 192.168.2.1;
from the subnet block, sequences boot is ok for me.

Change with it, using the origin Raspberry mac address :

subnet 192.168.2.0 netmask 255.255.255.0 {
option broadcast-address 192.168.2.255;
option domain-name-servers 192.168.2.1;
option routers 192.168.2.1;
option domain-search "testbed";
option tftp-server-name "192.168.2.1";
option option-66 "192.168.2.1";
option option-43 "Raspberry Pi Boot ";
default-lease-time 60;
max-lease-time 3600;
next-server 192.168.2.1;

pool {
range 192.168.2.10 192.168.2.254;
}

host rpi-1 {
hardware ethernet b8:27:eb:1e:1e:1e;
fixed-address 192.168.2.5;
## Information from subnet block, copy
option broadcast-address 192.168.2.255;
option domain-name-servers 192.168.2.1;
option routers 192.168.2.1;
option domain-search "testbed";
option tftp-server-name "192.168.2.1";
next-server 192.168.2.1;
###################################
}
}

To debug use "tcpdump -vv -i eth0 port bootpc" on the DHCP/BOOTP server, and look at the response with this 2 cases
Case 1
host rpi-1 {
hardware ethernet b8:27:eb:1e:1e:1e;
fixed-address 192.168.2.5;
}
Case 2
host rpi-1 {
hardware ethernet b8:27:eb:1e:1e:1e;
fixed-address 192.168.2.5;
option broadcast-address 192.168.2.255;
option domain-name-servers 192.168.2.1;
option routers 192.168.2.1;
option domain-search "testbed";
option tftp-server-name "192.168.2.1";
next-server 192.168.2.1;
}

[dhenry]

After more and more tests, just repeat this two lines from the subnet block in your host bloc
option tftp-server-name "192.168.2.1";
next-server 192.168.2.1;

Like this :
host rpi-1 {
hardware ethernet b8:27:eb:1e:1e:1e;
fixed-address 192.168.2.5;
option tftp-server-name "192.168.2.1";
next-server 192.168.2.1;
}
Sometimes you have too wait a long time between the DHCP response and starting TFPT transfert... i dont know why
Debugged with tcpdump -vv -i eth0 port bootpc

[tempest766]

came across this post in my quest to get the pi3 to respond to tftpboot messages from isc-dhcpd. Like the op, I found that the vendor specific attributes were a problem...

The thing that has me very hot under the collar though, is that for no apparent reason, the pi will no longer even send an initial DHCP DISCOVER message. This is after removing power and and cold booting after 30+ seconds. If I install the USB raspian image it boots fine, but mysteriously has lost the ability to net boot.

If it is possible to break a pi3 unit by playing with network hdcp messages (with no usb or sd card installed) then that says a lot about the quality of the system design.

As I write this...literally 10 MINUTES after powering the unit on, wireshark just captured a single dhcp request and offer.

[cbka]

same here - after booting from SD Card PXE not longer availible
RPI 3+b