Wrong time because of no DNS, and no DNS due to bad time.

[bubu314]

Hi,

I believe every thing is in the subject, if not here comes the longer version:

I got a new rpi4 (very nice btw). But I run into a bit of a problem. DNS won't resolve. After some searching (it took me a while....), Here is what I found:

DNS would not accept any request because my system's time is not properly. I you remember well, the lack of RTC makes it happen every time you switch it off and on again (while waiting for some time in between). Wrong time messes with security and certificates.

Then any time synchronization will fail because the client (ntp I believe, I have an almost, clean install using raspberry pi OS -not lite, not full- which is up to date). Here is one the many message I got in my syslog:

systemd-resolved[323]: DNSSEC validation failed for question 3.debian.pool.ntp.org IN A: signature-expired

They all look alike with different servers.

My first idea was to set a hard coded ip address for the ntp server, but then after some digging I found out that ntp server are rotated through some kind of pool (which makes it murky on how to proceed, cleanly).

So here is my question. How do I make my pi get some network time in this configuration. And of course I don't intend to set the time by hand on every reboot, nor do I wish to let it up 24/7.

Thanks for your advice.
And long live the pi, I love it.

Bubu.

[Underlyingglitch]

I presume you are using some version of Rasbian??

Have you tried installing a new fresh version? I work a lot with all kinds of raspberry pi's and OS' and never had this problem before

[spectrum6]

Is fake-hwclock service running? What does "systemctl status fake-hwclock" say? Do you leave the PI turned off for prolonged periods?

If fake-hwclock is working properly, this blog post may help you, take a look:
https://warlord0blog.wordpress.com/2019 ... on-failed/

[pidd]

I didn't think stub-resolvers needed the correct time for exactly this reason?

[spectrum6]

I didn't think stub-resolvers needed the correct time for exactly this reason?

From what I gathered DNSSEC does use time to validate the signature. I guess like an SSL certificate there is a validity time. But this is a local/client-side verification it appears. My guess is Debian changed the default or later versions of systemd changed the default. Perhaps most users do not use nameservers capable of DNSSEC.

According to this doc, you can also turn off the validation by setting it to false.

Code: Select all

DNSSEC=

    Takes a boolean argument or "allow-downgrade". If true all DNS lookups are DNSSEC-validated locally (excluding LLMNR and Multicast DNS). If the response to a lookup request is detected to be invalid a lookup failure is returned to applications. Note that this mode requires a DNS server that supports DNSSEC. If the DNS server does not properly support DNSSEC all validations will fail. If set to "allow-downgrade" DNSSEC validation is attempted, but if the server does not support DNSSEC properly, DNSSEC mode is automatically disabled. Note that this mode makes DNSSEC validation vulnerable to "downgrade" attacks, where an attacker might be able to trigger a downgrade to non-DNSSEC mode by synthesizing a DNS response that suggests DNSSEC was not supported. If set to false, DNS lookups are not DNSSEC validated.

    Note that DNSSEC validation requires retrieval of additional DNS data, and thus results in a small DNS look-up time penalty.

    DNSSEC requires knowledge of "trust anchors" to prove data integrity. The trust anchor for the Internet root domain is built into the resolver, additional trust anchors may be defined with dnssec-trust-anchors.d(5). Trust anchors may change at regular intervals, and old trust anchors may be revoked. In such a case DNSSEC validation is not possible until new trust anchors are configured locally or the resolver software package is updated with the new root trust anchor. In effect, when the built-in trust anchor is revoked and DNSSEC= is true, all further lookups will fail, as it cannot be proved anymore whether lookups are correctly signed, or validly unsigned. If DNSSEC= is set to "allow-downgrade" the resolver will automatically turn off DNSSEC validation in such a case.

    Client programs looking up DNS data will be informed whether lookups could be verified using DNSSEC, or whether the returned data could not be verified (either because the data was found unsigned in the DNS, or the DNS server did not support DNSSEC or no appropriate trust anchors were known). In the latter case it is assumed that client programs employ a secondary scheme to validate the returned DNS data, should this be required.

    It is recommended to set DNSSEC= to true on systems where it is known that the DNS server supports DNSSEC correctly, and where software or trust anchor updates happen regularly. On other systems it is recommended to set DNSSEC= to "allow-downgrade".

    In addition to this global DNSSEC setting systemd-networkd.service(8) also maintains per-link DNSSEC settings. For system DNS servers (see above), only the global DNSSEC setting is in effect. For per-link DNS servers the per-link setting is in effect, unless it is unset in which case the global setting is used instead.

    Site-private DNS zones generally conflict with DNSSEC operation, unless a negative (if the private zone is not signed) or positive (if the private zone is signed) trust anchor is configured for them. If "allow-downgrade" mode is selected, it is attempted to detect site-private DNS zones using top-level domains (TLDs) that are not known by the DNS root server. This logic does not work in all private zone setups.

    Defaults to "allow-downgrade"

[epoch1970]

You can use the time sources you want. NIST, among many organisations, publishes a list of public NTP servers.
https://tf.nist.gov/tf-cgi/servers.cgi

Some routers offer a local time service. You could use a stable LAN IP or name instead of a public name.

[GlowInTheDark]

BTW and FYI: In regards to the usual "It works for me; what is wrong with you?" type responses: I think the root problem here is that OP's DNS is "security conscious". Most DNS's aren't, so for "most of us", it *will* "just work".

Anyway, the idea has to be to get the system time to something that is "close to right" before trying to sync to the network. Why "fake-hwclock" isn't working to that end is the (next) mystery to be solved.

[spectrum6]

Anyway, the idea has to be to get the system time to something that is "close to right" before trying to sync to the network. Why "fake-hwclock" isn't working to that end is the (next) mystery to be solved.

I don't know if anyone has direct experience with DNSSEC so far, it would be good to know how much time skew is a problem for it. But this question is out there, along with whether he is running Raspbian. My Raspbian setups do not even use systemd-resolved, so I am guessing it might be Ubuntu or some other derivative that doesn't have fake-hwclock installed.

[GlowInTheDark]

have an almost, clean install using raspberry pi OS -not lite, not full- which is up to date...

It says he is running Raspberry Pi OS (the OS formerly known as Raspbian).

Of course, unclear what "not lite, not full" means. Sort of the Goldilocks of OSes, I guess...

[hippy]

BTW and FYI: In regards to the usual "It works for me; what is wrong with you?" type responses: I think the root problem here is that OP's DNS is "security conscious". Most DNS's aren't, so for "most of us", it *will* "just work".

Thanks for that. I could understand the OP's issue, but not why DNS wouldn't work. I don't recall DNS Query packets even including a time field, but if it's more than plain Good Old Fashioned DNS that would explain it.

[DougieLawson]

So here is my question. How do I make my pi get some network time in this configuration. And of course I don't intend to set the time by hand on every reboot, nor do I wish to let it up 24/7.

If you can't resolve because you don't have a good clock and can't get a good clock because you can't resolve you've got five choices.

1. Drop DNSSEC.
2. Get the time from a LAN based time server (with a fixed IP) that can resolve.
3. Hard-code the IP address of an internet time server. But read http://pages.cs.wisc.edu/~plonka/netgear-sntp/ for a laugh first.
4. Add RTC hardware.
5. Use GPS (it'll need PPS) and run a time server on your RPi.

If option two isn't available then option four is probably easiest as there's a load of hardware add-on RTCs.

[hippy]

Here's a thought ... Does a response to a failed DNSSEC request include a timestamp which could be used to set the time to make the next DNSSEC request succeed ?

[bubu314]

Hi,

I'll try to answer all the questions here.

I installed "Raspberry Pi OS (32-bit) with desktop" from the raspberrypi.org web site (not full nor lite were referring to the desktop and recommended software and the no desktop version of raspberry pi os).
systemctl status fake-hwclock gave me:

fake-hwclock.service - Restore / save the current clock
Loaded: loaded (/lib/systemd/system/fake-hwclock.service; enabled; vendor preset: enabled)
Active: active (exited) since Fri 2020-07-03 15:17:01 CEST; 23min ago
Docs: man:fake-hwclock(8)
Process: 117 ExecStart=/sbin/fake-hwclock load $FORCE (code=exited, status=0/SUCCESS)
Main PID: 117 (code=exited, status=0/SUCCESS)

Jul 03 15:17:01 XXXXX fake-hwclock[117]: Fri 03 Jul 2020 01:17:01 PM UTC
Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.

https://warlord0blog.wordpress.com/2019 ... on-failed/ seems to describe the same issue. The solution offered there is to disable (or rather downgrade dnssec). Could be a solution.
The documentation form spectrum6 seems to concurs.

The links from epoch1970 gives names *and* ip addresses of ntp server, which is one of the things I was looking for. However I will not use this solution, I'll keep those close to hand is just in case.

Considering the time of the rpi at boot time is seems fake-hwclock is working, but it cannot know whether reboot is right after shutdown or two days later (which can be the case here).

I think DougieLawson sums it about right.


Then there is hippy proposition, havn't seen any (time of response) in syslog, could be else where. But definitively more work than downgrading dnssec maybe another time.

Thanks for all those answers. For future reader check DougieLawson 5 solutions.

After some doc searching from the clues you gave me, my solution is to set
FallbackDNS=9.9.9.10 in /etc/systemd/resolved.conf . This sets a fall back dns here Quad9 without DNSSEC (see https://wiki.archlinux.org/index.php/Al ... ices#Quad9 ). It seems to work. I'll let you know if it does not in the long term.

NB: dns caching may prevent this from happening too often. To reproduce make sure to resolvectl flush-caches .

Cheers.

Bubu314

[pascalfbmartin]

5. Use GPS (it'll need PPS) and run a time server on your RPi.

You would need PPS if you need sub-second accuracy, otherwise this can be overkill. And GPS with PPS are either a bit more expensive, or cumbersome to use (e.g. serial port). When I checked, there was also some kernel tinkering needed, which always makes me nervous (my goal is not to spend all my time maintaining the kernel).

Even while I never had the DNS problem, I have had a somewhat similar chicken-and-egg issues: after each power outage (this is America..) my Internet link takes some time to come backup. The Raspberry Pi board boots fast and ntpd (did not try chronyd) starts. After a while ntpd gives up and the time will never be synchronized again. This is a problem because I use a Raspberry Pi to control my sprinklers, and watering in the middle of the day violates city ordinance (this is California..). Not to mention if frightens the Cat . I also have security IP cameras that were off by more than a day when we were burglarized. Embarrassing.. So I decided to do something.

Shameless plug starts here..

I was interested in NTP and had experience with GPS, so I wrote my own NTP server: https://github.com/pascal-fb-martin/houseclock. I made it as simple to install as possible, with minimal configuration and most defaults work when using a cheap USB GPS mouse. It built it with a web console, using my own HTTP server library (https://github.com/pascal-fb-martin/echttp), because I like to monitor my services from a web browser. Monitoring is also easy to automate this way.

It might be too simplistic for any particle physics experiment, and I should probably not ignore second adjustments.. but none of my watering has started off since then, and the camera timestamps have been OK.

Caution: as I have multiple small web services applications, so I created a small web portal that use HTTP redirect to share port 80 between multiple applications: https://github.com/pascal-fb-martin/houseportal. That is one more dependency.

[DougieLawson]

Shameless plug starts here..

I was interested in NTP and had experience with GPS, so I wrote my own NTP server: https://github.com/pascal-fb-martin/houseclock. I made it as simple to install as possible, with minimal configuration and most defaults work when using a cheap USB GPS mouse. It built it with a web console, using my own HTTP server library (https://github.com/pascal-fb-martin/echttp), because I like to monitor my services from a web browser. Monitoring is also easy to automate this way.

I'll have a look at that.

I've got a cheapo USB GPS (no PPS) that doesn't do much connected to a RPi running gpsd. My current time set-up is one RPi (with an RTC) running as an ntp server and all the other RPis sync to that (it's both IPv4 & IPv6).

[pidd]

Shameless plug starts here..

Wow, that is comprehensive, its the type of project I start, get as far as the bare minimum to work then lose the will. Well done!