beta-tester
Posts: 1385
Joined: Fri Jan 04, 2013 1:57 pm
Location: de_DE

any chance to fix dependency ordering cycle of systemd?

Fri Nov 13, 2015 2:37 pm

hi, i have a problem, that i got a dependency ordering cycle for unknown reason recently after an apt-get upgrade.
because of that systemd tried to resolve the cycle by deleting the rpcbind.service some needed resources services are not available after system boot.

is there a way to fix that problem?

Code: Select all

...
[    3.494252] systemd[1]: Found ordering cycle on basic.target/start
[    3.506955] systemd[1]: Found dependency on sysinit.target/start
[    3.519265] systemd[1]: Found dependency on rpcbind.service/start
[    3.531550] systemd[1]: Found dependency on network-online.target/start
[    3.544316] systemd[1]: Found dependency on network.target/start
[    3.556489] systemd[1]: Found dependency on dhcpcd.service/start
[    3.568562] systemd[1]: Found dependency on basic.target/start
[    3.580419] systemd[1]: Breaking ordering cycle by deleting job rpcbind.service/start
[    3.594361] systemd[1]: Job rpcbind.service/start deleted to break ordering cycle starting with basic.target/start
...
i am using the official raspbian jessie image and added several packages.
all was working perfectly well until i got a dhcpcd5 package update via "apt-get upgrade"
from [dhcpcd5 6.7.1-1+rpi3] to (dhcpcd5 6.7.1-1+rpi5)
{ I only give negative feedback }
RPi B (256MB), B (512MB), B+, ZeroW; 2B; 3B, 3B+; 4B (4GB)

ShiftPlusOne
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 6234
Joined: Fri Jul 29, 2011 5:36 pm
Location: The unfashionable end of the western spiral arm of the Galaxy

Re: any chance to fix dependency ordering cycle of systemd?

Fri Dec 25, 2015 1:41 am

Hey, just stumbled upon this thread. Would you be able to find out what you need to install onto a clean image to get this problem? It's hard to investigate this without being able to reproduce it.

weboide
Posts: 1
Joined: Sat Dec 26, 2015 3:19 am

Re: any chance to fix dependency ordering cycle of systemd?

Sat Dec 26, 2015 3:22 am

Installing nfs-common triggered it (or made it worse) for me. I haven't looked after doing a clean install to see if there's already dependency cycles.

[ 3.206659] systemd[1]: Found ordering cycle on basic.target/start
[ 3.215051] systemd[1]: Found dependency on sysinit.target/start
[ 3.223178] systemd[1]: Found dependency on kbd.service/start
[ 3.230924] systemd[1]: Found dependency on remote-fs.target/start
[ 3.239209] systemd[1]: Found dependency on mnt-nas.mount/start
[ 3.247214] systemd[1]: Found dependency on network.target/start
[ 3.255123] systemd[1]: Found dependency on dhcpcd.service/start
[ 3.262958] systemd[1]: Found dependency on basic.target/start
[ 3.270518] systemd[1]: Breaking ordering cycle by deleting job kbd.service/start
[ 3.281399] systemd[1]: Job kbd.service/start deleted to break ordering cycle starting with basic.target/start
[ 3.298003] systemd[1]: Found ordering cycle on basic.target/start
[ 3.306175] systemd[1]: Found dependency on sysinit.target/start
[ 3.314134] systemd[1]: Found dependency on raspi-config.service/start
[ 3.322582] systemd[1]: Found dependency on remote-fs.target/start
[ 3.330677] systemd[1]: Found dependency on mnt-nas.mount/start
[ 3.338467] systemd[1]: Found dependency on network.target/start
[ 3.346300] systemd[1]: Found dependency on dhcpcd.service/start
[ 3.354099] systemd[1]: Found dependency on basic.target/start
[ 3.361624] systemd[1]: Breaking ordering cycle by deleting job raspi-config.service/start
[ 3.373313] systemd[1]: Job raspi-config.service/start deleted to break ordering cycle starting with basic.target/start
[ 3.390077] systemd[1]: Found ordering cycle on basic.target/start
[ 3.398201] systemd[1]: Found dependency on sysinit.target/start
[ 3.406150] systemd[1]: Found dependency on console-setup.service/start
[ 3.414724] systemd[1]: Found dependency on remote-fs.target/start
[ 3.422825] systemd[1]: Found dependency on mnt-nas.mount/start
[ 3.430707] systemd[1]: Found dependency on network.target/start
[ 3.438677] systemd[1]: Found dependency on dhcpcd.service/start
[ 3.446610] systemd[1]: Found dependency on basic.target/start
[ 3.454285] systemd[1]: Breaking ordering cycle by deleting job console-setup.service/start
[ 3.466139] systemd[1]: Job console-setup.service/start deleted to break ordering cycle starting with basic.target/start
[ 3.483119] systemd[1]: Found ordering cycle on basic.target/start
[ 3.491219] systemd[1]: Found dependency on sysinit.target/start
[ 3.499192] systemd[1]: Found dependency on rpcbind.service/start
[ 3.507244] systemd[1]: Found dependency on network-online.target/start
[ 3.515827] systemd[1]: Found dependency on network.target/start
[ 3.523802] systemd[1]: Found dependency on dhcpcd.service/start
[ 3.531736] systemd[1]: Found dependency on basic.target/start
[ 3.539491] systemd[1]: Breaking ordering cycle by deleting job rpcbind.service/start
[ 3.550976] systemd[1]: Job rpcbind.service/start deleted to break ordering cycle starting with basic.target/start

larch
Posts: 1
Joined: Mon Dec 28, 2015 4:41 pm

Re: any chance to fix dependency ordering cycle of systemd?

Mon Dec 28, 2015 4:43 pm

ShiftPlusOne wrote:Hey, just stumbled upon this thread. Would you be able to find out what you need to install onto a clean image to get this problem? It's hard to investigate this without being able to reproduce it.
Hi Shift - yes this is easy a clean install plus:

apt-get install -y nfs-kernel-server nfs-common

Breaks this feature once you add an export.

chme
Posts: 37
Joined: Tue May 06, 2014 7:53 pm

Re: any chance to fix dependency ordering cycle of systemd?

Thu Dec 31, 2015 8:34 am

I can also reproduce this issue with a clean jessie-lite image on my rpi 1 model B.
I just have to add a nfs mount in /etc/fstab and enable the rpcbind service with "sudo systemctl enable rpcbind".

The entry in /etc/fstab looks like:

Code: Select all

192.168.178.24:/DataVolume/Public /public nfs defaults,noatime,noauto,x-systemd.automount 0 0
On reboot the rpcbind service does not start automatically and dmesg shows following error:

Code: Select all

[    3.408744] systemd[1]: Found dependency on network.target/start
[    3.416628] systemd[1]: Found dependency on dhcpcd.service/start
[    3.424401] systemd[1]: Found dependency on basic.target/start
[    3.432044] systemd[1]: Breaking ordering cycle by deleting job rpcbind.service/start
[    3.443360] systemd[1]: Job rpcbind.service/start deleted to break ordering cycle starting with basic.target/start
Starting the service manually succeeds.

There is another report on this forum, that seems to be related:
viewtopic.php?f=66&t=128816

beta-tester
Posts: 1385
Joined: Fri Jan 04, 2013 1:57 pm
Location: de_DE

Re: any chance to fix dependency ordering cycle of systemd?

Fri Jan 01, 2016 3:39 pm

ShiftPlusOne wrote:Hey, just stumbled upon this thread. Would you be able to find out what you need to install onto a clean image to get this problem? It's hard to investigate this without being able to reproduce it.
i just noticed, that there is activity on this thread...
this is ma initial post nfs-kernel-server not starting anymore since few days/weeks.

as workaround, i installed a fresh Raspbian Jessie image, and marked the dhcpcd5 package as HOLD with

Code: Select all

sudo apt-mark hold dhcpcd5
then it is save to install everything newly and get rid of the dependency ordering circle,
even after an apt-get upgrade.
but that means i do not get any updates for dhcpcd5.

i not figured out, what steps exactly triggers that issue, because i use a bigger script, to install/configure my RPi as PXE server from scratch,
but as larch sayed
larch wrote:
ShiftPlusOne wrote:Hey, just stumbled upon this thread. Would you be able to find out what you need to install onto a clean image to get this problem? It's hard to investigate this without being able to reproduce it.
Hi Shift - yes this is easy a clean install plus:
apt-get install -y nfs-kernel-server nfs-common
Breaks this feature once you add an export.
those kind of steps are also included on my script.

Code: Select all

...
sudo apt-get -y install nfs-kernel-server
sudo apt-get -y install pxelinux syslinux-common
sudo apt-get -y install dnsmasq
sudo apt-get -y install samba
...
plus some fstab and exports.

Code: Select all

## /etc/fstab
/srv/tftp/iso/win-pe-x86.iso                  /srv/tftp/nfs/winpe         auto  loop,ro  0  0
/srv/tftp/iso/ubuntu-15.10-desktop-amd64.iso  /srv/tftp/nfs/ubuntu/amd64  auto  loop,ro  0  0
...

Code: Select all

## /etc/exports
/srv/tftp                   *(ro,no_subtree_check,async,no_root_squash)
/srv/tftp/nfs/winpe         *(ro,no_subtree_check,async,no_root_squash)
/srv/tftp/nfs/ubuntu/amd64  *(ro,no_subtree_check,async,no_root_squash)
...
{ I only give negative feedback }
RPi B (256MB), B (512MB), B+, ZeroW; 2B; 3B, 3B+; 4B (4GB)

beta-tester
Posts: 1385
Joined: Fri Jan 04, 2013 1:57 pm
Location: de_DE

Re: any chance to fix dependency ordering cycle of systemd?

Sun Jan 03, 2016 7:08 pm

dropped way down in the threads again...
bump... :o
{ I only give negative feedback }
RPi B (256MB), B (512MB), B+, ZeroW; 2B; 3B, 3B+; 4B (4GB)

ShiftPlusOne
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 6234
Joined: Fri Jul 29, 2011 5:36 pm
Location: The unfashionable end of the western spiral arm of the Galaxy

Re: any chance to fix dependency ordering cycle of systemd?

Sun Jan 03, 2016 11:59 pm

Thanks for all the extra info. It will get sorted.

sielenk
Posts: 1
Joined: Mon Jan 04, 2016 12:12 pm

Re: any chance to fix dependency ordering cycle of systemd?

Mon Jan 04, 2016 12:18 pm

As I found out trying to mount a windows (samba) share:
The cycle problem with systemd gets triggered by adding a cifs mount in fstab, too.

ShiftPlusOne
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 6234
Joined: Fri Jul 29, 2011 5:36 pm
Location: The unfashionable end of the western spiral arm of the Galaxy

Re: any chance to fix dependency ordering cycle of systemd?

Thu Jan 14, 2016 5:58 pm


beta-tester
Posts: 1385
Joined: Fri Jan 04, 2013 1:57 pm
Location: de_DE

Re: any chance to fix dependency ordering cycle of systemd?

Fri Jan 15, 2016 9:07 am

ShiftPlusOne wrote:At this stage, it appears to be a bug in the way Debian handles sysvinit scripts in systemd.

https://bugs.debian.org/cgi-bin/bugrepo ... bug=763315
https://bugs.debian.org/cgi-bin/bugrepo ... bug=622394
https://bugs.debian.org/cgi-bin/bugrepo ... bug=761951
thank you for the links and spending your time to this...
but all these links were initially opened between 2011 and 2014...
i would expect, that these should be fixed in the meantime.
as i said, if i install everything to a virgine Raspbian Jessie (2015-11-21) with marked dhcpcd5 on hold, everything is working without any problems.
i run in trouble only, when the dhcpcd5 package is updating.
i mean, it looks to me like a very new issue, and not an old one.
{ I only give negative feedback }
RPi B (256MB), B (512MB), B+, ZeroW; 2B; 3B, 3B+; 4B (4GB)

ShiftPlusOne
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 6234
Joined: Fri Jul 29, 2011 5:36 pm
Location: The unfashionable end of the western spiral arm of the Galaxy

Re: any chance to fix dependency ordering cycle of systemd?

Sun Jan 17, 2016 2:15 pm

http://unix.stackexchange.com/questions ... ycle-start

It is a debian bug that we've hit by making a change to dhcpcd. Fundamentally, the problem is as described in those bug reports. The system wants all the sysvinit-style scripts from that runlevel started first, and one of them relies on having networking ready, which the dhcpcd service starts before. This can't happen, since the network won't be up before dhcpcd is started. The fix might be replacing all the nfs-related services with dhcpcd units.

Don't know if this will be addressed in the upcoming image, the one after that or when it's fixed in debian. Still looking at the options.

dagon666
Posts: 2
Joined: Wed Oct 01, 2014 10:20 am

Re: any chance to fix dependency ordering cycle of systemd?

Wed Jan 20, 2016 6:42 pm

Until we'll get a true solution for this problem, I've hacked a simple python script which restarts the rpcbind and the nfs-kernel-server after reboot, if it's not up. Seems to do the job for me and maybe it will help some of you.

https://github.com/dagon666/rpcbindplumber

beta-tester
Posts: 1385
Joined: Fri Jan 04, 2013 1:57 pm
Location: de_DE

Re: any chance to fix dependency ordering cycle of systemd?

Wed Feb 10, 2016 4:28 pm

beta-tester wrote:... if i install everything to a virgine Raspbian Jessie (2015-11-21) with marked dhcpcd5 on hold, everything is working without any problems.
i run in trouble only, when the dhcpcd5 package is updating.
damn, with the new Raspbian from 2016-02-XX i run immediately into trouble with no chance to do my workaround of last time with hold dhcpcd5 package to keep things running... :x
ok, as new WORKAROUND i now (re)start my broken services via rc.local command entry after boot... :(

Code: Select all

# /etc/rc.local
...
sudo service nfs-kernel-server restart &
exit 0
when is debian fixing that... :?: :x
{ I only give negative feedback }
RPi B (256MB), B (512MB), B+, ZeroW; 2B; 3B, 3B+; 4B (4GB)

359
Posts: 16
Joined: Thu May 29, 2014 7:55 pm

Re: any chance to fix dependency ordering cycle of systemd?

Sun Mar 13, 2016 10:14 am

Hi.. After tearing out hair, I found the post above from beta-tester, I'm up and running with nfs server running automatically, thank you to you beta-tester!

The post above from February 11th, wondering if there is any other better solution, I'm on the latest Jessie Lite on Pi3 now.

Thank you if anyone can update the situation

DirkS
Posts: 10398
Joined: Tue Jun 19, 2012 9:46 pm
Location: Essex, UK

Re: any chance to fix dependency ordering cycle of systemd?

Sun Mar 13, 2016 11:49 am

359 wrote:The post above from February 11th, wondering if there is any other better solution, I'm on the latest Jessie Lite on Pi3 now.
Not AFAIK. Only workarounds, no solutions

vahagn
Posts: 1
Joined: Mon May 02, 2016 12:15 am

Re: any chance to fix dependency ordering cycle of systemd?

Mon May 02, 2016 12:29 am

I am absolute novice in systemd and had to spend quite time to figure out this solution. But here is my workaround.
Copy rpcbind.service (and possibly other dependent services e.g. nfs-common.service, nfs-kernel-server.service) from /run/systemd/generator.late/ to /lib/systemd/system. These are the services that sysv generator creates automatically for systemd. Next edit the files to remove dependency from sysinit.target which causes the loop. Instead I added:

[Install]
WantedBy=multi-user.target

Then install these services through "sudo systemctl enable <service name>". Basically, that's it. It worked for me.

charango
Posts: 5
Joined: Tue May 10, 2016 4:00 am

Re: any chance to fix dependency ordering cycle of systemd?

Tue May 10, 2016 4:10 am

Hello there, I stjumbled upon this thread while looking for a solution to a problem I have which is very similar to the one posted here. Let me expalain:
I am a fairly new Debian user, I installed Jessie on a dual boot system with Win 10. After installing a few packages I got a systemd ordering cycle error message at boot which I am trying to understand and solve. It is very similar to the one evoked here. Could some good soul have a look at the log and point me in the right direction for a fix? Thanks very much!
Here is the message in the log:

May 9 05:34:48 debian systemd[1]: Found ordering cycle on sysinit.target/stop
May 9 05:34:48 debian systemd[1]: Found dependency on rpcbind.service/stop
May 9 05:34:48 debian systemd[1]: Found dependency on network-online.target/stop
May 9 05:34:48 debian systemd[1]: Found dependency on NetworkManager-wait-online.service/stop
May 9 05:34:48 debian systemd[1]: Found dependency on basic.target/stop
May 9 05:34:48 debian systemd[1]: Found dependency on sysinit.target/stop
May 9 05:34:48 debian systemd[1]: Breaking ordering cycle by deleting job rpcbind.service/stop
May 9 05:34:48 debian systemd[1]: Job rpcbind.service/stop deleted to break ordering cycle starting with sysinit.target/stop

beta-tester
Posts: 1385
Joined: Fri Jan 04, 2013 1:57 pm
Location: de_DE

Re: any chance to fix dependency ordering cycle of systemd?

Tue May 10, 2016 10:18 am

at the moment, my workaround to get my network work again is to restart the nfs-kernel-server service after each reboot.
for that i patched the /etc/rc.local file by adding an addition line with the restart of the service...

Code: Select all

# /etc/rc.local
...

# fix for systemd dependency cycle
sudo service nfs-kernel-server restart &

exit0
but i think it depends, what service is broken with the dependency ordering cycle...
{ I only give negative feedback }
RPi B (256MB), B (512MB), B+, ZeroW; 2B; 3B, 3B+; 4B (4GB)

charango
Posts: 5
Joined: Tue May 10, 2016 4:00 am

Re: any chance to fix dependency ordering cycle of systemd?

Tue May 10, 2016 10:52 am

Thanks for the reply, yes of course I understand it depends on the service that is "broken" so can't you tell by looking at my specific error message where the problem lies? Everything seems to be working when I boot, I mean I can go on the internet, so my network is not down. So I don't know if your fix (nfs-kernel-server) will be suitable for my system, will it?

DirkS
Posts: 10398
Joined: Tue Jun 19, 2012 9:46 pm
Location: Essex, UK

Re: any chance to fix dependency ordering cycle of systemd?

Tue May 10, 2016 10:56 am

charango wrote:Thanks for the reply, yes of course I understand it depends on the service that is "broken" so can't you tell by looking at my specific error message where the problem lies? Everything seems to be working when I boot, I mean I can go on the internet, so my network is not down. So I don't know if your fix (nfs-kernel-server) will be suitable for my system, will it?
I have the same error and to make sure I actually restart both services.
I added

Code: Select all

systemctl restart rpcbind
systemctl restart nfs-kernel-server
before the 'exit 0' in rc.local. (note that 'sudo' is not needed)

charango
Posts: 5
Joined: Tue May 10, 2016 4:00 am

Re: any chance to fix dependency ordering cycle of systemd?

Tue May 10, 2016 6:31 pm

Thanks again for the help but those two commands don't seem to change anything. After executing the first one there was no message and the second one said there was no such file.

beta-tester
Posts: 1385
Joined: Fri Jan 04, 2013 1:57 pm
Location: de_DE

Re: any chance to fix dependency ordering cycle of systemd?

Tue May 10, 2016 8:45 pm

charango wrote:Everything seems to be working when I boot, I mean I can go on the internet, so my network is not down. So I don't know if your fix (nfs-kernel-server) will be suitable for my system, will it?
does it mean, you do not notice any downsides/impact in your work...
nothing seems to be broken or you do not run into any problems...
the only thing you can see is the bool log...
so for you it is more or less a cosmetical issue only?

what did you installed, that caused that dependency ordering cycle?

if you don't see any impacts, i wouldn't care about...
{ I only give negative feedback }
RPi B (256MB), B (512MB), B+, ZeroW; 2B; 3B, 3B+; 4B (4GB)

charango
Posts: 5
Joined: Tue May 10, 2016 4:00 am

Re: any chance to fix dependency ordering cycle of systemd?

Tue May 10, 2016 9:08 pm

beta-tester wrote:
charango wrote: what did you installed, that caused that dependency ordering cycle?
That's a good question. If I knew how it happened I would be nearer a fix. ;)
It seems only a cosmetic thing as you say but I hate to have something nagging at me like that, I like to know why things happen.
I have only installed Debian since Sunday and there is a certain instability sometimes, for example the usb mouse will act strangely or the window I am working on will freeze up...I don't know if any of these problems are related..I am slowly trying to get accustomed to a linux system again after a few years away from it.

beta-tester
Posts: 1385
Joined: Fri Jan 04, 2013 1:57 pm
Location: de_DE

Re: any chance to fix dependency ordering cycle of systemd?

Tue May 10, 2016 10:04 pm

me too, i also like to know, why things happen and i hate error messages and warnings.
but sometimes you can't do anything as normal user :x

PS.: i don't think that the issues you have are related to the dependency order cycle issue. otherwise, you would see more and other additional errors in the logs...

PPS.: once i installed a package, that caused the issue, even with uninstalling that installed package(s) completely again didn't fix that issue.
only a new clean installation of the OS solved the issue. but i needed to install the package again, because i need its service. so i have to live with that issue and the workaround until a fix is available...
{ I only give negative feedback }
RPi B (256MB), B (512MB), B+, ZeroW; 2B; 3B, 3B+; 4B (4GB)

Return to “Raspberry Pi OS”