HELP! GUI no longer working on RPi4B 4GB unit

[Bosse_B]

I have had this RPi4 working for quite some time now and I usually access it via PuTTY for command line work) and RealVNC (for development in th GUI environment). It has constantly (weekly at least) been updated so it should be up-to-date.

Today I set out to experiment with an RTC time module using the DS3231 chip.
But I did not succeed with this so I reverted the changes I had done for it and also removed the RTC module itself from the GPIO connector.
Now my RPi4 behaves strangely on boot, it hangs for 1-2 minutes waiting for wpa-supplicant, then continues to the login prompt.
But when I do log in I am left in the text mode anyway!
And when I use RealVNC to get to the desktop it works OK (no error messages or such) but it only displays a gray background and there is nowhere to click the mouse...

NOTE:
Using sudo raspi-config I have set boot options B3 (Desktop GUI requiring user login).
I have also set vnc to create a virtual desktop to which I connect from my Win7 PC.
It has worked really well for many weeks.

I am completely baffled by this since I have not touched the GUI stuff at all.
sudo apt update reports all packages are up to date.

Where can I look in order to restore the GUI? I see nothing I can relate to in syslog.
What has caused this strange behaviour.

[PhatFil]

If your unsure about the path to backtrack your recent changes to undo any error, its possibly quicker in the long run to pull your data off the system sd card and reinstall raspbian from scratch. (and when done use the sdcard copier to create an emergency system sd card and your likelihood of ever needing it again will drop dramatically. Sods law..

If you can post the changes you made and the changes you made to backtrack you may spot the issue yourself thanks to the process and if not the collective wisdom may be able to shed some light? Though you could just be delaying the inevitable

[Bosse_B]

Not so long ago (some weeks) I hosed the same system when I tried to make the WiFi operate as an Access Point (without being a router) such that a person nearby could connect to its AP and then access the web based configuration system for the unit.
After I failed to fix that I was left with a system that no longer recognized the WiFi at all - no matter what I did.

So in the end I pulled out a new SD-card and started with a fresh full Buster image.
Ran all of the apt updates etc until it was up-to-date.

Then I put the old SD-card in a USB caddy and mounted that so I could copy over the work files I had in home/pi.
To get there I also had to install a set of tools I need such as SVN etc.
In the end I got it going but it took a day to figure out all I had done to get where I was and which was not in home/pi, like configuring history so it could be used as a tracker for what I had done via apt.

So now I am probably facing the same issue here, sigh.
I cannot for the life of me understand how the desktop could just disappear when I was playing with the I2C RTC device, though...

LATER:
I forgot to send the above when I was researching my options...
Turns out I had made notes on the previous restore and I had also saved the history command output from the old system.
So I did a history dump now on the messed up system too so I could see what I had used apt install for after the previous restore.

Then I proceeded to repeat the building from scratch and copying of data basically the same as the first time I did this including installing the useful later added software (history is really useful in situations like this).
Now all is well (I hope) with my main stuff working as before and no apparent problems anymore.

This brings up one final question, though:

Archiving backups?
I know there is an SD-Card copier shipped with the GUI of Buster and that is fine for creating a working clone of the current system.
But it only operates on an SD-card as target. I have found it to be very difficult to keep these tiny memory cards organized as backups, so I wonder if there is a utility to create an image file from an SD-card on an attached or networked hard drive (NAS)?
I.e. something like the following procedure:
1) Use SDCard Copier in the GUI towards a physical SDcard in an USB reader
2) Use something else (what?) to create an image file from the copied SD-card targeting a NAS or attached hard drive file.

This way one could keep a range of configured images on a backup drive as files...

[Heater]

I'm curious, with all that detective work on your history and such, did you ever discover what it was that you did that messed it up?

Typically I do not make backups of Pi SD card images. I started out doing that years ago by popping the card into an SD reader attached to a Linux PC and using the "dd" command to copy all the SD card to an image file. I could then use "dd" to write that image file back to a card if need be. This procedure has a number of problems:

0) It's slow and tedious.

1) Generally there is a lot of empty space on the SD card file system. One is spending a lot of time and disk space backing up nothing!

2) Those image files are big. Wasting a lot of space on my PC drive.

3) When it comes time to restore you run into problems if the SD you want to put the image on is not big enough. Or if it is bigger you end up not using it all.

Now I don't do that. The main bulk of what you want to write back to a new SD card can be had from a fresh download and install of Raspbian pretty quickly. The stuff that you need backed up is often reproduced from a bunch of apt-get install commands. Or it it's my own code a quick pull from the github repository it all lives on. Configuration changes are a matter of keeping records of what you have done/and or backing up copies of the changed config files. Often saved in github again for quick recovery.

Ideally one would script all this. Creating a personal configuration script that would download, install and configure everything just as you like. But that's a bit much to maintain when one is hacking and experimenting all the time.

[Bosse_B]

I'm curious, with all that detective work on your history and such, did you ever discover what it was that you did that messed it up?

Unfortunately not! I have been down this route twice now in the last few weeks and both times I had intended to check something I thought was safe (setting up WiFi as access point and testing the I2C connection to an RTC module). In the process somethings were installed via apt and probably changed basic settings I have no idea about. Result: broken system.

0) It's slow and tedious.
1) Generally there is a lot of empty space on the SD card file system. One is spending a lot of time and disk space backing up nothing!
2) Those image files are big. Wasting a lot of space on my PC drive.
3) When it comes time to restore you run into problems if the SD you want to put the image on is not big enough. Or if it is bigger you end up not using it all.

Now when I had to do the resurrection for the 2nd time I followed my notes from the first time plus used the history command output to see what apt installs I had done. So I got to a new working system with my home/pi data all restored.
But now I used the GUI program to copy the image to an SD card, in this case of half the size of what I normally use (16 vs 32 GB).
This will serve as my rescue in the future if I am hit again.
And you are right it IS very slow and tedious. And the final img file is very big and contains a lot of zero space.
So I used 7zip to compress the image file from 15 down to 4.5 GB as a zip archive.
Procedure:
1) SD Card Copier used in the GUI onto a smaller SD than the original
2) Win32DiskImager on Windows7 to create an image file from the SDcard
3) 7zip to reduce the size of the img file from 15 to 4.5 GB
And this all took a loooong time.
But should I ever have to re-image to an SD card Etcher can use a zip file as input so no need to expand the compressed file anyway.

Now I don't do that. The main bulk of what you want to write back to a new SD card can be had from a fresh download and install of Raspbian pretty quickly. The stuff that you need backed up is often reproduced from a bunch of apt-get install commands. Or it it's my own code a quick pull from the github repository it all lives on. Configuration changes are a matter of keeping records of what you have done/and or backing up copies of the changed config files. Often saved in github again for quick recovery.

Ideally one would script all this. Creating a personal configuration script that would download, install and configure everything just as you like. But that's a bit much to maintain when one is hacking and experimenting all the time.

I agree in principle (I use Subversion though), but maybe I should have created a new system as described above from the downloaded image, but then only performed the actions that pertain to the system itself like installing the stuff via apt install and doing nothing at all about the user files yet.
Then after that was done I could have created the backup system disk sans the /home/pi stuff.
Such an image would have been pretty much smaller and would be simpler to create and used.
Then I would also maybe use tar to archive the complete /home/pi directory once it had been properly set up as a home.tgz file.
In such a case the actual file changes in the future would probably only affect the /home/pi dir and could be easier handled unless they were caused by new apt installs, which would have to be kept track of.

If I am hit again in the future I will probably head that way.
Is there a way to archive away (using tar for instance) only changed files in the home directory?
Thus in the future one could back up more easily what has actually changed....