The long-time X-perts out there already know this, but, for those who have arrived more recently, to be completely technically accurate, an X server actually runs on what would normally be considered a client system. That is, when you run X on your local PC/Mac/workstation/mobile-device/Pi/etc., you're actually running an X server, not a client. You can then create connections to remote systems via the network to share the resources between systems. That's why there can be multiple user views into other X systems (e.g., 192.168.1.2:0, 192.168.1.2:1, 192.168.1.2:2, etc.), while with technologies such as Windoze Remote Desktop Protocol (RDP), the display on the remote system being logged into disappears locally and is instead displayed on the remote system's video output.
Knowing this makes it much easier to wrap your head around what's actually happening with X and, as mentioned above, why the X "client" on your local system seems so heavyweight, code-wise - it's really a server running a pretty demanding stack of services, whether you need all of them, or not (most of them have to do with reconstruction of the remote views (plural, optionally) from amazingly sparse streams of display element descriptors and values). I would highly recommend going back to the original MIT Project Athena papers (or historical summaries that contain the project goals) and reading about what they were attempting to do and why it was so challenging over relatively slow networks (compared with today's) as opposed to the solely text-oriented network protocols (e.g., telnet, ftp, mail, etc.) running on the early Unix systems up to that point. If you delve into the network sockets details, you will learn more about socket-based networking fundamentals than you could doing probably anything else.
The best things in life aren't things ... but, a Pi comes pretty darned close!
"Education is not the filling of a pail, but the lighting of a fire." -- W.B. Yeats
In theory, theory & practice are the same - in practice, they aren't!!!