Distributed webservers with "some kind of HA"

[jors]

Hi there,

I have a dream. Or at least, an idea

I am trying to set up some kind of HA (High Availability) system on a LWMP (Linux + Webserver + MySQL + PHP) stack running on top of N Raspberry Pi (RPi from now on). I say "some kind of HA" because I plan to do this vía DNS + multiple A registers, a simple way to do this without the need for additional services checking availability [1][2]. The way I see this working is:

- N dns A/CNAME records pointing to N different geographically located RPis.
- The browser's dns request will return one of these RPi ip addresses.
- If the web service is working on that ip, all is happiness and ends here.
- In other case, there will be a lapse time until a timeout is reached (about 1 minute?) and the browser tries to reach the next chosen ip.

This is on the dns + web side. On the MySQL side, the hardest thing to do seems to be the database replication setup, which I guess the best option in this case is master-master or multiple master replication [3] (although I am not sure how this will work on more than 2 hosts). I am also open to suggestions about using other database servers for this purpose.

I am aware of the following:

- This is not real HA, but it is always better than downtime.
- Maybe this won't work on all browsers (I expect it to work in modern ones).
- This can surely break user web sessions / SSL (I don't need them).

Thoughts on this? Does it make sense? If so, someone with the same needs willing to try?

[1] http://serverfault.com/questions/60553/ ... ecommended
[2] http://serverfault.com/questions/189290 ... my-website
[3] http://brendanschwartz.com/post/1270290 ... eplication

[broo0ose]

The pro way of doing this would be using Anycast DNS. That is a mixture of DNS and routing. You have the same IP address on more than one server in different locations, the routing protocols running on the server itself select the server that will serve a request. But that requires access to routers and IP addresses that I'm guessing you don't have.

Unfortunately DNS does not allow for a flip to the next available IP address if the first is unresponsive, I wish it did. You would have to code that into the client app but In this case it's the browser. DNS can be used for load balancing, which would reduce the hit if one server went down.

[jors]

The pro way of doing this would be using Anycast DNS. That is a mixture of DNS and routing. You have the same IP address on more than one server in different locations, the routing protocols running on the server itself select the server that will serve a request. But that requires access to routers and IP addresses that I'm guessing you don't have.

Sounds good, but unfortunatelly you are right: I don't have that access

Unfortunately DNS does not allow for a flip to the next available IP address if the first is unresponsive, I wish it did. You would have to code that into the client app but In this case it's the browser. DNS can be used for load balancing, which would reduce the hit if one server went down.

Well, maybe not at the DNS protocol level, but it seems it does at application (browser) level! This are the steps of my test:

1) Created the following dns registers:

Code: Select all

  test.enchufado.com	A 	79.158.83.87 # User DSL ip: answers to ping, no webservice running
  test.enchufado.com	A 	173.194.41.23 # Google Search node

2) Requested the created subdomain with different OS/browser. If the Google page was returned inmediately, test was discarded, dns + browser cache flushed & process repeated. In all cases, Google page was returned after some time. I guess this is due to application/browser logic. This are the times taken for loading Google in different situations:

Code: Select all

  Windows XP + IE 8 = ~ 25 secs
  Windows XP + Firefox 20 = ~ 25 secs
  Debian GNU/Linux + Iceweasel 10 = ~ 1 minute
  Debian GNU/Linux + Google Chrome 26 = ~ 1 minute

So the idea still seems useful

[broo0ose]

It looks like you are correct, I found this..
http://www.zytrax.com/books/dns/info/failover.html
I did not know browsers did this, which is interesting.
As you stated it has limitations, but better than nothing as long as it does not cause more problems.

[sadme]

On the MySQL side, the hardest thing to do seems to be the database replication setup, which I guess the best option in this case is master-master or multiple master replication [3] (although I am not sure how this will work on more than 2 hosts).

Well, MySQL replication is a sysadmin's worst nightmare. Master-Master configurations work fine when you're doing load balancing instead of HA. If for some reason MySQL servers go out of sync (i.e. the connection between them breaks down), you'll be in trouble. For example, if you have an auto-increment attribute on some indexes, you'll start to have duplicate identifiers with different content, and you won't be able to merge new entries without any effort. Of course there are some tricks to avoid it, but you'd better think about master-slaves configuration with one Write-only server and many Read-only ones. Implementing a failover mechanism for the scenario where the Write-only connection fails won't be a trivial task, but it will make a disaster recovery easier.

[jors]

Well, MySQL replication is a sysadmin's worst nightmare. Master-Master configurations work fine when you're doing load balancing instead of HA. If for some reason MySQL servers go out of sync (i.e. the connection between them breaks down), you'll be in trouble. For example, if you have an auto-increment attribute on some indexes, you'll start to have duplicate identifiers with different content, and you won't be able to merge new entries without any effort. Of course there are some tricks to avoid it, but you'd better think about master-slaves configuration with one Write-only server and many Read-only ones. Implementing a failover mechanism for the scenario where the Write-only connection fails won't be a trivial task, but it will make a disaster recovery easier.

Thanks for your comment. Sadly, I think you hit the nail here. Going this way means that:

• A single RPi has more power than the rest. It has to be defined which one will act as Master (preferably the most reliable in order to enabling failover the least possible).
• When failover is active (i.e. Master is down), websites will work in readonly mode and need to be code-ready (avoiding database writes by failing silently, showing an informative message or any other desired handling).

I think I cannot ask for more by using this kind of failover.