In this daily work journal, I set up a Rackspace loadbalancer and copy an existing cloud server to have two copies in the cloud, and finally test that the load balancing is actually working. By the way, if you thought you needed to be a sysadmin—or even technical—to do this sort of stuff, then then think again. Server management is becoming a lot like managing Word docs… actually, easier!
I have so much difficulty focusing, but It’s put-up or shut-up time. I need to use my daily work journal to keep me on-track. I’ve been reading up on Rackspace’s Zeus cloud load-balancing solution. I’m up to that point in getting my work ready for the public. It’s time to load-balance, and be 100% confident that the public version of Tiger will meet capacity. It’s supposed to be load-balancing in 30 seconds, for about $15/mo. Since a load-balancer is a single point-of-failure, almost all Rackspace clients buy 2. But for today’s exercise, because this is all on my personal Rackspace account, I will be keeping everything as small and cheap as possible. When the time comes to scale, I can simply up-size the virtual instances, add more virtual instances, and presumably add a new load-balancer.
So, what next? Jump in head-first, of course! Log into Rackspace… cloud control panel… done.
Hosting / Load Balancers / Create a Load Balancer
- Name it
- Choose a protocol: HTTP(80)
- Virtual IP type: Public
- Choose Algorithm
- Round Robin
- Weighted Round Robin
- Least Connections
- Weighted Least Connections
I’m choosing Least Connections. I believe the weighted stuff really only applies if you have a larger server instance that you’d like prefer to send the traffic to.
You also now select the region. I have a choice of ORD vs DFW. I’m going to choose ORD, which is the same region as my current server instance. This appears to be an opportunity to load-balance between servers in different regions. Maybe I’ll make the second Tiger instance in a different region.
The last thing is a list of your servers, to which you simply check the ones for which you want the load balancer enabled. I check my one instance. It’s sort of silly right now using a load balancer on one server instance, but my next step will naturally be creating a new server instance, so the load-balancer will at least be balancing the load between two servers. You can also add external nodes, which are apparently any external IP address.
Now, click Create Loadbalancer… done. Now I’ve got a loadbalancer. Ho ho ho.
I have to give some thought to how to test this thing. I will need some semblance of before and after. But for the next step, I will probably be using my existing cloud server as a template instance for the next server… either that, or I have to build the new server, which seems sort of silly when working with cloud servers. Let’s see…
Wow, this looks ridiculously simple. It’s taking me more time to think about and document it than to do the actual work, which is fine, because documenting and thinking about it is half the purpose here.
My existing server RAM Amount is 512MB, which is the second tier up from entry level. I’m paying about $20/mo for this. I would prefer to switch it to the first-tier, which is their el-cheap-o $10/mo virtual hosting, for which Rackspace is so known and has won so much loyalty. So, I will copy off a clone, then try to down-scale them both to the $10/mo tier, to which I will add the $15/mo load-balancer, and only be paying about $15/mo more than currently, and will be forcing the load-balancer to actually come into play, balancing between two very small cloud servers.
Okay, Hosting / Cloud Servers / My Server Images… check my single server, and click “New On-Demand Image”… oops… this isn’t going to “copy” a cloud instance. This is setting me up to configure the whole server again… okay… call that fanatical Rackspace support… done. Thank you, Darrin! These guys are GREAT!
Okay, Darrin walked me through clicking on the active server name through Cloud Servers / Server Instances, and then clicking the Images tab, and then New On-Demand Image. So I basically made myself my new on-demand image, so that now when I go to Cloud Server / Add Server, I have a Linux, Windows and My Server Images tab. Under My Server Images, I have the image I just created. Booyah!
Now, I have 2 cloud servers: one at the 512 $20/mo tier, and one at the 256 $10/mo tier. I’m going to add the new server to the load balancer…
Click Hosting / Load Balancers… click on My Loadbalancer… click Nodes tab… click Add Nodes… select the new server… Click Add Selected Nodes… done!
Okay, so now I have 2 servers on a loadbalancer, and it is presumably working. Of course, things aren’t REALLY switched over until I change the DNS to point to the loadbalancer instead of the old IP—which still exists, but shouldn’t be addressed directly anymore. This is very interesting. As complicated as a loadbalancer can be made out to be in our minds, really it’s a transparent layer just forwarding traffic. I guess there must be some communication between the load balancer and the servers, based on the connections method of balancing… but maybe not, since it initiated the connections, it could be doing its own housekeeping with no notion of the actual load being put on the servers. Hmmmmm. The algorithm does say Directs traffic to the node with the fewest open connections to the load balancer, so there must be some back-and-forth there. So long as a connection is held open to the server, I suppose it’s held open to the balancer as well. If so, that is good.
So, I should be able to plug the public “Virtual IP” given to me under the Hosting / Load Balancers / Overview tab
Okay, so plug the IP into a browser… D’ohhhhh… a username and password prompt appear! Of course, because I’m using virtual hosts in Apache. It just doesn’t know which website to serve me based on IP. And providing a domain name won’t help yet, because I haven’t changed DNS yet. Okay, I think this is a case of throwing caution to the wind, and just making the DNS entry and make it happen. Testing will be solid after DNS propegation, because I can pull up the Apache logs on both servers and watch round-robin balancing do it’s thing. Or I can put a file in the same location on both servers, with “foo” as the text content of one and “bar” in the other, surf to the file, and hit refresh, and watch the browser alternate between foo and bar.
It’s also worth noting that because I’m using virtual hosts, I don’t have to load-balance the entire server at once. I can keep the main bookmarklet that the company’s working off of untouched, and do it just to another “throw-away” domain. In fact, both of these domains are serving out of the very same directory, but one will be load-balanced, while the other is not. Very nice.
So, edit the DNS of one of the domains… done!
Wow, DNS propegation was nearly instant. I should now be able to put a different file in that directory on each server, hit F5 and see it alternate. Oops, I have to change the Hosting / Load Balancers / Configuration / Algorithm to Round Robin first… done.
Interesting! I used to ssh into these servers using domain names, but I have to use IPs now in order to be explicit. Okay, ssh’d into each. I had to re-negotiate SSL keys on the new server, and even then, I had to use the newly assigned password from Rackspace. Clearly, new images made from stored Cloud Files are not identical in every respect. New passwords are automatically generated by Rackspace. So for a first-time login, you have both the SSH key exchange confirmation, and a new password.
I logged into both, and made test.txt with foo in one file, and bar in the other, just as planned.
Now, I surf to the website… foo! Refresh… bar! foo… f5… bar… f5… foo… f5… bar!
Woot! Load balancer created, activated, configured and tested!
Now, I’m going to try to down-size my 512MB RAM server to only 256, so I’m only spending $15/mo more instead of $35…
Hosting / Cloud Servers / click server name / Resize
Hmmmmm. Clearly, the server is going to be offline for some time. I took a screenshot of that message. I’m thinking that since I have the second server, I should really use it to cover for the downtime. It would be silly not to. First, I need to remove the original server from the loadbalancer, which will force all traffic from the domain I changed DNS for to go to the new server. Then, I test that the app is working correctly on the new server. Do that before the resize…
Okay, I can just “disable” that node instead of remove it… done. And I do a refresh, and I only get “bar” which is the file on the new server. Great, so far! Now, get the bookmarklet from the app, and test the app from the new server.
BAM! Tested, and the app is running from the new server just fine. Now to edit the DNS for all the other bookmarklet domains.
Okay, I did it for the main bookmarklet the company works off of. Frankly, I don’t have to do it yet for all the other domain names used for development work, because as soon as I have Rackspace servers paid for by my company, we’ll be switching all these domains to point over there anyway, so I don’t have to keep going and editing all these DNS records. Two is enough—the old one that’s still sort-of used, and the new one that’s in mainstream use. Both of these map into the same directory through different Apache virtual server entries, but all the developer sites map into their own personal directories.
So, load balancing can be sliced and diced however you by domain name (or subdomain, whatever).
Ugh, for the main domain, DNS is managed from a bottom-tier web host, in order to get some drupal feature on a subdomain, and it’s no propagating nearly as fast as it did from GoDaddy… oh wait, I’m wrong. Something’s stubbornly caching DNS on my Ubuntu box. I tested from my Macbook Air, and I got the new IP. I’m using nslookup, so the cache is not in Chrome. Tested Firefox to confirm. Only getting foo. Ugh! Tried flusing DNS by restarting network services on Ubuntu:
…but still the old IP via nslookup. Also tested hitting the site through a co-workers machine, and it comes up “foo”. So the internal network is likely where the caching is occurring (my Macbook Air is on Guest WiFi, and likely different DNS servers). So, I can’t really resize the original Tiger server until the internal network comes around, or else I will be causing an outage of my app, and it’s pretty relied-upon these days.
Well, it’s taking forever for DNS to update here on our internal network. That’s strange, because when I did the same thing with GoDaddy, it was instant. It was instant with Hostmonster as well, but only on our Guest WiFi network. Maybe it was just the luck of the timing in hitting a scheduled refresh on the first update. Oh well, I have a 1-hour meeting coming up. That’ll help me kill time. Maybe I’ll publish this, and do an update. This is basically mission accomplished. I’m just waiting to shrink a server down from 512 to 256. Loadbalancing is actually implemented and tested.
- New Features Now Available for Rackspace Cloud Load Balancers (rackspace.com)
- Scaling for the Holidays Part 4: Load Balancers (rackspace.com)
- Howto setup a haproxy as fault tolerant / high available load balancer for multiple caching web proxies on RHEL/Centos/SL (robert.penz.name)