I’m sure many of you have seen this statue before, perhaps not the very same one in the picture, but possibly similar statues around the world. This one is located in New York City.
Statue of Atlas in NYC
This particular statue is the Titan Atlas (a God from ancient Greek Mythology) who was supposedly burdened with carrying the weight of the world – or the weight of the heavens – on his shoulders as a punishment from Zeus. Whether it was the weight of the world or something else is unclear, but most people seem to follow this same observation. In general it’s nothing more than a myth, but the lesson history teaches us is that it constantly likes to repeat itself. Clearly, no one can bear the entire weight of the world on their shoulders just like no one computer can either. If you are running SaaS (or Software as a Service) you are online 24/7 and so is your service. The problem is there are over two-billion users online (or with Internet access) today. What happens when too many of those users all start using your service at once?
What Is Load Balancing
The idea behind load balancing is that a single machine can only handle so much work at one time and you can only go vertical for so high. Notice that even in large cities you can only build so high before you have to start building out. Since on the Internet virtually anyone can be using your server at any time you run the risk of overloading without warning. If too many users all send requests to your server too quickly, the server will reach a point where the load is higher than its capacity and eventually crash. This particular vulnerability of typical client-server relationships on a network is exploited by what is commonly referred to as a DDoS attack or a Distributed Denial of Service attack. Basically, a number of clients (sometimes a bot-net controlled by one or more users) will attempt to send a lot of requests to a server or number of servers very fast in order to overload the server and prevent its intended users from being able to access the service. Sometimes this is done just to destabilize the service running on the server or for other malicious intents. There are ways to mitigate DoS attacks with firewall software/hardware or through other means depending on the service, but not all DoS attacks are malicious or even intentional in nature. Google, for example, experienced what was at first glance considered a DoS attack on its search service during one afternoon on June 25th of 2009. This actually wasn’t a malicious user or users at all. It was the world receiving the tragic breaking news of the death of Michael Jackson. Literally, millions and millions of users from all around the world flooded Google Search all at once with the same search phrase “Michael Jackson”. Google had never seen such a tremendous amount of traffic coming in all-at-once on a single search query, before, so their first thought was “ohnoes, we’re getting DdoSed!”
Scaling Out - SaaS
Why Do I Need It
The fact remains that any number of users can suddenly surge the number of requests coming in to your servers at any given time and whether that is malicious or not is unimportant. What is important is that you are better prepared to handle such situations so that your service will suffer as little downtime and degradation as possible. So load balancing allows you to distribute the load on a particular service or services over a larger array of resources. It’s basically making your service, as a whole, more tolerant of failure by being able to efficiently make use of all available resources.
If you are running any kind of high availability service over the Internet you need load balancing. Though, even small applications with just a few thousand users can benefit deeply from load balancing, as well. The only potential down-side is that you may need more than just one node to it. This isn’t always necessary as load-balancing can come in many shapes and sizes. For example, you might be doing load balancing on the same host node using multiple guest nodes on the same machine. All of the major services you probably use on a regular basis like your email, search engines, or popular social networking apps all make use of load balancing because it keeps things running a lot more smoothly as the number of users grow. If you’re not on-board with this yet – you probably should get on board quick.
How Do I Use It
There are few broad categories you can place load balancing techniques in. The easiest form of load balancing relies on existing system already built on top of how most systems function over the Internet (or large networks in general) and that’s DNS. DNS is a distributed system so it relies on multiple components in the network to do their job in order to make things more efficient. It reduces bottle-necks like those created by routing enormous amounts of packets across the planet in fractions of a second. Like most complex systems everything starts off small and simple and grows both horizontally and vertically, but at the core the protocols are fundamentally very simple.
DNS Load Balancing is simply relying on the DNS system to take care of the most basic problems for you. The way this works is you set the DNS record for a particular domain name to multiple IP addresses (usually one for each server) using low TTL (or Time to Live). Since DNS is cached at various levels this makes things like geographical loads efficient for services like name servers. A name server tells the DNS where to send the request for a particular domain name and can route packets to different locations depending on the geographical origin of the request thus alleviating network latency and allowing packets to travel shorter distances. Once the request comes in and is routed effectively the DNS is cached at multiple levels so that future requests are made to the same place. This can be cached at the local level, the ISP level and other levels in the parent zone. The name server then doesn’t become a bottle-neck since not every single request has to rely on that name server entirely. There is a TTL involved that will let the caching servers know when the cache has become stale and that it’s time to refresh. Also when requests to a particular server are no longer getting through the DNS server will know to try a different IP. So if you have different servers with different IPs in the DNS record that ultimately means if one server becomes unresponsive (potentially having gone down) the load is directed to a different server. The inherent problems with this approach are that it isn’t making very efficient use of all of your resources. It doesn’t take into account which servers are currently busy and if the DNS record has already been cached to a server that is now down you end up potentially being stuck with a poorly responsive server until the cache is refreshed. Additionally, you are exposing your infrastructure to the outside world by revealing the public IPs of your servers with no way to control the flow of traffic to an internal network. It’s very easy to have an unstable system this way. Most services that use this approach are usually just creating what is known as mirrors (servers that back each other up so that in case one goes down a backup can still be reached).
Software Load Balancing is another approach to solve some of the short-comings of the DNS offloading techniques described earlier. Software load balancers attempt to keep track of the available resources and when an incoming request is received it determines how to best allocate those resources in-order-to service that request. The benefits of this technique are that you don’t have to reveal your network setup to the outside world. Everything can be done on the internal networking configuration setup (whether that’s a local area network or otherwise), or in other words, you won’t expose your communication channels directly. Also, you have a tighter hand on security and distribution since you can more easily control the flow of traffic over the network. Some examples of common open-source load balancing software are Pound, Varnish, mod_proxy for Apache’s httpd, and Gearman. There are all sorts of nifty ways to balance the load across your network. You can have the load balancers poll the servers and check on resources like CPU usage, available memory, storage space, network traffic or open TCP connection, etc… The load balancer can then use this information to figure out how to best direct the incoming requests and serve up the responses as quickly and as efficiently as possible. There are still a few problems inherent to this technique depending on how you use it. If you’re only relying on a single machine you have a single point of failure. If the host node goes down the load balancer and all of your resources go with it. If you’ve only got one load balancer and multiple servers you still have a single point of failure. Additionally the load balancer itself can be DoSed given an attack of enough magnitude and proficiency. Not only that, but you have to worry about things like session storage consistency across multiple servers, file-system access, database synchronization between different database servers, and some network bottle-necks that might not always be easy to resolve with load balancing – to name a few.
Hardware Load Balancing there are some hardware load balancers as well. You can actually buy very expensive firewall/routers that take care of many of these things for you. Most people usually just setup a dedicated node or two with software load balancers that pretty much do the same thing. These hardware load balancers might do a better job of handling security and high bandwidth loads like Cisco’s ASA, but they do come with a heavy price tag.
Some Load Balancing Tips
There are some pretty common approaches to some of the problem inherent to distributing a service over multiple servers. For example, take your session storage as the most obvious problem. If you’re using PHP you are probably using the built in session handler, which makes use of file-based sessions. If you have users being directed to different servers by the load balancer you end up with the user having multiple sessions across those servers (that might be a little problematic for your application and annoying to the user). Some people will try to avoid this by creating what’s called a sticky session. Once the session is generated for that user they’re sent a cookie that lets the load balancer know upon subsequent requests to direct the user to this particular server. There are a few minor problems with that, but nothing you couldn’t work out through a well-planned architectural approach. Another way to approach this is by creating a centralized session storage server where all the requests will look for the session. Depending on your infrastructure this may or may not be a good idea and keep in mind it also creates a single point of failure. For example, if your servers are built on stacks (you have several software-based servers running on the same node like a webserver, database server, application server, etc…) it takes some tinkering to configure each stack to work from a centralized session storage. You can use something like Redis where you can have master/slave replication across all stacks. This takes a little less configuration and puts the dynamic into the software stack layer – thereby removing it from the load-balancing layer.
The other obvious problem is file system storage. If you allow your users to upload files to your server, or you store large amounts of files that your application relies on heavily, there needs to be some system whereby your application layer can access those files considering the load balancing may send requests to different servers. Again there is a centralized approach like with session storage, but even with a replication approach – to avoid the single-point of failure down side – you might create the problem of over redundancy. If your servers are set up in stacks having four or five copies of each file (or more depending on how many servers you have) on each server stack is a bit of a waste, especially if you’re already using RAID arrays for redundancy. Even if you have a centralized set of servers for storage you still face the problem of network overload. For example, consider that if your backbone bandwidth capacity is at 100Mbps but your central network bandwidth capcity is at 10x100Mbps you eventually create a bottleneck with increased usage as your backbone can only serve up to 100 megabits per second of traffic at any given time.
Using a CDN (or Content Delivery Network) is one solution often used when large amounts of files need be shared across a network, but this can also be a bit costly depending on your needs. In its simplest form a CDN is really just a group of servers that store files or data objects for you and replicate them across multiple nodes allowing many other servers on the network to access that data with improvements in caching and high bandwidth to reduce latency. The servers in the CDN clusters are usually strategically located on the edges of the core network to minimize the bottlenecks involved in the centralized network loop. So you are redirecting the traffic to access file storage away from the central network and off to the edge servers expanding on bandwidth and minimizing on bottle neck traffic. This solves both the single-point of failure problem as well as taking the complexity mechanism away from the server stack which can ultimately help reduce loads and create more efficient load balancing. Most services that utilize CDNs are usually ones that need to offer high-bandwidth access to a large user base with consistency. For example, a service that offer Hi-Definition video streaming, large photo sharing web sites, or other media services with high availability needs. You don’t always have to build this infrastructure yourself. You can rely on services like amazon Cloud Front which is a pay-as-you-go CDN service offered by amazon. There are many other competitors, of course, that can offer cheap CDN solutions. Depending on the sensitivity of your data this may or may not be an option for your particular SaaS needs. Still something to consider.
Besides just file storage you probably have a lot of database concerns in a system that scales horizontally, as well. If you’re just using a single LAMP stack with little more than PHP, MySQL and Apache running your back-end it might seem easy to scale wide at first. The problem you’re likely to run into head-on is the data-replication across your MySQL servers. The database is almost always the biggest bottleneck in SaaS. It usually contains tons of data that virtually every one of your users will access with each hit. There’s only so much traffic a single database server can handle, but setting up two or more database servers can show some significant improvement. Your load balancer can also play a role in this. There can be data object caching mechanisms in place to ease off some of the load for the most frequented queries. There can also be network latency issues to deal with once you have several database servers all replicating (especially if these servers are geographically spaced out across different data centers, cities, countries or even on different continents). Chunking is definitely not something I’d advice. It throws way too many variables into the equation and presents more problems than solutions – for the most applications.