[Roosh]

Matt's tattoo article last week caused us to hit the ROK's server capacity and so it's time to upgrade the hardware.

The software has already been optimized. Here's the current setup:

Nginx + PHP-FPM + APC PHP + Pagespeed + Varnish + Wordpress + W3 Total Cache

The weak point is W3 Total Cache, which is bulky compared to the HyperCache plugin, but it has an Amazon cloud CDN option that I can turn on in times of heavy usage to take pressure off the server. It's like a power boost.

With the CDN turned on, our dual processor server (16 cpus) can handle about 18 page views per second before the server load becomes too high. With the CDN turned off (i.e. the server processing and displaying everything), we can do about 10 page views a second.

I want to add a second server to our setup. How should I set up that server to maximize our capacity without breaking the bank?

Option 1: Use the second server as a CDN server. Problem with this is we have no "power boost" option by flipping on Amazon if we hit capacity again. This may be a good option if we want to add a 3rd server in the future. As long as I have the Amazon CDN option, I don't think this is the best option right now.

Option 2: Use the second server as an SQL database server. For a Wordpress site, how strong does the database server have to be? How much would it increase capacity?

Option 3: Hook up a second server and a load balancer. From what I understand we'd have to get some type of syncing software. Theoretically this option would double our capacity, but it's also the most expensive, because in addition to the second server, we have to pay monthly for the load balancer.

Maybe there is another option I'm missing?

Whatever option we take should take into account that we'll probably have to add a third server in the future. I'm looking at the best combination of capacity and cost. I don't mind sacrificing speed (page loading time) a bit.

[JustJay]

If TravelerKai is still hanging around you should check with him.

[ryanf]

I'll post a longer answer later tonight, I'm working till around 9. In the meantime, do you have snapshots of the server load statistics at the time it got very busy?

[Roosh]

(10-21-2014 02:13 PM)ryanf Wrote:  

I'll post a longer answer later tonight, I'm working till around 9. In the meantime, do you have snapshots of the server load statistics at the time it got very busy?

What kind of statistics are you looking for? The server is still busy at the moment from Matt's article... I can run commands as root in Putty.

[DJ-Matt]

I would go with option #3 if you can afford the expense, PHP is notorious for being CPU and RAM hungry, another mirrored server would solve that.

Because of caching I can't imagine the SQL being hit very hard. Depending on what hardware the current server uses, the problem might be easily solved by installing newer generation CPUs. I assume the RAM isn't getting maxed out?

Our website's server started to chug after we implemented a flash/html5 station player, changing out the CPU to one with more cores solved the problem.

Back when we ran our own streaming server with built in ad serving, I used the Linux "top" program to monitor things in realtime.

[Roosh]

Quote:

Depending on what hardware the current server uses, the probable might be easily solved by just newer generation CPUs. I assume the RAM isn't getting maxed out?

RAM is okay... I haven't seen our memory pass 50% usage.

Current server:

Processor: Dual Quad-Core 2.26GHz E5520 Nehalem Xeon
Memory: 12GB DDR
Hard Drive: 480GB Intel Solid State Drive S3500 series / SSD (Intel DC S3500)
Bandwidth: 1 Gbps port

[ryanf]

Are you able to tell if it's CPU that's pegging or disk access? I can get together some commands to ascertain this if it's not readily available through your server admin

[ryanf]

Okay, now that I'm home, I have about 30 minutes before I have to leave for martial arts.

I'm willing to bet, based on your server stats, that mysql is not the bottleneck here. Tuned correctly, your entire dataset from ROK probably fits into memory, and then mysql would rarely have to go to disk. If the memory is set very low for mysql, then it's frequently going to disk, although with that small of a dataset, linux is probably caching most of the disk reads anyway. So unless you see very high disk I/O, moving mysql to another server isn't likely to help you much.

If like last time, however, the bottleneck is caused by the sheer number of http requests to nginx (and eventually to php), then your best bet is option #3. Nginx is pretty efficient, but at high capacity, it will start to throttle or drop requests. PHP itself, when it has to be accessed, is a notorious memory and CPU hog. I've been writing in it for 15+ years, it's always the easiest to get set up with the drawback of resource contention. By load balancing between 2 servers, you get the benefit of the traffic being routed to the least busy server. Plus, if one machine goes down, a load balancer will detect that and send 100% of traffic to the other server, so you don't end up denying requests (unless server #2 gets overloaded also). 18 page view per second seems a bit low, even for wordpress. Do you know how to run the "top" command?

[plague]

Regarding your options:

1) I think in general you want to move as much as possible to Amazon's CDN (or any external CDN that you like). From glancing at the page source it looks like you are already doing for the CSS, Javascript, and images

2) It's hard to say whether you should use your 2nd server as a database without any numbers to support it. Do you have access to any statistics on your current database server? Are you able to collect any metrics from within Wordpress as to how many queries per second you are executing, and the average/median latency? My guess is that this isn't your problem.

3) I think making your 2nd server essentially a clone of the first and putting them both behind a load balancer is the best option. I don't think you necessarily need any syncing software, but depending on how W3 Total Cache is operating, you may end up with some duplicate caching across both servers as, for any given user, the first request may go to server 1, the 2nd request may go to server 2, and so on.

Thinking about the situation when your server became overloaded - what would have been ideal in that case is if you could have just served your entire site from cache for a period of time, setting a reasonable TTL/expiry so that updates would still propagate every few minutes. Imagine if every 5 minutes, the first person to hit a page would result in that page going into cache, and then for the next 5 minutes everyone would just load the cached version. It may be be worth considering having returnofkings.com always point to CloudFront instead of just cdn.returnofkings.com. Sometimes this can be a pain in the ass if you have to purge something from the cache, but it's worth a thought.

I admittedly don't know much about Wordpress but I deal a lot with caching and traffic management in my job.

[ryanf]

Also, and feel free to PM this (potential security risk), are you running any plugins to the wordpress installation?

[ryanf]

One more very important thing to note: if you have any plugins that do session work in wordpress, when you move to 2+ servers, you will have to change the session storage to database from filesystem. This should only apply if you have plugins that use session level data, which you probably don't use unless you have a shopping cart or another plugin that relies on keeping state from page to page.

[FunctionalPsycho]

Third option seems best. You don't need dedicated expensive hardware to act as a load balancer. You can just take a regular linux running computer and install a load balancer such as HAProxy on it.

Caching seems to be the best approach since ROK handles reads far more than writes and changes in data.

[Paracelsus]

Guys, at the risk of being paranoid I'd suggest you not talk intimate details of your setups other than in PM.

I have a horrible (and probably paranoid) suspicion given the previous viral article that the next move from the combined swarm of White Knighting Manginas will be a DDOS (or worse) by the guys in the V for Vendetta mask.

[HawkWrites]

Time out, what's the budget here, Roosh?

The budget determines the further rack set-up so that will be an important tidbet when it comes to trying to predict future abilities since the number of users to be managed depends directly on the server specs and that has a direct bearing on budget.

What are we running server-wise here?

If you're running a 1Gbps switch then you're limited by the amount of data each client (connection) is requesting and retrieving. For example, if you have 10 clients requesting 100 mb/s of data each then you'd have 10 possible connections. I know that's an extreme example but it illustrates that the end-point you're operating on could use some upgrading. For now, you should be fine but at some point you'll need to upgrade your infrastructure.

[thatGuy]

(10-21-2014 12:36 PM)Roosh Wrote:  

Nginx + PHP-FPM + APC PHP + Pagespeed + Varnish + Wordpress + W3 Total Cache

This is a good stack. Probably as good as you can do with wordpress.

(10-21-2014 12:36 PM)Roosh Wrote:  

Option 1: Use the second server as a CDN server. Problem with this is we have no "power boost" option by flipping on Amazon if we hit capacity again. This may be a good option if we want to add a 3rd server in the future. As long as I have the Amazon CDN option, I don't think this is the best option right now.

You could maybe offload static resources to a very cheap server (or two, or three -- at $10/mo you can go wild) since serving them is not very CPU-intensive and they're small enough to be held completely in memory so that would be fast. It would be a poorman's CDN. Expect similar gains as from the regular CDN.

(10-21-2014 12:36 PM)Roosh Wrote:  

Option 2: Use the second server as an SQL database server. For a Wordpress site, how strong does the database server have to be? How much would it increase capacity?

How busy is MySQL? I suspect that with as much caching as you do, it barely does anything. You don't have a hugely dynamic site: comments are handles by Disqus, articles change rarely(?)... But maybe you have some plugin which constantly molests the db?

Option 2 probably wouldn't do much for you.

(10-21-2014 12:36 PM)Roosh Wrote:  

Option 3: Hook up a second server and a load balancer. From what I understand we'd have to get some type of syncing software. Theoretically this option would double our capacity, but it's also the most expensive, because in addition to the second server, we have to pay monthly for the load balancer.

You don't really need a load balancer. You can just add another A record to your domain, and clients will randomly choose one of the two servers. When you're running a webapp with very many backend servers then you need to balance to avoid sending traffic to a server which is swamped but here you could probably get away with el cheapo solution. It won't help when one server goes down completely (half of the people will get nothing) but will help with the load.

(10-21-2014 12:36 PM)Roosh Wrote:  

Maybe there is another option I'm missing?

Option 4: You could turn to static files. Scrape your wordpress, and just serve it as plain html/css/js. It should be a boost but also pretty desperate, potentially inconvenient, and not necessarily much better than varnish already is.

But could be a massive win, hard to say from the outside. It would also make it very easy to distribute those files to multiple servers. Might even decrease your hosting bill while improving performance.

Option 5: Put your site on a diet. It takes 187 requests to load http://www.returnofkings.com/45275/5-mis...sian-girls it takes 50 to load yahoo.com. You're not yahoo.

You could combine all those js scripts into one file, many of the css into one file, minify them, maybe drop some images that don't matter. There's some other trickery to be done here to improve load speed but you don't care about that.

Option 6: Depending on your budget, consider Cloudflare. Maybe you could get away with the $20 plan but you might need the $200 one. The problem is that CF is a bit pozzed, though they do host 4chan. It would also make you immune from DDoSes and even some more sophisticated attacks.

Option 7: You probably already did that but have someone look if there are no performance bottlenecks in your current setup. Maybe it's running out of file descriptors, has too few workers/threads, or something easy to fix like that. Maybe a new kernel? There were a couple of optimizations around TCP lately.

Note that I don't use Wordpress any more so those are based on general web experience. But you should do something about 5 regardless of any other steps you take.

[loki]

why not use a few AWS virtual machines setup to auto scale so when you need more resource, they just spawn more servers when resources run low?

this type of config would run as a cluster, and you should keep SQL on a separate system if you follow best practice design rules for these types of deployments.

You can write the rules to make AWS servers do this and do away with physical hardware all together, just auto scale

that's what i would do.

[HawkWrites]

(10-22-2014 07:55 AM)loki Wrote:  

why not use a few AWS virtual machines setup to auto scale so when you need more resource, they just spawn more servers when resources run low?

this type of config would run as a cluster, and you should keep SQL on a separate system if you follow best practice design rules for these types of deployments.

You can write the rules to make AWS servers do this and do away with physical hardware all together, just auto scale

that's what i would do.

He'll still need to create more VMs when the resources run low; servers don't just pop up magically even with AWS.

Do you have any load balancer running, Roosh? You can control the number of requests being sent to each server that way and can balance the overall load.

[Roosh]

Quote:

Do you have any load balancer running, Roosh?

Nope.

[loki]

Actually you can make AWS auto spawn.

I will pm you some links on it in case you have a professional interest in this type of thing.

(10-22-2014 06:20 PM)HawkWrites Wrote:  

(10-22-2014 07:55 AM)loki Wrote:  

why not use a few AWS virtual machines setup to auto scale so when you need more resource, they just spawn more servers when resources run low?

this type of config would run as a cluster, and you should keep SQL on a separate system if you follow best practice design rules for these types of deployments.

You can write the rules to make AWS servers do this and do away with physical hardware all together, just auto scale

that's what i would do.

He'll still need to create more VMs when the resources run low; servers don't just pop up magically even with AWS.

Do you have any load balancer running, Roosh? You can control the number of requests being sent to each server that way and can balance the overall load.

[Handsome Creepy Eel]

I see that CloudFlare is now protecting our forum. Good work, Roosh