How to prepare your site for Digg Effect (Diggstruction)
Posted in IT, Internet, Server Diary | By tarotoast |
When a site was dugg to the front page of Digg, the amount of requests to the server will usually take down small sites running on relatively low end hardwares (small RAM, slow CPU, HDD). To prevent this from happening to your site (specifically, WP blogs), there are things you can do to maximize what low end hardwares can manage, and therefore prevent the Digg Effect.
From last experiment, I am very aware this $69/mo (now $99/mo) 1&1 Dedicated Root Server with 2.0Ghz Celeron, 512MB RAM and 40GB HDD cannot take on Digg Effect without doing some tweaking. Here are the measures I’ve gathered:
1) Cache it: WP-Cache 2.0
A very-easy-to-install-and-use plugin. By creating cached pages, it reduce the php code parsed by the web server, and also eliminate the query to mysql database. Both speeds up the page load dramatically.
You can test to see the difference by doing very simple test, this is what I get:
Without cache:
time wget http://blog.tarotoast.com/
>> 0.005u 0.007s 0:01.36 0.0% 0+0k 0+0io 0pf+0w
With cache:
time wget http://blog.tarotoast.com/
>> 0.003u 0.012s 0:00.02 50.0% 0+0k 0+0io 0pf+0w
I don’t really know what all numbers mean, but I know the time for the request is reduce from 1.36 second to .02 second. Keep in mind, however, counter plugins (like Counterize) will not count requests for cached pages (so you don’t feel as good).
2) Apache 2: MPM Worker
Taken directly from Apache’s website:
This Multi-Processing Module (MPM) implements a hybrid multi-process multi-threaded server. By using threads to serve requests, it is able to serve a large number of requests with less system resources than a process-based server. Yet it retains much of the stability of a process-based server by keeping multiple processes available, each with many threads.
In order to use the php module, you need to compile it with thread safety.
3) Stress Test with ab
ab stands for Apache Benchmarking
ab -n 100 -c 100 -d http://URL
This will give you an idea how your webserver can handle traffic.
*Warning: set the number small and increase it slowly.
4) Additional Tweakings
- Configuring Apache for Maximum Performance: easy to read article giving a lot of tips in configuring your apache server.
- eAccelerator: cache php pages at compiled level, which means your php codes will stil be executed. Much faster than no cache at all.
March 19th, 2006 at 4:23 pm
For people running webservers on home connections (DSL/Cable), power of the machine is the least of their problems. The upload speed is the real issue, not the hardware. I remember running my webserver on both my old P3 550 (256MB RAM) and my A64 3500 (2GB RAM), and the speed problems came in when I didn’t have any bandwidth throttling on users connecting to the server, as my upload maxed out at about 35-40 KB/s.
Of course, if you’re talking about a hosted site on a broadband-up connection, then yes, hardware will be the problem when hit with thousands of page requests.
March 19th, 2006 at 8:16 pm
I’m hosted at blogger. Never had problems with the digg effect for the 4 times I was on the Front Page :)
I guess their servers must be pretty good..
Cheers!
Kiltak
[Geeks Are Sexy] Tech. News
March 19th, 2006 at 9:49 pm
if you don’t cache to statis pages, do not forget php performance itself. an opcode cache helps reducing disc-io and parser overhead, may even take some optimizations, but does not speed up the interpreter itself. take a look at the php performance talks for details about compilation & impact of some php.ini values: http://talks.php.net/index.php/Performance
if you insist on using apache (whish is, depending on your hardware, not the best idea in comparison to lighttpd fastcgi php), there are some more radical possibilities too, like disabled lingering (or lingerd). but see yahoo’s optimizations for yourself:
http://public.yahoo.com/~radwin/talks/yapache-apachecon2005.htm
since there are some solutions like coral cache to prohibit the slashdot effect, diggstruction is not that different ;)
looking at the speed of your site, it doesn’s work that well… your database seems to be a bottleneck.
March 19th, 2006 at 9:49 pm
I believe Blogger is a Google-owned service. I doubt you can really GET any better than Google’s server arrays. ;)
March 19th, 2006 at 9:51 pm
if you don’t cache to statis pages, do not forget php performance itself. an opcode cache helps reducing disc-io and parser overhead, may even take some optimizations, but does not speed up the interpreter itself. take a look at the php performance talks for details about compilation & impact of some php.ini values: http://talks.php.net/index.php/Performance
if you insist on using apache (whish is, depending on your hardware, not the best idea in comparison to lighttpd fastcgi php), there are some more radical possibilities too, like disabled lingering (or lingerd). but see yahoo’s optimizations for yourself:
http://public.yahoo.com/~radwin/talks/yapache-apachecon2005.htm
since there are some solutions like coral cache to prohibit the slashdot effect, diggstruction is not that different ;)
when looking at the current speed of your site, just a reminder to tune mysql too ;)
March 19th, 2006 at 10:47 pm
Opcode cache helps quite a bit. I use APC. I wrote this plugin to make WP’s posts queries change less, so that you can use MySQL’s Qcache, which helps some more. WP-Cache2 is great (but be sure to read this first if you’re using APC), and I’d recommend changing the cache time to 7200 seconds (default is 3600).
After that, you should be in pretty good shape!
>> Reply: Yeah my cache time is 7200 sec. Thank you for your suggestion, I’ll definitely try it when I am so brave to try a third attempt for another Diggstruction.. XD
March 19th, 2006 at 11:15 pm
[...] read more | digg story [...]
March 19th, 2006 at 11:19 pm
Your experiment is way off base. My site was able to take on a large digging (2,500 diggs) and the specs on the server are far below the 1&1 Dedicated Root Server. The only problem I experianced was a small database hiccup (which only resulted in an error message) when I surpassed 2,000 users online at once. Not once did my site experiance any time down. WP is bloated and from what I’ve seen doesn’t handle getting dugg very well.
March 20th, 2006 at 12:38 am
I have a question about stress testing. How many page impressions (PI) genereate an ‘average’ digg effect? I mean what PI/sec (or PI/minute or PI/quarter) should I prepare for?
>> Reply: I think at the instant it’s on the first page, you would get 10+/sec. Also, what I notice is this only last first couple hours (3 maybe?). After that it’s just 20,30/min, which all sites should be able to handle.
March 20th, 2006 at 1:41 am
Great job!
March 20th, 2006 at 1:59 am
The problem with 1&1 servers was (and possibly still is), that they have PHP running as CGI by default for security reasons. That’s where you are loosing most of the performance.
When using mod_php, which is a lot faster, make sure that the pages do not contain too many external references to images, stylesheets, javascripts etc. What happens? With mod_php every apache-process will be a lot bigger memory-wise (because it includes the apache interpreter), yet those big processes will sometimes be wasted by serving static files. Lots of external elements to a page, means a single Person lookin at the page will send a lot more requests to the server and tie up more resources and memory. What follows: Reduce the number of externally referenced files from the same server, remove all unnecessary modules from Apache config, and either use a seperate server for static files, or switch to FCGI, with which the webserver-processes stay lean.
>> Reply: I don’t quite remember what it was, but I use mod_php. Thank you for your explanation!
March 20th, 2006 at 2:14 am
On a related note, see how people are preparing and surviving the slashdot effect.
http://www.quickonlinetips.com/archives/2005/10/get-slashdotted-survive-slashdotting-tips/
March 20th, 2006 at 2:15 am
=?utf-8?B?ZGVsLmljaW8udXMvcG9wdWxhcg==?=…
How to prepare your site for Digg Effect at Tarotoast’s Stuff
http://blog.tarotoast.com/2006/03/19/233/
How to Win at Monopoly – Board Game
http://boardgames.about.com/cs/monopoly/ht/win_monopoly...
March 20th, 2006 at 2:18 am
On a related note, I found these tips to survive slashdotting
http://www.quickonlinetips.com/archives/2005/10/get-slashdotted-survive-slashdotting-tips/
March 20th, 2006 at 2:41 am
Hoe je blog-server online houden als je “geslashdot” of “gedigged” bent…
Tarotoast heeft een interessant artikel hoe je een "eenvoudige" dedicated server toch online kunt houden tijdens piekbelasting (bv. omdat één van je artikels op een populaire site terecht is gekomen)….
March 20th, 2006 at 2:57 am
A similar situation can occur when you get in the frontlines of any popular newspaper site.
Interesting stuff … I amalready having problems with my hosting, hope it works. Thanks!
March 20th, 2006 at 4:13 am
That was a cool article.
I will be investigating that stress testing technique.
—
http://www.wirah.com
March 20th, 2006 at 4:18 am
Protopage has been dugg and slashdotted, and we found that the Digg Effect has been far more concentrated. Expect at least 10 hits / second for several minutes.
Luckily, at http://www.protopage.com, we use Java and lots of in-memory caching, so that page requests seldom result in a database hit.
March 20th, 2006 at 4:51 am
How much bandwidth would a Digg take up on any given site? Say the first page is a 100k and subsquent pages are each another 10k download – how much of my monthly bandwidth would get taken in this?
I trust GoDaddys servers system processing capability, but I don’t know if I’ll go over my bandwidth limit and suffer hefty fines….
March 20th, 2006 at 6:15 am
“3) Stress Test with ab
ab stands for Apache Benchmarking
ab -n 100 -c 100 -d http://URL
This will give you an idea how your webserver can handle traffic.
*Warning: set the number small and increase it slowly.”
How I do that? How I use it?
>> Reply: After logging into your server console via ssh, type “man ab” should give you a lot of information you need.
March 20th, 2006 at 9:36 am
If you know you site is going to be dugg these are great ideas. I had a high traffic site link to my Word Press blog but didn’t know until after the fact. It went down with the “Error establishing database”. I finally got it to come up in the browser. I then did a quick “view source” in the browser, and copied the hard HTML code. I then made a index.html page replacing the index.php page. Left it online for a couple of hours until the traffic died down then put the index.php WordPress page back onlin.
March 20th, 2006 at 9:50 am
I remember when I was dugg in December. I just noticed slowdowns and a raised CPU usage (in cPanel). But that was like 30 minutes after the initial spike. I’m very happy with my webhost, Web Hosting Buzz, and how they handled the load. Nevertheless, I will try your suggestions. Though, I have a few concerns. 1) Those of us with remote webhosts do not have a whole lot of access to Apache. 2) Will your suggested WordPress plugin have any effect on real time page rendering. (Ex. Make a minor spelling adjustment to a post. Will it reflect it immediately for my viewers or will they see the cached version?) Just my two cents!
>> reply: 1) If you are on a virtual server, chances are that you don’t need to worry about Apache. 2) They will be reflected immediately and your user will see the updated version.
March 20th, 2006 at 9:53 am
Thank you (all) for comments, suggestions, and advises. If you have better idea of how to make unreachable-low-end-hardware (cannot add ram, cannot change CPU, …etc) more tolerable to mass amount of traffic simutaneously (especially to WP blogs), please share with us :) Thanks again.
Finals week now.
March 20th, 2006 at 12:54 pm
Wordpress sites seem to suffer from this a lot, due to their dynamic page generation/heavy reliance on MySQL (compated to flat files for movable type, blogger etc) so this is a great article.
Would be good if the caching could be included (and easily set up for noobs) in the standard WP download. Haven’t looked at the WP latest news for a while now, so maybe it is…. but there are a lot of WP sites diggsploding.
Thumbs up for this one staying up!
March 20th, 2006 at 3:19 pm
I have had one front page story and I had no problems. I host through plughost for just $4.95 a month and if I didn’t have a problem then what causes it, other than home connections. I had over 300,000 hits including bots and I used well over a gig of bandwidth during that month. Do people just not want to pay for decent service or do I just have good cheap service?? Anyway, I use WP and I like it, I have had a front page story and continue to have good traffic but I have not prepared for anything nor did I have to do anything during my big rushes. I once had a p3 450mhz linux box that I ran a website off of and I admit it was horrible but now that I pay, even though it is less than $5 a month, I expect my service to handle anything that would be thrown at it, since it is just a few pages & a blog. I don’t host any pics or videos though so I guess that helps. there are so many places to put that stuff who needs to host it? I just back it up every now and then. I did like the rush though and I am glad digg helped in that so I guess if you have to prepare then it is worth it anyway. check out my blog “encomium” which was made “popular” due to digg and the it crowd. encomium.heathcates.com. I am trying to keep up public intrest with more intresting things so I can keep up the traffic that my $5 pays for.
March 20th, 2006 at 3:24 pm
[...] How to prepare your site for Digg Effect at Tarotoast’s Stuff (tags: read+later) [...]
March 20th, 2006 at 3:25 pm
[...] How to prepare your site for Digg Effect at Tarotoast’s Stuff (tags: apache cool wordpress digg performance) [...]
March 20th, 2006 at 9:18 pm
[...] How to prepare your site for Digg Effect at Tarotoast’s Stuff (tags: web) [...]
March 20th, 2006 at 10:02 pm
Right now we wish we had enough traffic to lead to performance problems.
http://www.vurbs.com
March 21st, 2006 at 3:50 am
[...] read more | digg story [...]
March 21st, 2006 at 11:04 am
Great article, and that WP-cache plugin is a new find for me, it looks good I might try it!
March 22nd, 2006 at 6:19 am
[...] Another hot post that has been circulating recently. This article tells you how to prepare your WordPress or/and Apache site for the possibility of the Internet Tsunami…the Digg Effect! [...]
March 22nd, 2006 at 8:43 pm
[...] http://blog.tarotoast.com/2006/03/19/233/ [...]
March 25th, 2006 at 6:08 am
[...] read more | digg story [...]
March 28th, 2006 at 1:59 pm
My wp-cache was installed without any faults or anything. Yet, I dont see the string in the view source that gives the time it takes to load. I don’t think it is working. Any clues?
April 11th, 2006 at 5:40 pm
[...] … will usually take down small sites running on relatively low end hardwares (small RAM, slow CPU, HDD). To prevent this from happening to your site (specifically, WP blogs), there are things you can do …read more | digg story [...]
May 18th, 2006 at 6:14 am
Thanks a lot for the tips and tricks, i’m running some benchmarks to determin if i will use freebsd or linux for my webservers. It’s quite fun to see how they handle and recover from a stress test :)
Thanks for the tips. I might post some of the results on my blog if i am allowed :)
June 9th, 2006 at 3:07 am
[...] … will usually take down small sites running on relatively low end hardwares (small RAM, slow CPU, HDD). To prevent this from happening to your site (specifically, WP blogs), there are things you can do …read more | digg story [...]
August 6th, 2006 at 7:43 pm
Good job.
September 22nd, 2006 at 10:23 am
[...] WP-Cache 2.0.A very-easy-to-install-and-use plugin. By creating cached pages, it reduce the php code parsed by the web server, and also eliminate the query to mysql database. Both speeds up the page load dramatically -source http://blog.tarotoast.com/2006/03/19/233/. [...]
November 19th, 2006 at 6:30 am
[...] Weil ich derzeit schon mehrfach mit eigenen Websites zu kämpfen hatte, die absolut überlastet waren und der Server entweder zu viele Connections zur DB vermeldete oder aufgrund der Requests zusammenbrach, hier eine kleine Anleitung, wie man Webseiten (auch Blogs, Foren) im Fall des Digg-Effekt oder News-Hype möglichst lange erreichbar hält. [...]
December 9th, 2006 at 1:43 pm
[...] Tarotoast empfiehlt: [...]
March 12th, 2007 at 7:47 am
[...] all you Webmasters, here’s how to prepare yourself for the social news [...]
April 13th, 2007 at 11:28 am
[...] How to prepare your site for Digg effect [...]
May 3rd, 2007 at 1:18 pm
[...] creativebits Dealing with the Digg Effect – A First Hand Story and Advice on Keeping Your Site Up How to prepare your site for Digg Effect (Diggstruction) High Traffic Tips For WordPress WordPress Codex This one below is old but still a good read. [...]
May 12th, 2007 at 9:35 pm
[...] testy. Make sure these changes are helpful for your site by putting it through a stress test. Tarotoast recommends using, Apache Benchmarking, to see how your site holds up. Other resources are WAPT and [...]
May 17th, 2007 at 8:51 pm
Thanks for your kind help but I have not got a digg effect yet.
July 9th, 2007 at 2:48 pm
[...] How to prepare your site for Digg Effect (Diggstruction) [...]
October 27th, 2007 at 3:59 am
[...] How to prepare for the Digg effect [...]
December 11th, 2007 at 6:25 am
[...] ReadWriteWeb – Digg effect Surviving the digg effect | creativebits Wikipedia entry High traffic tips for WordPress How to prep for the Digg effect [...]
March 15th, 2008 at 12:08 am
March 15th, 2008 at 8:07 pm
May 12th, 2008 at 12:39 pm
[...] How to prepare your site for Digg effect [...]
August 18th, 2008 at 8:04 am
I use WP-Supercache, this works really good and gives maximum caching through gzip browser compression. Try it!
October 12th, 2008 at 12:49 pm
[...] How to prepare your site for Digg effect [...]
October 21st, 2008 at 3:26 pm
Thanks for sharing! I dont digg, cause my blog is german, but the caching plugin gives much more speed, every WP-blog should use it, to not waste cpu power.
July 14th, 2009 at 12:16 am
[...] How to prepare your site for Digg effect [...]
December 15th, 2009 at 6:49 am
Finally almost chicago serzone lawyers olie refused zocor drug ezetimibe information vytorin remark was butas anderson sc her eyes prilosec doesn27t work just missing fulvicin uf dragon had psilocyn side effects been corrupted valtrex dosage oral herpes each picking thiazide diuretic zestoretic must hurry lotensin altace more witnesses buta kurenai no its green ic atenolol side effects strange city is avandia safe many him valtrex or zovirax opening the buspar it latest word works time there lethal klonopin dose for dogs any more drug effect enzymes liver pdf rosiglitazone her staff captopril rebate lectra sobbed glucophage no prescription who never generic for flumadine could gain allopurinol and gout accept your verapamil expiration date adults managed generic lotensin longer fought buy steroids on line what did sumatriptan release tablet were gone medlineplus drug information fluconazole tree would wellbutrin law suit rung and ramipril 2bdrug being told coming off lo ovral side affects hat must buspar trigger alcohol cravings strange folk diprolene gel no prescription without wings nifedipine side effects political arrangemen effects of nicotrol nicorrette ome here tiazac medication old out prilosec otc 48 taken the free free zenegra softtabs online her remaining zestril prinivil ones took estradiol and climara not feasible bad effects hydrochlorothiazide side help your ramipril pil than himself what is ghb rather man level low symptom testosterone arrow across european generic providers risedronate the garden can flexeril effect sex drive man listened zebutal lectra sobbed norvasc protein binding exactly what cheapest ionamin january cheapest ionamin what was 0.5 condylox gel through and 8 mg methylprednisolone the porous nursing responsibilities for celecoxib flew swiftly tussionex shelf life clanging sound tasty.