that's it for me
|
| Author | Content |
|---|---|
| incinerator Jul 11, 2006 6:23 AM EDT |
I've read this one and the articles svjn links to. Imho the test results look way to unreal to be true, anyways. I think lxer should stop linking to ziff davis stuff generally, and svjn stuff particularly. They're just not credible anymore. |
| sbergman27 Jul 11, 2006 6:41 AM EDT |
I disagree. I cannot confirm or disprove his results. But he says he did do his own tests to check eweek's results and I believe him. His conclusions about just where the problem lies may or may not be on track. But it looks like honest constructive criticism to me. Sticking one's head in the sand is rarely a productive action. In the end, the Mindcraft "study" was a big win for the Linux kernel. It may have been maliciously intended. But it showed up some real shortcomings in the 2.2 kernel. Steven's results ring true to me because FOSS is not always so hot when it comes to setting defaults. (Just look at PostgreSQL's defaults.) Microsoft really is better at that. With Linux, it is simply assumed that depployment will involve an experienced administrator. And with the rise of Linux's popularity, that's just not always the case anymore. SJVN rights some good, honest, informed stuff. And some of it is constructive criticism. You can agree with him... or not. But dropping his articles? Get real! Edit: I'm a PostgreSQL fan, btw. Pgsql is not alone in having questionable defaults. Plus, sometimes there are factors other than raw performance to be considered. |
| tuxchick2 Jul 11, 2006 7:32 AM EDT |
I don't find anything to dispute in the article. Probably the most important point to take away is that smart, skilled admins are more important than super-duper perfect telepathic software. |
| incinerator Jul 11, 2006 7:54 AM EDT |
Well, I don't know if you have actually read the eweek l;abs test report svjn links to at the beginning, but I have. And it is dodgy right at from start: They claim they have created a test environment with 1000 simulated clients. According to their slideshows, these 1000 clients were capable of only 1 transaction per second, whereas the win stacks were somewhat close to 17 transactions per second. That would mean that each virtual client would have to wait ONE THOUSAND SECONDS to get his transaction completed. That either means they set up a test for a slashdotting-like environment or they did know what they were doing at all, or they simply invented the figures. Either way, the result is absolutely disastrous for all stacks. Similiar things apply to the second slide "average throughput per second". Remember, the tests were performed on a Gigabit ethernet network. O would seriously doubt the result that one of the stacks is capable of putting out several megabytes per second whereas the worst only trickles out a few hundred kbs per second. Again, this is unreal. I could go on criticising each slide they put up at http://www.eweek.com/slideshow/0,1206,pg=0&s=26708&a=182823,... along with the article at http://www.eweek.com/article2/0,1895,1983364,00.asp but I guess you got the message by now. The benchmarks eweek performed are rubbish, and to jump to any conclusions from these rubbish benchmarks is more than silly. If I were svjn, I would have arranged for these rubbish benchmarks to be deleted from the eweek page at once, and done them properly again, with somewhat realistic results. If he's the expert he claims to be he would have checked the figures before writing about them. |
| devnet Jul 11, 2006 8:29 AM EDT |
Agreed...having just read through the slides and article, I have to second this notion incinerator... These tests go against every single benchmark I've ever conducted as an IT Manager. Tell me, when was the last time your home LAMP stack on a 100MB LAN had a throughput of less than 500kb? http://www.eweek.com/slideshow_viewer/0,1205,l=&s=26708&a=18... Incinerator is right, these be fishy results. |
| sbergman27 Jul 11, 2006 8:49 AM EDT |
It's unclear to me what the virtual clients are doing. Are they pummelling the server full out? Or are they trying to simulate a real user who loads a page and reads it, loads a page and reads it. |
| devnet Jul 11, 2006 9:08 AM EDT |
sbergman27,
I think its up to silkperformer, which is what they're using as their benchmarking utility. I'm pretty sure it can be configured to 'hit' a page or 'pummel' it. The question remains, what setting did they use? |
| sbergman27 Jul 11, 2006 9:15 AM EDT |
The question remains, how was the test set up? Without detailed specs, it's not a benchmark. It's a press release. |
| dinotrac Jul 11, 2006 9:48 AM EDT |
Looking at the line graphs (as opposed to the bar charts) I would guess their methodology is probably ok, key words being guess and probably. Without a definitive description of the methodology, it is hard to be sure. It looks like they defined a pile of simulated users sitting at computers and executing a series of scripted actions. If done correctly, there are reasonable and randomized waits between actions that reflect the way a real user would interact with the system. Presuming that they ran the same load against each server, the charts probably provide a decent relative measure of the servers they tested -- against the test load. The real problem is that, no matter how well they did their job, we can't rely on it for anything else because the details matter in something like this. What was the use cycle for an individual user? Were all the users set on at once, or was it ramped up? What randomizing distribution was used? Etc, etc, etc. |
Posting in this forum is limited to members of the group: [ForumMods, SITEADMINS, MEMBERS.]
Becoming a member of LXer is easy and free. Join Us!