Advanced

Is the MSN search engine being used in the battle to kill Linux?

Posted by tadelste on · 7 comments
LXer; By Tom Adelstein
Mail this story
Print this story

Author's experience with MSN bot and google research says MSN bot is bombarding open source sites. I kept wondering why my best read articles sunk lower on google and now I believe I know.

I began writing articles about Linux in April 1999. My first assignment involved a Howto Consulting Series for "Linux Today". My editor at the time, Dave Whitinger, felt that we needed some basic Big 5 consulting skills in an industry with approximately 2 million users.



Over the years, I had some top ten hits. Two articles that stand out the most include "How to Build a Low Cost Linux Personal Computer" and "How to Build a 1u Linux Rack Server". The former found itself on every major search engine directory globally. In 2000, we had several hundred thousand page views and it never had a mention on Slashdot. The second came at a time when no one could find the components to build their own 1u computers. It took me three weeks of research to find the case, low profile memory, cdrom and floppy. At the time, I had reseller agreements with Ingram Micro and Tech Data but still could not find the pieces. That article also got hundreds of thousands of page views in 2000. At some point, both articles disappeared from my web site as have other articles.



The past two years, I have devoted much of my writing efforts to influencing Linux use in Government. I have spoken to many groups, dug for obscure case studies. I have also championed the desktop.



In late 2002, I began working with a state Senator in Texas and helped sponsor SB 1579 so Texas government agencies could use open source software. That's when I started getting a letter a day from the BSA and/or Microsoft offering me amnesty if I would let them into my house to audit my computers. Lucky for me, I use Linux. Of course, I didn't take Microsoft up on their offer.



Over the last few weeks, I have noticed that a search on my name brings up articles written in 1999 and an obscure set of articles related to a music career that ended a couple of decades ago. I even have a spot on MSN entertainment for an album that hasn't been available in years. I really like the Japanese tech articles from 1999.



So what's going on? MSN Bot has become the preferred browser on some of my web sites creating links and generating hits. On sites I could control, I excluded it from my blog, for example, with a robots.txt file and entries. But as Tom Cruise said in the movie "A Few Good Men": The hits just keep coming.



A search on Google with the terms "robots.txt stop msnbot" will yield results asking mostly how to keep the MSN Bot from using up bandwidth and repeatedly hitting sites. As Fox News says "You Decide".

Last week, O'Reilly released a new book co-authored by me about "Exploring the JDS Linux Desktop". The real danger to Microsoft may be the fact that the book comes with a Morphix CD version of JDS. Of course, any good Linux user could build their own Morphix CD image and see how JDS really kicks. If you aren't familiar with Morphix, it's Debian based and free software. Once installed it's basically a Debian distro.



I think that if word got out, we'd have a memogate of our own and discover that Sun really isn't trying to kill Linux as George Colony claimed in an Opt-Ed piece in ZDNet UK. We'd also see how committed to Linux Sun remains and how little their ties to Microsoft remain. Also, we'd see why Sun's Linux continues to kick Microsoft in China.



If anyone does a search on my name, and winds up with an obscure Japanese Link from 1999, I can't do much about that. But, we did discover how to use the MSN bot against them --quite by accident. But these are desperate times and call for desperate measure such as luck.



One of the things I have discovered about Microsoft is how innovative they can be. In my research and activities, I have uncovered some remarkable things about a lot of companies and people. Would you find it surprising if some open source advocates were on the payrolls of commercial companies? What about discovering that all those articles about MSN search technology competing with Google simply gave MSN a search bot for manipulating the knowledge base of the Internet? The question is strictly one of conjecture; just a question for the sake of argument and to stimulate discussion about things people can do with technology.



I mean if this were politics instead of religion, you wouldn't be surprised to find that some people still believe that some questions remain about the forged documents on CBS. In case you're one of those people - CBS admitted they were forged. Of course this is about perception and no one really believes that MSN Bot would be used to manipulate what people read on web searches. Would they?







  Nav
» Read more about: Story Type: LXer Features; Groups:

» Get new stories by e-mail: nightly or as they are published

« Return to the newswire homepage

  Related stories
» More like this

Comments (7)

I incline to give good credibility scores to your speculation.

However, there is another possibility: perhaps MSN Bot is just plain stupid, and needs to be a lot more pedantic (i.e. to generate a lot more hits) than Google Bot in order to achieve the same results. Keep in mind that Google still is the best search engine on the Internet while Microsoft is still playing catch-up, so don't be surprised to see Microsoft emphasizing volume to achieve what they can't achieve through smartness. The old "i don't have to be smarter, i just need bigger muscles" thing.

I am involved with open source but my website ( http://florin.myip.org/ ) doesn't have a lot of material that would be interesting to the MSN Bot from the perspective of your assumptions. Yet i do see MSN Bot railgunning my site all the time, while Google Bot barely touches it every now and then. My first thought was: "morons! :-) this bot is really stupid if it has to thrash around that much."

Well, i dunno...
I can see your point of view about the MSNbot being just stupid. But just to check it out, take one of my dumb and old links and run a search on it. Then ask yourself, how did this page (which in my logs barely gets a hit) wind up with so many links?

I found 4000 + links on my blog embedded in comment sections. How did all those links get there? I changed the default to no comments, cleared out hundreds and hundreds more appeared. Something is very fishy - I'm not saying it's a conspiracy, but it eats like a horse, smells like a horse, rides like a horse -- so what is it? An elephant?

I agree that it's speculation and you certainly have a point about it being a possibility - maybe a probablity that MSNBot is just stupid. If so, they should pay us back for all the bandwidth they're using. This message was edited Sep 29, 2004 2:03 PM
Isn't it amazing that in 2 hours, the results on google changed. The MSN entertainment page is back several to page 3 as is the 1999 Japanese article. They've stripped the MSN entertainment pictures and data, but when I looked in Googles cache, it was just like it was this AM.

Amazing. Do you wish you had $60 billion? But you know, with all their money, they still haven't stopped Linux. Bucky Fuller said, "There's nothing as powerful as an idea whose time has come!"

This message was edited Sep 29, 2004 3:56 PM This message was edited Sep 29, 2004 3:58 PM
"I found 4000 + links on my blog embedded in comment sections. "

How exactly is the MSN bot creating these links in the comments section? Do you mean that MSN bot hit your site 4,000+ times? This message was edited Sep 29, 2004 4:54 PM
If you have a blog and you permit comments, spammers can send out a bot to automatically write a comment and add a link. If a search engine chooses to use those spambot links in their lexicon, it makes the site more popular. The Seachbot - in this case MSNbot can make a site rise in its ratings by checking those links and counting them as valid permalinks. It's one of many ways to harvest linked pages.

Since the search engine does not (necessarily) count hits, it can only know if a page or site is popular by the quantity of links.

Earlier today, I erased over 50 spammer links on one of the blogs I manage which has comments turned off. On the blog with 4000+ spammer links, I had to erase 18 months of content because I couldn't delete just the comments.

This message was edited Sep 29, 2004 8:13 PM This message was edited Sep 29, 2004 8:14 PM
tadelste, you could perhaps have used a binary search-and-replace to corrupt the spammer links, if they were all pretty much to the same place or otherwise easily identifiable.

If your blog lives in a database you can probably do something like UPDATE commenttable SET commenttext=REPLACE(commenttext,'oldurl','yoururl') WHERE commentext LIKE '%oldurl%';
AC:

I really appreciate your message. I tried to remove the comments programmatically. The text existed in the blog entry string and varied so widely that I gave up. The bots also used varying ip addresses - you know the drill: Spam hell.

Post a comment

A free account is needed to post. Write your comment here first: your text is kept in this browser while you sign up or log in, and is waiting for you when you come back.

Your message:
|||||
10
Markdown: **bold**, *italic*, [text](https://link), - lists, > quotes, $x^2$ for math. Formatting help

Sign up and post (it is free)  or  Log in and post