Homepage has PR, rest of site none

[ThePais]

Originally Posted by PPP

Not having one leads to dozens or hundreds of 404 errors a day. Search bots always look for it first, so you want to invite them in. There is no reason why you should ever not have one. I'd say this is especially true if you just had one that said something unhelpful. You want to correct that as soon as possible, when the bots come by next.

5 websites, no robots.txt on any of them, and a total of 2 404 errors. The only time you're going to see a 404 is if you've got a broken link or an out-dated sitemap.

The presence of an empty "/robots.txt" file has no explicit associated semantics, it will be treated as if it was not present, i.e. all robots will consider themselves welcome.

i.e. not having one present == having one present with 'User-agent: *'

[PPP]

I don't know what a broken link or outdated sitemap have to do with it, but how can you have had only two bot visits to five websites, and how can that be a good thing? Do you not have a favicon.ico, and not get 404s for that too?

You should have a robots.txt for the obvious reason you want to deny as many scrapper scumbots as you can... saying all bots are welcome is a terrible idea. But aside from that there is no possible downside to having one, and putting one up takes 15 seconds or less.

[corpfan1]

LOL...

My websites have been up for about 5 years...all indexed well...all doing fine...never ever had a robots.txt.

In fact, they did well super quick, indexed in less than 3 days by Google every time and often on first page for my chosen keywords within about 8 days.

[PPP]

Why don't you have a robots.txt file? Why do you want to make it easier for people to steal your content?

[ThePais]

Originally Posted by PPP

I don't know what a broken link or outdated sitemap have to do with it, but how can you have had only two bot visits to five websites, and how can that be a good thing? Do you not have a favicon.ico, and not get 404s for that too?

You should have a robots.txt for the obvious reason you want to deny as many scrapper scumbots as you can... saying all bots are welcome is a terrible idea. But aside from that there is no possible downside to having one, and putting one up takes 15 seconds or less.

Ah, I've just misinterpreting your post of "Not having one leads to dozens or hundreds of 404 errors a day" - thinking you meant the crawlers, rather than our logs.

Anyway, the point was that a lack of robots.txt is identical to having one allowing all user-agents.

[eugene]

robots.txt doesn't influence on page rank. !

[BootyCraze]

Without a lot of the teckie stuff, sometimes the reason is lack of content, lack of unique content, more outbound links than inbound links (and that counts your internal links)

I should have sold my internal PR 5 poker page to PR enthusiasts, and laughed. It dropped to 2 then 0.

The bots? yeah, it must have some validity but its not really a factor short term. I would put it down to lack of unique content, more outbound than inbound links or its just not updated yet> either way look at your stats b4 the PR rating

[offyourface]

Originally Posted by PPP

Why don't you have a robots.txt file? Why do you want to make it easier for people to steal your content?

Truth be told if they are scrapper bots they are not going to honor your robots.txt file anyhow, htaccess would be the way to block scrapper bots and I would recommend a bad bot trap. But yea, it takes 5 seconds to get a robots.txt file up and the bots are expecting to find it when they visit, so no sense in throwing them through a loop.

[PPP]

It's true pure thieves won't obey robots.txt, but less sophisticated ones will run bots that by default do obey it. So even if you only get 1 out of 10 scrappers to not scrap, that more than makes up for the few seconds it takes to put up the file. Also, there are much more benign pots that uselessly eat up bandwidth and server load that you can do without. For instance some of the link exchange bots could take thousands of pages every single day, and if you have no interest in that particular link network there is no reason to just have them stumbling around your sites every day. Those bots generally do obey robots.txt.

[Trikkur]

So what is a quick robots.txt file I should put up. I didn't ever have one on my sites, but I figure if it's gonna be just a few seconds, I might as well.