No search engine traffic wanted

[Justin]

Ive been lookin around this morning for this, but cant seem to find what I need. How do I put the robots.txt file in so that the spiders dont go through my site? Im workin on a new site, and dont want any spiders goin throught it right now.

[Nick]

User-agent: *
Disallow: /

Put the above in a file named robots.txt in the root directory of your site.

[Webgator]

If it's a brand new site, and you havn't submitted it to the SE's and have no inbound links then you will have nothing to worry about.

[mikebrad]

If it's a brand new site, and you havn't submitted it to the SE's and have no inbound links then you will have nothing to worry about.

That depends. I had a site get indexed before I did anything with it. Now, that doesn't mean that someone else (like my hosting company) didn't submit it automatically or something. I added the above robots.txt file and it did not get indexed any further.

Point is: Add the robots.txt file

[Justin]

Originally Posted by Webgator

If it's a brand new site, and you havn't submitted it to the SE's and have no inbound links then you will have nothing to worry about.

Yeh don't do this. I have a new site that got indexed with no back links. Apaprently it has happened to a lot of webmasters recently, I believe it might have somethin to do with the Google toolbar. Theres a thread on Digital Point I think that talks about it.

[mikebrad]

interesting, I didn't think about that.

[Webgator]

Well I have pages on domains that are not in the SE's, I have the Google toolbar running in Firefox and can honestly say that I have never experienced domains or sites showing up in the SE's index unless I submitted them via link exchanges, blogs, etc.

But yes, putting in the robots.txt should block them...if they listen. Not all spiders follow this protocal and will still follow any and all links it can find on your domain. The same applies to the noindex,nofollow tags.