If you want to exclude all the search engine spiders from your entire domain, you would write just the following into the robots.txt file: User-agent: * Disallow: / If you want to exclude all the spiders from a certain directory within your site, you would write the following: User-agent: * Disallow: /aboutme/ If you want to do this for multiple directories, you add on more Disallow lines: User-agent: * Disallow: /aboutme/ Disallow: /stats/ If you want to exclude certain files, then type in the rest of the path to the files you want to exclude: User-agent: * Disallow: /aboutme/album.html Disallow: /stats/refer.htm