User-agent: * Disallow: /images/ # To remove directories or individual pages of your website, you can place a robots.txt file at the root of your server. For # information on how to create a robots.txt file, see the The Robot Exclusion Standard. When creating your robots.txt file, please # keep the following in mind: When deciding which pages to crawl on a particular host, Googlebot will obey the first record # in the robots.txt file with a User-agent starting with "Googlebot." If no such entry exists, it will obey the first entry # with a User-agent of "*". Additionally, Google has introduced increased flexibility to the robots.txt file standard through # the use asterisks. Disallow patterns may include "*" to match any sequence of characters, and patterns may end in "$" to indicate # the end of a name. # To remove all pages under a particular directory (for example, lemurs), you'd use the following robots.txt entry: # User-agent: Googlebot # Disallow: /lemurs # To remove all files of a specific file type (for example, .gif), you'd use the following robots.txt entry: # User-agent: Googlebot # Disallow: /*.gif$ # To remove dynamically generated pages, you'd use this robots.txt entry: User-agent: Googlebot Disallow: /*? # Tell Googlenot to use any .do pages anymore User-agent: Googlebot Disallow: /*.do