# Disallow all crawlers access to certain pages. User-agent: * Disallow: /companies/ Disallow: /sotest/ Disallow: /css/ Disallow: /js/ Disallow: /img/ Disallow: /lib/ Disallow: /web_imgs/