User-agent: * Disallow: /waterpic/ Disallow: /airpic/ Disallow: /CAcountymaps/ Disallow: /Connections/ Disallow: /Docs/ Disallow: /Connections/ Disallow: /CountryMaps/ Disallow: /images/ Disallow: /shared/ Disallow: /WEB-INF/ Disallow: /Ref_Aquire.jsp Disallow: /List_AquireAll.jsp Disallow: /Detail_Poisoning.jsp Disallow: /List_NTPStudies.jsp Disallow: /Ref_Reg.jsp Disallow: /Detail_ChemUse.jsp Disallow: /Detail_ChemReg.jsp # # Too many ways to search the products # Disallow: /List_Products.jsp?PestName Disallow: /List_Products.jsp?SiteName Disallow: /List_Products.jsp?ProdUseList # # This still allows /ListProducts.jsp?RecId # # /DetailChemical.jsp # /DetailProduct.jsp # seem to be the most critical pages to allow User-agent: TurnitinBot Disallow: / #Will disallow all urls on your site