# Jim was here # $Id: robots.txt,v 1.18 2003/08/20 15:42:41 krusch Exp $ # # This is a file retrieved by webwalkers a.k.a. spiders that # conform to a defacto standard. # See # # Comments to the webmaster should be posted at # # Format is: # User-agent: # Disallow: | # Flag Date By Reason # $l1- 19950130 epc finally understood what the file was for! # $L2= 19960909 epc fixed url since mak moved to Webcrawler... # $L3= 19970811 epc drop /Stretch # $L4= 19991102 krusch fixed User-agent capitalization and contact info # $L5= 20010327 krusch Updated disallow rules # ----------------------------------------------------------------------------- User-agent: Twitterbot Disallow: * Allow: /common/images/socialmedia/* User-agent: * Allow: /common/images/socialmedia/* Disallow: /common/ Disallow: /Common/ Disallow: /ebrokers/ Disallow: /Ebrokers/ Disallow: /ebrokers/advisorpage.aspx Disallow: /Ebrokers/AdvisorPage.aspx Disallow: /Common/ContentFileLoader.aspx?ContentGUID=8421ff42-6146-42d8-959a-088eb0299141 Disallow: /Common/ContentFileLoader.aspx?ContentGUID=7518bf29-ebf3-4414-82b5-0072434c5a5f