# ROBOTS.TXT # DO NOT INDEX THIS DIRECTORY User-agent: * Disallow: /admin/ Disallow: /library/services/exams/files/ Disallow: /library/marc/ Disallow: /moakley/ Disallow: /cjrg/ Disallow: /CFIDE/ Disallow: /moakley/ Disallow: /cjrg/ Disallow: /CFIDE_0/ Disallow: /admin/ Disallow: /cfdocs/ Disallow: /admin_bak_52704/ Disallow: /cd/ Disallow: /images/ Disallow: /lib/ Disallow: /stulogin/ Disallow: /styles/ Disallow: /Templates/ Disallow: /TMP984219vij/ Disallow: /WebAnalysis/ Disallow: /Working_docs/ Disallow: /media/ Disallow: /includes/ Disallow: /examples/ Disallow: /database/ Disallow: /union/ Disallow: /_notes/ Disallow: /_mm/ Disallow: /dev/ Disallow: /PortalFiles/ Disallow: /highlights/stuorgs/jhtl/docs/pdf/etc/files/ User-agent: Googlebot User-agent: InfoNaviRobot User-agent: TV33_Mercator User-agent: AVSearch User-agent: Mercator User-agent: Scooter User-agent: Slurp User-agent: SearchengineLicenceSheep User-agent: shadow User-agent: MultiText User-agent: FAST-WebCrawler User-agent: Lycos_Spider User-agent: Atomz User-agent: htdig User-agent: spider00.logika.net User-agent: NetMechanic User-agent: libwww-perl User-agent: Teleport Pro # 2004-08-02 (disallow folders) # Updated 2004-11-05 (disallow ) # updated 2005-02-02 (disallow /dev) # updated 2005-10-07 (allow union for Google Scholar to crawl)