Web Resources
URL: http://www.soclair.ch/resources/useragents/spiders.html
Last updated: July 1, 1998 by webmaster@soclair.ch

Copyright © 1997, 1998 by Soclair Electronic. All rights reserved.

 

Alexa

Spider Name (HTTP_USER_AGENT) Host(s)
ia_archive (NewsFetcher 1.0) (http://www.alexa.com/  crawler@alexa.com) barker.alexa.com
[206.14.154.193]
ia_archiver/1.6 firestone.alexa.com
[206.14.154.183]
green.alexa.com
[206.14.154.191]

AltaVista

Spider Name (HTTP_USER_AGENT) Host(s)
Scooter/1.0 ww2.altavista.digital.com
[204.123.2.67]
add-url.altavista.digital.com
[204.123.9.76]
Scooter/1.0 scooter@pa.dec.com scooter.pa-x.dec.com
[204.123.9.19]
scooter.pa-x.dec.com
[204.123.9.20]
Scooter/1.1 (custom) vscooter.av.pa-x.dec.com
[204.123.9.47]
Scooter/2.0 G.R.A.B. X2.0 204.74.103.39
[204.74.103.39]
scooter3.av.pa-x.dec.com
[204.123.9.105]

ANZWERS

Spider Name (HTTP_USER_AGENT) Hosts
AnzwersCrawl/2.0 (anzwerscrawl@anzwers.com.au; http://faq.anzwers.com.au/anzwerscrawl.html) crawler2.anzwers.ozemail.net
[203.108.10.42]
crawler3.anzwers.ozemail.net
[203.108.10.43]

crawler.de

Spider Name (HTTP_USER_AGENT) Hosts
libwww-perl/5.xx obiwan.schlund.de
[195.4.183.9]
yoda.schlund.de
[195.4.183.12]
jabba.schlund.de
[195.4.183.28]
luke.schlund.de
[195.4.183.41]
sarlacc.schlund.de
[195.4.183.42]
vader.schlund.de
[195.4.183.50]
account.schlund.de
[195.4.183.58]
chewie.schlund.de
[195.4.183.59]
tarkin.schlund.de
[195.4.183.75]
smtp.schlund.de
[195.4.183.109]
homepage.schlund.de
[195.4.183.110]

EuroSeek

Spider Name (HTTP_USER_AGENT) Host
Arachnoidea (arachnoidea@euroseek.com) infra.euroseek.net
[195.242.46.55]

Excite

Spider Name (HTTP_USER_AGENT) Hosts
ArchitextSpider poncho.atext.com
[198.3.97.17]
crimpshrine.atext.com
[204.62.245.32]
jazzhole.atext.com
[204.62.245.85]
crawl1.atext.com
[204.62.245.166]
crawl2.atext.com
[204.62.245.167]
crawl4.atext.com
[204.62.245.169]

Fireball

Spider Name (HTTP_USER_AGENT) Hosts
KIT_Fireball/2.0 heavymetal.fireball.de
[193.189.227.33]
heavymetal.fireball.de
[193.189.227.22]

HotBot

Spider Name (HTTP_USER_AGENT) Hosts
Slurp/2.0 (slurp@inktomi.com; http://www.inktomi.com/slurp.html) i1.inktomi.com
[209.1.12.34]
i12.inktomi.com
[209.1.12.45]
i18.inktomi.com
[209.1.12.51]
j11.inktomi.com
[209.1.12.108]
j12.inktomi.com
[209.1.12.109]
j16.inktomi.com
[209.1.12.113]
j2005.inktomi.com
[209.1.12.251]
j2008.inktomi.com
[209.1.12.254]

Infoseek

Spider Name (HTTP_USER_AGENT) Hosts
InfoSeek Sidewinder/0.9 galore-bbn.infoseek.com
[204.162.96.45]
wilbur-bbn.infoseek.com
[204.162.96.92]
templeton-bbn.infoseek.com
[204.162.96.93]
cca26226.infoseek.com
[204.162.98.38]
204.162.98.60
[204.162.98.60]
cca2625a.infoseek.com
[204.162.98.90]
Mozilla/3.01 (Win95; I) backdraft-bbn.infoseek.com
[204.162.96.42]
blue-bbn.infoseek.com
[204.162.96.73]

Kolibri Online

Spider Name (HTTP_USER_AGENT) Host
Mozilla/4.04 [de] (Win95; I ;Kolibri gncwebbot) batman.iosys.de
[194.121.108.66]

Lycos

Spider Name (HTTP_USER_AGENT) Host(s)
[Empty] spider4.srv.pgh.lycos.com
[207.77.91.184]
Lycos_Spider_(T-Rex)/3.0 www.lycos.com
[207.77.90.12]
www.lycos.com
[207.77.90.14]
207.77.90.20
[207.77.90.20]
www-es.lycos.com
[207.77.90.136]
lycosidae.lycos.com
[207.77.90.186]

Messe Basel

Spider Name (HTTP_USER_AGENT) Host
AltaVista Intranet V1.0 AltaVista Search Intranet PX V1.0 webmaster@messebasel.ch 194.124.202.8
[194.124.202.8]

Muscat EuroFerret

Spider Name (HTTP_USER_AGENT) Host
Mozilla/3.0 (compatible; MuscatFerret/1.4.1; olly@muscat.co.uk) trousers.muscat.com
[193.114.89.43]
Mozilla/3.0 (compatible; MuscatFerret/1.5.2; olly@muscat.co.uk) grommit.muscat.com
[193.131.74.131]
Mozilla/3.0 (compatible; MuscatFerret/1.5.3; olly@muscat.co.uk)

NetNames

Spider Name (HTTP_USER_AGENT) Host
Namecrawler/1.0 shadow.netnames.net
[195.40.151.254]

Northern Light Search

Spider Name (HTTP_USER_AGENT) Host
Gulliver/1.2 taz.northernlight.com
[208.219.77.9]

PlanetSearch

Spider Name (HTTP_USER_AGENT) Host
fido/1.0 Harvest/1.4.pl2 fido.planetsearch.com
[206.64.113.2]

sear.ch

Spider Name (HTTP_USER_AGENT) Host(s)
AltaVista Intranet V1.0 bluewindow.ch webmaster@bluewin.ch search.tbw.ch
[195.186.1.46]
AltaVista Intranet V1.0 Re-Index webmaster@bluewin.ch search.tbw.ch
[195.186.1.46]
scooter.sear.ch
[195.186.1.53]

search.at

Spider Name (HTTP_USER_AGENT) Host(s)
AltaVista Intranet V1.0 pta.at webmaster@pta.at search.pta.at
[195.3.97.3]
AltaVista Intranet V2.0 pta.at webmaster@pta.at search.pta.at
[195.3.97.3]
195.3.97.4
[195.3.97.4]
search.at V1.2 dill.relog.ch
[194.191.121.42]

seednet

Spider Name (HTTP_USER_AGENT) Host
GAIS Robot/1.0B2 gais3.seed.net.tw
[139.175.250.22]

Sexsearcher!

Spider Name (HTTP_USER_AGENT) Host
sexsearcher petersserver.sexsearcher.com
[209.73.233.226]

Swiss Search

Spider Name (HTTP_USER_AGENT) Hosts
SwissSearch V1.2 efeu.search.ch
[194.191.121.17]
nzz-temp.relog.ch
[194.191.121.42]
dill.relog.ch
[194.191.121.42]
wurmerl.search.at
[194.191.121.105]

Thunderstone

Spider Name (HTTP_USER_AGENT) Host
Mozilla/2.0 (compatible; T-H-U-N-D-E-R-S-T-O-N-E) shop.thunderstone.com
[198.49.220.62]

WebCrawler

Spider Name (HTTP_USER_AGENT) Host(s)
WebCrawler/3.0 Robot libwww/5.0a tripod.webcrawler.com
[204.62.245.110]
WebCrawler-AddURL/2.0 info.webcrawler.com
[198.3.99.63]
wc-s11.webcrawler.com
[198.3.101.77]

WiseWire

Spider Name (HTTP_USER_AGENT) Hosts
WiseWirexx

xx stands for one of the following strings:

  • [Empty]
  • -Alpha-1.0
  • -Alpha-Spider
  • -Alpha12-Spider971219a
  • -Alpha12-Spider(971223a)
  • -HotSpider-1.0
  • -Spider-1.0
  • -Spider2
  • -Widow-1.0
  • -Widow-1.0r
  • -Widow-1.0-ALPHA12
spider1.wisewire.com
[204.192.112.114]
204.192.112.130
[204.192.112.130]
204.192.112.178
[204.192.112.178]
nat-20-149.wisewire.com
[206.181.238.149]
nat-20-151.wisewire.com
[206.181.238.151]
nat-20-156.wisewire.com
[206.181.238.156]
spider2.wisewire.com
[208.215.47.115]
spider2-uu.wisewire.com
[208.215.47.115]
spider3.wisewire.com
[208.215.47.130]