百度 | 神马 | 搜狗 | 技术文档 | 学习资料分享 - 记录帝国CMS及运维技术的点点滴滴
你的位置:首页 > 实用IT技术 » 正文

【爬虫相关】2018各大搜索引擎蜘蛛一览

发布时间:2020-09-17

作品分类:爬虫相关  蜘蛛  抓取  名称  各大  蜘蛛  搜索引擎

360Spider
Sosospider
SogouNewsSpider
Sogouwebspider
Sogouinstspider
Sogouspider2
Sogoublog
SogouOrionspider
sogouspider
Sogou web spider
YodaoBot
Googlebot
Bingbot
Slurp
Teoma
ia_archiver
twiceler
MSNBot
Scrubby
Robozilla
Gigabot
googlebot-image
googlebot-mobile
yahoo-mmcrawler
yahoo-blogs/v3.9
psbot
Nutch
naverbot
yeti
asterias
AhrefsBot
BLEXBot
MJ12bot
bingbot
netEstate
YisouSpider
Yahoo!SlurpChina
YoudaoBot
msnbot
msnbot-media
EasouSpider
JikeSpider
EtaoSpider
YandexBot
Google-Test
MegaIndex.ru

1、百度蜘蛛:Baiduspider
网上的资料百度蜘蛛名称有BaiduSpider、baiduspider等,都洗洗睡吧,那是旧黄历了。百度蜘蛛最新名称为Baiduspider。日志中还发现了Baiduspider-image这个百度旗下蜘蛛,查了下资料(其实直接看名字就可以了……),是抓取图片的蜘蛛。
常见百度旗下同类型蜘蛛还有下面这些:Baiduspider-mobile(抓取wap)、Baiduspider-image(抓取图片)、Baiduspider-video(抓取视频)、Baiduspider-news(抓取新闻)。
注:以上百度蜘蛛目前常见的是Baiduspider和Baiduspider-image两种。

2、谷歌蜘蛛:Googlebot
这个争议较少,但也有说是GoogleBot的。谷歌蜘蛛最新名称为“compatible; Googlebot/2.1;”。还发现了Googlebot-Mobile,看名字是抓取wap内容的。

3、360蜘蛛:360Spider,它是一个很“勤奋抓爬”的蜘蛛。
4、SOSO蜘蛛:Sosospider,也可为它颁一个“勤奋抓爬”奖的蜘蛛。
5、雅虎蜘蛛:“Yahoo! Slurp China”或者Yahoo! 
名称中带“Slurp”和空格,名称有空格robots里名称可以使用“Slurp”或者“Yahoo”单词描述,不知道有效无效。

6、有道蜘蛛:YoudaoBot,YodaoBot(两个名字都有,中文拼音少了个U字母读音差别很大嘎,这都会少?)
7、搜狗蜘蛛:Sogou News Spider
搜狗蜘蛛还包括如下这些:Sogou web spider、Sogou inst spider、Sogou spider2、Sogou blog、Sogou News Spider、Sogou Orion spider,
(参考一些网站的robots文件,搜狗蜘蛛名称可以用Sogou概括,无法验证不知道有没有效)
看看最权威的百度的robots.txt ,http://www.baidu.com/robots.txt 就为Sogou搜狗蜘蛛费了不少字节,占了一大块领地。
“Sogou web spider;Sogou inst spider;Sogou spider2;Sogou blog;Sogou News Spider;Sogou Orion spider”目前6个,名称都带空格。
线上常见”Sogou web spider/4.0″ ;”Sogou News Spider/4.0″ ;”Sogou inst spider/4.0″ 可以为它颁个“占名为王”奖。
8、MSN蜘蛛:msnbot,msnbot-media(只见到msnbot-media在狂爬……)

9、必应蜘蛛:bingbot 
线上(compatible; bingbot/2.0;)
10、一搜蜘蛛:YisouSpider

11、Alexa蜘蛛:ia_archiver

12、宜sou蜘蛛:EasouSpider

13、即刻蜘蛛:JikeSpider

14、一淘网蜘蛛:EtaoSpider
“Mozilla/5.0 (compatible; EtaoSpider/1.0; http://省略/EtaoSpider)”
根据上述蜘蛛中选择几个常用的允许抓取,其余的都可以通过robots屏蔽抓取。如果你暂时空间流量还足够使用,等流量紧张了就保留几个常用的屏蔽掉其它蜘蛛以节省流量。至于那些蜘蛛抓取对网站能带来有利用的价值,网站的管理者眼睛是雪亮的。

另外还发现了如 YandexBot、AhrefsBot和ezooms.bot这些蜘蛛,据说这些蜘蛛国外,对中文网站用处很小。那不如就节省下资源。

最后再附上360搜索 so.com的蜘蛛IP段

101.226.166.195101.226.169.228180.153.236.113182.118.25.105182.118.20.228
101.226.166.196101.226.169.229180.153.236.114169.254.95.120182.118.20.215
101.226.166.197101.226.169.230180.153.236.115182.118.25.20182.118.20.216
101.226.166.198180.153.231.111180.153.236.116182.118.29.148182.118.20.217
101.226.166.199101.226.166.219180.153.236.59182.118.35.16182.118.20.218
101.226.166.200101.226.166.220180.153.236.60182.118.35.17182.118.20.219
101.226.166.201101.226.166.221180.153.236.61182.118.35.18182.118.22.144
101.226.166.202101.226.166.222180.153.236.62182.118.35.19182.118.25.216
180.153.231.56101.226.166.223180.153.236.63182.118.25.15182.118.25.217
101.226.169.215101.226.166.224180.153.236.64182.118.29.147182.118.25.218
101.226.169.216101.226.166.225180.153.236.65182.118.35.11182.118.25.219
101.226.169.217101.226.166.226180.153.236.66182.118.35.12182.118.25.220
101.226.169.218180.153.231.77180.153.227.108182.118.35.13182.118.22.145
101.226.169.219101.226.166.243180.153.236.189182.118.35.14182.118.25.221
101.226.169.220101.226.166.244180.153.236.190169.254.95.120182.118.25.222
101.226.169.221101.226.166.245180.153.236.191182.118.29.150182.118.25.223
101.226.169.222101.226.166.246180.153.236.192182.118.35.26182.118.25.224
180.153.231.108101.226.166.247180.153.236.193182.118.35.27182.118.25.225
101.226.166.251101.226.166.248180.153.236.194182.118.35.28182.118.22.221
101.226.166.252101.226.166.249180.153.236.195182.118.35.29182.118.22.222
101.226.166.253101.226.166.250180.153.236.196182.118.35.30182.118.22.223
101.226.166.254180.153.231.89180.153.227.105169.254.95.120182.118.22.224
101.226.167.195101.226.167.207180.153.236.165182.118.25.25182.118.22.225
101.226.167.196101.226.167.208180.153.236.166182.118.29.149182.118.22.231
101.226.167.197101.226.167.209180.153.236.167182.118.35.21182.118.22.232
101.226.167.198101.226.167.210180.153.236.168182.118.35.22182.118.22.233
180.153.231.90101.226.167.211180.153.236.169182.118.35.23182.118.22.234
101.226.168.195101.226.167.212180.153.236.170182.118.35.24182.118.22.235
101.226.168.196101.226.167.213180.153.236.171182.118.25.186182.118.22.236
101.226.168.197101.226.167.214180.153.236.172182.118.25.187182.118.22.237
101.226.168.198180.153.231.92180.153.236.51182.118.25.188182.118.22.238
101.226.168.199101.226.166.203180.153.236.52182.118.25.189182.118.22.239
101.226.168.200101.226.166.204180.153.236.53182.118.25.190182.118.22.240
101.226.168.201101.226.166.205180.153.236.54182.118.25.181182.118.22.241
101.226.168.202101.226.166.206180.153.236.55182.118.25.182182.118.22.242
180.153.231.98101.226.166.207180.153.236.56182.118.25.183182.118.22.243
101.226.167.247101.226.166.208180.153.236.57182.118.25.184182.118.22.244
101.226.167.248101.226.166.209180.153.236.58182.118.25.185182.118.22.245
101.226.167.249101.226.166.210180.153.227.104182.118.25.136182.118.22.246
101.226.167.250180.153.231.57180.153.236.157182.118.25.137182.118.22.247
101.226.167.251101.226.167.215180.153.236.158182.118.25.138182.118.22.248
101.226.167.252101.226.167.216180.153.236.159182.118.25.139182.118.22.249
101.226.167.253101.226.167.217180.153.236.160182.118.25.140182.118.22.250
101.226.167.254101.226.167.218180.153.236.161182.118.25.141182.118.22.141
180.153.231.97101.226.167.219180.153.236.162182.118.25.142182.118.25.201
101.226.168.203101.226.167.220180.153.236.163182.118.25.143182.118.25.202
101.226.168.204101.226.167.221180.153.236.164182.118.25.144182.118.25.203
101.226.168.205101.226.167.222180.153.236.149182.118.25.145182.118.25.204
101.226.168.206180.153.231.93180.153.236.150182.118.25.176182.118.25.205
101.226.168.207101.226.169.199180.153.236.151182.118.25.177182.118.20.203
101.226.168.208101.226.169.200180.153.236.152182.118.25.178182.118.20.229
101.226.168.209101.226.169.201180.153.236.153182.118.25.179182.118.20.230
101.226.168.210101.226.169.202180.153.236.154182.118.25.180182.118.20.231
180.153.231.99101.226.169.203180.153.236.155182.118.25.171182.118.20.232
101.226.167.223101.226.169.204180.153.236.156182.118.25.172182.118.20.202
101.226.167.224101.226.169.205180.153.236.133182.118.25.173182.118.20.220
101.226.167.225101.226.169.206180.153.236.134182.118.25.174182.118.20.221
101.226.167.226180.153.231.106180.153.236.135182.118.25.175182.118.20.222
101.226.167.227101.226.168.243180.153.236.136182.118.25.196182.118.20.223
101.226.167.228101.226.168.244180.153.236.137182.118.25.197182.118.22.146
101.226.167.229101.226.168.245180.153.236.138182.118.25.198182.118.25.226
101.226.167.230101.226.168.246180.153.236.139182.118.25.199182.118.25.227
180.153.231.94101.226.168.247180.153.236.140182.118.25.200182.118.25.228
101.226.168.211101.226.168.248180.153.227.107182.118.25.191182.118.25.229
101.226.168.212101.226.168.249180.153.236.181182.118.25.192182.118.25.230
101.226.168.213101.226.168.250180.153.236.182182.118.25.193182.118.22.226
101.226.168.214180.153.231.104180.153.236.183182.118.25.194182.118.22.227
101.226.168.215101.226.167.231180.153.236.184182.118.25.195182.118.22.228
101.226.168.216101.226.167.232180.153.236.185182.118.22.143182.118.22.229
101.226.168.217101.226.167.233180.153.236.186182.118.25.211182.118.22.230
101.226.168.218101.226.167.234180.153.236.187182.118.25.212182.118.22.216
180.153.231.100101.226.167.235180.153.236.188182.118.25.213182.118.22.217
101.226.166.211101.226.167.236180.153.236.101182.118.25.214182.118.22.218
101.226.166.212101.226.167.237180.153.236.102182.118.25.215182.118.22.219
101.226.166.213101.226.167.238180.153.236.103182.118.20.233182.118.22.220
101.226.166.214180.153.231.95180.153.236.104182.118.20.234182.118.22.142
101.226.166.215101.226.167.199180.153.236.105182.118.20.235182.118.25.206
101.226.166.216101.226.167.200180.153.236.106182.118.20.236182.118.25.207
101.226.166.217101.226.167.201180.153.236.107182.118.20.237182.118.25.208
101.226.166.218101.226.167.202180.153.236.108182.118.22.147182.118.25.209
180.153.231.58101.226.167.203180.153.236.117182.118.25.231182.118.25.210
101.226.166.235101.226.167.204180.153.236.118182.118.25.232182.118.20.201
101.226.166.236101.226.167.205180.153.236.119182.118.25.233182.118.20.211
101.226.166.237101.226.167.206180.153.236.120182.118.25.234182.118.20.212
101.226.166.238180.153.231.91180.153.236.121182.118.25.235182.118.20.213
101.226.166.239101.226.167.239180.153.236.122182.118.21.231182.118.20.214
101.226.166.240101.226.167.240180.153.236.123182.118.21.232182.118.22.211
101.226.166.241101.226.167.241180.153.236.124182.118.21.233182.118.22.212
101.226.166.242101.226.167.242180.153.236.67182.118.21.234182.118.22.213
180.153.231.88101.226.167.243180.153.236.68182.118.21.235182.118.22.214
101.226.168.235101.226.167.244180.153.236.69182.118.21.236182.118.22.215
101.226.168.236101.226.167.245180.153.236.70182.118.21.237182.118.20.206
101.226.168.237101.226.167.246180.153.236.71182.118.21.238182.118.21.220
101.226.168.238180.153.231.96180.153.236.72182.118.21.239182.118.21.221
101.226.168.239101.226.168.251180.153.236.73182.118.21.240182.118.21.222
101.226.168.240101.226.168.252180.153.236.74182.118.20.205182.118.21.223
101.226.168.241101.226.168.253180.153.236.35182.118.21.201182.118.21.224
101.226.168.242101.226.168.254180.153.236.36182.118.21.202182.118.21.225
180.153.231.103101.226.169.195180.153.236.37182.118.21.203182.118.21.226
101.226.166.227101.226.169.196180.153.236.38182.118.21.204182.118.21.227
101.226.166.228101.226.169.197180.153.236.39182.118.21.205182.118.21.228
101.226.166.229101.226.169.198180.153.236.40182.118.21.206182.118.21.229
101.226.166.230180.153.231.105180.153.236.41182.118.21.207182.118.21.230
101.226.166.231180.153.236.19180.153.236.42182.118.21.208182.118.21.245
101.226.166.232180.153.236.20180.153.227.106182.118.21.209182.118.21.246
101.226.166.233180.153.236.21180.153.236.173182.118.21.210182.118.21.247
101.226.166.234180.153.236.22180.153.236.174182.118.21.211182.118.21.248
180.153.231.78180.153.236.23180.153.236.175182.118.21.212182.118.21.249
101.226.169.207180.153.236.24180.153.236.176182.118.21.213182.118.21.250
101.226.169.208180.153.236.25180.153.236.177182.118.21.214182.118.21.251
101.226.169.209180.153.236.26180.153.236.178182.118.21.215182.118.21.252
101.226.169.210180.153.236.125180.153.236.179182.118.21.216182.118.21.253
101.226.169.211180.153.236.126180.153.236.180182.118.21.217182.118.21.254
101.226.169.212180.153.236.127180.153.236.141182.118.21.218182.118.22.148
101.226.169.213180.153.236.128180.153.236.142182.118.21.219182.118.25.236
101.226.169.214180.153.236.129180.153.236.143182.118.20.207182.118.25.237
180.153.231.107180.153.236.130180.153.236.144182.118.20.249182.118.25.238
101.226.168.227180.153.236.131180.153.236.145182.118.20.250182.118.25.239
101.226.168.228180.153.236.132180.153.236.146182.118.20.251182.118.25.240
101.226.168.229180.153.231.113180.153.236.147182.118.20.252182.118.25.131
101.226.168.230180.153.236.11180.153.236.148182.118.20.253182.118.25.132
101.226.168.231180.153.236.12182.118.25.151182.118.20.254182.118.25.133
101.226.168.232180.153.236.13182.118.25.152182.118.21.241182.118.25.134
101.226.168.233180.153.236.14182.118.25.153182.118.21.242182.118.25.135
101.226.168.234180.153.236.15182.118.25.154182.118.21.243182.118.22.149
180.153.231.102180.153.236.16182.118.25.155182.118.21.244182.118.25.241
101.226.168.219180.153.236.17182.118.25.161182.118.22.201182.118.25.242
101.226.168.220180.153.236.18182.118.25.162182.118.22.202182.118.25.243
101.226.168.221180.153.236.43182.118.25.163182.118.22.203182.118.25.244
101.226.168.222180.153.236.44182.118.25.164182.118.22.204182.118.25.245
101.226.168.223180.153.236.45182.118.25.165182.118.22.205182.118.25.146
101.226.168.224180.153.236.46182.118.22.196182.118.22.206182.118.25.147
101.226.168.225180.153.236.47182.118.22.197182.118.22.207182.118.25.148
101.226.168.226180.153.236.48182.118.22.198182.118.22.208182.118.25.149
180.153.231.101180.153.236.49182.118.22.199182.118.22.209182.118.25.150
101.226.169.223180.153.236.50182.118.22.200182.118.22.210182.118.25.156
101.226.169.224180.153.236.109182.118.25.101182.118.20.224182.118.25.157
101.226.169.225180.153.236.110182.118.25.102182.118.20.225182.118.25.158
101.226.169.226180.153.236.111182.118.25.103182.118.20.226182.118.25.159
101.226.169.227180.153.236.112182.118.25.104182.118.20.227182.118.25.160
42.236.99.242.236.99.12242.236.99.6242.236.99.18542.236.99.56
42.236.99.342.236.99.12342.236.99.6342.236.99.18642.236.99.57
42.236.99.442.236.99.12442.236.99.6442.236.99.18742.236.99.58
42.236.99.542.236.99.12542.236.99.6542.236.99.18842.236.99.59
42.236.99.642.236.99.12642.236.99.6642.236.99.18942.236.99.60
42.236.99.742.236.99.13042.236.99.6742.236.99.19042.236.99.61
42.236.99.842.236.99.13142.236.99.6842.236.99.19442.236.99.250
42.236.99.942.236.99.13242.236.99.6942.236.99.19542.236.99.253
42.236.99.1042.236.99.13342.236.99.7042.236.99.19642.236.99.179
42.236.99.1142.236.99.13442.236.99.7142.236.99.19742.236.99.180
42.236.99.1242.236.99.13542.236.99.7242.236.99.19842.236.99.181
42.236.99.1342.236.99.13642.236.99.7342.236.99.19942.236.99.182
42.236.99.1442.236.99.13742.236.99.7442.236.99.20042.236.99.183
42.236.99.1542.236.99.13842.236.99.7542.236.99.20142.236.99.184
42.236.99.1642.236.99.13942.236.99.7642.236.99.20242.236.99.251
42.236.99.1742.236.99.14042.236.99.7742.236.99.20342.236.99.252
42.236.99.1842.236.99.14142.236.99.7842.236.99.20442.236.99.116
42.236.99.1942.236.99.14242.236.99.7942.236.99.20542.236.99.117
42.236.99.2042.236.99.14342.236.99.8042.236.99.20642.236.99.118
42.236.99.2142.236.99.14442.236.99.8142.236.99.20742.236.99.119
42.236.99.2242.236.99.14542.236.99.8242.236.99.20842.236.99.120
42.236.99.2342.236.99.14642.236.99.8342.236.99.20942.236.99.121
42.236.99.2442.236.99.14742.236.99.8442.236.99.21042.236.99.254
42.236.99.2542.236.99.14842.236.99.8542.236.99.21142.236.99.249
42.236.99.2642.236.99.14942.236.99.8642.236.99.21242.236.99.242
42.236.99.2742.236.99.15042.236.99.8742.236.99.21342.236.99.243
42.236.99.2842.236.99.15142.236.99.8842.236.99.21442.236.99.244
42.236.99.2942.236.99.15242.236.99.8942.236.99.21542.236.99.245
42.236.99.3042.236.99.15342.236.99.9042.236.99.21642.236.99.246
42.236.99.3142.236.99.15442.236.99.9142.236.99.21742.236.99.247
42.236.99.3242.236.99.15542.236.99.9242.236.99.21842.236.99.248
42.236.99.3342.236.99.15642.236.99.9342.236.99.21942.236.99.53
42.236.99.3442.236.99.15742.236.99.9442.236.99.22042.236.99.54
42.236.99.3542.236.99.15842.236.99.9542.236.99.22142.236.99.55
42.236.99.3642.236.99.15942.236.99.9642.236.99.22242.236.99.176
42.236.99.3742.236.99.16042.236.99.9742.236.99.22342.236.99.177
42.236.99.3842.236.99.16142.236.99.9842.236.99.22442.236.99.178
42.236.99.3942.236.99.16242.236.99.9942.236.99.22542.236.99.113
42.236.99.4042.236.99.16342.236.99.10042.236.99.22642.236.99.114
42.236.99.4142.236.99.16442.236.99.10142.236.99.22742.236.99.115
42.236.99.4242.236.99.16542.236.99.10242.236.99.22842.236.99.239
42.236.99.4342.236.99.16642.236.99.10342.236.99.22942.236.99.240
42.236.99.4442.236.99.16742.236.99.10442.236.99.23042.236.99.241
42.236.99.4542.236.99.16842.236.99.10542.236.99.23142.236.99.52
42.236.99.4642.236.99.16942.236.99.10642.236.99.23242.236.99.175
42.236.99.4742.236.99.17042.236.99.10742.236.99.23342.236.99.112
42.236.99.4842.236.99.17142.236.99.10842.236.99.23442.236.99.238
42.236.99.4942.236.99.17242.236.99.10942.236.99.23542.236.99.237
42.236.99.5042.236.99.17342.236.99.11042.236.99.23642.236.99.111
42.236.99.5142.236.99.174
Top