【爬虫相关】YisouSpider是什么蜘蛛爬虫？

YisouSpider是什么蜘蛛爬虫？

Yisouspider是神马搜索的网页爬虫，同时也在为CNZZ云推荐提供服务。网页爬虫是搜索引擎访问互联网，自动化抓取网页内容的程序。很多人反应，神马搜索的蜘蛛爬虫占用CPU非常高，爬的非常卡，所以有时候可以根据需求屏蔽

IP段

42.156.139.32 YisouSpider
42.156.136.32 YisouSpider
42.156.137.32 YisouSpider
42.156.138.32 YisouSpider
42.120.161.32 YisouSpider
42.120.160.32 YisouSpider

42.120.160.97
42.120.161.97
42.156.136.97
42.156.137.97
42.156.138.97
42.156.139.97

42.120.160.51
42.120.161.51
42.156.136.51
42.156.137.51
42.156.138.51
42.156.139.51
106.11.152.198
106.11.153.204
106.11.156.193
106.11.157.220
106.11.158.214
106.11.159.197

42.156.136.23
42.156.137.23
42.156.138.23
42.156.139.23
42.120.160.23
42.120.161.23
106.11.154.8
106.11.155.28
106.11.156.9
106.11.157.19
106.11.159.20

蜘蛛屏蔽方式如下

robots.txt
User-agent: YisouSpider
Disallow: /
User-agent: EasouSpider
Disallow: /

.htaccess
RewriteEngine on
RewriteCond %{HTTP_USER_AGENT} ^YisouSpider [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^EasouSpider [NC,OR]
RewriteCond %{HTTP_USER_AGNET} ^spbot [NC]
RewriteRule .* – [F,L]

<?php
$ua=$_SERVER[‘HTTP_USER_AGENT’];
$ua=strtolower($ua);
if($ua!=”” &&strpos($ua,”YisouSpider”)){

header(‘HTTP/1.1 404 Not Found’);
header(“status: 404 Not Found”);
}else{
}
?>