
平台论坛博客文库

› 论坛 › 程序设计 › Perl › 新玩具《JSON》Perl如何抓取数据？

12 / 2 页下一页

新玩具《JSON》Perl如何抓取数据？ [复制链接]

墨迹哥

富足长乐

论坛徽章:: 3

电梯直达

1楼 [收藏(0)] [报告]

发表于 2013-04-10 22:13 |只看该作者 |倒序浏览

之前发过一个抓取页面的代码.可是我发现能抓取页面不能抓取json数据,
现在还在研究当中...发出原始的过程.找大伙一块研究研究...之前没玩过这东西..
我还在搜论坛,看有没有前人做过或问过的.在这里mark一下..

请问：如何抓取返回的json数据？？？。。。。

数据包:

POST /tools/web-sites-on-web-server/php/get-web-sites-on-web-server-json-data.php HTTP/1.1
Host: www.yougetsignal.com
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:19.0) Gecko/20100101 Firefox/19.0 Iceweasel/19.0.2
Accept: text/javascript, text/html, application/xml, text/xml, */*
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
X-Requested-With: XMLHttpRequest
X-Prototype-Version: 1.6.0
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
Referer: http://www.yougetsignal.com/tools/web-sites-on-web-server/
Content-Length: 22
Cookie:
Connection: keep-alive
Pragma: no-cache
Cache-Control: no-cache
remoteAddress=sohu.com

复制代码

返回的数据:

HTTP/1.1 200 OK
Date: Wed, 10 Apr 2013 13:48:43 GMT
Server: Apache
Vary: Accept-Encoding
Content-Length: 16773
Keep-Alive: timeout=2, max=100
Connection: Keep-Alive
Content-Type: text/html
{"status":"Success", "resultsMethod":"scrape", "lastScrape":"2013-04-10 06:48:47", "domainCount":"516", "remoteAddress":"sohu.com", "remoteIpAddress":"61.135.181.175", "domainArray":[["029love.sohu.com.cn", ""], ["0755qq.com", ""], ["17173.com", ""], ["17173.net", ""], ["17173.net.cn", ""], ["2004.sohu.com", ""], ["2006.sports.sohu.com", ""], ["2008.sohu.com.cn", ""], ["2008.sohu.comwww.sohu.com.cn", ""], ["2010.s.sohu.com", ""], ["286818458.blog.sohu.com.cn", ""], ["394003276.sohu.com.cn", ""], ["512.sohu.com", ""], ["51center.com", ""], ["55039106.blog.sohu.com.cn", ""], ["60.sohu.com", ""], ["60269.show.sohu.com.cn", ""], ["656979.com", ""], ["8888love-me.blog.sohu.com.cn", ""], ["91hdw.com", ""], ["abcddfe.blog.sohu.com.cn", ""], ["ablum.chinaren.com.cn", ""], ["add.sohu.com", ""], ["ai-weiyang.blog.sohu.com.cn", ""], ["air.sohu.com", ""], ["aiyits.sohu.com.cn", ""], ["akitsuki.blog.sohu.com.cn", ""], ["alooflau.blog.sohu.com.cn", ""], ["alumni.chinaren.com.cn", ""], ["angel.2010.sohu.com", ""], ["apple.sohu.com.cn", ""], ["apple86.blog.sohu.com.cn", ""], ["arcticsnowfanily.blog.sohu.com.cn", ""], ["art.2008.sohu.com", ""], ["art.sohu.com", ""]

复制代码

文库|博客

墨迹哥

富足长乐

论坛徽章:: 3

2楼 [报告]

发表于 2013-04-10 22:23 |只看该作者

翻了下，貌似论坛没有相关的资料。。。我继续gg。。。

实战分享：从技术角度谈机器学习入门| 【大话IT】RadonDB低门槛向MySQL集群下战书 | ChinaUnix打赏功能已上线！ | 新一代分布式关系型数据库RadonDB知多少？

墨迹哥

富足长乐

论坛徽章:: 3

3楼 [报告]

发表于 2013-04-10 22:27 |只看该作者

该地址上，iamlimeng 说，需要分析ajax，然后顺腾模瓜。。可以试试看。。。

http://bbs.chinaunix.net/thread-1819833-1-1.html

iakuf

小富即安

论坛徽章:: 1

4楼 [报告]

发表于 2013-04-10 23:00 |只看该作者

本帖最后由 iakuf 于 2013-04-10 23:34 编辑

回复 1# 墨迹哥

perl -Mojo -E 'say f("http://www.yougetsignal.com/tools/web-sites-on-web-server/php/get-web-sites-on-web-server-json-data.php" => { remoteAddress => "sina.com",key => "" } => {Referer => "http://www.yougetsignal.com/tools/web-sites-on-web-server/"})->body'

复制代码

因为实现过于简单，所以大家很少讲到这个。。。来测试一下。

墨迹哥

富足长乐

论坛徽章:: 3

5楼 [报告]

发表于 2013-04-10 23:01 |只看该作者

目前找到JS了。。还得研究研究。。。

function reverseIpDomainCheck(remoteAddress) {
document.getElementById('results').innerHTML = "<img src=\"/img/loader.gif\" alt=\"Please wait...\" title=\"Please wait...\" style=\"width: 2.6875em; height: 0.6875em;\" /> Checking "+remoteAddress+". This may take a up to three minutes..." ;
ajaxRequest.transport.abort(); // Cancel the previous request
var url = "/tools/web-sites-on-web-server/php/get-web-sites-on-web-server-json-data.php";
new Ajax.Request(url, {
method: 'post',
parameters: {'remoteAddress': remoteAddress, 'key': key},
onFailure: function(transport) {
document.getElementById('results').innerHTML = "<img src=\"/img/flag_red.gif\" alt=\"\" style=\"height: 1em; width: 1em;\" /> Service currently unavailable.";
},
onException: function(transport) {
document.getElementById('results').innerHTML = "<img src=\"/img/flag_red.gif\" alt=\"\" style=\"height: 1em; width: 1em;\" /> Service currently unavailable.";
},
onSuccess: function(transport) {
reverseIpDomainJsonData = transport.responseText.evalJSON(); // Place the results into JSON
if(reverseIpDomainJsonData.status != "Fail" && reverseIpDomainJsonData.domainCount != 0) { // The request was successful
if(reverseIpDomainJsonData.domainCount > 999) {
domainCountText = "over ";
}
else {
domainCountText = "";
}
if(reverseIpDomainJsonData.domainCount == 1) {
domainPluralityText = "domain";
}
else {
domainPluralityText = "domains";
}
if(reverseIpDomainJsonData.remoteAddress != reverseIpDomainJsonData.remoteIpAddress) {
resultsSummaryHTML = "<img src=\"/img/flag_green.gif\" alt=\"\" style=\"height: 1em; width: 1em;\" /> Found "+domainCountText+""+reverseIpDomainJsonData.domainCount+" "+domainPluralityText+" hosted on the same web server as <a href=\"http://"+remoteAddress+"\" target=\"_blank\">"+reverseIpDomainJsonData.remoteAddress+"</a> ("+reverseIpDomainJsonData.remoteIpAddress+").";
}
else {
resultsSummaryHTML = "<img src=\"/img/flag_green.gif\" alt=\"\" style=\"height: 1em; width: 1em;\" /> Found "+domainCountText+""+reverseIpDomainJsonData.domainCount+" "+domainPluralityText+" hosted on the same web server as <a href=\"http://"+remoteAddress+"\" target=\"_blank\">"+reverseIpDomainJsonData.remoteAddress+"</a>.";
}
// List all of the domains
profaneDomainExists = false;
resultsHTML = "";
for(i = 0; i < reverseIpDomainJsonData.domainArray.length; i++) {
if(reverseIpDomainJsonData.domainArray[i][1] == 1) { // Check to see if it is a profane domain
//resultsHTML += "<a href=\"http://"+reverseIpDomainJsonData.domainArray[i][0]+"\" target=\"_blank\" style=\"color: #DF454B;\">"+reverseIpDomainJsonData.domainArray[i][0]+"</a> (<a href=\"/tools/web-sites-on-web-server/?remoteAddress="+reverseIpDomainJsonData.domainArray[i][0]+"\">linkback</a>)\n";
resultsHTML += "<a href=\"http://"+reverseIpDomainJsonData.domainArray[i][0]+"\" target=\"_blank\" style=\"color: #DF454B;\">"+reverseIpDomainJsonData.domainArray[i][0]+"</a>\n";
profaneDomainExists = true;
}
else {
//resultsHTML += "<a href=\"http://"+reverseIpDomainJsonData.domainArray[i][0]+"\" target=\"_blank\">"+reverseIpDomainJsonData.domainArray[i][0]+"</a> (<a href=\"/tools/web-sites-on-web-server/?remoteAddress="+reverseIpDomainJsonData.domainArray[i][0]+"\">linkback</a>)\n";
resultsHTML += "<a href=\"http://"+reverseIpDomainJsonData.domainArray[i][0]+"\" target=\"_blank\">"+reverseIpDomainJsonData.domainArray[i][0]+"</a>\n";
}
}
if(profaneDomainExists == true) {
resultsSummaryHTML = resultsSummaryHTML + "It appears that the web server located at "+reverseIpDomainJsonData.remoteIpAddress+" may be hosting one or more web sites with explicit content. The web sites in question are highlighted in red below. There is a possibility that all of the web sites on this web server may be blocked by web filtering software. Search engine rankings for these web sites may be affected as well.";
}
document.getElementById('results').innerHTML = resultsSummaryHTML + resultsHTML;
}
else { // There was a problem
document.getElementById('results').innerHTML = "<img src=\"/img/flag_red.gif\" alt=\"\" style=\"height: 1em; width: 1em;\" /> " + reverseIpDomainJsonData.message + "";
}
}
});
}

复制代码

墨迹哥

富足长乐

论坛徽章:: 3

6楼 [报告]

发表于 2013-04-10 23:13 |只看该作者

回复 4# iakuf

貌似有问题。。。。

root@smart:/var/module# perl -Mojo -E 'say f("http://www.yougetsignal.com/tools/web-sites-on-web-server/php/get-web-sites-on-web-server-json-data.php" => { remoteAddress => sohu.com })->json'
ojo::f is DEPRECATED in favor of ojo::p at -e line 1
Mojo::UserAgent::build_form_tx is DEPRECATED in favor of Mojo::UserAgent::build_tx at /usr/local/share/perl/5.14.2/ojo.pm line 46
Mojo::UserAgent::Transactor::form is DEPRECATED in favor of Mojo::UserAgent::Transactor::tx at /usr/local/share/perl/5.14.2/Mojo/UserAgent.pm line 66
root@smart:/var/module# perl -Mojo -E 'say n("http://www.yougetsignal.com/tools/web-sites-on-web-server/php/get-web-sites-on-web-server-json-data.php" => { remoteAddress => sohu.com })->json'
ojo::n is DEPRECATED in favor of ojo::p at -e line 1
Mojo::UserAgent::build_json_tx is DEPRECATED in favor of Mojo::UserAgent::build_tx at /usr/local/share/perl/5.14.2/ojo.pm line 50
Mojo::UserAgent::Transactor::json is DEPRECATED in favor of Mojo::UserAgent::Transactor::tx at /usr/local/share/perl/5.14.2/Mojo/UserAgent.pm line 73

复制代码

iakuf

小富即安

论坛徽章:: 1

7楼 [报告]

发表于 2013-04-10 23:37 |只看该作者

本帖最后由 iakuf 于 2013-04-10 23:37 编辑

回复 6# 墨迹哥

在测试一下上面那个，之所以得不到结果是因为源站对 referer 做了控制。对于 Perl 来讲，抓这种东西太小意思了。
另外，和 JS 没关系，你直接抓 HTTP 的 header ，不要信别人看 js. 整个互联网的交互都可以通过 HTTP 来搞定。推荐你使用 HttpFox 这个 Firefox 中的插件来看 Header .

墨迹哥

富足长乐

论坛徽章:: 3

8楼 [报告]

发表于 2013-04-10 23:40 |只看该作者

回复 7# iakuf

这个是ref的地址。。

http://www.yougetsignal.com/tools/web-sites-on-web-server/

复制代码

你说的header抓到的话该怎么操作呢？

iakuf

小富即安

论坛徽章:: 1

9楼 [报告]

发表于 2013-04-10 23:45 |只看该作者

回复 8# 墨迹哥

perl -Mojo -E 'say f("http://www.yougetsignal.com/tools/web-sites-on-web-server/php/get-web-sites-on-web-server-json-data.php" => { remoteAddress => "sina.com",key => "" } => {Referer => "http://www.yougetsignal.com/tools/web-sites-on-web-server/"})->body'

复制代码

使用 ojo 模块和使用 Mojo::UserAgent 模块，都可以很方便的加参数和 Http 的 Header 。
象
f("http://www.yougetsignal.com/tools/web-sites-on-web-server/php/get-web-sites-on-web-server-json-data.php" => { remoteAddress => "sina.com",key => "" } => {Referer => "http://www.yougetsignal.com/tools/web-sites-on-web-server/"})->body

相当于

my $ua = Mojo::UserAgent->new;
say $ua->post_form(
"http://www.yougetsignal.com/tools/web-sites-on-web-server/php/get-web-sites-on-web-server-json-data.php"
=> {
remoteAddress => "sina.com",
key => ""
}
=> {
Referer => "http://www.yougetsignal.com/tools/web-sites-on-web-server/"
})->res->body

复制代码

可以见到，在 Mojo 的函数 new 完后，当调用 POST 时，需要传三个参数，第二个第三个都可以不传，如果传的话，都需要是 hash 的引用。
第一个参数是 URL ，第二个参数是 POST 的键值对，第三个参数是 HTTP 的 header 的键值对。

墨迹哥

富足长乐

论坛徽章:: 3

10楼 [报告]

发表于 2013-04-10 23:46 |只看该作者

回复 9# iakuf

明白了，我知道怎么弄了。。哈哈！谢谢哈！

12 / 2 页下一页

返回列表

Chinaunix › 论坛 › 程序设计 › Perl › 新玩具《JSON》Perl如何抓取数据？