通常抓取级联数数据情况不多,但要是真需要时,确多了一些麻烦,比如抓取商品分类级别信息等。本内容将讲述如何采集无限级联联数据,并以GoldData来抓取2019年最新的省市县三级为示例。
在数据集管理里,添加数据集area。如下图所示:
数据集相当于数据库中的表,只是字段是灵活的,可以随着需要而添加和变更。
在规则管理里,添加规则arearule,并将地址http://xzqh.mca.gov.cn/map 填为抓取入口地址。
我们通过分析可知,我们可从http://xzqh.mca.gov.cn/map 获取省级数据,然后通过级级数据请求http://xzqh.mca.gov.cn/selectJson 带有省级名称来获得市级数据,最后再通过市级数据分别请求http://xzqh.mca.gov.cn/selectJson 带有市级数据来获得县级数据。
并且发现请求 http://xzqh.mca.gov.cn/selectJson 是需要发POST请求的,因此我们需要将url加个前缀fake:,然后用规则里用JavaScript去请求URL。
我们在此编写的area数据集,有以下字段:
名称 | 说明 |
---|---|
sn | 取区域编码作为记录唯一字段 |
name | 名称 |
code | 取区域编码 |
abbr | 省名简写 |
parent_code | 取父区域编码 |
因此编写规则如下:
(注:具体规则内容请见文章最后)
然后编写完成后,我们就可以启动抓取器进行抓取。
打开数据管理,选择area数据集查看如下图所示:
回到数据管理,选择条件,选择需要导出的字段,进行导出。像这里数据比较多,GoldData将会以打包excel文件并压缩为zip文件下载。解压到本地,然后打开excel就可以看到抓取的数据,如下图所示:
通过这节内容,可以了解了如何通过GoldData 抓取级联数据。但是下一个问题是如何将数据导入自关联列表呢,且看下一往篇将会讲述如何将级联数据融合到数据库自关联表当中。
(抓取规则)
[
{
__sample: http://xzqh.mca.gov.cn/map
match0: http\:\/\/xzqh\.mca\.gov\.cn\/map
fields0:
{
__model: true
__node: js
__js:
'''
var exp11=/json\s=\s(.+)\s+\$\(doc/
var ret=exp11.exec(html)
var ss=eval(ret[1])
for(var i=0;i<ss.length;i++){
var ele=ss[i]
var exp12=/\((.+)\)/.exec(ele.shengji)
var exp13=/(.+)\(/.exec(ele.shengji)
out.add({sn:ele.quHuaDaiMa,name:exp13[1],code:ele.quHuaDaiMa,abbr:exp12[1],
parent_code:null,
})
}
'''
name:
{
expr: ""
attr: ""
js: ""
__label: ""
__showOnList: false
__type: ""
down: "0"
accessPathJs: ""
uploadConf: ""
}
code:
{
expr: ""
attr: ""
js: ""
__label: ""
__showOnList: false
__type: ""
down: "0"
accessPathJs: ""
uploadConf: ""
}
abbr:
{
expr: ""
attr: ""
js: ""
__label: ""
__showOnList: false
__type: ""
down: "0"
accessPathJs: ""
uploadConf: ""
}
sn:
{
expr: ""
attr: ""
js: ""
__label: ""
__showOnList: false
__type: ""
down: "0"
accessPathJs: ""
uploadConf: ""
}
__dataset: area
parent_code:
{
expr: ""
attr: ""
js: ""
__label: ""
__showOnList: false
__type: ""
down: "0"
accessPathJs: ""
uploadConf: ""
}
}
fields1:
{
__node: js
__js:
'''
var exp11=/json\s=\s(.+)\s+\$\(doc/
var ret=exp11.exec(html)
var ss=eval(ret[1])
for(var i=0;i<ss.length;i++){
var ele=ss[i]
var url='fake:http://xzqh.mca.gov.cn/selectJson?shengji='+ele.shengji+"&code="+ele.quHuaDaiMa
//var $ajax(url,[__method:'POST',data:'shengji='+ele.shengji]).content
out.add({href:url})
}
'''
href:
{
expr: ""
attr: ""
js: ""
__label: ""
__showOnList: false
__type: ""
down: "0"
accessPathJs: ""
uploadConf: ""
}
}
}
{
__sample: fake:http://xzqh.mca.gov.cn/selectJson?shengji=北京市(京)&code=110000
match0: fake\:http\:\/\/xzqh\.mca\.gov\.cn\/selectJson\?shengji=.+\&code=\d+
fields0:
{
__model: true
__dataset: area
sn:
{
expr: ""
attr: ""
js: ""
__label: ""
__showOnList: false
__type: ""
down: "0"
accessPathJs: ""
uploadConf: ""
}
__node: js
__js:
'''
var urlRet= /http\:\/\/xzqh\.mca\.gov\.cn\/selectJson\?shengji=(.+)\&code=(\d+)/.exec(baseUri);
var shengji=urlRet[1]
var sjCode=urlRet[2]
var url='http://xzqh.mca.gov.cn/selectJson'
var content=$ajax(url,{__method:'POST',data:'shengji='+shengji }).content
var arr=eval(content);
for(var i=0;i<arr.length;i++){
var dj=arr[i];
var area={
name:dj.diji,
code:dj.quHuaDaiMa,
parent_code:sjCode,
sn:dj.quHuaDaiMa,
}
out.add(area);
}
'''
name:
{
expr: ""
attr: ""
js: ""
__label: ""
__showOnList: false
__type: ""
down: "0"
accessPathJs: ""
uploadConf: ""
}
code:
{
expr: ""
attr: ""
js: ""
__label: ""
__showOnList: false
__type: ""
down: "0"
accessPathJs: ""
uploadConf: ""
}
abbr:
{
expr: ""
attr: ""
js: ""
__label: ""
__showOnList: false
__type: ""
down: "0"
accessPathJs: ""
uploadConf: ""
}
sn:
{
expr: ""
attr: ""
js: ""
__label: ""
__showOnList: false
__type: ""
down: "0"
accessPathJs: ""
uploadConf: ""
}
parent_code:
{
expr: ""
attr: ""
js: ""
__label: ""
__showOnList: false
__type: ""
down: "0"
accessPathJs: ""
uploadConf: ""
}
}
fields1:
{
__node: js
__js:
'''
var urlRet= /http\:\/\/xzqh\.mca\.gov\.cn\/selectJson\?shengji=(.+)\&code=(\d+)/.exec(baseUri);
var shengji=urlRet[1]
var sjCode=urlRet[2]
var url='http://xzqh.mca.gov.cn/selectJson'
var content=$ajax(url,{__method:'POST',data:'shengji='+shengji }).content
var arr=eval(content);
for(var i=0;i<arr.length;i++){
var dj=arr[i];
var url='fake:http://xzqh.mca.gov.cn/selectJson?shengji='+shengji+"&sjcode="+sjCode+"&diji="+dj.diji+"&djcode="+dj.quHuaDaiMa
out.add({href:url})
}
'''
href:
{
expr: ""
attr: ""
js: ""
__label: ""
__showOnList: false
__type: ""
down: "0"
accessPathJs: ""
uploadConf: ""
}
}
}
{
__sample: fake:href :fake:http://xzqh.mca.gov.cn/selectJson?shengji=北京市(京)&sjcode=110000&diji=北京市&djcode=110000
match0: fake\:http\:\/\/xzqh\.mca\.gov\.cn/selectJson\?shengji=([^&]+)\&sjcode=([^&]+)\&diji=([^&]+)\&djcode=([^&]+)
fields0:
{
__model: true
__dataset: area
__node: js
sn:
{
expr: ""
attr: ""
js: ""
__label: ""
__showOnList: false
__type: ""
down: "0"
accessPathJs: ""
uploadConf: ""
}
__js:
'''
var urlRet=/fake\:http\:\/\/xzqh\.mca\.gov\.cn\/selectJson\?shengji=([^\&]+)\&sjcode=([^\&]+)\&diji=([^\&]+)\&djcode=([^\&]+)/.exec(baseUri)
var shengji=urlRet[1]
var sjCode=urlRet[2]
var dj=urlRet[3]
var djCode=urlRet[4]
var url='http://xzqh.mca.gov.cn/selectJson'
var content=$ajax(url,{__method:'POST',data:'shengji='+shengji+'&diji='+dj }).content
var arr=eval(content);
for(var i=0;i<arr.length;i++){
var dj=arr[i];
var area={
name:dj.xianji,
code:dj.quHuaDaiMa,
parent_code:djCode,
sn:dj.quHuaDaiMa,
}
out.add(area);
}
'''
name:
{
expr: ""
attr: ""
js: ""
__label: ""
__showOnList: false
__type: ""
down: "0"
accessPathJs: ""
uploadConf: ""
}
code:
{
expr: ""
attr: ""
js: ""
__label: ""
__showOnList: false
__type: ""
down: "0"
accessPathJs: ""
uploadConf: ""
}
abbr:
{
expr: ""
attr: ""
js: ""
__label: ""
__showOnList: false
__type: ""
down: "0"
accessPathJs: ""
uploadConf: ""
}
sn:
{
expr: ""
attr: ""
js: ""
__label: ""
__showOnList: false
__type: ""
down: "0"
accessPathJs: ""
uploadConf: ""
}
parent_code:
{
expr: ""
attr: ""
js: ""
__label: ""
__showOnList: false
__type: ""
down: "0"
accessPathJs: ""
uploadConf: ""
}
}
}
]
免责声明:本站发布的内容(图片、视频和文字)以原创、转载和分享为主,文章观点不代表本网站立场,如果涉及侵权请联系站长邮箱:is@yisu.com进行举报,并提供相关证据,一经查实,将立刻删除涉嫌侵权内容。