求个分页采集思路。

复制内容到剪贴板

代码:

function collect_content($collect,$rulecontent,$page_mode=''){
    //ini_set("memory_limit" , "128M");
    static $tmpcontent = array();
    static $tmpurl = array();
    static $data = array();
    $decode = $collect['lang'] == 'gb2312'? 1:0;
    $html = fetch($collect['url'],$decode); //fetch是一个自定义函数，使用CURL
    //echo $html;
    if($rulecontent['content']){
        preg_match("/".$rulecontent['content']."/is",$html,$content);
        if(empty($content[1]))    collect_error('内容','content');
    }
    //echo $html;
    //print_r($content);exit;
    $data = array(
        'content'=>$content[1]
    );
    if($rulecontent['page']){
        preg_match_all("/".$rulecontent['page']."/is",$html,$page);
        //分页部分列表
        //exit($rulecontent['page_mode']);
        //echo($html);
        //print_r($page[1]);exit;
        if($page[1]){
            
            $tmpcontent[] = $data['content'];
            $tmpurl[] = $collect['url'];
            
            if($rulecontent['page_mode'] == 'page_part'){
                foreach($page[1] as $key=>$val){
                    //判断网址是否是完整路径
                    $host = parse_url($collect['url']);
                    if(!strpos($val,$host['host'])) $val = 'http://'.$host['host'].$val;
                    
                    $collect['url'] = $val;
                    $tmpurl = array_unique($tmpurl);
                    if(in_array($collect['url'],$tmpurl))    continue;
                    $tmpurl[] = $collect['url'];
                    $results = collect_content($collect,$rulecontent,'page_part');
                    if(!in_array($results['content'], $tmpcontent)) $tmpcontent[] = $results['content'];
                }
            }
            //合并内容
            $data['content'] = implode('[page]', $tmpcontent);
        }
    }
    return $data;
}

引用:

1.取内容
2.分析内容做数据采集
3.取页码的显示部分,根据当前页的规则分析是否有下一页.
在这里 当前页的形式可以用正则表示为 然后根据这个位置,向下找一个,如果能找到,那就是还有下一页,
然后得到下一页的地址,传递给程序,继续从第一部开始,如果没有取到
那就是证明已经到了最后一页了,整个数据采集完成.