forked from andeya/pholcus
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
一、界面升级: 1. Windows下编译时自动添加图标 2. web及cmd版的命令行窗口添加软件名称 二、下载器相关升级: 1. 初步增加PhantomJS下载器,实现双下载引擎,可在请求中指定 2. 下载控制转移至Request中,从而下载更灵活 3. Request增加重定向跳转次数控制功能 4. Request可以支持无限重载 5. Request增加序列化方法 6. 下载器实例改为全局唯一 三、规则语法升级: 1. 文本结果输出方法改为Output(ruleName string, resp *context.Response, item interface{}) 2. 文件输出方法建议用FileOutput(resp *context.Response, name ...string) 3. 更改解析函数Parse(resp *context.Response, ruleName ...string),ruleName为空时调用Root函数 4. OutFeild()方法改为IndexOutFeild() 5. Spider结构体新增CreatItem(ruleName string, item map[int]interface{}) map[string]interface{},将map索引自动转换为OutFeild字段 6. Spider结构体新增代理服务器列表 6. AddQueue()接收*Request参数,UseCookie改为EnableCookie 7. 规则方法库中升级timer计时器 四、数据库操作升级: 1. 优化更新mongodb操作方法库 2. 数据库连接池增加空闲连接定时回收
- Loading branch information
Showing
14 changed files
with
202 additions
and
132 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,40 @@ | ||
package downloader | ||
|
||
import ( | ||
"net/http" | ||
|
||
"github.com/henrylee2cn/pholcus/app/downloader/context" | ||
"github.com/henrylee2cn/pholcus/config" | ||
"github.com/henrylee2cn/surfer" | ||
) | ||
|
||
type Surfer struct { | ||
surf surfer.Surfer | ||
phantom surfer.Surfer | ||
} | ||
|
||
var SurferDownloader = &Surfer{ | ||
surf: surfer.New(), | ||
phantom: surfer.NewPhantom(config.SURFER_PHANTOM.FULL_APP_NAME, config.SURFER_PHANTOM.FULL_TEMP_JS), | ||
} | ||
|
||
func (self *Surfer) Download(cReq *context.Request) *context.Response { | ||
cResp := context.NewResponse(nil) | ||
|
||
var resp *http.Response | ||
var err error | ||
|
||
if cReq.GetUsePhantomJS() { | ||
resp, err = self.phantom.Download(cReq) | ||
} else { | ||
resp, err = self.surf.Download(cReq) | ||
} | ||
|
||
cResp.SetRequest(cReq) | ||
|
||
cResp.SetResponse(resp) | ||
|
||
cResp.SetError(err) | ||
|
||
return cResp | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.