asp.net c#采集需要登錄頁(yè)面的實(shí)現(xiàn)原理及代碼
更新時(shí)間:2013年02月01日 16:39:02 作者:
當(dāng)我們采集頁(yè)面的時(shí)候,如果被采集的網(wǎng)站需要登錄才能采集,原理搞清楚了,就好辦了,我們所要做的僅僅是在采集的時(shí)候(或者說(shuō)HttpWebRequest提交數(shù)據(jù)的時(shí)候),將Cookie信息放入Http請(qǐng)求頭里面就可以了,感興趣的朋友可以了解下,或許對(duì)你有所幫助
首先說(shuō)明:代碼片段是從網(wǎng)絡(luò)獲取,然后自己修改。我想好的東西應(yīng)該拿來(lái)分享。
實(shí)現(xiàn)原理:當(dāng)我們采集頁(yè)面的時(shí)候,如果被采集的網(wǎng)站需要登錄才能采集。不管是基于Cookie還是基于Session,我們都會(huì)首先發(fā)送一個(gè)Http請(qǐng)求頭,這個(gè)Http請(qǐng)求頭里面就包含了網(wǎng)站需要的Cookie信息。當(dāng)網(wǎng)站接收到發(fā)送過(guò)來(lái)的Http請(qǐng)求頭時(shí),會(huì)從Http請(qǐng)求頭獲取相關(guān)的Cookie或者Session信息,然后由程序來(lái)處理,決定你是否有權(quán)限訪問(wèn)當(dāng)前頁(yè)面。
好了,原理搞清楚了,就好辦了。我們所要做的僅僅是在采集的時(shí)候(或者說(shuō)HttpWebRequest提交數(shù)據(jù)的時(shí)候),將Cookie信息放入Http請(qǐng)求頭里面就可以了。
在這里我提供2種方法。
第一種,直接將Cookie信息放入HttpWebRequest的CookieContainer里??创a:
protected void Page_Load(object sender, EventArgs e)
{
//設(shè)置Cookie,存入Hashtable
Hashtable ht = new Hashtable();
ht.Add("username", "youraccount");
ht.Add("id", "yourid");
this.Collect(ht);
}
public void Collect(Hashtable ht)
{
string content = string.Empty;
string url = "http://www.ibest100.com/需要登錄后才能采集的頁(yè)面";
string host = "http://www.ibest100.com";
try
{
//獲取提交的字節(jié)
byte[] bs = Encoding.UTF8.GetBytes(content);
//設(shè)置提交的相關(guān)參數(shù)
HttpWebRequest req = (HttpWebRequest)HttpWebRequest.Create(url);
req.Method = "POST";
req.ContentType = "application/json;charset=utf-8";
req.ContentLength = bs.Length;
//將Cookie放入CookieContainer,然后再將CookieContainer添加到HttpWebRequest
CookieContainer cc = new CookieContainer();
cc.Add(new Uri(host), new Cookie("username", ht["username"].ToString()));
cc.Add(new Uri(host), new Cookie("id", ht["id"].ToString()));
req.CookieContainer = cc;
//提交請(qǐng)求數(shù)據(jù)
Stream reqStream = req.GetRequestStream();
reqStream.Write(bs, 0, bs.Length);
reqStream.Close();
//接收返回的頁(yè)面,必須的,不能省略
WebResponse wr = req.GetResponse();
System.IO.Stream respStream = wr.GetResponseStream();
System.IO.StreamReader reader = new System.IO.StreamReader(respStream, System.Text.Encoding.GetEncoding("utf-8"));
string t = reader.ReadToEnd();
System.Web.HttpContext.Current.Response.Write(t);
wr.Close();
}
catch (Exception ex)
{
System.Web.HttpContext.Current.Response.Write("異常在getPostRespone:" + ex.Source + ":" + ex.Message);
}
}
第二種,每次打開(kāi)采集程序時(shí),需要先到被采集的網(wǎng)站模擬登錄一次,獲取CookieContainer,然后再采集??创a:
protected void Page_Load(object sender, EventArgs e)
{
try
{
CookieContainer cookieContainer = new CookieContainer();
string formatString = "username={0}&password={1}";//***************
string postString = string.Format(formatString, "youradminaccount", "yourpassword");
//將提交的字符串?dāng)?shù)據(jù)轉(zhuǎn)換成字節(jié)數(shù)組
byte[] postData = Encoding.UTF8.GetBytes(postString);
//設(shè)置提交的相關(guān)參數(shù)
string URI = "http://www.ibest100.com/登錄頁(yè)面";//***************
HttpWebRequest request = WebRequest.Create(URI) as HttpWebRequest;
request.Method = "POST";
request.KeepAlive = false;
request.ContentType = "application/x-www-form-urlencoded";
request.CookieContainer = cookieContainer;
request.ContentLength = postData.Length;
// 提交請(qǐng)求數(shù)據(jù)
System.IO.Stream outputStream = request.GetRequestStream();
outputStream.Write(postData, 0, postData.Length);
outputStream.Close();
//接收返回的頁(yè)面,必須的,不能省略
HttpWebResponse response = request.GetResponse() as HttpWebResponse;
System.IO.Stream responseStream = response.GetResponseStream();
System.IO.StreamReader reader = new System.IO.StreamReader(responseStream, Encoding.GetEncoding("gb2312"));
string srcString = reader.ReadToEnd();
//打開(kāi)您要訪問(wèn)的頁(yè)面
URI = "http://www.ibest100.com/需要登錄后才能采集的頁(yè)面";//***************
request = WebRequest.Create(URI) as HttpWebRequest;
request.Method = "GET";
request.KeepAlive = false;
request.CookieContainer = cookieContainer;
// 接收返回的頁(yè)面
response = request.GetResponse() as HttpWebResponse;
responseStream = response.GetResponseStream();
reader = new System.IO.StreamReader(responseStream, Encoding.GetEncoding("gb2312"));
srcString = reader.ReadToEnd();
//輸出獲取的頁(yè)面或者處理
Response.Write(srcString);
}
catch (WebException we)
{
string msg = we.Message;
Response.Write(msg);
}
}
也許有人會(huì)問(wèn),如果對(duì)方登錄的時(shí)候要驗(yàn)證碼怎么辦?那你就用第一種方式吧,只不過(guò)需要你分析對(duì)方的Cookie。
應(yīng)用范圍:采集數(shù)據(jù)、論壇發(fā)帖、博客發(fā)文。
實(shí)現(xiàn)原理:當(dāng)我們采集頁(yè)面的時(shí)候,如果被采集的網(wǎng)站需要登錄才能采集。不管是基于Cookie還是基于Session,我們都會(huì)首先發(fā)送一個(gè)Http請(qǐng)求頭,這個(gè)Http請(qǐng)求頭里面就包含了網(wǎng)站需要的Cookie信息。當(dāng)網(wǎng)站接收到發(fā)送過(guò)來(lái)的Http請(qǐng)求頭時(shí),會(huì)從Http請(qǐng)求頭獲取相關(guān)的Cookie或者Session信息,然后由程序來(lái)處理,決定你是否有權(quán)限訪問(wèn)當(dāng)前頁(yè)面。
好了,原理搞清楚了,就好辦了。我們所要做的僅僅是在采集的時(shí)候(或者說(shuō)HttpWebRequest提交數(shù)據(jù)的時(shí)候),將Cookie信息放入Http請(qǐng)求頭里面就可以了。
在這里我提供2種方法。
第一種,直接將Cookie信息放入HttpWebRequest的CookieContainer里??创a:
復(fù)制代碼 代碼如下:
protected void Page_Load(object sender, EventArgs e)
{
//設(shè)置Cookie,存入Hashtable
Hashtable ht = new Hashtable();
ht.Add("username", "youraccount");
ht.Add("id", "yourid");
this.Collect(ht);
}
public void Collect(Hashtable ht)
{
string content = string.Empty;
string url = "http://www.ibest100.com/需要登錄后才能采集的頁(yè)面";
string host = "http://www.ibest100.com";
try
{
//獲取提交的字節(jié)
byte[] bs = Encoding.UTF8.GetBytes(content);
//設(shè)置提交的相關(guān)參數(shù)
HttpWebRequest req = (HttpWebRequest)HttpWebRequest.Create(url);
req.Method = "POST";
req.ContentType = "application/json;charset=utf-8";
req.ContentLength = bs.Length;
//將Cookie放入CookieContainer,然后再將CookieContainer添加到HttpWebRequest
CookieContainer cc = new CookieContainer();
cc.Add(new Uri(host), new Cookie("username", ht["username"].ToString()));
cc.Add(new Uri(host), new Cookie("id", ht["id"].ToString()));
req.CookieContainer = cc;
//提交請(qǐng)求數(shù)據(jù)
Stream reqStream = req.GetRequestStream();
reqStream.Write(bs, 0, bs.Length);
reqStream.Close();
//接收返回的頁(yè)面,必須的,不能省略
WebResponse wr = req.GetResponse();
System.IO.Stream respStream = wr.GetResponseStream();
System.IO.StreamReader reader = new System.IO.StreamReader(respStream, System.Text.Encoding.GetEncoding("utf-8"));
string t = reader.ReadToEnd();
System.Web.HttpContext.Current.Response.Write(t);
wr.Close();
}
catch (Exception ex)
{
System.Web.HttpContext.Current.Response.Write("異常在getPostRespone:" + ex.Source + ":" + ex.Message);
}
}
第二種,每次打開(kāi)采集程序時(shí),需要先到被采集的網(wǎng)站模擬登錄一次,獲取CookieContainer,然后再采集??创a:
復(fù)制代碼 代碼如下:
protected void Page_Load(object sender, EventArgs e)
{
try
{
CookieContainer cookieContainer = new CookieContainer();
string formatString = "username={0}&password={1}";//***************
string postString = string.Format(formatString, "youradminaccount", "yourpassword");
//將提交的字符串?dāng)?shù)據(jù)轉(zhuǎn)換成字節(jié)數(shù)組
byte[] postData = Encoding.UTF8.GetBytes(postString);
//設(shè)置提交的相關(guān)參數(shù)
string URI = "http://www.ibest100.com/登錄頁(yè)面";//***************
HttpWebRequest request = WebRequest.Create(URI) as HttpWebRequest;
request.Method = "POST";
request.KeepAlive = false;
request.ContentType = "application/x-www-form-urlencoded";
request.CookieContainer = cookieContainer;
request.ContentLength = postData.Length;
// 提交請(qǐng)求數(shù)據(jù)
System.IO.Stream outputStream = request.GetRequestStream();
outputStream.Write(postData, 0, postData.Length);
outputStream.Close();
//接收返回的頁(yè)面,必須的,不能省略
HttpWebResponse response = request.GetResponse() as HttpWebResponse;
System.IO.Stream responseStream = response.GetResponseStream();
System.IO.StreamReader reader = new System.IO.StreamReader(responseStream, Encoding.GetEncoding("gb2312"));
string srcString = reader.ReadToEnd();
//打開(kāi)您要訪問(wèn)的頁(yè)面
URI = "http://www.ibest100.com/需要登錄后才能采集的頁(yè)面";//***************
request = WebRequest.Create(URI) as HttpWebRequest;
request.Method = "GET";
request.KeepAlive = false;
request.CookieContainer = cookieContainer;
// 接收返回的頁(yè)面
response = request.GetResponse() as HttpWebResponse;
responseStream = response.GetResponseStream();
reader = new System.IO.StreamReader(responseStream, Encoding.GetEncoding("gb2312"));
srcString = reader.ReadToEnd();
//輸出獲取的頁(yè)面或者處理
Response.Write(srcString);
}
catch (WebException we)
{
string msg = we.Message;
Response.Write(msg);
}
}
也許有人會(huì)問(wèn),如果對(duì)方登錄的時(shí)候要驗(yàn)證碼怎么辦?那你就用第一種方式吧,只不過(guò)需要你分析對(duì)方的Cookie。
應(yīng)用范圍:采集數(shù)據(jù)、論壇發(fā)帖、博客發(fā)文。
您可能感興趣的文章:
- C# Winform中實(shí)現(xiàn)主窗口打開(kāi)登錄窗口關(guān)閉的方法
- C#實(shí)現(xiàn)簡(jiǎn)單的登錄界面
- div彈出層的ajax登錄(Jquery版+c#)
- C#.NET實(shí)現(xiàn)網(wǎng)頁(yè)自動(dòng)登錄的方法
- C#實(shí)現(xiàn)的三種模擬自動(dòng)登錄和提交POST信息的方法
- C#實(shí)現(xiàn)登錄窗口(不用隱藏)
- .NET C#使用微信公眾號(hào)登錄網(wǎng)站
- C#中登錄窗體和歡迎窗體關(guān)閉方法分析
- c#通過(guò)進(jìn)程調(diào)用cmd判斷登錄用戶權(quán)限代碼分享
- c#通用登錄模塊分享
- C#實(shí)現(xiàn)的WINDOWS登錄功能示例
相關(guān)文章
ASP.NET仿新浪微博下拉加載更多數(shù)據(jù)瀑布流效果
本篇文章介紹了如何實(shí)現(xiàn)下拉加載更多數(shù)據(jù)瀑布流的效果,這種效果最近很流行,文中示例代碼介紹的非常詳細(xì),具有一定的參考價(jià)值,感興趣的小伙伴們可以參考一下2015-07-07ASP.NET單選按鈕控件RadioButton常用屬性和方法介紹
RadioButton又稱(chēng)單選按鈕,其在工具箱中的圖標(biāo)為 ,單選按鈕通常成組出現(xiàn),用于提供兩個(gè)或多個(gè)互斥選項(xiàng),即在一組單選鈕中只能選擇一個(gè)2014-04-04Entity?Framework生成DataBase?First模式
本文詳細(xì)講解了Entity?Framework生成DataBase?First模式的方法,文中通過(guò)示例代碼介紹的非常詳細(xì)。對(duì)大家的學(xué)習(xí)或工作具有一定的參考借鑒價(jià)值,需要的朋友可以參考下2022-03-03asp.net 關(guān)于==?:和if()else()條件判斷等效例子
關(guān)于==?:和if()else() 等效例子2010-03-03使用.net core3.0 正式版創(chuàng)建Winform程序的方法(圖文)
這篇文章主要介紹了使用.net core3.0 正式版創(chuàng)建Winform程序的方法,文中通過(guò)示例代碼介紹的非常詳細(xì),對(duì)大家的學(xué)習(xí)或者工作具有一定的參考學(xué)習(xí)價(jià)值,需要的朋友們下面隨著小編來(lái)一起學(xué)習(xí)學(xué)習(xí)吧2020-03-03