asp.net c#采集需要登錄頁面的實現(xiàn)原理及代碼

更新時間：2013年02月01日 16:39:02 作者：

當我們采集頁面的時候，如果被采集的網(wǎng)站需要登錄才能采集，原理搞清楚了，就好辦了，我們所要做的僅僅是在采集的時候（或者說HttpWebRequest提交數(shù)據(jù)的時候），將Cookie信息放入Http請求頭里面就可以了，感興趣的朋友可以了解下，或許對你有所幫助

首先說明：代碼片段是從網(wǎng)絡獲取，然后自己修改。我想好的東西應該拿來分享。

實現(xiàn)原理：當我們采集頁面的時候，如果被采集的網(wǎng)站需要登錄才能采集。不管是基于Cookie還是基于Session，我們都會首先發(fā)送一個Http請求頭，這個Http請求頭里面就包含了網(wǎng)站需要的Cookie信息。當網(wǎng)站接收到發(fā)送過來的Http請求頭時，會從Http請求頭獲取相關的Cookie或者Session信息，然后由程序來處理，決定你是否有權限訪問當前頁面。

好了，原理搞清楚了，就好辦了。我們所要做的僅僅是在采集的時候（或者說HttpWebRequest提交數(shù)據(jù)的時候），將Cookie信息放入Http請求頭里面就可以了。

在這里我提供2種方法。
第一種，直接將Cookie信息放入HttpWebRequest的CookieContainer里?？创a：

復制代碼代碼如下:

 
protected void Page_Load(object sender, EventArgs e) 
{ 
//設置Cookie，存入Hashtable 
Hashtable ht = new Hashtable(); 
ht.Add("username", "youraccount"); 
ht.Add("id", "yourid"); 
this.Collect(ht); 
} 
public void Collect(Hashtable ht) 
{ 
string content = string.Empty; 
string url = "http://www.ibest100.com/需要登錄后才能采集的頁面"; 
string host = "http://www.ibest100.com"; 
try 
{ 
//獲取提交的字節(jié) 
byte[] bs = Encoding.UTF8.GetBytes(content); 
//設置提交的相關參數(shù) 
HttpWebRequest req = (HttpWebRequest)HttpWebRequest.Create(url); 
req.Method = "POST"; 
req.ContentType = "application/json;charset=utf-8"; 
req.ContentLength = bs.Length; 
//將Cookie放入CookieContainer，然后再將CookieContainer添加到HttpWebRequest 
CookieContainer cc = new CookieContainer(); 
cc.Add(new Uri(host), new Cookie("username", ht["username"].ToString())); 
cc.Add(new Uri(host), new Cookie("id", ht["id"].ToString())); 
req.CookieContainer = cc; 
//提交請求數(shù)據(jù) 
Stream reqStream = req.GetRequestStream(); 
reqStream.Write(bs, 0, bs.Length); 
reqStream.Close(); 
//接收返回的頁面，必須的，不能省略 
WebResponse wr = req.GetResponse(); 
System.IO.Stream respStream = wr.GetResponseStream(); 
System.IO.StreamReader reader = new System.IO.StreamReader(respStream, System.Text.Encoding.GetEncoding("utf-8")); 
string t = reader.ReadToEnd(); 
System.Web.HttpContext.Current.Response.Write(t); 
wr.Close(); 
} 
catch (Exception ex) 
{ 
System.Web.HttpContext.Current.Response.Write("異常在getPostRespone:" + ex.Source + ":" + ex.Message); 
} 
} 

第二種，每次打開采集程序時，需要先到被采集的網(wǎng)站模擬登錄一次，獲取CookieContainer，然后再采集?？创a：

復制代碼代碼如下:

 
protected void Page_Load(object sender, EventArgs e) 
{ 
try 
{ 
CookieContainer cookieContainer = new CookieContainer(); 
string formatString = "username={0}&password={1}";//*************** 
string postString = string.Format(formatString, "youradminaccount", "yourpassword"); 
//將提交的字符串數(shù)據(jù)轉(zhuǎn)換成字節(jié)數(shù)組 
byte[] postData = Encoding.UTF8.GetBytes(postString); 
//設置提交的相關參數(shù) 
string URI = "http://www.ibest100.com/登錄頁面";//*************** 
HttpWebRequest request = WebRequest.Create(URI) as HttpWebRequest; 
request.Method = "POST"; 
request.KeepAlive = false; 
request.ContentType = "application/x-www-form-urlencoded"; 
request.CookieContainer = cookieContainer; 
request.ContentLength = postData.Length; 
// 提交請求數(shù)據(jù) 
System.IO.Stream outputStream = request.GetRequestStream(); 
outputStream.Write(postData, 0, postData.Length); 
outputStream.Close(); 
//接收返回的頁面，必須的，不能省略 
HttpWebResponse response = request.GetResponse() as HttpWebResponse; 
System.IO.Stream responseStream = response.GetResponseStream(); 
System.IO.StreamReader reader = new System.IO.StreamReader(responseStream, Encoding.GetEncoding("gb2312")); 
string srcString = reader.ReadToEnd(); 
//打開您要訪問的頁面 
URI = "http://www.ibest100.com/需要登錄后才能采集的頁面";//*************** 
request = WebRequest.Create(URI) as HttpWebRequest; 
request.Method = "GET"; 
request.KeepAlive = false; 
request.CookieContainer = cookieContainer; 
// 接收返回的頁面 
response = request.GetResponse() as HttpWebResponse; 
responseStream = response.GetResponseStream(); 
reader = new System.IO.StreamReader(responseStream, Encoding.GetEncoding("gb2312")); 
srcString = reader.ReadToEnd(); 
//輸出獲取的頁面或者處理 
Response.Write(srcString); 
} 
catch (WebException we) 
{ 
string msg = we.Message; 
Response.Write(msg); 
} 
} 