面包会有的

... ...

日志

关于我

加菲

覆巢之下焉有完卵。出来道上混,迟早是要还的。

文章分类

采集需要登陆的内容,即破session

2008-08-06 15:56:31| 分类：默认分类 | 标签： |举报 |字号大中小订阅

下载LOFTER 我的照片书 |

1.在下载数据的开始,有些网站是要登录了才能看到相应的数据,这个就需要我们发送登录用户名和密码了,但我是登录了,但他服务器也不是垃圾,在他那里重定向了,共产生了2个SESSION,这第2个SESSION我就不知道如何捕抓.于是我就投机^-^,用软件将SESSION捕抓下来了1个叫Ethereal的软件,用以下代码加入到HTTP请求的头部

WebClient myWebClient = new WebClient();

string sessionkey=textBox78.Text;

string refererurl=textBox77.Text;

myWebClient.Headers.Clear();

myWebClient.Headers.Add("Cookie",sessionkey);

myWebClient.Headers.Add("Referer", refererurl);

myWebClient.Headers.Add("User-agent", "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.5) Gecko/20031107 Debian/1.5-3");

这样就欺骗了服务器了,哈哈

2.第二部就是代码下载

byte[] myDataBuffer = myWebClient.DownloadData(remoteUri);

download = Encoding.Default.GetString(myDataBuffer);

3.第3部就是数据的匹配了,我是将流读取到数据里,然后用IndexOf得到2个关键字段的位置,然后用Substring取出来的,我知道这很笨,但用正则表达式难啊(谁会的指点我下),匹配完了得到的字符串我就用以下的函数去掉了HTML代码:

private string StripHTML(string strHtml)

{

string [] aryReg ={

@"<script[^>]*?>.*?</script>",

@"<(\/\s*)?!?((\w+:)?\w+)(\w+(\s*=?\s*(([""'])(\\[""'tbnr]|[^\7])*?\7|\w+)|.{0})|\s)*?(\/\s*)?>",

@"([\r\n])[\s]+",

@"&(quot|#34);",

@"&(amp|#38);",

@"&(lt|#60);",

@"&(gt|#62);",

@"&(nbsp|#160);",

@"&(iexcl|#161);",

@"&(cent|#162);",

@"&(pound|#163);",

@"&(copy|#169);",

@"&#(\d+);",

@"-->",

@"<!--.*\n"

};

string [] aryRep = {

"",

"\"",

"&",

"<",

">",

" ",

"\xa1",//chr(161),

"\xa2",//chr(162),

"\xa3",//chr(163),

"\xa9",//chr(169),

"",

"\r\n",

""

};

string newReg =aryReg[0];

string strOutput=strHtml;

for(int i = 0;i<aryReg.Length;i++)

{

Regex regex = new Regex(aryReg[i],RegexOptions.IgnoreCase );

strOutput = regex.Replace(strOutput,aryRep[i]);

}

strOutput.Replace("<","");

strOutput.Replace(">","");

strOutput.Replace("\r\n","");

return strOutput;

}

4.到了后面就是入库了,这个大家都懂了吧.但是我还有点问题就是,在我写数据的时候,出了EXCEPTION,说我的字段太长了,不能写进到数据库,我用的是ACCESS,我试验下用SQL吧.

5.大家有什么好的建议给我留个言赛.共同进步嘛.

评论这张

转发至微博

阅读(895)| 评论(1)

历史上的今天

this.p={  m:2,
              b:2,
              loftPermalink:'',
              id:'fks_084074093080084066093083085095087094083071081081087071',
              blogTitle:'采集需要登陆的内容,即破session',
              blogAbstract:'<P style=\"TEXT-INDENT: 2em;\"  \>1.在下载数据的开始,有些网站是要登录了才能看到相应的数据,这个就需要我们发送登录用户名和密码了,但我是登录了,但他服务器也不是垃圾,在他那里重定向了,共产生了2个SESSION,这第2个SESSION我就不知道如何捕抓.于是我就投机^-^,用软件将SESSION捕抓下来了1个叫Ethereal的软件,用以下代码加入到HTTP请求的头部</P\>  <P style=\"TEXT-INDENT: 2em;\"  \>WebClient myWebClient = new WebClient();</P\>  <P style=\"TEXT-INDENT: 2em;\"  \>string sessionkey=textBox78.Text;</P\>  <P style=\"TEXT-INDENT: 2em;\"  \>     string refererurl=textBox77.Text;</P\>',
              blogTag:'',
              blogUrl:'blog/static/3824571420087635631211',
              isPublished:1,
              istop:false,
              type:2,
              modifyTime:1317372324063,
              publishTime:1218009391211,
              permalink:'blog/static/3824571420087635631211',
              commentCount:1,
              mainCommentCount:1,
              recommendCount:0,
              bsrk:-100,
              publisherId:0,
              recomBlogHome:false,
              currentRecomBlog:false,
              attachmentsFileIds:[],
              vote:{},
              groupInfo:{},
              friendstatus:'none',
              followstatus:'unFollow',
              pubSucc:'',
              visitorProvince:'',
              visitorCity:'',
              visitorNewUser:false,
              postAddInfo:{},
              mset:'000',
              mcon:'',
              srk:-100,
              remindgoodnightblog:false,
              isBlackVisitor:false,
              isShowYodaoAd:true,
              hostIntro:'覆巢之下焉有完卵。\n出来道上混,迟早是要还的。',
              hmcon:'1',
              selfRecomBlogCount:'0',
              lofter_single:'<iframe width="140" height="560" style="overflow:hidden;" src="http://www.lofter.com/mailEntry.do?blogad=1&blog" frameBorder="0"></iframe>'
            }

{list a as x}
    {if !!x}
    <div class="iblock nbw-fce nbw-f40">
      <a class="fc03 noul" target="_blank" hidefocus="true" href="http://blog.163.com/${x.visitorName}/">
      {if x.visitorName==visitor.userName}
      <img alt="${x.visitorNickname|escape}" onerror="this.src=location.f40" class="cwd bdwa bdc0" src="${fn1(x.visitorName)}&r=${visitor.imageUpdateTime}"/>
      {else}
      <img alt="${x.visitorNickname|escape}" onerror="this.src=location.f40" class="cwd bdwa bdc0" src="${fn1(x.visitorName)}"/>
      {/if}
      </a>
      <div class="cwd vname thide">
        {if x.moveFrom=='wap'}
          <a class="noul pnt" target="_blank" href="http://blog.163.com/services/wapblog.html?frompersonalbloghome"><span title="来自网易手机博客" class="iblock wapIcon"> </span></a>
        {elseif x.moveFrom=='iphone'}
          <a class="noul pnt" target="_blank"><span title="来自iPhone客户端" class="iblock iphoneIcon"> </span></a>
        {elseif x.moveFrom=='android'}
          <a class="noul pnt" target="_blank"><span title="来自Android客户端" class="iblock androidIcon"> </span></a>
        {elseif x.moveFrom=='mobile'}
          <a class="noul pnt" target="_blank" href="http://blog.163.com/services/emsblog.html?frompersonalbloghome"><span title="来自网易短信写博" class="iblock wapIcon"> </span></a>
        {/if}
        <a class="fc03 m2a"  target="_blank" hidefocus="true" href="http://blog.163.com/${x.visitorName}/">
          ${fn(x.visitorNickname,8)|escape}
        </a>
      </div>
    </div>
    {/if}
    {/list}

<#--最新日志，群博日志--> <#--推荐日志-->

<p class="fc06">推荐过这篇日志的人：</p>
    <div>
      {list a as x}
      {if !!x}
      <div class="iblock nbw-fce nbw-f40">
        <a class="fc03 noul" target="_blank" hidefocus="true" href="http://blog.163.com/${x.recommenderName}/">
        <img alt="${x.recommenderNickname|escape}" onerror="this.src=location.f40" class="cwd bdwa bdc0" src="${fn1(x.recommenderName)}"/>
        </a>
        <div class="cwd thide">
          <a class="fc03 m2a" target="_blank" hidefocus="true" href="http://blog.163.com/${x.recommenderName}/">
            ${fn(x.recommenderNickname,6)|escape}
          </a>
        </div>
      </div>
      {/if}
      {/list}
    </div>
    {if !!b&&b.length>0}
    <p  class="fc06">他们还推荐了：</p>
    <ul>
    {list b as y}
      {if !!y}
        <li class="rrb"><span class="iblock">·</span><a class="fc03 m2a" target="_blank" href="http://blog.163.com/${y.recommendBlogPermalink}/?from=blog/static/3824571420087635631211">${y.recommendBlogTitle|escape}</a></li>
      {/if}
    {/list}
    </ul>
    {/if}

<#--引用记录--> <#--博主推荐--> <#--随机阅读--> <#--首页推荐--> <#--历史上的今天--> <#--被推荐日志--> <#--上一篇，下一篇--> <#-- 热度 -->

{list a as x}
    {if !!x}
    <div class="hotItem iblock nbw-fce nbw-f40">
      <a class="fc03 noul" target="_blank" hidefocus="true" href="http://blog.163.com/${x.publisherUsername}/">
      {if x.publisherUsername==visitor.userName}
      <img alt="${x.publisherNickname|escape}" onerror="this.src=location.f40" class="cwd bdwa bdc0" src="${fn1(x.publisherUsername)}&r=${visitor.imageUpdateTime}"/>
      {else}
      <img alt="${x.publisherNickname|escape}" onerror="this.src=location.f40" class="cwd bdwa bdc0" src="${fn1(x.publisherUsername)}"/>
      {/if}
      </a>
      <div class="cwd vname thide">
        <a class="fc03 m2a"  target="_blank" hidefocus="true" href="http://blog.163.com/${x.publisherUsername}/">
          ${fn(x.publisherNickname,8)|escape}
        </a>
      </div>
      <a class="f-myLikeIcons hottype {if x.type==1} js-liketype{elseif x.type==2} js-reblogtype{elseif x.type==3} js-sharetype{else}{/if}" target="_blank" hidefocus="true" href="http://blog.163.com/${x.publisherUsername}/"> </a>
    </div>
    {/if}
    {/list}

<#-- 网易新闻广告 --> <#--右边模块结构--> <#--评论模块结构--> <#--引用模块结构--> <#--博主发起的投票-->

页脚

我的照片书 - 手机博客 - 下载LOFTER APP - 订阅此博客

面包会有的

导航

日志

采集需要登陆的内容,即破session

历史上的今天

最近读者

热度

评论

页脚