Behind firewall- Crawl a web page

This below program helps to crawl a web page,especially if we are behind the firewall..

Proxy.py

import urllib2
#Browser proxy setting
proxy_info = {
‘user’ : ‘vignesh_v’,
‘pass’ : ‘passwd’,
‘host’ : “10.16.65.13”,
‘port’ : 80
}

# build a new opener that uses a proxy requiring authorization
proxy_support = urllib2.ProxyHandler({“http” : \
“http://%(user)s:%(pass)s@%(host)s:%(port)d” % proxy_info})
opener = urllib2.build_opener(proxy_support, urllib2.HTTPHandler)
# Install the opener
urllib2.install_opener(opener)
#Open the url
f = urllib2.urlopen(‘http://www.python.org/’)
print f.headers
print f.read()

Advertisements