urllib.request

Collapse
X
 
  • Time
  • Show
Clear All
new posts
  • gintare
    New Member
    • Mar 2007
    • 103

    urllib.request

    I do not understand what additional parameters i have to pas to urllib.request to download a full webpage.

    1)

    For instance, if you open
    http://www.vocabulary. com/dictionary/reiterate
    on the right bottom you will find "usage examples".

    If you try to save wabpage and than reopen it:
    file:///C:/Users/Gintare/Downloads/reiterate%20-%20Dictionary%2 0Definition%20% 20%20Vocabulary .com.htm
    the "usage examples" will be displayed.

    But if you use:
    Code:
    res="http://www.vocabulary.com/dictionary/reiterate"
    resp = urllib.request.urlopen(res)
    html = resp.read()             
    soup=BeautifulSoup(html)                          shtml=BeautifulSoup.prettify(soup)                           fFI=open('requested_file.html','w',encoding='utf-8')
    fFI.writelines(shtml)
    fFI.close()
    The 'requested_file .html' will not have the section "usage examples".

    How to download full webpage with the section "usage examples" using urllib.request?

    2) How to retrieve other examples, which normally are accessible by pressing hyperlink "next". There are more than 40 examples for this word, but hey are displayed only in portions by 4.
  • numberwhun
    Recognized Expert Moderator Specialist
    • May 2007
    • 3467

    #2
    Have you tried simply using:

    Code:
    urllib.urlopen
    Take out the 'request' portion and try that instead.

    Comment

    • gintare
      New Member
      • Mar 2007
      • 103

      #3
      I am getting error:
      AttributeError at /search1

      'module' object has no attribute 'urlopen'

      Request Method: GET
      Request URL: http://127.0.0.1:8000/search1
      Django Version: 1.6.2
      Exception Type: AttributeError
      Exception Value:


      on line 1000: resp = urllib.urlopen( res)

      Comment

      Working...