{"generator":"Jekyll","link":[{"@attributes":{"href":"https:\/\/tatianashavrina.github.io\/\/feed.xml","rel":"self","type":"application\/atom+xml"}},{"@attributes":{"href":"https:\/\/tatianashavrina.github.io\/\/","rel":"alternate","type":"text\/html","hreflang":"en, ru"}}],"updated":"2022-08-10T10:53:00+00:00","id":"https:\/\/tatianashavrina.github.io\/\/feed.xml","title":"Natural Language Processing","subtitle":"Tatiana Shavrina's blog about NLP\n","author":{"name":"Tatiana Shavrina","email":""},"entry":[{"title":"How to Make a Youtube Comment Corpus (without API)","link":{"@attributes":{"href":"https:\/\/tatianashavrina.github.io\/\/2020\/01\/08\/youtube\/","rel":"alternate","type":"text\/html","title":"How to Make a Youtube Comment Corpus (without API)"}},"published":"2020-01-08T00:00:00+00:00","updated":"2020-01-08T00:00:00+00:00","id":"https:\/\/tatianashavrina.github.io\/\/2020\/01\/08\/youtube","content":"

All ways to do it<\/strong><\/p>\n\n

Of course, Youtube has its own useful API<\/a><\/p>\n\n

But if you are reading this article, you probably already know it has strict limitations, including the number of comments collected per day.<\/p>\n\n

\"\"<\/p>\n\n

If you are a researcher who would like to collect a small dataset, it is possible to reproduce all the manual behavour in a way described below.<\/p>\n\n

Collecting Comments<\/h2>\n\n

Since we aren\u2019t using proposed functionality, the script should pretend it is a user scrolling through the trends.<\/p>\n\n

Our solution will be based on Selenium + CSSSelector<\/strong> - let\u2019s start!<\/p>\n\n

import<\/span> os<\/span>,<\/span> sys<\/span>,<\/span> time<\/span>,<\/span> datetime<\/span>\nimport<\/span> json<\/span>\nimport<\/span> requests<\/span>\nimport<\/span> lxml.html<\/span>\nimport<\/span> io<\/span>,<\/span> codecs<\/span>\nfrom<\/span> tqdm<\/span> import<\/span> tqdm<\/span>\nfrom<\/span> lxml.cssselect<\/span> import<\/span> CSSSelector<\/span>\n#to make the Browser Working\n<\/span>from<\/span> selenium<\/span> import<\/span> webdriver<\/span>\n#Send keycodes to Elements\n<\/span>from<\/span> selenium.webdriver.common.keys<\/span> import<\/span> Keys<\/span>\n#scrape the url's and comments\n<\/span>from<\/span> bs4<\/span> import<\/span> BeautifulSoup<\/span>\n<\/code><\/pre><\/div><\/div>\n\n
Selenium - rolling through trends<\/h3>\n\n
Firstly, we will scroll the trends page with Selenium. \nYou should have Firefox installed to use the code below \u2013 or change for your browser<\/a>.<\/p>\n\n
# The List where the links to the videos are stored\n<\/span>links<\/span> =<\/span> set<\/span>()<\/span>\ncomments<\/span> =<\/span> list<\/span>()<\/span>\nhomePage<\/span> =<\/span> 'https:www.youtube.com'<\/span>\nlinksSize<\/span> =<\/span> 10<\/span>\ndriver<\/span> =<\/span> webdriver<\/span>.<\/span>Firefox<\/span>()<\/span>\noutput_path<\/span> =<\/span> os<\/span>.<\/span>path<\/span>.<\/span>join<\/span>(<\/span>\"\/media\/user\/youtube\/\"<\/span>,<\/span> str<\/span>(<\/span>datetime<\/span>.<\/span>date<\/span>.<\/span>today<\/span>()),<\/span>\".txt\"<\/span>)<\/span>\n \ndef<\/span> loadFullPage<\/span>(<\/span>Timeout<\/span>):<\/span>\n    reachedbottom<\/span> =<\/span> None<\/span>\n    while<\/span> not<\/span> reachedbottom<\/span>:<\/span>\n        #scroll one page down\n<\/span>        driver<\/span>.<\/span>execute_script<\/span>(<\/span>\"window.scrollTo(0,Math.max(document.documentElement.scrollHeight,document.body.scrollHeight,document.documentElement.clientHeight));\"<\/span>);<\/span>\n        time<\/span>.<\/span>sleep<\/span>(<\/span>Timeout<\/span>)<\/span>\n        #check if the bottom is reached\n<\/span>        a<\/span> =<\/span> driver<\/span>.<\/span>execute_script<\/span>(<\/span>\"return document.documentElement.scrollTop;\"<\/span>)<\/span>\n        b<\/span> =<\/span> driver<\/span>.<\/span>execute_script<\/span>(<\/span>\"return document.documentElement.scrollHeight - document.documentElement.clientHeight;\"<\/span>)<\/span>\n        relativeHeight<\/span> =<\/span> a<\/span> \/<\/span> b<\/span>\n        if<\/span>(<\/span>relativeHeight<\/span>==<\/span>1<\/span>):<\/span>\n            reachedbottom<\/span> =<\/span> True<\/span>\ndef<\/span> getComments<\/span>(<\/span>link<\/span>):<\/span>\n    driver<\/span>.<\/span>get<\/span>(<\/span>url<\/span>=<\/span>'https:youtube.com'<\/span>+<\/span>link<\/span>)<\/span>\n    loadFullPage<\/span>(<\/span>1<\/span>)<\/span>\n\ndef<\/span> main<\/span>():<\/span>\n    driver<\/span>.<\/span>get<\/span>(<\/span>url<\/span>=<\/span>homePage<\/span>)<\/span>\n    enoughLinks<\/span> =<\/span> None<\/span>\n    while<\/span> not<\/span> enoughLinks<\/span>:<\/span>\n        loadFullPage<\/span>(<\/span>1<\/span>)<\/span>\n        soup<\/span> =<\/span> BeautifulSoup<\/span>(<\/span>driver<\/span>.<\/span>page_source<\/span>,<\/span> 'html.parser'<\/span>)<\/span>\n        for<\/span> link<\/span> in<\/span> soup<\/span>.<\/span>find_all<\/span>(<\/span>\"a\"<\/span>,<\/span>class_<\/span>=<\/span>\"yt-simple-endpoint style-scope ytd-grid-video-renderer\"<\/span>,<\/span> href<\/span>=<\/span>True<\/span>):<\/span>\n            if<\/span> link<\/span> not<\/span> in<\/span> links<\/span>:<\/span>\n                links<\/span>.<\/span>add<\/span>(<\/span>link<\/span>[<\/span>'href'<\/span>])<\/span>\n        if<\/span> len<\/span>(<\/span>links<\/span>)<\/span> <<\/span> linksSize<\/span>:<\/span>\n            driver<\/span>.<\/span>refresh<\/span>()<\/span>\n        else<\/span>:<\/span>\n            #for i in range(len(links)-1000):\n<\/span>                #links.pop()\n<\/span>            enoughLinks<\/span> =<\/span> True<\/span>\n    output<\/span> =<\/span> open<\/span>(<\/span>output_path<\/span>,<\/span> 'a'<\/span>)<\/span>\n    for<\/span> link<\/span> in<\/span> links<\/span>:<\/span>\n        output<\/span>.<\/span>write<\/span>(<\/span>link<\/span>+<\/span>\"<\/span>\\n<\/span>\"<\/span>)<\/span>\n    output<\/span>.<\/span>close<\/span>()<\/span>\n    print<\/span>(<\/span>links<\/span>)<\/span>\n\nif<\/span> __name__<\/span> ==<\/span> '__main__'<\/span>:<\/span>\n    main<\/span>()<\/span>\n<\/code><\/pre><\/div><\/div>\n\n
CSSSelector - extracting comments<\/h3>\n\n
Now as we have saved the unique ids of the videos in trends, we will start to gather comments with requests and CSSSelector.\nCSSSelector is quite optional here, you can use either lxml or regular expressions, but the library in concern IMHO is most convenient for collecting the trees of comments.<\/p>\n\n
We are using user agent still pretending to be Mozilla Firefox.<\/p>\n\n
YOUTUBE_COMMENTS_URL<\/span> =<\/span> 'https:\/\/www.youtube.com\/all_comments?v={youtube_id}'<\/span>\nYOUTUBE_COMMENTS_AJAX_URL<\/span> =<\/span> 'https:\/\/www.youtube.com\/comment_ajax'<\/span>\nUSER_AGENT<\/span> =<\/span> 'Mozilla\/5.0 (Windows NT 6.1; WOW64) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/48.0.2564.116 Safari\/537.36'<\/span>\n\ndef<\/span> find_value<\/span>(<\/span>html<\/span>,<\/span> key<\/span>,<\/span> num_chars<\/span>=<\/span>2<\/span>):<\/span>\n    pos_begin<\/span> =<\/span> html<\/span>.<\/span>find<\/span>(<\/span>key<\/span>)<\/span> +<\/span> len<\/span>(<\/span>key<\/span>)<\/span> +<\/span> num_chars<\/span>\n    pos_end<\/span> =<\/span> html<\/span>.<\/span>find<\/span>(<\/span>'\"'<\/span>,<\/span> pos_begin<\/span>)<\/span>\n    return<\/span> html<\/span>[<\/span>pos_begin<\/span>:<\/span> pos_end<\/span>]<\/span>\n\ndef<\/span> extract_comments<\/span>(<\/span>html<\/span>):<\/span>\n    tree<\/span> =<\/span> lxml<\/span>.<\/span>html<\/span>.<\/span>fromstring<\/span>(<\/span>html<\/span>)<\/span>\n    item_sel<\/span> =<\/span> CSSSelector<\/span>(<\/span>'.comment-item'<\/span>)<\/span>\n    text_sel<\/span> =<\/span> CSSSelector<\/span>(<\/span>'.comment-text-content'<\/span>)<\/span>\n    time_sel<\/span> =<\/span> CSSSelector<\/span>(<\/span>'.time'<\/span>)<\/span>\n    author_sel<\/span> =<\/span> CSSSelector<\/span>(<\/span>'.user-name'<\/span>)<\/span>\n\n    for<\/span> item<\/span> in<\/span> item_sel<\/span>(<\/span>tree<\/span>):<\/span>\n        yield<\/span> {<\/span>'cid'<\/span>:<\/span> item<\/span>.<\/span>get<\/span>(<\/span>'data-cid'<\/span>),<\/span>\n               'text'<\/span>:<\/span> text_sel<\/span>(<\/span>item<\/span>)[<\/span>0<\/span>].<\/span>text_content<\/span>(),<\/span>\n               'time'<\/span>:<\/span> time_sel<\/span>(<\/span>item<\/span>)[<\/span>0<\/span>].<\/span>text_content<\/span>().<\/span>strip<\/span>(),<\/span>\n               'author'<\/span>:<\/span> author_sel<\/span>(<\/span>item<\/span>)[<\/span>0<\/span>].<\/span>text_content<\/span>()}<\/span>\n\n\ndef<\/span> extract_reply_cids<\/span>(<\/span>html<\/span>):<\/span>\n    tree<\/span> =<\/span> lxml<\/span>.<\/span>html<\/span>.<\/span>fromstring<\/span>(<\/span>html<\/span>)<\/span>\n    sel<\/span> =<\/span> CSSSelector<\/span>(<\/span>'.comment-replies-header > .load-comments'<\/span>)<\/span>\n    return<\/span> [<\/span>i<\/span>.<\/span>get<\/span>(<\/span>'data-cid'<\/span>)<\/span> for<\/span> i<\/span> in<\/span> sel<\/span>(<\/span>tree<\/span>)]<\/span>\n\ndef<\/span> ajax_request<\/span>(<\/span>session<\/span>,<\/span> url<\/span>,<\/span> params<\/span>,<\/span> data<\/span>,<\/span> retries<\/span>=<\/span>10<\/span>,<\/span> sleep<\/span>=<\/span>20<\/span>):<\/span>\n    for<\/span> _<\/span> in<\/span> range<\/span>(<\/span>retries<\/span>):<\/span>\n        response<\/span> =<\/span> session<\/span>.<\/span>post<\/span>(<\/span>url<\/span>,<\/span> params<\/span>=<\/span>params<\/span>,<\/span> data<\/span>=<\/span>data<\/span>)<\/span>\n        if<\/span> response<\/span>.<\/span>status_code<\/span> ==<\/span> 200<\/span>:<\/span>\n            response_dict<\/span> =<\/span> json<\/span>.<\/span>loads<\/span>(<\/span>response<\/span>.<\/span>text<\/span>)<\/span>\n            return<\/span> response_dict<\/span>.<\/span>get<\/span>(<\/span>'page_token'<\/span>,<\/span> None<\/span>),<\/span> response_dict<\/span>[<\/span>'html_content'<\/span>]<\/span>\n        else<\/span>:<\/span>\n            time<\/span>.<\/span>sleep<\/span>(<\/span>sleep<\/span>)<\/span>\n\ndef<\/span> download_comments<\/span>(<\/span>youtube_id<\/span>,<\/span> sleep<\/span>=<\/span>1<\/span>):<\/span>\n    session<\/span> =<\/span> requests<\/span>.<\/span>Session<\/span>()<\/span>\n    session<\/span>.<\/span>headers<\/span>[<\/span>'User-Agent'<\/span>]<\/span> =<\/span> USER_AGENT<\/span>\n\n    # Get Youtube page with initial comments\n<\/span>    response<\/span> =<\/span> session<\/span>.<\/span>get<\/span>(<\/span>YOUTUBE_COMMENTS_URL<\/span>.<\/span>format<\/span>(<\/span>youtube_id<\/span>=<\/span>youtube_id<\/span>))<\/span>\n    html<\/span> =<\/span> response<\/span>.<\/span>text<\/span>\n    reply_cids<\/span> =<\/span> extract_reply_cids<\/span>(<\/span>html<\/span>)<\/span>\n    ret_cids<\/span> =<\/span> []<\/span>\n    for<\/span> comment<\/span> in<\/span> extract_comments<\/span>(<\/span>html<\/span>):<\/span>\n        ret_cids<\/span>.<\/span>append<\/span>(<\/span>comment<\/span>[<\/span>'cid'<\/span>])<\/span>\n        yield<\/span> comment<\/span>\n\n    page_token<\/span> =<\/span> find_value<\/span>(<\/span>html<\/span>,<\/span> 'data-token'<\/span>)<\/span>\n    session_token<\/span> =<\/span> find_value<\/span>(<\/span>html<\/span>,<\/span> 'XSRF_TOKEN'<\/span>,<\/span> 4<\/span>)<\/span>\n    first_iteration<\/span> =<\/span> True<\/span>\n    # Get remaining comments (the same as pressing the 'Show more' button)\n<\/span>    while<\/span> page_token<\/span>:<\/span>\n        data<\/span> =<\/span> {<\/span>'video_id'<\/span>:<\/span> youtube_id<\/span>,<\/span>\n                'session_token'<\/span>:<\/span> session_token<\/span>}<\/span>\n\n        params<\/span> =<\/span> {<\/span>'action_load_comments'<\/span>:<\/span> 1<\/span>,<\/span>\n                  'order_by_time'<\/span>:<\/span> True<\/span>,<\/span>\n                  'filter'<\/span>:<\/span> youtube_id<\/span>}<\/span>\n        if<\/span> first_iteration<\/span>:<\/span>\n            params<\/span>[<\/span>'order_menu'<\/span>]<\/span> =<\/span> True<\/span>\n        else<\/span>:<\/span>\n            data<\/span>[<\/span>'page_token'<\/span>]<\/span> =<\/span> page_token<\/span>\n        response<\/span> =<\/span> ajax_request<\/span>(<\/span>session<\/span>,<\/span> YOUTUBE_COMMENTS_AJAX_URL<\/span>,<\/span> params<\/span>,<\/span> data<\/span>)<\/span>\n        if<\/span> not<\/span> response<\/span>:<\/span>\n            break<\/span>\n        page_token<\/span>,<\/span> html<\/span> =<\/span> response<\/span>\n        reply_cids<\/span> +=<\/span> extract_reply_cids<\/span>(<\/span>html<\/span>)<\/span>\n        for<\/span> comment<\/span> in<\/span> extract_comments<\/span>(<\/span>html<\/span>):<\/span>\n            if<\/span> comment<\/span>[<\/span>'cid'<\/span>]<\/span> not<\/span> in<\/span> ret_cids<\/span>:<\/span>\n                ret_cids<\/span>.<\/span>append<\/span>(<\/span>comment<\/span>[<\/span>'cid'<\/span>])<\/span>\n                yield<\/span> comment<\/span>\n\n        first_iteration<\/span> =<\/span> False<\/span>\n        time<\/span>.<\/span>sleep<\/span>(<\/span>sleep<\/span>)<\/span>\n\n    # Get replies (the same as pressing the 'View all X replies' link)\n<\/span>    for<\/span> cid<\/span> in<\/span> reply_cids<\/span>:<\/span>\n        data<\/span> =<\/span> {<\/span>'comment_id'<\/span>:<\/span> cid<\/span>,<\/span>\n                'video_id'<\/span>:<\/span> youtube_id<\/span>,<\/span>\n                'can_reply'<\/span>:<\/span> 1<\/span>,<\/span>\n                'session_token'<\/span>:<\/span> session_token<\/span>}<\/span>\n        params<\/span> =<\/span> {<\/span>'action_load_replies'<\/span>:<\/span> 1<\/span>,<\/span>\n                  'order_by_time'<\/span>:<\/span> True<\/span>,<\/span>\n                  'filter'<\/span>:<\/span> youtube_id<\/span>,<\/span>\n                  'tab'<\/span>:<\/span> 'inbox'<\/span>}<\/span>\n        response<\/span> =<\/span> ajax_request<\/span>(<\/span>session<\/span>,<\/span> YOUTUBE_COMMENTS_AJAX_URL<\/span>,<\/span> params<\/span>,<\/span> data<\/span>)<\/span>\n        if<\/span> not<\/span> response<\/span>:<\/span>\n            break<\/span>\n        _<\/span>,<\/span> html<\/span> =<\/span> response<\/span>\n        for<\/span> comment<\/span> in<\/span> extract_comments<\/span>(<\/span>html<\/span>):<\/span>\n            if<\/span> comment<\/span>[<\/span>'cid'<\/span>]<\/span> not<\/span> in<\/span> ret_cids<\/span>:<\/span>\n                ret_cids<\/span>.<\/span>append<\/span>(<\/span>comment<\/span>[<\/span>'cid'<\/span>])<\/span>\n                yield<\/span> comment<\/span>\n        time<\/span>.<\/span>sleep<\/span>(<\/span>sleep<\/span>)<\/span>\n\n\ndef<\/span> comments_main<\/span>(<\/span>youtube_id<\/span>,<\/span> output<\/span>,<\/span> limit<\/span>=<\/span>100<\/span>):<\/span>\n    try<\/span>:<\/span>\n        if<\/span> not<\/span> youtube_id<\/span> or<\/span> not<\/span> output<\/span>:<\/span>\n            parser<\/span>.<\/span>print_usage<\/span>()<\/span>\n            raise<\/span> ValueError<\/span>(<\/span>'you need to specify a Youtube ID and an output filename'<\/span>)<\/span>\n        print<\/span>(<\/span>'Downloading Youtube comments for video:'<\/span>,<\/span> youtube_id<\/span>)<\/span>\n        count<\/span> =<\/span> 0<\/span>\n        with<\/span> io<\/span>.<\/span>open<\/span>(<\/span>output<\/span>,<\/span> 'w'<\/span>,<\/span> encoding<\/span>=<\/span>'utf8'<\/span>)<\/span> as<\/span> fp<\/span>:<\/span>\n            for<\/span> comment<\/span> in<\/span> download_comments<\/span>(<\/span>youtube_id<\/span>):<\/span>\n                sys<\/span>.<\/span>stdout<\/span>.<\/span>write<\/span>(<\/span>json<\/span>.<\/span>dumps<\/span>(<\/span>comment<\/span>,<\/span> ensure_ascii<\/span>=<\/span>False<\/span>))<\/span>\n                count<\/span> +=<\/span> 1<\/span>\n                sys<\/span>.<\/span>stdout<\/span>.<\/span>write<\/span>(<\/span>'Downloaded %d comment(s)<\/span>\\r<\/span>'<\/span> %<\/span> count<\/span>)<\/span>\n                sys<\/span>.<\/span>stdout<\/span>.<\/span>flush<\/span>()<\/span>\n                if<\/span> limit<\/span> and<\/span> count<\/span> >=<\/span> limit<\/span>:<\/span>\n                    break<\/span>\n        print<\/span>(<\/span>'<\/span>\\n<\/span>Done!'<\/span>)<\/span>\n    except<\/span> Exception<\/span> as<\/span> e<\/span>:<\/span>\n        print<\/span>(<\/span>'Error:'<\/span>,<\/span> str<\/span>(<\/span>e<\/span>))<\/span>\n        sys<\/span>.<\/span>exit<\/span>(<\/span>1<\/span>)<\/span>\n<\/code><\/pre><\/div><\/div>\n\n
Saving comments for parsed video ids<\/h3>\n\n
The code above has some magic numbers, like limit=100 default parameter in comments_main \u2013 consciously not so big as the crawler is illustrating a collection of a small scientific corpus. \nCollecting comments in a loop:<\/p>\n\n
wdir<\/span> =<\/span> r<\/span>'\/media\/user\/youtube\/comments'<\/span>\nuids<\/span> =<\/span> open<\/span>(<\/span>r<\/span>'\/media\/user\/youtube\/'<\/span>+<\/span>str<\/span>(<\/span>datetime<\/span>.<\/span>date<\/span>.<\/span>today<\/span>())<\/span>+<\/span>'.txt'<\/span>,<\/span> 'r'<\/span>,<\/span> encoding<\/span>=<\/span>'utf8'<\/span>).<\/span>readlines<\/span>()<\/span>\nuids<\/span> =<\/span> [<\/span>i<\/span>.<\/span>strip<\/span>(<\/span>'\/watch?v='<\/span>)<\/span> for<\/span> i<\/span> in<\/span> uids<\/span>]<\/span>\nfor<\/span> i<\/span> in<\/span> tqdm<\/span>(<\/span>uids<\/span>):<\/span>\n    uid<\/span> =<\/span> i<\/span>.<\/span>strip<\/span>(<\/span>'<\/span>\\n<\/span>'<\/span>)<\/span>\n    output<\/span> =<\/span> os<\/span>.<\/span>path<\/span>.<\/span>join<\/span>(<\/span>wdir<\/span>,<\/span> uid<\/span>+<\/span>'_'<\/span>+<\/span>str<\/span>(<\/span>datetime<\/span>.<\/span>date<\/span>.<\/span>today<\/span>())<\/span>+<\/span>'.txt'<\/span>)<\/span>\n    comments_main<\/span>(<\/span>uid<\/span>,<\/span> output<\/span>)<\/span>\n<\/code><\/pre><\/div><\/div>\n\n
Power-up!<\/h3>\n\n
Finally, what you can (optionally) do is<\/p>\n