feat(route): add 北京大学学生就业指导服务中心#7904
Merged
TonyRL merged 10 commits intoDIYgod:masterfrom Apr 24, 2022
Merged
Conversation
request content in script tag to get full text
|
This pull request is being automatically deployed with Vercel (learn more). 🔍 Inspect: https://vercel.com/diy/rsshub-do-not-use/2ZNZEZA4SZosZ6QfdFWdkw1DRhMg |
Contributor
|
Successfully generated as following: https://rsshub-do-not-use-jo6lckd7z-diy.vercel.app/pku/scc/recruit - **Failed**https://rsshub-do-not-use-jo6lckd7z-diy.vercel.app/pku/scc/recruit/xwrd - Successhttps://rsshub-do-not-use-jo6lckd7z-diy.vercel.app/pku/scc/recruit/tzgg - Successhttps://rsshub-do-not-use-jo6lckd7z-diy.vercel.app/pku/scc/recruit/zpxx - Successhttps://rsshub-do-not-use-jo6lckd7z-diy.vercel.app/pku/scc/recruit/gfjgxx - Successhttps://rsshub-do-not-use-jo6lckd7z-diy.vercel.app/pku/scc/recruit/sxxx - Successhttps://rsshub-do-not-use-jo6lckd7z-diy.vercel.app/pku/scc/recruit/cyxx - Success |
SettingDust
suggested changes
Jul 20, 2021
| Connection: 'keep-alive', | ||
| 'proxy-connection': 'keep-alive', | ||
| Accept: 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9', | ||
| 'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36', |
| }) | ||
| .get(); | ||
|
|
||
| const sorted = list.sort((a, b) => b.pubDate.getTime() - a.pubDate.getTime()).slice(0, 10); |
Contributor
Author
There was a problem hiding this comment.
get 请求一次会拿到1000多条数据，其中前几条是置顶信息，之后的按时间顺序排列。因此需要重新按时间排序，过滤一下置顶信息。
| const detail_page = await got({ method: 'get', url: item.link, headers }); | ||
| const detail = cheerio.load(detail_page.data); | ||
| const script = detail('script', 'div#content-div').html(); | ||
| const content_route = script.match(/\$\("#content-div"\).load\("(\S+)"\)/)[1]; |
Contributor
Author
There was a problem hiding this comment.
本地没能复现这个问题，自己部署了一下，发现每次部署完后第一次请求会报错，之后的请求正常，可能和请求header有关，我再看一看。
|
|
||
| ctx.state.data = { | ||
| title: `北京大学学生就业指导服务中心 - ${feed_title}`, | ||
| link: baseUrl, |
Contributor
|
Successfully generated as following: https://rsshub-do-not-use-db4bhamgw-diy.vercel.app/pku/scc/recruit - Successhttps://rsshub-do-not-use-db4bhamgw-diy.vercel.app/pku/scc/recruit/xwrd - Successhttps://rsshub-do-not-use-db4bhamgw-diy.vercel.app/pku/scc/recruit/tzgg - Successhttps://rsshub-do-not-use-db4bhamgw-diy.vercel.app/pku/scc/recruit/zpxx - Successhttps://rsshub-do-not-use-db4bhamgw-diy.vercel.app/pku/scc/recruit/gfjgxx - Successhttps://rsshub-do-not-use-db4bhamgw-diy.vercel.app/pku/scc/recruit/sxxx - Successhttps://rsshub-do-not-use-db4bhamgw-diy.vercel.app/pku/scc/recruit/cyxx - Success |
DylanXie123
commented
Jul 24, 2021
|
|
||
| const sorted = list.sort((a, b) => b.pubDate.getTime() - a.pubDate.getTime()).slice(0, 10); | ||
| await got({ method: 'get', url: sorted[0].link, headers }); | ||
| await wait(500); |
Contributor
Author
There was a problem hiding this comment.
试了很多次，发现只有这样先请求一次，延迟一段时间后再请求才行
|
好像这个完成了一半吗，谢谢各位大佬的辛苦工作啊 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
该 PR 相关 Issue / Involved issue
Close #7767
完整路由地址 / Example for the proposed route(s)
新RSS检查列表 / New RSS Script Checklist
Puppeteer? Make use ofPuppeteer?说明 / Note