Skip to content

feat: Support RSS3 UMS output format#13900

Merged
TonyRL merged 31 commits intoDIYgod:masterfrom
HenryQW:feat/rss3-ums
Nov 28, 2023
Merged

feat: Support RSS3 UMS output format#13900
TonyRL merged 31 commits intoDIYgod:masterfrom
HenryQW:feat/rss3-ums

Conversation

@HenryQW
Copy link
Copy Markdown
Collaborator

@HenryQW HenryQW commented Nov 28, 2023

Involved Issue / 该 PR 相关 Issue

Close #

Example for the Proposed Route(s) / 路由地址示例

NOROUTE

Note / 说明

This is the first step to onboard all RSSHub instances to the RSS3 Network. The PR enables the output of all existing routes in the RSS3 Unified Metadata Schemas format.

Background

The RSS3 Network is the Open Information Layer, structuring information and making it universally available and valuable, fostering the development of decentralized projects akin to Twitter, Google, and OpenAI (read more here). We are working on a plan to incentivize all RSSHub instances that join our mission to promote the free flow of information on the Open Web.

We see RSS as the de facto very first Open Standard for information dissemination on the Internet, and should be rightfully included as one of the Open Information sources on the RSS3 Network.

The original RSS however, was slowly strangled by data oligarchies, the Internet we see today has its openness eroded. One of the reasons this is happening, in our opinion, is that RSS did/does not generate incentives to motivate content creators (one of the works left behind by Arron Swartz, RSS 3.0, was to establish the proper ownership for the creators).

RSSHub as one of (if not the largest) sources of RSS Feeds, could expedite onboarding more creators to the RSS3 Network. We understand that there are hard-to-overcome barriers and steep learning curves for creators to join Web3, therefore by including RSS, content creators can join the RSS3 Network and the Open Web by enabling RSS on their websites.

What RSS3 is Planning to Do

  1. We submit this PR to enable all RSSHub instances to output in the UMS format.
  2. We are working on a decentralized RSS3 Node, where RSSHub will be an integral part of the Node to serve information to users of the decentralized RSS3 Network. Anyone can freely spin up an RSS3 Node and join the Network. An RSS3 Node will support more data sources other than RSS.
  3. We will try our best to make the RSS3 Node deployment process as smooth as possible. There will be an incentive to encourage existing RSSHub instances to join the Network.
  4. The RSS3 Network, by its design in the whitepaper, will distribute requests to all Nodes and record these requests for calculating incentives. Those incentives will be distributed to Node hosts periodically in the form of RSS3 tokens on a blockchain.
  5. The RSS Network will be fully decentralized, this means any RSSHub instance hosts will be able to join and leave the Network in a permissionless fashion.
  6. Your existing RSSHub and its operation will be unaffected regardless of your participation in the RSS3 Network.
RSS3 Network

A simplified architecture of the RSS3 Network.

What Does This Mean to an RSSHub Instance

TL;DR

Your RSSHub instance will receive more requests, based on the served requests, you will receive incentives in the form of RSS3 tokens.

Participating in the RSS3 Network means that your RSSHub instance must be accessible by the RSS3 Global Indexer (aka the GI; see our whitepaper for more details) through your RSS3 Node.

Requests will be routed to qualified instances by the GI, so expect more traffic coming to your instance.

The GI will record the requests served by all instances, and allocate incentives periodically to instance hosts.

The onboarding mechanism will be announced soon (no later than 2024Q1).

An RSS Request

A simplified RSS request lifecycle on the RSS3 Network.

What Do You Have to Do

Once the onboarding mechanism is live, if you plan to participate, you will have to:

  1. Discovery Stage: either make your RSSHub instance completely public (we encourage you to do this), or implement one of the authentication methods supported by RSSHub. Spin up an RSS3 Node and make sure that your RSSHub is accessible by your RSS3 Node. Join the RSS3 Network with your RSS3 Node.
  2. Servicing Stage: Your RSS3 Node will receive requests routed from the GI, it will then relay them to your RSSHub.
  3. Clearing Stage: The RSS3 Network will allocate incentives periodically based on serviced requests.

These are subject to changes as the RSS3 Network evolves. And more tutorials will be ready in the future.

What This PR Does

This PR enables all RSSHub instances to output the response in the UMS format. See https://rsshub3.henry.wang/abc.ums?key=rss3 for an example. This is the first step to preparing your RSSHub to join the RSS3 Network. This PR does not introduce any breaking change, so your existing RSSHub and its operation are not affected in any way.

Everything is handled by RSSHub middleware to render the output in the UMS, where the actual feed is now in metadata

{
  "data": [
    {
      "owner": "www.abc.net.au",
      "id": "https://www.abc.net.au/news/2023-11-10/when-labor-states-dont-dance-to-albanese-government-tune-grattan/103087218",
      "network": "RSS",
      "from": "www.abc.net.au",
      "to": "www.abc.net.au",
      "tag": "RSS",
      "type": "article",
      "direction": "out",
      "feeValue": "0",
      "actions": [
        {
          "tag": "RSS",
          "type": "article",
          "platform": "www.abc.net.au",
          "from": "www.abc.net.au",
          "to": "www.abc.net.au",
          "metadata": {
            "authors": [
              {
                "name": "Michelle Grattan"
              }
            ],
            "description": "<div><p>It's helpful for the Albanese Government to have all mainland states in Labor hands – but only up to a point.</p><p>This week we've seen the Queensland government bite back at federal plans to curb the national infrastructure program, while Victorian resistance to changes to the Murray-Darling water plan prompted Environment Minister Tanya Plibersek to lash out.</p><p>Infrastructure is always a vexed issue. The program is full of pork-barrelling, whoever is in power. Even when that's not involved, what to build and when it should be built is often contested.</p><p>In May, the government announced a 90-day review of the $120 billion infrastructure pipeline it inherited from the Coalition.</p><p>Infrastructure Minister Catherine King said projects had increased from about 150 to 800. The government's aim was to reduce the number of projects (many of them small) and rearrange priorities.</p><p>High inflation, cost overruns and shortages of labour and materials are plaguing the program.</p><p>The political difficulties of abolishing or changing projects, often involving negotiation with states and territories, are obvious enough. Now they have become significantly worse.</p>\n  <figure>\n    <img src=\"https://live-production.wcms.abc-cdn.net.au/7a1bfd275c558bbecd11265666471f4a\" alt=\"The cost of the Coalition-inherited infrastructure pipeline has blown out by billions, Jim Chalmers says.(AAP Image: Lukas Coch)\" referrerpolicy=\"no-referrer\">\n    <figcaption>The cost of the Coalition-inherited infrastructure pipeline has blown out by billions, Jim Chalmers says.(AAP Image: Lukas Coch)</figcaption>\n  </figure>\n\n\n\n\n<p>The government has received its stocktake, and Treasurer Jim Chalmers says the overall cost of the program has blown out by some $33 billion.</p><p>Also, <a href=\"https://www.imf.org/en/News/Articles/2023/10/31/cs103123-australia-staff-concluding-statement-of-the-2023-article-iv#:%7E:text=Australia's%20economy%20has%20been%20resilient,after%20a%20correction%20in%202022.\">an International Monetary Fund report</a> last week said infrastructure projects should be rolled out at a \"more measured and co-ordinated pace, given supply constraints, to alleviate inflationary pressures\".</p><p>Chalmers is pushing this message, but it's not being received well in Queensland.</p><p>State Treasurer Cameron Dick was blunt. \"Queensland is Australia's growth state and we need more infrastructure, not less,\" <a href=\"https://twitter.com/camerondickqld/status/1720977443337691323\">he said in a tweet</a>.</p><p>\"If infrastructure cuts are needed, they should be made to southern states with low growth and high debt.\" (Fun fact: the electorate offices of Queenslanders Chalmers and Dick share a common wall.)</p><p>Queensland Police Minister Mark Ryan said: \"I've got a clear message for Jim. Jim's a mate of mine. Jim, those projects better not be in Queensland.\"</p><p>The last thing the Palaszczuk government wants is for projects to be cancelled, slashed or delayed. It is in a particularly precarious position – it faces an election in a year's time and will be fighting for survival.</p><p>Queensland has an obvious political self-interest in resisting infrastructure cuts, but there's a national point too. With large numbers of migrants coming into Australia, the demand for transport and other infrastructure will be increasing, rather than decreasing. Whatever cuts and slowdowns are made will need to be well judged.</p><p>The federal government argues the existing pipeline is unrealistic and without change could not be delivered anyway. But even if the decisions about what to cut, scale back or defer are economically sound, in political terms they could store up electoral time bombs for the government.</p><p>Even minor and unworthy projects can be sensitive in marginal seats. Scrapping them could open opportunities for the opposition. Also, available funds for new projects presumably will be limited.</p><p>When the government finishes its negotiations with the states and the outcomes are announced, King will be the main minister defending the decisions.</p><p>As we saw in the row over the rejection of Qatar Airways' bid for extra flights, she struggles when under pressure. She could find the task challenging.</p><p>The fight over the government's water changes centre on its planned amendments to the Murray-Darling Basin plan.</p><p>The legislation, soon to be considered by the Senate, broadens the activities that can be funded and extends the times for delivery of water-recovery projects. Most importantly, it removes the cap on the federal government's \"buybacks\" of extra water for the environment.</p><p>The Murray-Darling plan is always fraught, because the interests of upstream and downstream users and their governments differ. Nevertheless, Queensland, South Australia and New South Wales have signed on – although NSW has done so reluctantly.</p><p>But Victoria, where the Andrews government has built a close relationship with irrigators, has held out, defending its position <a href=\"https://www.water.vic.gov.au/our-programs/murray-darling-basin/social-and-economic-impacts-of-the-basin-plan-in-victoria\">on the basis of work done by Frontier Economics</a>.</p><p>Its report argues that \"previous water recovery has resulted in less irrigation […] putting the viability of major irrigation districts and the industries and communities they support under pressure\".</p><p>\"Further water recovery from irrigators (buybacks and on-farm projects) will add to the impacts already being felt and undermine the ability of irrigation communities to plan for the future.\"</p><p>Plibersek declared, in an interview with the ABC, that it was \"extraordinary that we've got a Labor government using dodgy modelling to join up with Barnaby Joyce and David Littleproud\".</p><p>Victoria's Water Minister Harriet Shing retorts: \"This isn't about party politics, and it's disappointing to see it framed that way. We don't apologise for standing up for Victorian communities and environments.\"</p>\n  <figure>\n    <img src=\"https://live-production.wcms.abc-cdn.net.au/fdc89d3b1f642287a93748de72cfa131\" alt=\"Victorian resistance to changes to the Murray-Darling water plan saw Tanya Plibersek lash out.(ABC News:&nbsp;Luke Stephenson)\" referrerpolicy=\"no-referrer\">\n    <figcaption>Victorian resistance to changes to the Murray-Darling water plan saw Tanya Plibersek lash out.(ABC News:&nbsp;Luke Stephenson)</figcaption>\n  </figure>\n\n\n\n\n<p>But Plibersek has backing from Jamie Pittock, from the Australian National University's Fenner School of Environment and Society. He says: \"The Victorian government can usually be relied on to make decisions based on solid data. In the case of the Murray-Darling Basin, bizarrely, it has relied on low-quality consultants' reports that exaggerate the socio-economic costs and ignore the benefits from water buybacks.\"</p><p>The legislation will come to a vote in the Senate this year, and there will be wrangling with the crossbench.</p><p>Assuming the legislation passes, the federal government can override Victoria and proceed with the buybacks of water for the environment. But it will still face the opposition of farming and irrigator groups, and some local communities.</p><p>It would be hard to find political observers who believe Peter Dutton can win the next election, due by May 2025. But there is increasing talk about the possibility that Labor, given it has a very narrow majority, could find itself in minority government. (Contrast a year ago, when all the talk was about Labor's prospects for increasing its majority.)</p><p>Being pushed into minority is something Albanese – a senior figure in the minority Gillard government – would want to avoid at all costs. It would hamper the government's flexibility to pursue its program, mean constant negotiation with crossbenchers including bolshie Greens, and encourage the Coalition to run maximum disruption.</p><p>The challenge of keeping out of minority increases the importance of the \"ground game\" in Labor's marginal electorates. And it could make controversies over local issues – scrapped infrastructure projects, or unpopular new ventures including ugly transmission lines for renewable energy – potentially dangerous for the incumbents in those seats.</p><p><strong>Michelle Grattan is a professorial fellow at the University of Canberra and chief political correspondent at <a href=\"https://theconversation.com/grattan-on-friday-when-labor-states-dont-dance-to-albanese-governments-tune-217280\" target=\"_blank\" rel=\"noopener noreferrer\">The Conversation</a>, where this article first appeared.</strong></p></div>\n",
            "pubDate": "2023-11-09T15:11:35.000Z",
            "tags": [
              "grattan",
              "michelle grattan",
              "anthony albanese",
              "infrastructure",
              "labor",
              "labor party",
              "annastacia palaszczuk",
              "jim chalmers",
              "catherine king",
              "tanya plibersek",
              "murray-darling"
            ],
            "title": "When Labor states don't dance to Albanese government's tune"
          },
          "related_urls": [
            "https://www.abc.net.au/news/2023-11-10/when-labor-states-dont-dance-to-albanese-government-tune-grattan/103087218"
          ]
        }
      ],
      "timestamp": 1699549161
    }
  ]
}

We hope to share more details with you in 2024Q1, stay tuned.

--
The RSS3 Team

HenryQW and others added 22 commits November 7, 2023 22:53
also optimized middleware/template
[pull] master from diygod:master
[pull] master from diygod:master
[pull] master from diygod:master
[pull] master from diygod:master
[pull] master from diygod:master
[pull] master from diygod:master
[pull] master from diygod:master
#2249)

Bumps [@types/react](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/react) from 18.2.37 to 18.2.38.
- [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases)
- [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/react)

---
updated-dependencies:
- dependency-name: "@types/react"
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [@types/markdown-it](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/markdown-it) from 13.0.6 to 13.0.7.
- [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases)
- [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/markdown-it)

---
updated-dependencies:
- dependency-name: "@types/markdown-it"
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [@types/react-dom](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/react-dom) from 18.2.15 to 18.2.16.
- [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases)
- [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/react-dom)

---
updated-dependencies:
- dependency-name: "@types/react-dom"
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
[pull] master from diygod:master
Bumps [@types/react-dom](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/react-dom) from 18.2.16 to 18.2.17.
- [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases)
- [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/react-dom)

---
updated-dependencies:
- dependency-name: "@types/react-dom"
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
[pull] master from diygod:master
[pull] master from diygod:master
[pull] master from diygod:master
[pull] master from diygod:master
[pull] master from diygod:master
@github-actions github-actions bot added core enhancement auto: route no found Automated test failed due to route can not be found in PR description body labels Nov 28, 2023
@github-actions github-actions bot closed this Nov 28, 2023
@HenryQW HenryQW removed the auto: route no found Automated test failed due to route can not be found in PR description body label Nov 28, 2023
@HenryQW HenryQW reopened this Nov 28, 2023
Repository owner deleted a comment from github-actions bot Nov 28, 2023
Copy link
Copy Markdown
Collaborator

@TonyRL TonyRL left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing documentation updates on website/docs/parameter.md and website/i18n/zh/docusaurus-plugin-content-docs/current/parameter.md

HenryQW and others added 2 commits November 28, 2023 17:15
@TonyRL TonyRL merged commit 3a5488a into DIYgod:master Nov 28, 2023
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Mar 25, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants