-
|
Hey guys, I'm having trouble to scrape a particular website listing job offers.
If anyone could have a look, I really don't know what I'm missing here. Is it possible that the website rejects the FreshRSS requests ? Anyway, thanks in advance for your time. |
Beta Was this translation helpful? Give feedback.
Replies: 4 comments 8 replies
-
|
At a glance I think the data's actually in the following element: <div data-baseline-vue-component-id="vacancy-overview" id="vue-a99613bc-432e-4e8a-919d-8dfc1be04b42" class="js-vue-component"><vacancy-overview :filters="{"degrees":[{"label":"Enseignement secondaire sup\u00e9rieur","id":"1530"},{"label":"Pas de dipl\u00f4me","id":"1531"}],"job_types":[{"label":"Poste sp\u00e9cifique","id":"1559"}]}" :items="[{"link":"https:\/\/environnement.brussels\/jobs\/ouvrieres-forestieres-ouvriers-forestiers-hfx-convention-de-premier-emploi-25-ans","degrees":["Pas de dipl\u00f4me"],"job_types":["Poste sp\u00e9cifique&quo etc.
Since it seems to be a Drupal website maybe they could enable RSS if you ask them? A few possible candidate URLs like |
Beta Was this translation helpful? Give feedback.
-
|
Hi, Tx for your reply. You mean that the code I see in the console is generated by Javascript ? When looking at the code, I selected Each of these <li ...> are actually under the node you mentioned
I've also emailed the webmaster in case of they'd accept to enable RSS feed, but I'd like to understand what's blocking the XPath scraping here. Thanks in advance |
Beta Was this translation helpful? Give feedback.
-
|
To rephrase what @Frenzie was saying: you cannot use your Web browser DOM inspector as you do, because this shows a page after JavaScript transformation. To accurately observe the response, use a View source instead, or download the HTML page and open it in a text editor. The relevant job informations are in a JSON structure, which you cannot scrape with just HTML+XPath, because it is not HTML. For now, the easiest would we to use a bridge for instance with https://github.com/RSS-Bridge/rss-bridge We might add a possibility to extract JSON from HTML pages in the future, but this is not something FreshRSS can at the moment (besides with an extension). |
Beta Was this translation helpful? Give feedback.
-
|
Please open a new feature request for a mode HTML+JSON to support your usecase |
Beta Was this translation helpful? Give feedback.
-
|
I have a patch coming:
Edit: See #6888 |
Beta Was this translation helpful? Give feedback.
-
|
See #6888 |
Beta Was this translation helpful? Give feedback.
-
|
Hi guys, Again, thanks a million to you all, This is really amazing ! Muche appreciated |
Beta Was this translation helpful? Give feedback.
-
|
It is already available in your rolling release edge branch, so you are welcome to start using it and give feedback |
Beta Was this translation helpful? Give feedback.
-
|
Hi @Alkarex and happy new year to all, So, I've updated my self-hosted FreshRSS instance to v1.25.0 and I now have the option to add an HTML+XPATH+JSON feed. So 'I'm trying to list all job offers posted here : https://environnement.brussels/offres-emploi. I know all posts are listed under Could someone help me to configure a feed to fetch all job offers from that page please? Thanks in advance |
Beta Was this translation helpful? Give feedback.
-
|
Save the example from #6888 (comment) as |
Beta Was this translation helpful? Give feedback.
-
|
thanks a lot @Frenzie @Alkarex I got it working
One last comment from a non-dev user like me. Do you think it would be possible to write a one-pager guide in the project documentation (or reference an existing one), to explain :
Anyway, for the time being, thanks a million for the support, and the change deployed. Have all a nice day ! |
Beta Was this translation helpful? Give feedback.
-
|
Beta Was this translation helpful? Give feedback.
I have a patch coming:
Edit: See #6888