Skip to content

Commit 31bceb1

Browse files
feat: add metascraper-citation (#873)
* feat: add metascraper-citation Highwire citation_* tags beat Open Graph on academic pages. The bundle no-ops unless those tags are present. Co-authored-by: Cursor <[email protected]> * test(citation): add PLOS ONE HTML fixture Co-authored-by: Cursor <[email protected]> * test(citation): snapshot publisher HTML fixtures Co-authored-by: Cursor <[email protected]> * test(citation): add five publisher HTML fixtures Co-authored-by: Cursor <[email protected]> * feat(citation): read Dublin Core meta tags Co-authored-by: Cursor <[email protected]> * fix(citation): prefer citation_date over online_date arXiv sets citation_date to the original announcement and citation_online_date to the latest version. Prefer the work date. Co-authored-by: Cursor <[email protected]> --------- Co-authored-by: Cursor <[email protected]>
1 parent d558336 commit 31bceb1

16 files changed

Lines changed: 11541 additions & 0 deletions

‎README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -238,6 +238,7 @@ const metascraper = require('metascraper')([
238238

239239
- [metascraper-audio](https://github.com/microlinkhq/metascraper/tree/master/packages/metascraper-audio) – Get audio property from HTML markup.
240240
- [metascraper-author](https://github.com/microlinkhq/metascraper/tree/master/packages/metascraper-author) – Get author property from HTML markup.
241+
- [metascraper-citation](https://github.com/microlinkhq/metascraper/tree/master/packages/metascraper-citation) – Get title, author, date, and publisher from Highwire Press `citation_*` and Dublin Core `dc.*` / `dcterms.*` tags.
241242
- [metascraper-date](https://github.com/microlinkhq/metascraper/tree/master/packages/metascraper-date) – Get date property from HTML markup.
242243
- [metascraper-defuddle](https://github.com/microlinkhq/metascraper/tree/master/packages/metascraper-defuddle) – A Defuddle connector for metascraper.
243244
- [metascraper-description](https://github.com/microlinkhq/metascraper/tree/master/packages/metascraper-description) – Get description property from HTML markup.
Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
# Change Log
2+
3+
All notable changes to this project will be documented in this file.
4+
See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.
Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
<div align="center">
2+
<br>
3+
<img style="width: 500px; margin:3rem 0 1.5rem;" src="https://metascraper.js.org/static/logo-banner.png" alt="metascraper">
4+
<br>
5+
<br>
6+
<p align="center"><strong>metascraper-citation</strong>: Get title, author, date, and publisher from Highwire Press <code>citation_*</code> and Dublin Core <code>dc.*</code> / <code>dcterms.*</code> meta tags.</p>
7+
<p align="center">See our <a href="https://metascraper.js.org" target='_blank' rel='noopener noreferrer'>website</a> for more information.</p>
8+
<br>
9+
</div>
10+
11+
## Install
12+
13+
```bash
14+
$ npm install metascraper-citation --save
15+
```
16+
17+
Load it **before** the generic property packages so these values win over Open Graph (e.g. a clean `citation_title` instead of `og:title` with a site suffix).
18+
19+
```js
20+
const metascraper = require('metascraper')([
21+
require('metascraper-citation')(),
22+
require('metascraper-author')(),
23+
require('metascraper-date')(),
24+
require('metascraper-publisher')(),
25+
require('metascraper-title')()
26+
])
27+
```
28+
29+
## Supported tags
30+
31+
The bundle is a no-op unless [`test`](#test) finds at least one Highwire or Dublin Core tag. Highwire selectors run first; Dublin Core fills gaps (and covers repositories that only emit DC). Missing fields fall through to the generic rules.
32+
33+
Only the four properties below are set. Bibliographic extras (`citation_doi`, `dc.identifier`, ISSN, volume, issue, pages, PDF URL, language, abstract) are ignored.
34+
35+
| Property | Tags (first match wins) |
36+
| --- | --- |
37+
| **title** | `citation_title`, `dc.title`, `dcterms.title` |
38+
| **author** | `citation_author`, `dc.creator`, `dcterms.creator`, `dc.contributor`, `dcterms.contributor` (first author only) |
39+
| **date** | `citation_publication_date`, `citation_date`, `citation_online_date`, `dcterms.issued`, `dcterms.created`, `dc.date`, `dcterms.date` |
40+
| **publisher** | `citation_publisher`, `citation_journal_title`, `dc.publisher`, `dcterms.publisher` |
41+
42+
Names are matched case-insensitively (`DC.Title` and `dc.title` are the same tag).
43+
44+
## API
45+
46+
### test
47+
48+
```js
49+
const { test } = require('metascraper-citation')
50+
```
51+
52+
Returns `true` when the Cheerio document has at least one `meta[name]` starting with `citation_`, `dc.`, or `dcterms.` (case-insensitive). Used as `rules.test` so the bundle skips pages with no scholarly markup.
53+
54+
## License
55+
56+
**metascraper-citation** © [Microlink](https://microlink.io), released under the [MIT](https://github.com/microlinkhq/metascraper/blob/master/LICENSE.md) License.<br>
57+
Authored and maintained by [Microlink](https://microlink.io) with help from [contributors](https://github.com/microlinkhq/metascraper/contributors).
58+
59+
> [microlink.io](https://microlink.io) · GitHub [microlinkhq](https://github.com/microlinkhq) · X [@microlinkhq](https://x.com/microlinkhq)
Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
{
2+
"name": "metascraper-citation",
3+
"description": "Metascraper rules for Highwire Press citation_* and Dublin Core dc.* / dcterms.* meta tags.",
4+
"homepage": "https://github.com/microlinkhq/metascraper/packages/metascraper-citation",
5+
"version": "5.55.2",
6+
"types": "src/index.d.ts",
7+
"main": "src/index.js",
8+
"author": {
9+
"email": "[email protected]",
10+
"name": "microlink.io",
11+
"url": "https://microlink.io"
12+
},
13+
"repository": {
14+
"directory": "packages/metascraper-citation",
15+
"type": "git",
16+
"url": "git+https://github.com/microlinkhq/metascraper.git"
17+
},
18+
"bugs": {
19+
"url": "https://github.com/microlinkhq/metascraper/issues"
20+
},
21+
"keywords": [
22+
"citation",
23+
"dublin-core",
24+
"extract",
25+
"google-scholar",
26+
"highwire",
27+
"html",
28+
"metadata",
29+
"metascraper",
30+
"scraper",
31+
"scholar"
32+
],
33+
"dependencies": {
34+
"@metascraper/helpers": "workspace:*"
35+
},
36+
"devDependencies": {
37+
"ava": "8",
38+
"cheerio": "latest"
39+
},
40+
"engines": {
41+
"node": ">= 22"
42+
},
43+
"files": [
44+
"src"
45+
],
46+
"scripts": {
47+
"test": "NODE_PATH=.. TZ=UTC ava --timeout 15s"
48+
},
49+
"license": "MIT"
50+
}
Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
declare function rules(): import('metascraper').Rules;
2+
3+
declare namespace rules {
4+
/**
5+
* `true` when the document has at least one Highwire (`citation_*`)
6+
* or Dublin Core (`dc.*` / `dcterms.*`) meta tag.
7+
*/
8+
function test(htmlDom: import('cheerio').CheerioAPI): boolean;
9+
}
10+
11+
export = rules;
Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
1+
'use strict'
2+
3+
const {
4+
author,
5+
date,
6+
memoizeOne,
7+
publisher,
8+
title,
9+
toRule
10+
} = require('@metascraper/helpers')
11+
12+
const toAuthor = toRule(author)
13+
const toDate = toRule(date)
14+
const toPublisher = toRule(publisher)
15+
const toTitle = toRule(title)
16+
17+
const meta = name => $ => $(`meta[name="${name}" i]`).attr('content')
18+
19+
const MARKUP =
20+
'meta[name^="citation_" i], meta[name^="dc." i], meta[name^="dcterms." i]'
21+
22+
const test = memoizeOne(
23+
$ => $(MARKUP).length > 0,
24+
memoizeOne.EqualityFirstArgument
25+
)
26+
27+
module.exports = () => {
28+
const rules = {
29+
title: [
30+
toTitle(meta('citation_title')),
31+
toTitle(meta('dc.title')),
32+
toTitle(meta('dcterms.title'))
33+
],
34+
author: [
35+
toAuthor(meta('citation_author')),
36+
toAuthor(meta('dc.creator')),
37+
toAuthor(meta('dcterms.creator')),
38+
toAuthor(meta('dc.contributor')),
39+
toAuthor(meta('dcterms.contributor'))
40+
],
41+
date: [
42+
toDate(meta('citation_publication_date')),
43+
toDate(meta('citation_date')),
44+
toDate(meta('citation_online_date')),
45+
toDate(meta('dcterms.issued')),
46+
toDate(meta('dcterms.created')),
47+
toDate(meta('dc.date')),
48+
toDate(meta('dcterms.date'))
49+
],
50+
publisher: [
51+
toPublisher(meta('citation_publisher')),
52+
toPublisher(meta('citation_journal_title')),
53+
toPublisher(meta('dc.publisher')),
54+
toPublisher(meta('dcterms.publisher'))
55+
]
56+
}
57+
58+
rules.test = ({ htmlDom }) => test(htmlDom)
59+
rules.pkgName = 'metascraper-citation'
60+
61+
return rules
62+
}
63+
64+
module.exports.test = test

0 commit comments

Comments
 (0)