You can download executable static binary from https://github.com/kitsuyui/scraper/releases/latest .
- typically Linux ... download scraper_linux_amd64.
- typically macOS ... download scraper_darwin_amd64.
- typically Windows ... download scraper_windows_amd64.exe
and then just you have to do is adding PATH and permitting it to executable.
$ target=scraper_darwin_amd64 # executable binary you downloaded.
$ cp "./$target" /usr/bin/scraper
$ chmod +x /usr/bin/scraper
$ go install github.com/kitsuyui/scraper@latest$ wget http://example.com/ -O index.html
$ cat <<'EOF'> scraper-config.json
[
{"type": "css", "label": "Title", "query": "h1"},
{"type": "xpath", "label": "LinkURL", "query": "//a/@href"}
]
EOF
$ scraper < index.html
[
{
"type": "css",
"label": "Title",
"query": "h1",
"results": [
"Example Domain"
]
},
{
"type": "xpath",
"label": "LinkURL",
"query": "//a/@href",
"results": [
"http://www.iana.org/domains/example"
]
}
]
Table element scraping is also available. Obviously it supports colspan/rowspan attributes.
$ cat <<'EOF'> table.html
<table border=1>
<tr><td>a</td><td>b</td><td>c</td><td>d</td></tr>
<tr><td>e</td><td rowspan="2" colspan="2">f</td><td>g</td></tr>
<tr><td>h</td><td>i</td></tr>
</table>
EOF
$ cat <<'EOF'> table-scraper-config.json
[
{"type": "table-xpath", "label": "Tables", "query": "//table"}
]
EOF
$ scraper -c table-scraper-config.json < table.html
[
{
"type": "table-xpath",
"label": "Tables",
"query": "//table",
"results": [
[
["a", "b", "c", "d"],
["e", "f", "f", "g"],
["h", "f", "f", "i"]
]
]
}
]
$ wget http://example.com/ -O index.html
$ cat <<'EOF'> scraper-config.json
[
{"type": "regex", "label": "Description", "query": "This domain is .+? to be used for .+?\\."}
]
EOF
$ scraper < index.html
[
{
"type": "regex",
"label": "Description",
"query": "This domain is .+? to be used for .+?\\.",
"results": [
"This domain is established to be used for illustrative examples in documents."
]
}
]
First match only. Unlike css and xpath types which return all matching nodes,
the regex type returns only the first match found in the input text.
Capture groups. When the pattern contains capture groups (parentheses), the results
array includes the full match at index 0 followed by each capture group:
query: "Cat, (.*?), Snake"
input: "Cat, Dog, Snake and Cat, Rabbit, Snake"
results: ["Cat, Dog, Snake", "Dog"]
^^^^^^^^^^^^^^^^^ ^^^
full match (idx 0) capture group 1 (idx 1)
Note that only the first occurrence (Cat, Dog, Snake) is returned even though
the input contains a second match (Cat, Rabbit, Snake).
Use css or xpath if you need all occurrences.
Obviously these recipes are mixable. You can scrape all at once with one-config by writing these target.
Configurations can be checked by using subcommand validate.
This sub-command also checks format of each types.
For example,
regextype will be checked the query is valid regular expression.xpathtype will be checked the query is valid XPath.
$ scraper validate
Error: Recipe[0]: expected identifier, found < instead
exit status 1
$ scraper server$ curl -X PUT localhost:8080/example.com -d@- <<'EOT'
[{"type": "xpath", "label": "LinkURL", "query": "//a/@href"}]
EOT$ curl -X GET localhost:8080/example.com
[{"type": "xpath", "label": "LinkURL", "query": "//a/@href"}]$ curl example.com | curl -X POST localhost:8080/example.com -d @-
[
{
"type": "xpath",
"label": "LinkURL",
"query": "//a/@href",
"results": [
"http://www.iana.org/domains/example"
]
}
]Docker image for server mode: kitsuyui/scraper
$ docker run -p 8080:8080 -it kitsuyui/scraper
$ scraper -h
scraper: Swiss army knife for web scraping
Usage:
scraper [-c=<conf>] [-i=<input>]
scraper validate [-c=<conf>]
scraper server [-d=<conf-dir>] [-H=<host>] [-p=<port>]
Options:
-c=<conf> --config=<conf> Configuration file [default: scraper-config.json].
-i=<input> --input=<input> Input file specified instead of STDIN.
-o=<output> --output=<output> Output file specified instead of STDOUT.
-H=<host> --host=<host> Server mode host [default: 127.0.0.1].
-p=<port> --port=<port> Server mode port [default: 8080].
-d=<conf-dir> --conf-dir=<conf-dir> Configuration directory for server mode [default: .].
CLI diagnostics are written to stderr. Successful commands exit with status 0.
Error statuses are:
2: configuration validation failed.3: server mode startup failed.4: input file could not be opened.5: output file could not be created.6: configuration file could not be opened.7: scraping failed.
$ go get -d ./src/...
$ go build -o build/scraper main.go
$ ./bin/build.sh
$ docker run --rm -v "$(pwd)":/root -w /root tcnksm/gox sh -c "./bin/build.sh"Install lefthook and run the following command to set up the Git hooks:
$ lefthook install- pre-commit: runs
go vet ./...to catch issues before each commit. - pre-push: runs
go vet ./...andgo test ./...before each push.
The CI pipeline still runs the full test suite on every pull request and push to main — the hooks bring the same feedback earlier, during local development.
The 3-Clause BSD License. See also LICENSE file.