Skip to content

Commit 58f49f2

Browse files
waybackarchiverwaybackarchiver
authoredMar 21, 2023
Several improvements (#335)
- Add client timeout for telegra.ph - Log pooling errors - Warcraft supports timeout and return waiting error - Telegra.ph performance improvement - Upload artifact remotely with timeout - Remotely file upload with separate function - Throw a fatal error if the command-line flag value is not specified - Replace os.Tempdir with testing.T.TempDir - Replace ioutil.ReadAll with io.ReadAll - Add storage testing - Change default ipfs port to 5001 - Place ipfs related environments for testing - Wrap testing using t.Run - Add `chromedp.NoModifyURL` compatibility
1 parent 142f095 commit 58f49f2

24 files changed

+324
-216
lines changed
 

‎.github/workflows/testing.yml

+8-3
Original file line numberDiff line numberDiff line change
@@ -50,6 +50,9 @@ jobs:
5050
SENDER_PWD: ${{ secrets.MATRIX_SENDER_PWD }}
5151
RECVER_UID: ${{ secrets.MATRIX_RECVER_UID }}
5252
RECVER_PWD: ${{ secrets.MATRIX_RECVER_PWD }}
53+
WAYBACK_IPFS_MODE: ${{ vars.WAYBACK_IPFS_MODE }}
54+
WAYBACK_IPFS_HOST: ${{ vars.WAYBACK_IPFS_HOST }}
55+
WAYBACK_IPFS_PORT: ${{ vars.WAYBACK_IPFS_PORT }}
5356
steps:
5457
- name: Harden Runner
5558
uses: step-security/harden-runner@2e205a28d0e1da00c5f53b161f4067b052c61f34 # v1.5.0
@@ -92,7 +95,7 @@ jobs:
9295
with:
9396
args: -h
9497

95-
- name: Install Packages
98+
- name: Install Packages for Linux
9699
if: matrix.os == 'ubuntu-latest'
97100
shell: bash
98101
run: |
@@ -105,7 +108,7 @@ jobs:
105108
you-get --version
106109
ffmpeg -version
107110
108-
- name: Install Packages
111+
- name: Install Packages for MacOS
109112
if: matrix.os == 'macos-latest'
110113
shell: bash
111114
run: |
@@ -115,7 +118,7 @@ jobs:
115118
you-get --version
116119
ffmpeg -version
117120
118-
- name: Install Packages
121+
- name: Install Packages for Windows
119122
if: matrix.os == 'windows-latest'
120123
shell: bash
121124
run: |
@@ -129,6 +132,8 @@ jobs:
129132
- name: Set environments
130133
shell: bash
131134
run: |
135+
ipfsMode="${{ vars.WAYBACK_IPFS_MODE }}"
136+
echo "WAYBACK_IPFS_MODE=${ipfsMode:-daemon}" >> $GITHUB_ENV
132137
# Set env to enable reduxer
133138
echo "WAYBACK_STORAGE_DIR=${{ runner.temp }}" >> $GITHUB_ENV
134139
# Append paths to environment path

‎README.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -225,7 +225,7 @@ You can also specify configuration options either via command flags or via envir
225225
| - | `LOG_LEVEL` | `info` | Log level, supported level are `debug`, `info`, `warn`, `error`, `fatal`, defaults to `info` |
226226
| - | `ENABLE_METRICS` | `false` | Enable metrics collector |
227227
| - | `WAYBACK_LISTEN_ADDR` | `0.0.0.0:8964` | The listen address for the HTTP server |
228-
| - | `CHROME_REMOTE_ADDR` | - | Chrome/Chromium remote debugging address, for screenshot |
228+
| - | `CHROME_REMOTE_ADDR` | - | Chrome/Chromium remote debugging address, for screenshot, format: `host:port`, `wss://domain.tld` |
229229
| - | `WAYBACK_POOLING_SIZE` | `3` | Number of worker pool for wayback at once |
230230
| - | `WAYBACK_BOLT_PATH` | `./wayback.db` | File path of bolt database |
231231
| - | `WAYBACK_STORAGE_DIR` | - | Directory to store binary file, e.g. PDF, html file |

‎cmd/wayback/serve.go

+1-1
Original file line numberDiff line numberDiff line change
@@ -181,7 +181,7 @@ func (srv *services) run(ctx context.Context, opts service.Options) *services {
181181
name: s,
182182
})
183183
default:
184-
logger.Error("unrecognize %s in `--daemon`", s)
184+
logger.Fatal("unrecognize %s in `--daemon`", s)
185185
}
186186
}
187187

‎config/options.go

+1-1
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,7 @@ const (
2323
defOverTor = false
2424

2525
defIPFSHost = "127.0.0.1"
26-
defIPFSPort = 4001
26+
defIPFSPort = 5001
2727
defIPFSMode = "pinner"
2828
defIPFSTarget = ""
2929
defIPFSApikey = ""

‎docs/changelog.md

+15
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,21 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
3636
- No longer build image for `linux/s390x`
3737
- Get rid of the Tor binary ([#336](https://github.com/wabarc/wayback/pull/336))
3838
- Adjusting lux to pluggable mode ([#337](https://github.com/wabarc/wayback/pull/337))
39+
- Several improvements ([#335](https://github.com/wabarc/wayback/pull/335))
40+
- Add client timeout for telegra.ph
41+
- Log pooling errors
42+
- Warcraft supports timeout and return waiting error
43+
- Telegra.ph performance improvement
44+
- Upload artifact remotely with timeout
45+
- Remotely file upload with separate function
46+
- Throw a fatal error if the command-line flag value is not specified
47+
- Replace os.Tempdir with testing.T.TempDir
48+
- Replace ioutil.ReadAll with io.ReadAll
49+
- Add storage testing
50+
- Change default ipfs port to 5001
51+
- Place ipfs related environments for testing
52+
- Wrap testing using t.Run
53+
- Add `chromedp.NoModifyURL` compatibility
3954

4055
### Fixed
4156
- Fix semgrep scan workflow ([#312](https://github.com/wabarc/wayback/pull/312))

‎docs/resources.md

+5
Original file line numberDiff line numberDiff line change
@@ -22,3 +22,8 @@ From the popular Wayback Machine to lesser-known platforms, there is something h
2222
## Wiki
2323

2424
- [Web archiving](https://en.wikipedia.org/wiki/Web_archiving)
25+
26+
## Tools
27+
28+
- [Browserless](https://www.browserless.io/): Web Automation & Headless Browser Automation Tool.
29+

‎go.mod

+10-10
Original file line numberDiff line numberDiff line change
@@ -5,16 +5,16 @@ module github.com/wabarc/wayback
55
go 1.18
66

77
require (
8-
github.com/PuerkitoBio/goquery v1.8.0
8+
github.com/PuerkitoBio/goquery v1.8.1
99
github.com/bwmarrin/discordgo v0.23.3-0.20210627161652-421e14965030
1010
github.com/cretz/bine v0.2.0
1111
github.com/davecgh/go-spew v1.1.1
1212
github.com/dghubble/go-twitter v0.0.0-20201011215211-4b180d0cc78d
1313
github.com/dghubble/oauth1 v0.7.1
1414
github.com/dstotijn/go-notion v0.6.1
1515
github.com/dustin/go-humanize v1.0.0
16-
github.com/fatih/color v1.13.0
17-
github.com/gabriel-vasile/mimetype v1.4.1
16+
github.com/fatih/color v1.15.0
17+
github.com/gabriel-vasile/mimetype v1.4.2
1818
github.com/go-shiori/go-readability v0.0.0-20220215145315-dd6828d2f09b
1919
github.com/go-shiori/obelisk v0.0.0-20221119111008-23c015a8fad7
2020
github.com/google/go-github/v40 v40.0.0
@@ -38,15 +38,15 @@ require (
3838
github.com/wabarc/archive.org v1.2.1-0.20210708220121-cb9b83ff9896
3939
github.com/wabarc/go-anonfile v0.1.0
4040
github.com/wabarc/go-catbox v0.1.0
41-
github.com/wabarc/helper v0.0.0-20230209075818-96584f1ebf9d
41+
github.com/wabarc/helper v0.0.0-20230318095659-969de9ddf4b6
4242
github.com/wabarc/imgbb v1.0.0
4343
github.com/wabarc/ipfs-pinner v1.1.1-0.20220126131044-16299c0dd43d
4444
github.com/wabarc/logger v0.0.0-20210730133522-86bd3f31e792
4545
github.com/wabarc/playback v0.0.0-20220715111526-90d0327d3f04
4646
github.com/wabarc/rivet v0.1.4-0.20221226142645-ebc8a29d914f
47-
github.com/wabarc/screenshot v1.6.0
48-
github.com/wabarc/telegra.ph v0.0.0-20221226141851-edf1cc14c076
49-
github.com/wabarc/warcraft v0.2.2-0.20211107142816-7beea5a75ab5
47+
github.com/wabarc/screenshot v1.6.1-0.20230315004517-7587f8bc14e0
48+
github.com/wabarc/telegra.ph v0.0.0-20230318134541-a0922e1ace3a
49+
github.com/wabarc/warcraft v0.3.1-0.20230308125707-3daa5592ba52
5050
go.etcd.io/bbolt v1.3.6
5151
golang.org/x/net v0.8.0
5252
golang.org/x/sync v0.1.0
@@ -68,8 +68,8 @@ require (
6868
github.com/cenkalti/backoff/v4 v4.2.0 // indirect
6969
github.com/cespare/xxhash/v2 v2.2.0 // indirect
7070
github.com/cheggaaa/pb/v3 v3.0.8 // indirect
71-
github.com/chromedp/cdproto v0.0.0-20221126224343-3a0787b8dd28 // indirect
72-
github.com/chromedp/chromedp v0.8.6 // indirect
71+
github.com/chromedp/cdproto v0.0.0-20230310204135-a6d692f2c96d // indirect
72+
github.com/chromedp/chromedp v0.9.1 // indirect
7373
github.com/chromedp/sysutil v1.0.0 // indirect
7474
github.com/crackcomm/go-gitignore v0.0.0-20170627025303-887ab5e44cc3 // indirect
7575
github.com/decred/dcrd/crypto/blake256 v1.0.0 // indirect
@@ -97,7 +97,7 @@ require (
9797
github.com/itchyny/timefmt-go v0.1.3 // indirect
9898
github.com/josharian/intern v1.0.0 // indirect
9999
github.com/json-iterator/go v1.1.12 // indirect
100-
github.com/kallydev/telegraph-go v1.0.0 // indirect
100+
github.com/kallydev/telegraph-go v1.0.1-0.20230318133700-df034d9eed50 // indirect
101101
github.com/kennygrant/sanitize v1.2.4 // indirect
102102
github.com/kkdai/youtube/v2 v2.7.18 // indirect
103103
github.com/klauspost/cpuid/v2 v2.2.2 // indirect

0 commit comments

Comments
 (0)