Hey @djkemmet, this is definitely something I'm interested in and I'm excited that you're willing to contribute towards seeing it happen.

This is good timing as I'm getting ready to write some grant applications for ArchiveBox and I'm hoping to fund my full-time development on it for the next few years. The API is definitely one of the main things I want to build in the next year, and I want to work on ArchiveBox as my main job if I can pull off finding a big sponsor.

The grants I'm aiming for are $150k+ so if your budget for this task is <$2k I wouldn't bother sponsoring this if you cant wait a few months, as I'll likely get around to it anyway. If your needs are more urgent than that or your budget is higher then lets discuss further via chat/call.

I agree either way we should finalize an API spec and framework choice, the work so far toward that is here: #496 (comment)

You can comment your thoughts on that draft API spec in that issue ^, and we can continue the discussion about timeline/sponsorship/and your specific needs for ML/NLP/etc. here.

Places I'm considering applying for grants:

• https://longnow.org
• https://www.arweave.org/funding (was offered grant in 2020, but was too busy then)
• https://grants.filecoin.io (have friends here, and they fund similar archiving tools)
• https://www.opentech.fund / https://www.opentech.fund/funds/internet-freedom-fund/ (specifically interested in censorship circumvention and internet freedom tooling)
• https://www.ripe.net/support/cpf (May 1st deadline)
• https://www.python.org/psf/grants/
• https://nlnet.nl/news/2023/20230201-call.html (April 1st deadline)
• https://www.nsf.gov/pubs/2023/nsf23556/nsf23556.htm (Sept 2023 deadline)
• Open Source Peer Bonus Program  |  Google Open Source
• MIECO: Advancing the future of the internet - Mozilla Innovations
• https://sfconservancy.org/projects/apply/ or https://opencollective.com/collectives (help with non-profit administration, legal stuff, and getting funding)
• more: https://github.com/ralphtheninja/open-funding, https://itsfoss.com/open-source-funding-platforms/, https://www.oss.fund

WOO! So excited to see that grant money might be a possibility! Btw, grant writing is becoming trivially easy with GPT.

heyy @pirate,

yeah my budget for this was $1000 USD If there's anyone in your network who might be suited for this work and assuming $1000 is a reasonable amount of money to get the API off the ground I would definitely be happy to sponsor someone based on your recommendation. that said, as far as grants go, have you looked at futo? https://futo.org/fellows/

Hello guys!
I can step up to work on this.

@pirate let me know if there's anything to consider while doing grant work for the project and what things to have in mind (like what you mentioned of grants already being discussed with other parties, and guidelines for the code to be more likely to be merged).

Hey @pe1uca thanks for the offer! I'd definitely accept help with grant applications, want to chat on a call sometime? lets find a calendar time on Zulip

A quick update for everyone here, I am currently in the process of setting up a contract with the United Nations to help with their internet archiving needs! That contract will focus directly on ArchiveBox development (keeping it all open source), as well as enhancing it for their specific needs. It's a slow moving bureaucratic process so it may not start for a few more months until we get all the approvals, but it's a major step forward in securing real money for part/full time ArchiveBox development going forward.

Just a call with @pe1uca and we discussed a good approach to get some API functionality before my event sourcing refactor is complete:

1. create a new app under archivebox/api (separate from core and CLI)
2. create an endpoint for each archivebox CLI subcommand (e.g. add, remove, list, schedule) and allow it to take config/env variables and CLI parameters as the POST request parameters
3. call the CLI code directly using the requests parameters, and return the stdout/stderr as the response

Notes:

• for long-running requests (e.g. an add that takes 20min), we can return a 200 and use this hack to run the job after the response is returned. later on we will use a proper job queue system, but because I am working on this in my refactor we don't want to depend on this just yet otherwise it will block the entire API development process. if the job is cancelled or times out early, it can be picked up automatically and resumed by an archivebox schedule ONLY_NEW=False job in the background, acting as a janky but cheap/easy background job system until my refactor is finished
• some off-the-shelf django library or snippet should be used to allow users to create and manage API tokens from the UI/CLI to be used with the new API endpoints. I think allowing the user to create named API tokens (that inherit the same permissions as that user) is enough for now, no need to do OAuth2 or JWT

We're also discussing this further on the ArchiveBox chat server here: https://zulip.archivebox.io/#narrow/stream/158-development/topic/REST.20API.20development/near/1166232

@pe1uca is going to take a look at doing this, @djkemmet if you're still willing to sponsor development it would be great to discuss it further between the three of us :)

Will this cover, say, returning a JSON payload of document content via a search? That would be the key thing for me.

Yes, certainly @rcarmo

I made a step towards an API by making the current CLI command available via API:

See #1397

The new REST API is rapidly evolving on dev! Big thanks to @Brandl for the first PR, keep an eye out for new releases!

https://github.com/ArchiveBox/ArchiveBox/releases/v0.8.0-rc

I'm going to close this issue for now and move these discussions to:

• REST API bugs and feature requests here: Question: what's the current status of the REST API? #496
• discuss ArchiveBox grants and funding here: https://zulip.archivebox.io/#narrow/stream/178-funding/topic/grants.20to.20apply.20to