ah interesting, all the files I tested with had an object at the top level instead of a list

I can add handling for lists pretty easily, there are so many different JSON formats to support haha

Instead of trying to figure out what is going on when the first line of the JSON file is garbage, it would be easier to try not skipping it first, and then try again after skipping it if that fails.

    try:
        links = json.load(json_file)
    except json.decoder.JSONDecodeError:
        # sometimes the first line is a comment or other junk, so try without
        json_file.seek(0)
        first_line = json_file.readline()
        #print('      > Trying JSON parser without first line: "', first_line.strip(), '"', sep= '')
        links = json.load(json_file)
        # we may fail again, which means we really don't know what to do

But maybe even this isn't necessary? It looks like the original "skip the first line" logic came about because ArchiveBox would add the filename to the file as the first line when putting in the sources directory, but that doesn't seem to happen any more (which seems like a much, much better way to go).

But maybe even this isn't necessary?

Yes but I want to keep the workaround logic as a fallback because users still have the old "filename as first line" style imports in their sources/ dir and they might want to re-import their sources again later on.

I do agree we should move it to a try: except: fallback though as you showed above.

Yes but I want to keep the workaround logic as a fallback because users still have the old "filename as first line" style imports in their sources/ dir and they might want to re-import their sources again later on.

I do agree we should move it to a try: except: fallback though as you showed above.

I don't see how the existing workaround ever worked for anything because it chops off everything before the first { which includes the [, and the rest of the parsing assumes that links is a list.

Ah sorry, it was long enough ago that I don't remember what it was for exactly... maybe it was to handle an extra newline at the start, or maybe I thought I was handling a JSON object at the top level instead of JSONL?

Either way I'm down to change it, this parser is broken enough that it's not useful in its current state anyway.

Handling JSONL wouldn’t be hard to add as a another fallback. We could try JSON, then JSONL, and then try them both again without the first line to handle old source lists that had that extra line added.

This is done, thanks again @jimwins for all your great work here!

Will be out in the next release, or pull :dev to get it early.