Read and write the Pipfile as UTF-8 everywhere - #6756

The issue

No issue filed; this is a follow-up to #6750.

Pipfile.read() decodes the Pipfile as utf-8-sig, but a few other places still open it with the locale encoding: the Pipfile hash (Pipfile.calculate_hash()), the lock data built from the Pipfile when there is no usable Pipfile.lock (Lockfile.as_dict() and Lockfile.load() when _meta is missing), and Pipfile.write_toml(). On Windows the locale encoding is a legacy code page, so a Pipfile with any non-ASCII text, even just a comment, breaks:

• pipenv lock on Japanese Windows (cp932) with # 開発用の依存関係 at the top of the Pipfile:

  File "...\pipenv\utils\lockfile.py", line 205, in as_dict
  ...
  UnicodeDecodeError: 'cp932' codec can't decode byte 0x82 in position 25: illegal multibyte sequence
  

• pipenv install colorama with the same Pipfile rewrites it in cp932 (the comment becomes \x8aJ\x94\xad...) and then fails with UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8a, because the next read is UTF-8. The Pipfile stays broken for every later command and on every other machine.

Separately, pipenv.utils.locking.Lockfile.load() still reads Pipfile.lock with the locale encoding. With a BOM, json raises JSONDecodeError, so the lockfile is reported as corrupt and moved to Pipfile.lock.bak (on Linux too).

The fix

Read the Pipfile as utf-8-sig and write it as UTF-8 in those places, and read Pipfile.lock as utf-8-sig in locking.Lockfile.load(), matching what #6750 did for the other readers. A Pipfile that pipenv rewrites still loses its BOM, as before.

tests/unit/test_pipfile_encoding.py adds 9 cases (Pipfile with non-ASCII text, saved with and without a BOM, through the hash, write_toml(), both Pipfile-derived lock paths, plus a Pipfile.lock with a BOM through locking.Lockfile.load()). On Windows with cp932 all 9 fail without the change; under UTF-8 (-X utf8, like Linux) the 4 BOM cases fail. All pass with it. tests/unit otherwise gives the same results as on main here (test_help and the test_release_tasks.py/test_vendor.py imports already fail locally).

The checklist

• Associated issue
• A news fragment in the news/ directory to describe this fix with the extension .bugfix.rst, .feature.rst, .behavior.rst, .doc.rst. .vendor.rst. or .trivial.rst (this will appear in the release changelog). Use semantic line breaks and name the file after the issue number or the PR #.

I used Claude Code to write this change.