- Modified SourceFile.Load() to detect UTF-8 BOM in files
- Modified SourceFile.Save() to preserve original BOM state
- Added HasUtf8Bom property to SourceFile struct
- Updated InternalAPI.Unshipped.txt with new API members
- Added tests to verify BOM preservation (no-BOM and with-BOM cases)

Fixes issue where dotnet add package command was adding BOM to files
that originally didn't have one, breaking shebang scripts.

Co-authored-by: DamianEdwards <[email protected]>

- Use SourceText's Encoding property instead of custom HasUtf8Bom tracking
- Use stackalloc for 3-byte buffer on .NET Core (with fallback for netstandard2.0)
- Update tests to use _testAssetsManager.CreateTestDirectory() infrastructure
- Add issue reference (#52054) to test docs
- Remove unnecessary API surface from InternalAPI.Unshipped.txt

Co-authored-by: jjonescz <[email protected]>

- Remove custom DetectEncoding() method and use SourceText.From(stream, encoding: null)
- SourceText automatically detects encoding including BOM when encoding is null
- Move <see href/> tags inside <summary> elements for better documentation
- Simplifies implementation while maintaining full functionality

Co-authored-by: jjonescz <[email protected]>

- Add PreservesNonUtf8Encoding test to verify UTF-16 encoding is preserved
- Tests that files with UTF-16 LE encoding (with BOM 0xFF 0xFE) maintain their encoding
- Ensures SourceText's auto-detection works for encodings beyond UTF-8
- All encoding preservation tests now pass (UTF-8 no BOM, UTF-8 with BOM, UTF-16)

Co-authored-by: jjonescz <[email protected]>

Co-authored-by: Fred Silberberg <[email protected]>

- Replace Assert.StartsWith with Assert.Equal to verify the entire file content
- Ensures the test validates that Save() correctly writes the complete content
- Makes the test more robust by checking the full output including the added package directive

Co-authored-by: jjonescz <[email protected]>

- Replace raw string literal with explicit \n characters in expected content
- Raw string literals with indentation were causing blank lines to be stripped
- Explicit newlines ensure consistent comparison across platforms
- Fixes test failure on Windows where line endings differ

Co-authored-by: jjonescz <[email protected]>