In #1678 I added an automated tool for scoring the conformance test suite, and #1682 and #1683 added some features to support edge cases.

Now I am working on going through the remaining files where the automated conformance judgment on a file differs from the manual one (open PRs: #1684, #1685, #1687, #1688, #1689, #1690, #1691).

I'll use this issue to track the remaining discrepancies (from conformance/src/unexpected_fails.py):

• mypy/dataclasses_usage.toml: Pass vs. Fail (Conformance tests: Fix automated scoring for dataclasses_usage.toml #1684)
• mypy/directives_version_platform.toml: Pass vs. Fail (Conformance test suite: Fix automation discrepancies in directives_version_platform #1685)
• mypy/namedtuples_define_functional.toml: Pass vs. Fail
• mypy/overloads_basic.toml: Pass vs. Fail (Conformance tests: Fix automated checking on overloads_basic.py #1687)
• mypy/specialtypes_never.toml: Pass vs. Fail (Conformance tests: Update Never test #1688)
• mypy/typeddicts_alt_syntax.toml: Pass vs. Fail (Conformance tests: Allow mypy error on typeddicts_alt_syntax.py #1689)
• mypy/typeddicts_type_consistency.toml: Pass vs. Fail (Conformance tests: Allow TD.get(known_key) to be T | None #1690)
• pyre/directives_reveal_type.toml: Unsupported vs. Pass
• pyre/namedtuples_define_functional.toml: Pass vs. Fail
• pyright/classes_override.toml: Pass vs. Fail (Conformance tests: Fix automated scoring in classes_override.py #1691)
• pyright/dataclasses_frozen.toml: Pass vs. Fail (Conformance tests: Fix automated scoring for pyright dataclasses tests #1699)
• pyright/dataclasses_slots.toml: Pass vs. Fail (Conformance tests: Fix automated scoring for pyright dataclasses tests #1699)
• pyright/dataclasses_transform_field.toml: Pass vs. Fail (Conformance tests: Fix automated scoring for pyright dataclasses tests #1699)
• pyright/dataclasses_transform_func.toml: Pass vs. Fail (Conformance tests: Fix automated scoring for pyright dataclasses tests #1699)
• pyright/generics_defaults.toml: Pass vs. Fail (Conformance tests: Fix generics_defaults scoring for pyright #1700)
• pyright/literals_literalstring.toml: Pass vs. Fail (Conformance tests: Ignore overlapping overload errors in LiteralString #1701)
• pyright/overloads_basic.toml: Pass vs. Fail (Conformance tests: Fix automated checking on overloads_basic.py #1687)
• pyright/protocols_variance.toml: Pass vs. Fail (Conformance tests: Fix pyright scoring for protocols_variance #1702)
• pyright/specialtypes_never.toml: Pass vs. Fail (Conformance tests: Update Never test #1688)
• pyright/tuples_unpacked.toml: Pass vs. Fail (Conformance tests: Fix tuples_unpacked.py #1703)
• pytype/dataclasses_usage.toml: Pass vs. Fail (Conformance tests: Fix automated scoring for dataclasses_usage.toml #1684)
• pytype/directives_assert_type.toml: Pass vs. Fail (Conformance tests: mark pytype as failing assert_type test #1706)
• pytype/directives_version_platform.toml: Pass vs. Fail (Conformance test suite: Fix automation discrepancies in directives_version_platform #1685)
• pytype/generics_scoping.toml: Pass vs. Fail (Conformance tests: mark pytype as failing generics_scope #1704)
• pytype/typeddicts_inheritance.toml: Pass vs. Fail (conformance tests: fix scoring for typeddicts_inheritance #1705)

Any help is appreciated.