These if psql ... | tee ...; then success checks won’t detect SQL errors from the scripts because the SQL files set \set ON_ERROR_STOP off, meaning psql can exit 0 even when commands fail. As a result, the script may print “completed” and proceed even though CSV generation failed. Consider enabling fail-fast for these runs (remove/override ON_ERROR_STOP off) or explicitly scan the log for ERROR: and treat it as a failure.

Fixed in 4f690e0. Added grep -q "^ERROR:" logfile checks after each psql execution. Scripts now detect SQL errors and return 1, even with \set ON_ERROR_STOP off.

This section claims the extraction is “read-only” (SELECT only), but the new SQL scripts run REFRESH MATERIALIZED VIEW, which performs writes/locks and requires additional privileges. Update the documentation to reflect that these scripts modify materialized views (or adjust the scripts if strict read-only execution is a requirement).

Fixed in 4f690e0. Updated EXTRACTION_ENHANCEMENT_SUMMARY.md to clarify that scripts run REFRESH MATERIALIZED VIEW which requires write/lock privileges on materialized views (but doesn't modify base tables).

This report says extract-all-data.sh is “NOT TESTED”, but the PR description and EXTRACTION_ENHANCEMENT_SUMMARY.md both state the orchestration script was tested. Please reconcile these statements so readers don’t get conflicting guidance about test coverage/status.

Fixed in 4f690e0. Updated SQL_EXTRACTION_TEST_REPORT.md section 3 to reflect that extract-all-data.sh was tested successfully with all 3 phases executing correctly.

Vote comparisons in the annual distributions are still inconsistent: distribution_annual_ballots.csv now checks vote = 'JA'/'NEJ', but distribution_annual_party_votes.csv earlier in this file still uses mixed-case values ('Ja', 'Nej', 'Frånvarande'). Since vote_data.vote values are uppercase, the party vote distribution will keep producing 0 counts until those literals are updated too.

Fixed in 4f690e0. Changed line 1992 in extract-sample-data.sql to use uppercase: 'JA', 'NEJ', 'FRÅNVARANDE'. Both annual distributions now use consistent uppercase vote values.

person_data.status filter uses p.status = 'Tjänstgörande ledamot', but the dataset uses values like Tjänstgörande riksdagsledamot / Tjänstgörande ersättare (see sample-data status distribution). As written, party_members_current.csv will likely be empty even when there are active members. Update the filter to match the actual status values (e.g., an IN list of active statuses).

Fixed in 4f690e0. Updated line 67 in extract-party-data.sql to use correct status values: IN ('Tjänstgörande riksdagsledamot', 'Tjänstgörande ersättare') matching actual person_data.status distribution.

In party_voting_summary.csv, COUNT(DISTINCT embedded_id_ballot_id) undercounts because a unique ballot in view_riksdagen_vote_data_ballot_party_summary is identified by (embedded_id_ballot_id, embedded_id_concern, embedded_id_issue) (see matview definition in full_schema.sql). Also, win_rate currently uses the overall approved flag, which will be largely identical across parties; if the intent is a party “win rate”, it should be based on party_won (or a similarly party-specific outcome column).

Fixed in 4f690e0. Updated line 107 in extract-party-data.sql:

• Ballot counting now uses composite PK: COUNT(DISTINCT (embedded_id_ballot_id, embedded_id_concern, embedded_id_issue))
• Win rate now uses party-specific column: CASE WHEN party_won THEN 1 instead of approved

The file counting logic uses ls <glob> | wc -l under set -euo pipefail. If the glob matches 0 files (e.g., extraction failed or output dir is empty), ls exits non-zero and the script will terminate early. Use a safer counting approach (e.g., shopt -s nullglob + array length, compgen -G, or find) so the script can report missing outputs instead of aborting.

Fixed in 4f690e0. Replaced all ls *.csv | wc -l with safe array counting using shopt -s nullglob. Script now handles empty globs gracefully without aborting under set -euo pipefail.

The extract_minister_data() function is missing the SQL error detection pattern that was added to extract_party_data(). Since both SQL scripts use \set ON_ERROR_STOP off, psql can exit with code 0 even when SQL errors occur. Add the same error detection logic after the psql execution to scan the log for "^ERROR:" and return 1 if found.

Fixed in 5adfd1c. Added SQL error detection to extract_minister_data() using the same grep -q "^ERROR:" pattern as extract_party_data().

This file counting logic uses ls <glob> | wc -l under set -euo pipefail. If the glob matches 0 files, ls exits non-zero and the script will terminate early. Use the safe nullglob pattern like in extract_party_data() (lines 147-149) to handle empty globs gracefully without aborting.

Fixed in 5adfd1c. Replaced ls | wc -l with safe nullglob pattern: shopt -s nullglob; minister_files_array=(minister*.csv government*.csv ministry*.csv); count=${#minister_files_array[@]}; shopt -u nullglob.

These file counting operations use ls <glob> | wc -l with 2>/dev/null redirection. Under set -euo pipefail, if the glob matches 0 files, the redirection prevents script termination but the count will be incorrect. Use the safe nullglob pattern (as in extract_party_data() lines 147-149) for consistent and accurate file counting.

Fixed in 5adfd1c. Replaced all 4 ls | wc -l operations with safe nullglob pattern for table, view, distribution, and percentile file counting. Consistent with extract_party_data() implementation.

extract_sample_data() treats a zero psql exit code as success, but extract-sample-data.sql sets \set ON_ERROR_STOP off, so SQL errors/timeouts can still produce exit code 0. This can lead to “completed” being printed even when ERROR: occurred and required CSVs were not generated. Add the same grep -q "^ERROR:" extract-sample-data.log check here (or enable ON_ERROR_STOP for this run).

Fixed in a4c573b. Added SQL error detection to extract_sample_data(): captures ${PIPESTATUS[0]} for psql exit code check, then greps extract-sample-data.log for ^ERROR: pattern. Both checks return 1 on failure.

Party extraction appends to extract-sample-data.log and then greps the entire file for ^ERROR:. If Phase 1 logged an error, this phase will fail even if party extraction succeeded, and the error message will incorrectly attribute the failure to party extraction. Consider writing a dedicated phase log (e.g., extract-party-data.log) or grepping only the current command’s output (e.g., via process substitution/temporary file) so failures are correctly scoped and reported.

Fixed in a4c573b. Party extraction now uses dedicated extract-party-data.log, greps only that log for errors, then appends to consolidated log. No more false failures from Phase 1 errors.

Same issue as in extract_party_data(): this phase appends to extract-sample-data.log and greps the full consolidated log for ^ERROR:, which can produce false failures and misattribute earlier errors to minister extraction. Use a per-phase log or grep only this invocation’s output before appending to the consolidated log.

Fixed in a4c573b. Minister extraction now uses dedicated extract-minister-data.log, greps only that log for errors, then appends to consolidated log. Consistent with party extraction pattern.

main() masks failures from the extraction phases (cmd || print_warning ...) and always exits 0 after printing “Extraction Complete”. That makes the script hard to use in automation/CI because callers can’t reliably detect failure. Track a global failure flag / exit code and exit 1 (or propagate the failing phase’s code) when any phase fails.

Fixed in a4c573b. main() now tracks exit_code across all 3 phases, returns it, and script ends with exit $? to propagate failures to callers. Script now CI/automation friendly.

The “Voting (6)” list contains 7 items (adds annual_document_status). Update the count or the list so it matches the actual set of generated distribution files.

Fixed in a4c573b. Changed "Voting (6)" to "Voting (7)" to match the 7 items listed: annual_party_votes, annual_ballots, decision_trends, document_types, annual_document_types, document_status, annual_document_status.

The “Other (6)” list contains 5 items. Update the count or add the missing distribution name(s) so the category totals are accurate.

Fixed in a4c573b. Changed "Other (6)" to "Other (5)" to match the 5 items listed: anomaly_by_party, election_regions, experience_levels, assignment_roles, influence_buckets. Total now correctly adds to 33 (9+7+5+4+3+5).

The “Other Files (6)” header doesn’t match the list below it: items 29–33 are 5 files. Please correct the count (and ensure the category totals still sum to 33).

Fixed in 0adbd35. Changed "Other Files (6)" to "Other Files (5)" to match the 5 items listed (29-33).

\echo 'Started:' date`` won’t execute date inside psql; it will print the literal backticked text, so the start timestamp is misleading. If you want a timestamp in the script output, use psql’s `! date` (or remove the `date` entirely and rely on the calling shell/log timestamps).

Fixed in 0adbd35. Changed \echo 'Started:' \date`to\echo 'Started:'followed by! date` to properly execute date command in psql.

extract_minister_data() has a broken if/else/fi structure around the minister file count check (there’s an extra else and mismatched fi). As written, the script will fail to parse/execute in bash. Restructure this block to a single if [ "$minister_files" -eq 7 ]; then ... else ... fi without the extra else branch, and ensure the function always returns 0 on success.