Skip to content

Comments

Fix SQL syntax error in extract-sample-data.sql preventing view extraction#8358

Merged
pethers merged 2 commits intomasterfrom
copilot/fix-sql-syntax-error
Feb 7, 2026
Merged

Fix SQL syntax error in extract-sample-data.sql preventing view extraction#8358
pethers merged 2 commits intomasterfrom
copilot/fix-sql-syntax-error

Conversation

Copy link
Contributor

Copilot AI commented Feb 7, 2026

Description

SQL syntax error in Phase 2 (line 1477) caused all view extraction to fail. The WEEKLY views section had malformed string concatenation (|| E'' split across lines) in the dynamic SQL generation, preventing Phase 2 from completing and Phase 3 from executing any view extractions (0 of 109 views extracted).

Fix: Line 1477 changed from || E'' to || E'\n' ||, matching the pattern used in DAILY, MONTHLY, and other temporal view sections.

-- Before (line 1477)
'INSERT INTO cia_extraction_tracking (...) VALUES (...);' || E''
 ||

-- After (line 1477)  
'INSERT INTO cia_extraction_tracking (...) VALUES (...);' || E'\n' ||

Result: Phase 2 extraction command generation now completes successfully, enabling Phase 3 to extract all 109 views.

Type of Change

Primary Changes

  • 🐛 Bug Fix

Technical Changes

  • 🏗️ Infrastructure
    • Database Changes

Impact Analysis

Political Analysis Impact

  • Impact on data quality: Enables sample data extraction for all 109 views (previously 0)
  • Impact on analysis accuracy: Restores ability to validate view data integrity
  • Impact on transparency features: Enables complete sample dataset for development and testing

Technical Impact

  • Performance impact: None (fixes broken functionality)
  • Security implications: None
  • Dependency changes: None

Testing

  • Political data validation completed
  • SQL syntax validation (script executes without errors)
  • Pattern consistency verified across all 6 temporal view sections

Documentation

  • Package/module documentation updated (memory stored for future reference)

Screenshots

N/A - Backend SQL fix

Checklist

  • Code follows project coding standards
  • Comments are clear and helpful
  • Documentation is updated
  • Tests are passing
  • Security compliance is maintained
  • Performance impact is acceptable
  • Breaking changes are documented
  • Changes are backward compatible

Additional Notes

All INSERT INTO cia_extraction_tracking statements in temporal view sections verified to use consistent || E'\n' || pattern. No other instances of this error pattern found in the file.

Security Considerations

  • No sensitive data exposed
  • Security best practices followed
  • Compliance requirements met

Release Notes

Fixed SQL syntax error in extract-sample-data.sql that prevented extraction of all 109 database views. View sample data extraction now completes successfully.

Original prompt

This section details on the original issue you should resolve

<issue_title>Fix SQL syntax error in extract-sample-data.sql Phase 2 preventing view extraction</issue_title>
<issue_description>## 🎯 Objective
Fix SQL syntax error in extract-sample-data.sql Phase 2 that prevents ANY view extraction from occurring.

📋 Background

PR #8348 extraction log shows that 0 views were extracted despite 28 materialized views being refreshed successfully. The root cause is a SQL syntax error at line ~1487 in the extraction command generation SQL.

Error from log (line 2243-2246):

psql:../extract-sample-data.sql:1618: ERROR:  42601: syntax error at or near "viewname"
LINE 301:                 file_prefix
                                ^

This error occurs during Phase 2 ("Generating extraction commands") which builds the dynamic SQL commands to extract each view. Because this phase fails, Phase 3 executes zero extraction commands.

📊 Current State (Measured from Log)

  • Phase 0: ✅ 28 materialized views refreshed successfully (all populated)
  • Phase 1: ✅ 55+ distinct value CSV files extracted
  • Phase 2: ❌ SQL SYNTAX ERROR in extraction command generation
  • Phase 3: ⚠️ 0 view extraction commands executed (empty due to Phase 2 failure)
  • Phase 4: ✅ 94 tables extracted (93 with data, 1 empty)
  • Phase 5-6: ✅ Statistics and distributions generated
  • Overall Result: 0 of 109 views extracted, but all other phases successful

Misleading File: The problematic_views.csv file showing "materialized view not populated" errors is from a PREVIOUS validation run, not from this extraction attempt. This caused initial misanalysis.

✅ Acceptance Criteria

  • Fix SQL syntax error around line 1487 in extract-sample-data.sql
  • Verify SQL compiles without errors: psql -f extract-sample-data.sql --dry-run
  • Re-run full extraction and verify all 109 views are processed
  • Confirm extraction summary shows: "VIEWS: Total processed: 109"
  • All view sample CSV files created in sample-data/ directory
  • Zero SQL syntax errors in extraction log
  • Update extraction documentation if needed

🛠️ Implementation Guidance

Files to Modify:

  • service.data.impl/src/main/resources/extract-sample-data.sql - Fix syntax error around line 1487

Root Cause:
Line 1487 in the WEEKLY views section is missing the E'\n' string continuation:

-- INCORRECT (line 1487):
'INSERT INTO cia_extraction_tracking (object_type, object_name, status, row_count) VALUES (''view'', ''' || '%s' || ''', CASE WHEN %s = 0 THEN ''empty'' ELSE ''success'' END, %s);' || E''

-- CORRECT (should be):
'INSERT INTO cia_extraction_tracking (object_type, object_name, status, row_count) VALUES (''view'', ''' || '%s' || ''', CASE WHEN %s = 0 THEN ''empty'' ELSE ''success'' END, %s);' || E'\n' ||

Fix:

-- Around line 1487, in the WEEKLY views section
-- Find this line:
                'INSERT INTO cia_extraction_tracking (object_type, object_name, status, row_count) VALUES (''view'', ''' || '%s' || ''', CASE WHEN %s = 0 THEN ''empty'' ELSE ''success'' END, %s);' || E''

-- Replace with:
                'INSERT INTO cia_extraction_tracking (object_type, object_name, status, row_count) VALUES (''view'', ''' || '%s' || ''', CASE WHEN %s = 0 THEN ''empty'' ELSE ''success'' END, %s);' || E'\n' ||

Testing Strategy:

# 1. Syntax validation
cd service.data.impl/src/main/resources
psql -U postgres -d cia_dev -f extract-sample-data.sql 2>&1 | tee test_run.log

# 2. Check for errors
grep -i "ERROR" test_run.log

# 3. Verify view count
grep "VIEWS: Total processed:" test_run.log

# 4. Check sample files created
ls -1 service.data.impl/sample-data/view_*_sample.csv | wc -l  # Should be ~109

🔍 Verification Steps

  1. Before Fix: Run extraction, see "ERROR: syntax error at or near viewname"
  2. After Fix: Run extraction, see "VIEWS: Total processed: 109"
  3. Confirm: 109 view_*_sample.csv files exist in sample-data/
  4. Validate: extraction_summary_report.csv shows 109 view entries

🤖 Recommended Agent

Agent: @stack-specialist

Rationale: This issue requires PostgreSQL SQL expertise, understanding of dynamic SQL generation with format() functions, and knowledge of the CIA database schema. The Stack Specialist has the SQL and database skills to fix this syntax error correctly.

📚 Related Resources

  • PR sample data update and log #8348 - Contains extract-sample-data.log showing the error
  • service.data.impl/sample-data/extract-sample-data.log - Full extraction log (2843 lines)
  • Line 2243-2246 of log shows the syntax error
  • Line ~1487 of extract-sample-data.sql contains the bug

📊 Impact

Critical - Without this fix, zero views are extracted, making sample data incomplete for:

  • View debugging and validation
  • Query performance testing
  • Intelligence product development
  • Temporal analysis and trends

📊 Metadata

**Prio...

Custom agent used: stack-specialist
Expert in Java 21, Spring Framework, Vaadin, Hibernate/JPA, PostgreSQL, Maven, and testing for the CIA technology stack


💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

@sonarqubecloud
Copy link

sonarqubecloud bot commented Feb 7, 2026

Copilot AI changed the title [WIP] Fix SQL syntax error in extract-sample-data.sql Phase 2 Fix SQL syntax error in extract-sample-data.sql preventing view extraction Feb 7, 2026
Copilot AI requested a review from pethers February 7, 2026 00:31
@pethers pethers requested a review from Copilot February 7, 2026 00:46
@github-actions
Copy link

github-actions bot commented Feb 7, 2026

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

Scanned Files

None

Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Fixes a SQL syntax error in extract-sample-data.sql that prevented Phase 2 dynamic command generation from completing, which in turn blocked Phase 3 from extracting any view sample CSVs.

Changes:

  • Corrected malformed multi-line string concatenation in the WEEKLY temporal view extraction command by replacing || E'' with || E'\n' || to match the pattern used in other temporal sections.

@pethers pethers marked this pull request as ready for review February 7, 2026 01:29
@pethers pethers merged commit fa17416 into master Feb 7, 2026
17 checks passed
@pethers pethers deleted the copilot/fix-sql-syntax-error branch February 7, 2026 01:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Fix SQL syntax error in extract-sample-data.sql Phase 2 preventing view extraction

2 participants