Conversation
…lication data synchronization
…ed data parsing for synchronization
…nt retrieval and refactor fetch logic
…tions, members, and research areas
… relationship synchronization
…ces and reorganize exports
… overview statistics
…uring sync operations
9 tasks
There was a problem hiding this comment.
Pull request overview
Adds an admin-facing publication sync system that updates external metadata (citations/keywords/authors) and rebuilds Publication↔Member↔ResearchArea relationships, with a new Sync UI and an SSE-backed API for live progress.
Changes:
- Introduces a publication sync engine (
runSync) with metadata and relationship rebuild phases, plus a matching engine for Member/Area linkage. - Expands OpenAlex/Crossref clients and adds a unified DOI sync service for “OpenAlex-first, Crossref-fallback”.
- Adds an Admin “Data Sync” page with stats, controls, and live progress via an SSE API route.
Reviewed changes
Copilot reviewed 19 out of 19 changed files in this pull request and generated 9 comments.
Show a summary per file
| File | Description |
|---|---|
| src/lib/publications/sync.ts | New orchestration for metadata sync + relationship rebuild with SSE events and a process-local lock/cache |
| src/lib/publications/matching.ts | New multi-signal matching logic for Publication↔Member and Publication↔Area |
| src/lib/publications/openalex.ts | Adds sync-focused OpenAlex fetch/parse including citations, keywords, and ORCID author info |
| src/lib/publications/crossref.ts | Adds citation-only fetch helper and refactors fetching into a shared helper |
| src/lib/publications/doi-sync.ts | New unified DOI sync service (OpenAlex → Crossref fallback) |
| src/lib/publications/shared.ts | Adds sync-related types/constants (AuthorInfo/SyncMetadata/SYNC_DELAY/MIN_KEYWORD_SCORE) |
| src/lib/publications/index.ts | Re-exports new sync services and types |
| src/lib/admin/actions/sync.ts | Adds getSyncStats server action for the admin sync dashboard |
| src/lib/admin/actions/index.ts | Re-exports sync action/types |
| src/app/api/admin/sync/route.ts | New SSE endpoint to run sync and stream progress events |
| src/components/admin/layout/sidebar-nav.tsx | Adds “Tools → Data Sync” nav entry |
| src/app/admin/(authenticated)/sync/page.tsx | New admin sync page wired to getSyncStats |
| src/app/admin/(authenticated)/sync/loading.tsx | Loading skeleton for sync page |
| src/app/admin/(authenticated)/sync/_components/* | Client UI for stats, controls, SSE state handling, and progress rendering |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
TKanX
deleted the
feature/25-implement-publication-data-sync-relationship-matching-system
branch
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary:
Implemented a sophisticated data synchronization system for the admin panel that automates the enrichment of publication metadata and rebuilds relational links between content. This feature integrates OpenAlex and CrossRef APIs to fetch citation counts and keywords, and employs a multi-signal matching engine to automatically link publications to members and research areas based on ORCID, name matching, and keyword analysis.
Changes:
Implemented Sync Dashboard (
src/app/admin/(authenticated)/sync/):SyncStatsCardsshowing current data health (synced vs. unsynced items).SyncControlsto trigger different sync modes: Metadata-only, Relationships-only, or Full Sync.SyncProgressPanelusing Server-Sent Events (SSE) to display live progress logs, phase updates, and error reporting.Developed Matching Engine (
src/lib/publications/matching.ts):Built Sync Logic (
src/lib/publications/sync.ts):polite-pool).Created API Infrastructure:
POST /api/admin/syncendpoint that streams SSE events for long-running sync processes.src/lib/publications/openalex.ts) to fetch rich metadata including author ORCIDs and topic keywords.Database Updates:
getSyncStatsServer Action to power the dashboard metrics.UI Components:
useSynchook to manage SSE connection lifecycle and state.