Guides: Deduplication & Cleaning | RevProHub

Guides: Deduplication & Cleaning

Duplicate Detection & Resolution Auto + Manual Resolution 45 minutes
Guides Objectives:
By the end of this hands-on session, users will be able to:
  • Navigate the RevProHub Deduplication Dashboard and understand duplicate status types
  • Run duplicate detection scans to identify potential duplicate records
  • Resolve duplicates using automatic resolution or manual side-by-side comparison
  • Mark records as Unique, Master Record, or Resolved Duplicate
  • Export and import deduplication data for offline processing
  • Understand how deduplication integrates with the screening workflow

Part 1: Deduplication Dashboard

Navigate to: Main Project Dashboard → Conducting Phase → Studies → Deduplication


77
Total Records Imported
0
Not Checked
0
Potential Duplicates
0
Resolved Duplicates
77
Unique Studies for Screening
📊 Duplicate Status Definitions:
  • Not Checked – Record not yet processed by duplicate detection
  • Potential Duplicate – System-identified possible duplicate awaiting review
  • Master Record – The primary/kept version of a duplicate set
  • Resolved Duplicate – Confirmed duplicate, linked to master record
  • Unique – Confirmed as the only copy of this study

Part 2: Three-Step Deduplication Workflow

Step 1: Detect Duplicates

Scan all records to flag potential duplicates

Step 2: Resolve Duplicates

Auto-resolve or manual side-by-side comparison

Step 3: Proceed to Screening

Unique studies ready for title/abstract screening

🔍 Detection Scan Logic: RevProHub compares records by title similarity (fuzzy matching), DOI exact match, and author/year combinations. Potential duplicates are flagged for review without automatic deletion.

Part 3: Manual Resolution Interface

Navigate to: Deduplication Dashboard → Step 2 → "Go to Manual Resolution"

Duplicate Group: "GLP-1 Agonists for Weight Loss"

Record A (ID: 5996)
Title: Comparison of Clinical Efficacy of Tirzepatide, Liraglutide and Semaglutide...
Authors: Ciudin A et al.
Year: 2026
DOI: 10.1007/s12325-026-03523-5
Source: PubMed
Record B (ID: 5890)
Title: Comparison of Clinical Efficacy of Tirzepatide, Liraglutide and Semaglutide...
Authors: Ciudin A; Sapin H; Fan L et al.
Year: 2026
DOI: (missing)
Source: Embase
✅ Resolution Options:
  • Select Master Record: Choose the most complete record to keep
  • Mark as Unique: Confirm records are not duplicates
  • Merge: Combine metadata from both records into master
Auto-Resolution: For large duplicate sets, RevProHub can automatically resolve by keeping the record with the most complete metadata (DOI present, longer abstract, more author details). Always review auto-resolved results before finalizing.

Part 4: Deduplication Studies Library

Navigate to: Deduplication Dashboard → Bulk Deduplication or click any stat card

📋 Library Features:
  • Bulk Actions: Select multiple studies → Mark as Unique, Mark as Duplicate, or Reverse to Not Checked
  • Search & Filter: Search by title, filter by duplicate status, sort by any column
  • Pagination: 25, 50, 100, or All entries per page
  • Sortable Columns: ID, Authors, Year, Title, Duplication Status
IDAuthorsYearTitleDuplication Status
5996Ciudin A et al.2026Comparison of Clinical Efficacy of Tirzepatide, Liraglutide and Semaglutide...Potential Duplicate
5890Ciudin A; Sapin H et al.2026Comparison of Clinical Efficacy of Tirzepatide, Liraglutide and Semaglutide...Potential Duplicate
5888Bailey B et al.2020Weight Loss Outcomes in a Real-World SettingUnique

Part 5: Import/Export Deduplication Data

Navigate to: Deduplication Dashboard → Import/Export Data

Export All Data
Full project data with deduplication status + screening decisions
Export Imported Studies
Bibliographic info + deduplication status only
Import Studies Data
Update duplicate_status and duplicate_of_study_id via CSV
CSV Import Format Requirements:
  • Required column: study_id (internal RevProHub ID)
  • Optional columns for deduplication:
    • duplicate_status: Must be 'Not Checked', 'Potential Duplicate', 'Master Record', 'Resolved Duplicate', or 'Unique'
    • duplicate_of_study_id: The ID of the master record (if status is 'Resolved Duplicate')
  • Optional for screening decisions: user_id, stage (TA/FT), decision (Include/Exclude/Maybe)

Part 6: Integration with Screening Workflow

1️⃣ Import studies → 2️⃣ Run duplicate detection → 3️⃣ Resolve duplicates → 4️⃣ Unique studies automatically appear in Screening Dashboard "Pending Assignment"
Only Unique and Master Record studies proceed to screening. Resolved duplicates are excluded from the screening queue to prevent double work.
🧪 Group Hands-On Activity: Deduplication & Cleaning (35 min)
  1. Access Deduplication Dashboard: In your training project, go to Conducting Phase → Studies → Deduplication.
  2. Review Dashboard Stats: Note Total Records Imported vs. Unique Studies for Screening (difference indicates duplicates).
  3. Run Detection Scan: Click "Run Detection Scan". Wait for completion – system flags potential duplicates.
  4. Review Potential Duplicates: Click "Potential Duplicates" stat card. Review the flagged records.
  5. Manual Resolution: Click "Go to Manual Resolution". For each duplicate group:
    • Compare side-by-side metadata (title, authors, year, DOI, source)
    • Select the most complete record as Master Record
    • Confirm other records as Resolved Duplicate
  6. Alternative: Bulk Marking: In the Deduplication Library, check multiple records → use bulk actions toolbar → "Mark as Unique" or "Mark as Duplicate".
  7. Export Deduplication Data: Go to "Import/Export Data" → Export Imported Studies. Open CSV to view duplicate_status column.
  8. Verify Screening Queue: Go to Screening Dashboard → Title/Abstract Stage → "Pending Assignment" count should reflect unique studies only.

Debrief Questions:

  • Why is deduplication a critical step before screening?
  • What are the risks of skipping duplicate detection?
  • When would you choose manual resolution over auto-resolution?
  • How does tracking duplicate_of_study_id maintain auditability?
  • What metadata fields are most reliable for duplicate detection (DOI, title, author combinations)?

Best Practices for Deduplication

Recommendations:
  • Run detection scan after each major import batch (not just once at the end)
  • Use DOI matching as the primary detection criterion when available
  • For records without DOI, rely on title + author + year combinations
  • When selecting a Master Record, prioritize records with: DOI present → complete abstract → correct author formatting
  • Document your deduplication approach in the review protocol (which fields used, manual vs. auto)
  • Export deduplication log for transparency in final report appendices
Deduplication Module Completion Checklist:
✔ Detection scan completed on imported studies
✔ All potential duplicate groups reviewed and resolved
✔ Master records selected for each duplicate set
✔ Unique status confirmed for non-duplicate records
✔ Unique study count verified matches screening queue total
✔ Deduplication data exported for documentation (optional)
✔ Ready to proceed to Title/Abstract screening

Next Guides: Screening & Selection – applying eligibility criteria to unique studies.

2026 RevProHub
Guides: Deduplication & Cleaning | Detection Scan | Auto/Manual Resolution | Master Record Selection