
Research Data Curation Workflow
Validate, standardize, and curate research datasets with automated quality workflows
What You Can Do
You can systematically validate research datasets, automatically standardize formats and schemas, and verify compliance requirements in one workflow. The skill performs comprehensive quality assessment, enriches metadata, detects duplicates and anomalies, and generates documentation—transforming raw datasets into publication-ready, audit-trail-enabled collections.
Features
Validates dataset structure, data types, and value ranges against your schema; identifies missing values, outliers, and format inconsistencies
Automatically generates and enriches metadata—provenance, collection methods, variable definitions, units, and licensing information
Checks datasets against GDPR, HIPAA, research ethics guidelines, and institutional policies; flags PII, restricted data, and consent gaps
Converts datasets to standard formats (CSV, JSON, Parquet); aligns column naming, encoding, and coding schemes across related datasets
Generates quality metrics—completeness, consistency, validity, and accuracy—with actionable feedback and remediation suggestions
Identifies duplicate records, near-duplicates, and statistical anomalies using fuzzy matching and outlier detection algorithms
Creates detailed audit logs of all transformations, validation steps, and decisions; tracks dataset lineage and version history
Example Output
Input: Raw survey dataset (500 rows, inconsistent date formats, missing value codes, unclear variable labels)
Output:
Quality Assessment Report
─────────────────────────
Completeness: 92% (46 cells flagged)
Consistency: 89% (8 column naming issues, 12 encoding mismatches)
Validity: 95% (23 values outside expected range)
Accuracy: 88% (5 likely duplicates, 3 logical errors)
Compliance Status: ⚠️ NEEDS REVIEW
- 12 rows flagged: contain potential PII (names, email patterns)
- Missing consent documentation for 15 respondents
- Compliance: GDPR Article 6 basis unclear
Metadata Generated
──────────────────
Collection Date: 2024-01-15 to 2024-03-22
Data Custodian: Research Team A
License: CC-BY-4.0
Variables Documented: 34/34
Dict: [age (integer, 18-85), gender (categorical), ...]
Recommendations
────────────────
✓ Standardize dates to ISO 8601
✓ Code missing values as 'NA' consistently
✓ Redact PII columns or pseudonymize
✓ Obtain explicit consent docs for 15 respondents
After curation: Clean, standardized dataset ready for publication with full metadata, compliance sign-off, and audit trail.
What's Included
- Data Validation Engine: Schema checking, type validation, range/pattern enforcement, and constraint verification
- Quality Assessment Module: Automated scoring for completeness, consistency, validity, and accuracy with recommendations
- Compliance Checker: GDPR, HIPAA, and institutional policy verification; PII detection; consent tracking
- Metadata Templates: Pre-built templates for Data Documentation Initiative (DDI), Dublin Core, and domain-specific schemas
- Transformation Workflows: Standardization templates for CSV-to-JSON, encoding conversion, column renaming, and deduplication
- Audit & Versioning System: Complete log of all changes, decisions, and curator actions with rollback capability
Who It's For
- Research Data Managers
- Academic Researchers & Scientists
- Data Stewards & Curators
- Compliance & Ethics Officers
- Data Quality Engineers
Best For
- Preparing datasets for public repositories or journals
- Ensuring compliance with funding agency requirements
- Standardizing multi-source research data
- Auditing data quality before analysis or publication
- Documenting dataset lineage and transformations







