AST 1205 adp to costpoint flow
Create ADP Employee to Costpoint Flow
Implement the main Prefect flow that orchestrates employee data sync from ADP to Costpoint using the connectors and mapping configuration.
Key details:
- Create in
workflows/adp_integration.py - Follow ETL pattern: load credentials → extract → transform → load
- Load ADP and Costpoint credentials from AWS Secrets Manager (see TBD-09)
- Pass credentials to connector instances
- Use
@task(retries=3)decorator for automatic retry logic - Use
.map()for parallel employee processing - Use
unmapped()for shared parameters in map operations - Handle partial failures: return error dict with
{"error": str(e), "failed_at": "stage"} - Track success/failure counts for reporting
- Create Prefect artifacts for observability
- Accept parameters: customer_id, workflow_name, source_config, destination_config, mapping_block_name
- See
workflows/examples/pokemon_demo.pyfor reference pattern
Open questions:
- Should this be incremental sync (changed records) or full sync?