MigryX parses every dfPower Studio and DMS job file — standardize, parse, match, encode, validate, and profile operations — and converts them to idiomatic Python, Snowflake UDFs, Databricks PySpark pipelines, and dbt tests. All DQ logic. Zero rewrites.
A purpose-built parser ingests every DataFlux and DMS artifact — from .dfm job files and DQ scheme definitions to SAS code calling DQPARSE() — and emits production-ready modern equivalents.
A structured, parser-driven approach that inventories every artifact, converts each DQ operation class-by-class, then validates output parity before cutover.
Full inventory and complexity profiling of every DataFlux artifact before any output code is generated.
Operation-class-aware code generation preserving all DQ logic with idiomatic open-source equivalents.
Side-by-side output comparison across a representative data sample before decommissioning DataFlux.
Structural parser for .dfm binary/XML job files used by dfPower Studio and DMS. Reads nodes, edges, scheme references, locale bindings, and job metadata with full fidelity before any conversion step begins.
Converts address standardization (USPS CASS-style), name parsing & standardization, date/phone/fax formatting, and custom standardization schemes to Python normalization pipelines using usaddress, nameparser, and regex equivalents.
Translates DataFlux match keys, blocking rules, frequency analysis tables, and probabilistic thresholds into py-recordlinkage comparison vectors or dedupe training configurations — preserving precision and recall targets.
Reverse-engineers DataFlux parse scheme logic — field splitting, token extraction, pattern recognition — into equivalent Python regular expressions, spaCy NLP rules, and structured parser calls (nameparser, usaddress).
Maps DataFlux Profile job configurations to ydata-profiling and Great Expectations profiling runs. Converts validate rules (regex, reference lookup, domain/range) into GE Expectation Suites and dbt-expectations tests.
Exports DFM repository artifacts — locale-specific schemes (US, UK, Canada, Germany), reference tables, and custom phonetic encoding schemes — into portable Python dictionaries, CSV lookup tables, and Snowflake staging tables.
| DataFlux Operation | Artifact / Format | Python / Open-Source Target | Cloud Target |
|---|---|---|---|
| Standardize — Address | Standardize Scheme (US/UK/CA locale) | usaddress + custom normalizer | Snowflake Python UDF |
| Standardize — Name | Name standardization scheme | nameparser HumanName | Snowflake Python UDF |
| Standardize — Date / Phone | Date / phone formatting scheme | dateutil, phonenumbers | Snowflake JS UDF |
| Parse — Name / Address | Parse Scheme (.dfm node) | nameparser, usaddress | Databricks UDF |
| Parse — Custom tokens | Custom parse scheme patterns | re + spaCy ruler | Snowflake Python UDF |
| Match — Deterministic | Exact match keys | py-recordlinkage Compare.exact() | dbt test / Snowflake SQL |
| Match — Probabilistic | Fuzzy match rules + thresholds | py-recordlinkage / dedupe | Databricks PySpark ML |
| Encode — Phonetic | Soundex, NYSIIS, Metaphone schemes | phonetics library | Snowflake JS UDF (soundex) |
| Profile | Profile job nodes | ydata-profiling + Great Expectations | Databricks profiling notebook |
| Validate — Regex | Validate rule (pattern match) | Great Expectations expect_column_values_to_match_regex | dbt-expectations |
| Validate — Reference Lookup | Reference data table lookup | Great Expectations expect_column_values_to_be_in_set | dbt test / Snowflake constraint |
| DQPARSE() | SAS DQ function in DATA step | nameparser / usaddress | Snowflake Python UDF |
| DQSTANDARDIZE() | SAS DQ function in DATA step | Custom normalizer Python function | Snowflake Python UDF |
| DQMATCH() | SAS DQ function in DATA step | py-recordlinkage match score | Databricks PySpark UDF |
| Process Job (Orchestration) | Job chain / schedule / event trigger | Apache Airflow DAG (Python) | Databricks Workflow JSON |
| Real-time Service | Web Service node (.dfm) | FastAPI endpoint wrapping DQ functions | AWS Lambda / Azure Function |
| DataFlux Product / Concept | MigryX Migration Scope | Primary Target | Secondary Target |
|---|---|---|---|
| dfPower Studio | All .dfm job files, DQ nodes, scheme bindings | Python + Great Expectations | Snowflake UDFs |
| SAS Data Management Studio (DMS) | Data Jobs, Process Jobs, job canvas metadata | Python pipelines + Airflow | Databricks Workflows |
| SAS Data Quality Server | DQ schemes, locales, reference tables | Python + open-source DQ libs | Snowflake Python UDFs |
| DataFlux DMP | End-to-end job orchestration, schedules | Airflow DAGs | Databricks Workflows |
| Real-time Services | Web service endpoint definitions, DQ functions | FastAPI microservices | AWS Lambda |