Skip to main content

Transformations

Transformations manipulate data between source and destination.

SQL Transforms​

transforms:
  - name: clean_data
    type: sql
    query: |
      SELECT
        id,
        TRIM(UPPER(email)) as email,
        CAST(created_at AS DATE) as date
      FROM raw_data
      WHERE status != 'deleted'
`

## Python Transforms

```yaml
transforms:
  - name: custom_logic
    type: python
    code: |
      def transform(df):
          df['full_name'] = df['first_name'] + ' ' + df['last_name']
          df['age_group'] = pd.cut(df['age'], bins=[0,18,35,60,100])
          return df
`

## Built-in Transforms

| Transform | Description |
|-----------|-------------|
| **Join** | Combine two or more datasets |
| **Filter** | Remove rows based on conditions |
| **Aggregate** | Group and summarize data |
| **Pivot** | Reshape rows to columns |
| **Deduplicate** | Remove duplicate records |
| **Cast** | Change column data types |

## See Also

- [Sources](/docs/etl-jobs/sources) — Source configuration
- [Destinations](/docs/etl-jobs/destinations) — Destination configuration