Sources

Sources define where ProAI extracts data from. Each source is configured via a connector profile.

Supported Source Types

Type	Connectors
Databases	PostgreSQL, MySQL, Oracle, SQL Server, BigQuery, Redshift
Cloud Storage	S3, GCS, Azure Blob, ADLS
Streaming	Kafka, Kinesis, Pulsar
APIs	REST, GraphQL, gRPC, Webhooks
Files	CSV, JSON, Parquet, Avro, XML

Configuration Example

sources:
  - name: production_users
    type: postgres
    connector: prod-postgres
    config:
      schema: public
      table: users
      incremental: true
      cursor_column: updated_at
      batch_size: 10000
`

## Incremental Loading

Enable incremental loading to only process new or updated records since the last run.

```yaml
incremental:
  enabled: true
  cursor_column: updated_at
  lookback_hours: 24
`

## See Also

- [Integrations](/docs/integrations/overview) — All integration options
- [Transformations](/docs/etl-jobs/transformations) — Transform reference