Key Concepts
Understanding these core concepts will help you work effectively with ProAI.
Pipelines
A pipeline is the fundamental unit of work in ProAI. It defines how data flows from sources through transformations to destinations.
| Concept | Description |
|---|---|
| Source | Where data comes from (database, API, file, stream) |
| Transform | Operations applied to the data (filter, join, aggregate) |
| Destination | Where processed data is written |
| Node | A single step in a pipeline |
| Edge | A connection between two nodes |
DAGs (Directed Acyclic Graphs)
ProAI represents pipelines as DAGs — acyclic graphs where each node is a task and edges define execution dependencies.
Important
DAGs cannot contain cycles. If you need cyclical processing, use iterative patterns within a single node.
Connectors
Connectors are pluggable adapters that integrate with external systems. Each connector handles:
- Schema discovery — Automatic detection of source/destination schemas
- Read/Write operations — Efficient data transfer with batching
- Type mapping — Automatic conversion between system-specific types
Execution Modes
| Mode | Use Case |
|---|---|
| Batch | Process bounded datasets on a schedule |
| Streaming | Process unbounded data in real-time |
| Micro-batch | Near-real-time processing with small batches |
See Also
- Terminology — Complete glossary
- Architecture — System architecture deep dive
- Release Notes — What's new