# Workflows

Build automated data pipelines using the dcupl Console's visual graph-based editor. Connect nodes, define data flow, and process data at scale. The [template reference](/console/workflows/reference) has the template JSON, the script globals, the run API and the limits.

## What are workflows?

Workflows are **directed acyclic graphs (DAGs)** where data flows through connected processing steps:

```mermaid
flowchart LR
  T[Trigger] --> F[Fetch Data]
  F --> TR[Transform]
  TR --> S[Save Results]
  F -->|error| E[Handle Error]
```

- **Nodes** are execution units that process data
- **Edges** define how data flows between nodes
- **Items** are discrete data units moving through the graph
- **Ports** are named connection points (`main`, `error`, custom)

## Creating a workflow

1. Open your project in the Console (the draft version)
2. Navigate to **Workflows** in the sidebar
3. On the **Templates** tab, click **+** (Create a new workflow template)
4. Enter a **Workflow Name** and **Workflow Key**, pick a starting template, and click **Create Workflow**

The key takes lowercase letters, numbers and hyphens, and the form fills it in from the name. A key written `<key>--<suffix>` (for example `product-sync--staging`) creates a variant of `<key>`: the **Templates** tab shows it on its base template's card. There, **Create Variant** in a template's menu adds another variant, and **Promote to Base** writes the variant over the base template under the base key (its nodes, edges and variables). The base keeps its name and description where it has them; otherwise it takes the variant's. Then it offers to delete the variant. A deployed base keeps running the old template until you deploy it again.

Deploying prefixes the key with your project ID (`<projectId>-<key>`, unless the runner is set up without the prefix), and the deployed key must be 4–49 characters long, so keep the workflow key to about 28 characters.

The workflow editor opens with the template you picked — the default **From Scratch (HTTP)** template is an HTTP trigger wired to a status response. Click **Edit Workflow** to change it.

## Triggers

Every workflow starts with a trigger that defines how it's activated:

| Trigger                        | Description                                                                               | Use Case                                |
| ------------------------------ | ----------------------------------------------------------------------------------------- | --------------------------------------- |
| **HTTP Request** (webhook)     | HTTP endpoint on the runner: a path, allowed methods, and optional API-key authentication | External integrations, real-time events |
| **Scheduled** (`trigger-cron`) | Runs on a cron expression (5 fields, or 6 with seconds) in a timezone (default UTC)       | Daily syncs, periodic updates           |

There is no separate manual trigger. To run a deployed workflow by hand, click **Execute** in the editor: it calls the HTTP trigger, or offers **Run now** for a scheduled trigger.

### Adding a trigger

1. From the node palette, drag a trigger onto the canvas
2. Configure the trigger type and settings
3. Connect it to your first processing node

## Node types

Drag nodes from the palette onto the canvas to build your workflow.

### Request node

Makes HTTP requests to external APIs:

- Configure URL, method, headers, and body
- Access workflow variables with `{{variables.apiToken}}`
- Success data flows to the `main` port
- Errors route to specialized ports (`error`, `client-error`, `server-error`, `timeout`)

### Script node

Executes JavaScript for data transformation in an isolated sandbox:

- Read input data with `$json()` (all items' data) or `$json.first()` / `$json.at(i)` (one item's data); `$items()` returns the raw items (`{ json, … }`)
- `return` a value to emit it on the `main` port; `throw` to route to the `error` port
- Route to multiple output ports with `_output.route()`
- Use the namespaced helpers: `_csv.toJSON()` / `_csv.fromJSON()`, `_json`, `_array`, `_string`, `_object`, `_datetime`, `_xml`
- Scripts cannot make HTTP calls or touch the filesystem — use a Request node for outbound HTTP and file nodes for I/O

All globals and how a return value becomes items: [Script globals](/console/workflows/reference#script-globals).

### File nodes

Work with files from various storage backends:

| Node            | Purpose                                                       |
| --------------- | ------------------------------------------------------------- |
| **dcupl-files** | Read/write files in dcupl cloud storage                       |
| **git-files**   | Read/write files in a GitHub repository (clone, commit, push) |
| **s3-files**    | Upload/download from AWS S3                                   |
| **azure-files** | Upload/download from Azure Blob Storage                       |

### Data Mapper node

Transform data structures with field mappings (`origin` → `target`):

- Map or rename top-level properties (`origin` and `target` are property names, not nested paths)
- Drop a property by leaving its target empty
- Keep all unmapped properties with a `*` → `*` mapping
- Or, with no mappings, run the items through a dcupl model definition (JSON)

### dcupl Instance node

Query and update dcupl data directly in workflows:

- Initialize dcupl instances
- Run queries against models
- Update data programmatically

A node added in the editor references the project's **Workflow Execution Key**: `initOptions.config.apiKey` holds the key's uid (in the **Init Options** JSON on the node's **Config** tab), and deploying replaces the uid with the key. New projects are private by default, so without a key the node can't load the project's loader: the CDN answers `403` and the node fails. If your project has no key named Workflow Execution Key, set `initOptions.config.apiKey` to the uid of another project key that can read the files your loader loads (Project Settings → API Keys). A `{{variables.…}}` placeholder is not filled in here (see [Node types](/console/workflows/reference#node-types)).

### dcupl API node

Sends reporting e-mails, fetches run digests, and lists recent runs, authenticated as the runner. See [The dcupl API node](/console/workflows/reporting#the-dcupl-api-node).

## Connecting nodes with edges

Edges define how data flows between nodes.

### Creating connections

1. Click on a node's output port
2. Drag to another node's input port
3. Release to create the connection

### Edge conditions and transforms

> **Watch out:** The editor lets you add a **Condition** or **Transform** to an edge (click the edge in edit mode), but the workflow runner currently **ignores both**: every item passes through unchanged, and the runner only logs a warning. Don't rely on them for routing or reshaping data.

To route items, use a Script node with custom output ports instead: add the ports on the node's **Ports & Schema** tab (**+ Add Output**), connect each port to its own node, and return `_output.route({ valid: [...], invalid: [...] })` from the script. To reshape items, use a Script or Data Mapper node.

## Data flow patterns

### Linear flow (ETL)

```mermaid
flowchart LR
  E[Extract] --> T[Transform] --> L[Load]
```

Simple sequential processing from source to destination.

### Parallel branches

```mermaid
flowchart LR
  S[Source] --> A[Process A]
  S --> B[Process B]
  A --> M[Merge]
  B --> M
```

Process data through multiple paths simultaneously, then combine results.

### Conditional routing

```mermaid
flowchart LR
  V[Validate] -->|valid| P[Process]
  V -->|invalid| E[Log Error]
```

Route items to different nodes from a Script node's custom output ports (see [Edge Conditions and Transforms](#edge-conditions-and-transforms)).

## Merge strategies

When multiple edges connect to a single step node, its `merge` setting decides how incoming data is combined. Without one, the items of all inputs are concatenated:

| Strategy (`mode`)                 | Behavior                                                              | Use Case                     |
| --------------------------------- | --------------------------------------------------------------------- | ---------------------------- |
| **Append** (`append`)             | Concatenate all items                                                 | Aggregating multiple sources |
| **Merge** (`merge`)               | Combine by index position                                             | Parallel enrichment          |
| **Merge by Key** (`merge-by-key`) | Left join on a common field (`key`), the first input is the base      | SQL-style joins              |
| **Wait All** (`wait-all`)         | Emit one item `{ inputs: [[…], […]] }` that keeps the inputs separate | Comparing datasets           |
| **First** (`first`)               | Use only the first input edge's items, ignore the rest                | Picking one source           |

### Configuring merge strategy

The editor shows a node's merge setting (in the node's **Config** tab) but has no control to choose one. Set it in the workflow template JSON (the project file `workflows/<key>.workflow-v3.json`) by adding a `merge` property to the receiving step node, for example `"merge": { "mode": "merge-by-key", "key": "id" }`. `merge-by-key` requires `key`, and a node with `merge` must declare at least two input ports (**+ Add Input** on its **Ports & Schema** tab), or validation rejects the template.

## Error handling

Workflows treat errors as data that can be routed and processed.

### Error ports

Every step node has an `error` output port that triggers when:

- An exception is thrown
- Validation fails
- A timeout occurs

**Connect error ports** to handle failures gracefully:

```mermaid
flowchart LR
  A[API Call] -->|success| P[Process]
  A -->|error| E[Handle Error]
```

- **Error port connected**: Workflow continues, error becomes data
- **Error port not connected**: Workflow stops, error propagates

### HTTP-specific error ports

Request nodes provide specialized error routing:

| Port           | Status | Description                          |
| -------------- | ------ | ------------------------------------ |
| `main`         | 2xx    | Success responses                    |
| `client-error` | 4xx    | Bad request, not found, unauthorized |
| `server-error` | 5xx    | Server problems (retry these)        |
| `timeout`      | -      | Request took too long                |
| `error`        | -      | Network errors, DNS failures         |

### Error handling patterns

**Fallback to backup:**

```mermaid
flowchart LR
  P[Primary] -->|error| B[Backup]
  P -->|success| R[Result]
  B -->|success| R
```

**Graceful degradation:**

```mermaid
flowchart LR
  F[Fetch Live] -->|success| P[Process]
  F -->|error| D[Use Defaults]
  D --> P
```

## Variables

Store configuration values and secrets that can be accessed throughout workflows.

### Adding variables

1. Click the key icon (**Project Variables**) in the workflow toolbar, or **Manage workflow variables** on the Workflows page
2. In **Global Workflow Variables**, enter a **Name**, **Value**, and optional **Description**
3. Click **Add Variable**

Values are masked in the list. There is no separate secret flag: every variable value is encrypted with the runner's key when you deploy.

### Using variables

Reference variables in node configurations:

- URLs: `{{variables.apiBaseUrl}}/users`
- Headers: `Bearer {{variables.apiToken}}`

Variables are resolved when you deploy — after changing one, redeploy. They resolve from (highest to lowest priority):

1. Deployment overrides (set per deployment with **Deploy with variables…**)
2. Project-level variables
3. Workflow template variables

## Response configuration

A response node (palette section **Response**) decides what an HTTP-triggered run returns:

| Node                                    | Description                                                                  |
| --------------------------------------- | ---------------------------------------------------------------------------- |
| **HTTP Response** (`response-status`)   | Returns the execution status only: `{ "status", "executionId", "duration" }` |
| **Script Response** (`response-script`) | Returns what your script returns, with optional custom response headers      |

Execution details for debugging (data trace, memory profile) are options of a test run in the editor, not a response type.

## Deploying workflows

### Save and deploy

1. Click **Save Changes** to save your workflow (or turn on **Deploy on Save**)
2. Select the target runner in the runner dropdown of the toolbar
3. Click **Deploy** (the gear next to it opens **Deploy with variables…**)
4. Wait for the runner's status to show **Up to date**

### Testing

- Click **Execute** to run the deployed workflow from the editor (**Start Test** for an HTTP trigger, **Run now** for a scheduled one)
- Send requests to the trigger endpoint: `{runner-url}/v3/workflow/{projectId}-{workflowKey}/{path}`, where `{path}` is the trigger's path without its leading slash (the [run API](/console/workflows/reference#calling-a-deployed-workflow) has the details)
- Click **History** to see past runs

## Running workflows locally

There is no local workflow runner: a workflow only runs on a runner your project can deploy to. What works from your machine today, with the [CLI](/cli/workflows):

- `dcupl workflow validate --file <file>` checks a template file against the template schema and the graph rules (unique ids, edges that point at declared ports, at least one trigger, `merge` with two inputs). It needs no account and no network.
- `dcupl workflow test --file <file> --runner <uid>` deploys your local file to a runner (use a dev runner) and runs it once, printing the node-by-node result. `--input` sends a request body; `--node <id>` runs a single node. It needs credentials (`dcupl config set`).
- `dcupl workflow run <key> --runner <uid>` triggers a workflow that is already deployed.

## Memory efficiency

Each node processes its whole input batch at once, so memory grows with the number and size of items. An edge's items are handed to the next node without a copy and released once that node has pulled them, so earlier results don't pile up across the graph. An edge holds at most 1,000,000 items; pushing more fails the run with `Buffer limit exceeded`.

## Best practices

### Node design

- Keep nodes focused on single responsibility
- Use descriptive names
- Handle errors with dedicated error ports

### Performance

- Keep runs within the [runtime limits](/console/workflows/reference#limits) (by default a run times out after 10 s, a script node after 500 ms, an HTTP request after 5 s)
- Check memory per node with **Memory Profiling** in a test run
- Monitor memory in production

### Testing

- Test with realistic data volumes
- Verify merge strategies work correctly
- Test error paths thoroughly

## Next steps

- [Getting started](/console/workflows/getting-started) — Build your first workflow step-by-step
- [Template reference](/console/workflows/reference) — Template JSON, script globals, run API and limits
- [Reporting](/console/workflows/reporting) — Get notified when workflows fail, and send run digests
- [CLI workflows](/cli/workflows) — Validate, test, and deploy workflow templates from the command line
- [REST APIs](/console/rapi) — Deploy auto-generated REST APIs from your data models