Workflows

Build automated data pipelines using the dcupl Console's visual graph-based editor. Connect nodes, define data flow, and process data at scale. The template reference has the template JSON, the script globals, the run API and the limits.

What are workflows?

Workflows are directed acyclic graphs (DAGs) where data flows through connected processing steps:

flowchart LR
  T[Trigger] --> F[Fetch Data]
  F --> TR[Transform]
  TR --> S[Save Results]
  F -->|error| E[Handle Error]

• Nodes are execution units that process data
• Edges define how data flows between nodes
• Items are discrete data units moving through the graph
• Ports are named connection points (main, error, custom)

Creating a workflow

1. Open your project in the Console (the draft version)
2. Navigate to Workflows in the sidebar
3. On the Templates tab, click + (Create a new workflow template)
4. Enter a Workflow Name and Workflow Key, pick a starting template, and click Create Workflow

The key takes lowercase letters, numbers and hyphens, and the form fills it in from the name. A key written <key>--<suffix> (for example product-sync--staging) creates a variant of <key>: the Templates tab shows it on its base template's card. There, Create Variant in a template's menu adds another variant, and Promote to Base writes the variant over the base template under the base key (its nodes, edges and variables). The base keeps its name and description where it has them; otherwise it takes the variant's. Then it offers to delete the variant. A deployed base keeps running the old template until you deploy it again.

Deploying prefixes the key with your project ID (<projectId>-<key>, unless the runner is set up without the prefix), and the deployed key must be 4–49 characters long, so keep the workflow key to about 28 characters.

The workflow editor opens with the template you picked — the default From Scratch (HTTP) template is an HTTP trigger wired to a status response. Click Edit Workflow to change it.

Triggers

Every workflow starts with a trigger that defines how it's activated:

There is no separate manual trigger. To run a deployed workflow by hand, click Execute in the editor: it calls the HTTP trigger, or offers Run now for a scheduled trigger.

Adding a trigger

1. From the node palette, drag a trigger onto the canvas
2. Configure the trigger type and settings
3. Connect it to your first processing node

Node types

Drag nodes from the palette onto the canvas to build your workflow.

Request node

Makes HTTP requests to external APIs:

• Configure URL, method, headers, and body
• Access workflow variables with {{variables.apiToken}}
• Success data flows to the main port
• Errors route to specialized ports (error, client-error, server-error, timeout)

Script node

Executes JavaScript for data transformation in an isolated sandbox:

• Read input data with $json() (all items' data) or $json.first() / $json.at(i) (one item's data); $items() returns the raw items ({ json, … })
• return a value to emit it on the main port; throw to route to the error port
• Route to multiple output ports with _output.route()
• Use the namespaced helpers: _csv.toJSON() / _csv.fromJSON(), _json, _array, _string, _object, _datetime, _xml
• Scripts cannot make HTTP calls or touch the filesystem — use a Request node for outbound HTTP and file nodes for I/O

All globals and how a return value becomes items: Script globals.

File nodes

Work with files from various storage backends:

Data Mapper node

Transform data structures with field mappings (origin → target):

• Map or rename top-level properties (origin and target are property names, not nested paths)
• Drop a property by leaving its target empty
• Keep all unmapped properties with a * → * mapping
• Or, with no mappings, run the items through a dcupl model definition (JSON)

dcupl Instance node

Query and update dcupl data directly in workflows:

• Initialize dcupl instances
• Run queries against models
• Update data programmatically

A node added in the editor references the project's Workflow Execution Key: initOptions.config.apiKey holds the key's uid (in the Init Options JSON on the node's Config tab), and deploying replaces the uid with the key. New projects are private by default, so without a key the node can't load the project's loader: the CDN answers 403 and the node fails. If your project has no key named Workflow Execution Key, set initOptions.config.apiKey to the uid of another project key that can read the files your loader loads (Project Settings → API Keys). A {{variables.…}} placeholder is not filled in here (see Node types).

dcupl API node

Sends reporting e-mails, fetches run digests, and lists recent runs, authenticated as the runner. See The dcupl API node.

Connecting nodes with edges

Edges define how data flows between nodes.

Creating connections

1. Click on a node's output port
2. Drag to another node's input port
3. Release to create the connection

Edge conditions and transforms

To route items, use a Script node with custom output ports instead: add the ports on the node's Ports & Schema tab (+ Add Output), connect each port to its own node, and return _output.route({ valid: [...], invalid: [...] }) from the script. To reshape items, use a Script or Data Mapper node.

Data flow patterns

Linear flow (ETL)

flowchart LR
  E[Extract] --> T[Transform] --> L[Load]

Simple sequential processing from source to destination.

Parallel branches

flowchart LR
  S[Source] --> A[Process A]
  S --> B[Process B]
  A --> M[Merge]
  B --> M

Process data through multiple paths simultaneously, then combine results.

Conditional routing

flowchart LR
  V[Validate] -->|valid| P[Process]
  V -->|invalid| E[Log Error]

Route items to different nodes from a Script node's custom output ports (see Edge Conditions and Transforms).

Merge strategies

When multiple edges connect to a single step node, its merge setting decides how incoming data is combined. Without one, the items of all inputs are concatenated:

Configuring merge strategy

The editor shows a node's merge setting (in the node's Config tab) but has no control to choose one. Set it in the workflow template JSON (the project file workflows/<key>.workflow-v3.json) by adding a merge property to the receiving step node, for example "merge": { "mode": "merge-by-key", "key": "id" }. merge-by-key requires key, and a node with merge must declare at least two input ports (+ Add Input on its Ports & Schema tab), or validation rejects the template.

Error handling

Workflows treat errors as data that can be routed and processed.

Error ports

Every step node has an error output port that triggers when:

• An exception is thrown
• Validation fails
• A timeout occurs

Connect error ports to handle failures gracefully:

flowchart LR
  A[API Call] -->|success| P[Process]
  A -->|error| E[Handle Error]

• Error port connected: Workflow continues, error becomes data
• Error port not connected: Workflow stops, error propagates

HTTP-specific error ports

Request nodes provide specialized error routing:

Error handling patterns

Fallback to backup:

flowchart LR
  P[Primary] -->|error| B[Backup]
  P -->|success| R[Result]
  B -->|success| R

Graceful degradation:

flowchart LR
  F[Fetch Live] -->|success| P[Process]
  F -->|error| D[Use Defaults]
  D --> P

Variables

Store configuration values and secrets that can be accessed throughout workflows.

Adding variables

1. Click the key icon (Project Variables) in the workflow toolbar, or Manage workflow variables on the Workflows page
2. In Global Workflow Variables, enter a Name, Value, and optional Description
3. Click Add Variable

Values are masked in the list. There is no separate secret flag: every variable value is encrypted with the runner's key when you deploy.

Using variables

Reference variables in node configurations:

• URLs: {{variables.apiBaseUrl}}/users
• Headers: Bearer {{variables.apiToken}}

Variables are resolved when you deploy — after changing one, redeploy. They resolve from (highest to lowest priority):

1. Deployment overrides (set per deployment with Deploy with variables…)
2. Project-level variables
3. Workflow template variables

Response configuration

A response node (palette section Response) decides what an HTTP-triggered run returns:

Execution details for debugging (data trace, memory profile) are options of a test run in the editor, not a response type.

Deploying workflows

Save and deploy

1. Click Save Changes to save your workflow (or turn on Deploy on Save)
2. Select the target runner in the runner dropdown of the toolbar
3. Click Deploy (the gear next to it opens Deploy with variables…)
4. Wait for the runner's status to show Up to date

Testing

• Click Execute to run the deployed workflow from the editor (Start Test for an HTTP trigger, Run now for a scheduled one)
• Send requests to the trigger endpoint: {runner-url}/v3/workflow/{projectId}-{workflowKey}/{path}, where {path} is the trigger's path without its leading slash (the run API has the details)
• Click History to see past runs

Running workflows locally

There is no local workflow runner: a workflow only runs on a runner your project can deploy to. What works from your machine today, with the CLI:

• dcupl workflow validate --file <file> checks a template file against the template schema and the graph rules (unique ids, edges that point at declared ports, at least one trigger, merge with two inputs). It needs no account and no network.
• dcupl workflow test --file <file> --runner <uid> deploys your local file to a runner (use a dev runner) and runs it once, printing the node-by-node result. --input sends a request body; --node <id> runs a single node. It needs credentials (dcupl config set).
• dcupl workflow run <key> --runner <uid> triggers a workflow that is already deployed.

Memory efficiency

Each node processes its whole input batch at once, so memory grows with the number and size of items. An edge's items are handed to the next node without a copy and released once that node has pulled them, so earlier results don't pile up across the graph. An edge holds at most 1,000,000 items; pushing more fails the run with Buffer limit exceeded.

Best practices

Node design

• Keep nodes focused on single responsibility
• Use descriptive names
• Handle errors with dedicated error ports

Performance

• Keep runs within the runtime limits (by default a run times out after 10 s, a script node after 500 ms, an HTTP request after 5 s)
• Check memory per node with Memory Profiling in a test run
• Monitor memory in production

Testing

• Test with realistic data volumes
• Verify merge strategies work correctly
• Test error paths thoroughly