Data flow

Understanding how data flows through dcupl helps you make better architectural decisions. This page traces data from external sources to query results.

Overview

Data in dcupl flows through four stages:

flowchart LR
  SOURCE["1. Source<br/>CSV, JSON, API"]
  LOAD["2. Load<br/>Parse & Transform"]
  STORE["3. Store<br/>In-Memory"]
  QUERY["4. Query<br/>Filter & Retrieve"]

  SOURCE --> LOAD
  LOAD --> STORE
  STORE --> QUERY

Each stage has specific responsibilities and components.

Stage 1: data sources

Data originates from external sources. dcupl supports various formats:

Data sources can be:

• Static files - Hosted on CDN or bundled with your app
• Dynamic endpoints - APIs that return fresh data
• Workflow outputs - Transformed data from Console workflows

Stage 2: loading

The loading stage fetches and parses raw data and stages it for the model. Processing against the model definitions happens afterwards, in dcupl.init() or dcupl.update().

flowchart TD
  subgraph loading["Loading Stage"]
    FETCH["Fetch Data"]
    PARSE["Parse Format"]
    TRANSFORM["Apply Loader Transformers"]
    STAGE["Stage Data Container"]
  end

  SOURCE[Data Source] --> FETCH
  FETCH --> PARSE
  PARSE --> TRANSFORM
  TRANSFORM --> STAGE
  STAGE --> PROCESS["dcupl.init() / update()<br/>Process & Validate Against Model"]
  PROCESS --> STORE[In-Memory Store]

Loader transformers are JavaScript resources: a raw-file transformer runs on the fetched file before parsing, a parsed-file transformer on the parsed items.

DcuplAppLoader

The DcuplAppLoader handles loading from a LoaderConfiguration:

import { Dcupl } from '@dcupl/core';
import { DcuplAppLoader } from '@dcupl/loader';

const dcupl = new Dcupl({
  config: { projectId: 'your-project', apiKey: 'your-api-key' },
});
const loader = new DcuplAppLoader();
dcupl.loaders.add(loader);

// 1. Fetch configuration
await loader.config.fetch();

// 2. Load the resources it lists
await loader.process();

// 3. Apply the loaded models and data
await dcupl.init();

The loader reads dcupl.lc.json which defines:

{
  "resources": [
    {
      "url": "${baseUrl}/models/product.dcupl.json",
      "type": "model",
      "tags": ["product"]
    },
    {
      "url": "${baseUrl}/data/products.json",
      "type": "data",
      "model": "Product",
      "tags": ["product"]
    }
  ]
}

loader.process() loads every resource. To load only some of them, pass resourceTags, or define applications (named sets of resources) as described in Loader.

Manual loading

You can also load data directly without a LoaderConfiguration:

const dcupl = new Dcupl();

// Add model
dcupl.models.set({
  key: 'Product',
  properties: [
    { key: 'name', type: 'string' },
    { key: 'price', type: 'int' },
  ],
});

// Add data
dcupl.data.set(
  [
    { key: 'p1', name: 'Laptop', price: 999 },
    { key: 'p2', name: 'Mouse', price: 29 },
  ],
  { model: 'Product' },
);

// Data calls are staged: init() (later: update()) applies them
await dcupl.init();

Data transformations

When init() (or update()) processes the staged data, dcupl applies transformations defined in models:

Type Coercion - Converts strings to proper types

// Raw CSV data: "29.99"
// Model property: { key: 'price', type: 'float' }
// Result: 29.99 (number)

Array Conversion - Splits delimited strings into arrays

// Raw data: "red,blue,green"
// Property: { key: 'colors', type: 'Array<string>', separator: ',' }
// Result: ["red", "blue", "green"]

Value Mapping - Translates codes to readable values

// Raw data: "1"
// Model: valueMappings: [{ attributes: ['status'], values: [{ from: ['1'], to: 'active' }] }]
// Result: "active"

Stage 3: in-memory storage

After loading, data lives in memory organized by model.

dcupl.init() processes every model after the models it references. It handles plain properties and references before the attributes computed from them (resolved, query, derived and grouped references, derived and expression properties), so a derived property or derived reference that reads a resolved reference still gets its value. dcupl.update() re-processes only the affected attributes when the change is small, no model was replaced with data.set(), no model definition changed since init() and no query reference is affected (a partial update), and runs the full pipeline otherwise.

Each model holds its items by their key (or the model's keyProperty). That key is how query.one({ itemKey }), data.update() and data.remove() find an item. When rows loaded together share a key, the last one wins (see Data Key Strategy).

Indices

dcupl indexes attributes to speed up queries. References and properties marked index: true are indexed while the data is processed; other properties are indexed on demand (for example when you request facets for them).

eq and in filters use an existing index instead of scanning all items; other operators scan the items.

Reference graph

dcupl processes a model after the models it references. A reference cycle between models (A → B → A) makes init() fail: it logs the cycle and resolves to false. A model may reference itself (for example a parent reference); that is not a cycle.

graph LR
  Product -->|"category"| Category
  Order -->|"customer"| Customer
  Order -->|"items"| Product

References link items across models, which enables deep queries such as category.name.

Stage 4: querying

Queries retrieve and filter data from the in-memory store.

flowchart TD
  subgraph query["Query Execution"]
    CREATE["Create List"]
    BUILD["Build Query"]
    EXECUTE["Execute Filter"]
    RESULT["Return Results"]
  end

  STORE[In-Memory Store] --> CREATE
  CREATE --> BUILD
  BUILD --> EXECUTE
  EXECUTE --> RESULT

Lists

A List is a filtered view of a model's data:

const productList = dcupl.lists.create({
  modelKey: 'Product',
});

Multiple lists can exist for the same model, each with different filters.

Query building

Build queries using the Catalog API:

// Add filter conditions
productList.catalog.query.addCondition({
  attribute: 'price',
  operator: 'gte',
  value: 100,
});

productList.catalog.query.addCondition({
  attribute: 'category',
  operator: 'eq',
  value: 'Electronics',
});

Query execution

A list filters the items with its conditions, then sorts them, then cuts out the page (start, count):

flowchart LR
  QUERY["Query Conditions"]
  FILTER["Filter Items"]
  SORT["Apply Sorting"]
  PAGE["Apply Pagination"]
  ITEMS["Return Items"]

  QUERY --> FILTER
  FILTER --> SORT
  SORT --> PAGE
  PAGE --> ITEMS

// Execute and get results
const items = productList.catalog.query.items();

Each list caches its query results and reuses them until an update() changes the list's data. The cache is bounded by performance.cache.maxSize (default 64 MiB; the least recently used entries are dropped first), see Caching.

Additional operations

Beyond filtering, the Catalog provides:

Facets - Count items by property values

const categoryFacets = productList.catalog.fn.facets({ attribute: 'category' });
const brandFacets = productList.catalog.fn.facets({ attribute: 'brand' });
// categoryFacets: [{ value: 'Electronics', size: 4, selected: true, ... }, { value: 'Clothing', size: 1, selected: false, ... }]
// brandFacets: [{ value: 'BrandA', size: 3, ... }, { value: 'BrandB', size: 1, ... }]

A facet ignores the list's own condition on the faceted attribute, so Clothing keeps its count while the list shows only Electronics. Conditions on other attributes (here price) do reduce the counts.

Aggregations - Calculate statistics

const stats = productList.catalog.fn.aggregate({
  attribute: 'price',
  types: ['min', 'max', 'avg', 'sum', 'count'],
});
// { attribute: 'price', types: [...], min: 149, max: 999, avg: 511.5, sum: 2046, count: 4 }

Sorting and Pagination - Order and limit results

// Page 1, 20 items per page, cheapest first
productList.catalog.query.applyOptions({
  sort: { attributes: ['price'], order: ['ASC'] },
  start: 0,
  count: 20,
});

Complete flow example

Here is the full data flow from source to query result:

sequenceDiagram
  participant Source as Data Source
  participant Loader as DcuplAppLoader
  participant Store as In-Memory Store
  participant List as DcuplList
  participant App as Your Application

  Source->>Loader: JSON/CSV data
  Loader->>Loader: Parse & transform
  Loader->>Store: Stage data
  Store->>Store: init() processes data, builds indices
  App->>List: Create list
  App->>List: Add conditions
  List->>Store: Execute query
  Store->>List: Filtered items
  List->>App: Results

import { Dcupl } from '@dcupl/core';
import { DcuplAppLoader } from '@dcupl/loader';

// 1. Initialize
const dcupl = new Dcupl({ config: { projectId: 'demo', apiKey: 'demo-api-key' } });
const loader = new DcuplAppLoader();
dcupl.loaders.add(loader);

// 2. Load (Source -> Parse -> Store)
await loader.config.fetch();
await loader.process();
await dcupl.init();

// 3. Query
const list = dcupl.lists.create({ modelKey: 'Product' });

list.catalog.query.addCondition({
  attribute: 'category',
  operator: 'eq',
  value: 'Electronics',
});

// 4. Results
const items = list.catalog.query.items();
const brandFacets = list.catalog.fn.facets({ attribute: 'brand' });

Performance considerations

Loading

• Load only needed data using resource tags
• Use CDN for faster delivery
• Consider data size: what fits depends on your data and the device (see Data size and limits); data too large for the browser belongs on a server-hosted REST API (RAPI) instance

Storage

• References and index: true properties are indexed at load time, other properties on demand (e.g. for facets)
• Memory usage scales with data size
• Reference resolution adds overhead for deeply nested queries

Querying

• eq and in conditions on references and index: true properties are fast index lookups
• Complex queries with many conditions may scan more items
• Facet calculations touch all matching items