# Data flow

Understanding how data flows through dcupl helps you make better architectural decisions. This page traces data from external sources to query results.

## Overview

Data in dcupl flows through four stages:

```mermaid
flowchart LR
  SOURCE["1. Source<br/>CSV, JSON, API"]
  LOAD["2. Load<br/>Parse & Transform"]
  STORE["3. Store<br/>In-Memory"]
  QUERY["4. Query<br/>Filter & Retrieve"]

  SOURCE --> LOAD
  LOAD --> STORE
  STORE --> QUERY
```

Each stage has specific responsibilities and components.

## Stage 1: data sources

Data originates from external sources. dcupl supports various formats:

| Format | Description                      | Use Case                |
| ------ | -------------------------------- | ----------------------- |
| JSON   | Native format, no parsing needed | APIs, exports           |
| CSV    | Comma-separated values           | Spreadsheets, databases |
| API    | REST endpoints                   | Live data               |

Data sources can be:

- **Static files** - Hosted on CDN or bundled with your app
- **Dynamic endpoints** - APIs that return fresh data
- **Workflow outputs** - Transformed data from Console workflows

## Stage 2: loading

The loading stage fetches and parses raw data and stages it for the model. Processing against the model definitions happens afterwards, in `dcupl.init()` or `dcupl.update()`.

```mermaid
flowchart TD
  subgraph loading["Loading Stage"]
    FETCH["Fetch Data"]
    PARSE["Parse Format"]
    TRANSFORM["Apply Loader Transformers"]
    STAGE["Stage Data Container"]
  end

  SOURCE[Data Source] --> FETCH
  FETCH --> PARSE
  PARSE --> TRANSFORM
  TRANSFORM --> STAGE
  STAGE --> PROCESS["dcupl.init() / update()<br/>Process & Validate Against Model"]
  PROCESS --> STORE[In-Memory Store]
```

Loader transformers are JavaScript resources: a raw-file transformer runs on the fetched file before parsing, a parsed-file transformer on the parsed items.

### DcuplAppLoader

The `DcuplAppLoader` handles loading from a LoaderConfiguration:

```typescript
import { Dcupl } from '@dcupl/core';
import { DcuplAppLoader } from '@dcupl/loader';

const dcupl = new Dcupl({
  config: { projectId: 'your-project', apiKey: 'your-api-key' },
});
const loader = new DcuplAppLoader();
dcupl.loaders.add(loader);

// 1. Fetch configuration
await loader.config.fetch();

// 2. Load the resources it lists
await loader.process();

// 3. Apply the loaded models and data
await dcupl.init();
```

The loader reads `dcupl.lc.json` which defines:

```json
{
  "resources": [
    {
      "url": "${baseUrl}/models/product.dcupl.json",
      "type": "model",
      "tags": ["product"]
    },
    {
      "url": "${baseUrl}/data/products.json",
      "type": "data",
      "model": "Product",
      "tags": ["product"]
    }
  ]
}
```

`loader.process()` loads every resource. To load only some of them, pass `resourceTags`, or define applications (named sets of resources) as described in [Loader](/sdk/api/loader).

### Manual loading

You can also load data directly without a LoaderConfiguration:

```typescript
const dcupl = new Dcupl();

// Add model
dcupl.models.set({
  key: 'Product',
  properties: [
    { key: 'name', type: 'string' },
    { key: 'price', type: 'int' },
  ],
});

// Add data
dcupl.data.set(
  [
    { key: 'p1', name: 'Laptop', price: 999 },
    { key: 'p2', name: 'Mouse', price: 29 },
  ],
  { model: 'Product' },
);

// Data calls are staged: init() (later: update()) applies them
await dcupl.init();
```

### Data transformations

When `init()` (or `update()`) processes the staged data, dcupl applies transformations defined in models:

**Type Coercion** - Converts strings to proper types

```typescript
// Raw CSV data: "29.99"
// Model property: { key: 'price', type: 'float' }
// Result: 29.99 (number)
```

**Array Conversion** - Splits delimited strings into arrays

```typescript
// Raw data: "red,blue,green"
// Property: { key: 'colors', type: 'Array<string>', separator: ',' }
// Result: ["red", "blue", "green"]
```

**Value Mapping** - Translates codes to readable values

```typescript
// Raw data: "1"
// Model: valueMappings: [{ attributes: ['status'], values: [{ from: ['1'], to: 'active' }] }]
// Result: "active"
```

## Stage 3: in-memory storage

After loading, data lives in memory organized by model.

`dcupl.init()` processes every model after the models it references. It handles plain properties and references before the attributes computed from them (resolved, query, derived and grouped references, derived and expression properties), so a derived property or derived reference that reads a resolved reference still gets its value. `dcupl.update()` re-processes only the affected attributes when the change is small, no model was replaced with `data.set()`, no model definition changed since `init()` and no [query reference](/sdk/api/models#reference) is affected (a partial update), and runs the full pipeline otherwise.

Each model holds its items by their `key` (or the model's `keyProperty`). That key is how `query.one({ itemKey })`, `data.update()` and `data.remove()` find an item. When rows loaded together share a key, the last one wins (see [Data Key Strategy](/learn/best-practices/data-modeling#data-key-strategy)).

### Indices

dcupl indexes attributes to speed up queries. References and properties marked `index: true` are indexed while the data is processed; other properties are indexed on demand (for example when you request facets for them).

`eq` and `in` filters use an existing index instead of scanning all items; other operators scan the items.

### Reference graph

dcupl processes a model after the models it references. A reference cycle between models (A → B → A) makes `init()` fail: it logs the cycle and resolves to `false`. A model may reference itself (for example a `parent` reference); that is not a cycle.

```mermaid
graph LR
  Product -->|"category"| Category
  Order -->|"customer"| Customer
  Order -->|"items"| Product
```

References link items across models, which enables deep queries such as `category.name`.

## Stage 4: querying

Queries retrieve and filter data from the in-memory store.

```mermaid
flowchart TD
  subgraph query["Query Execution"]
    CREATE["Create List"]
    BUILD["Build Query"]
    EXECUTE["Execute Filter"]
    RESULT["Return Results"]
  end

  STORE[In-Memory Store] --> CREATE
  CREATE --> BUILD
  BUILD --> EXECUTE
  EXECUTE --> RESULT
```

### Lists

A List is a filtered view of a model's data:

```typescript
const productList = dcupl.lists.create({
  modelKey: 'Product',
});
```

Multiple lists can exist for the same model, each with different filters.

### Query building

Build queries using the Catalog API:

```typescript
// Add filter conditions
productList.catalog.query.addCondition({
  attribute: 'price',
  operator: 'gte',
  value: 100,
});

productList.catalog.query.addCondition({
  attribute: 'category',
  operator: 'eq',
  value: 'Electronics',
});
```

### Query execution

A list filters the items with its conditions, then sorts them, then cuts out the page (`start`, `count`):

```mermaid
flowchart LR
  QUERY["Query Conditions"]
  FILTER["Filter Items"]
  SORT["Apply Sorting"]
  PAGE["Apply Pagination"]
  ITEMS["Return Items"]

  QUERY --> FILTER
  FILTER --> SORT
  SORT --> PAGE
  PAGE --> ITEMS
```

```typescript
// Execute and get results
const items = productList.catalog.query.items();
```

Each list caches its query results and reuses them until an `update()` changes the list's data. The cache is bounded by `performance.cache.maxSize` (default 64 MiB; the least recently used entries are dropped first), see [Caching](/sdk/advanced/caching).

### Additional operations

Beyond filtering, the Catalog provides:

**Facets** - Count items by property values

```typescript
const categoryFacets = productList.catalog.fn.facets({ attribute: 'category' });
const brandFacets = productList.catalog.fn.facets({ attribute: 'brand' });
// categoryFacets: [{ value: 'Electronics', size: 4, selected: true, ... }, { value: 'Clothing', size: 1, selected: false, ... }]
// brandFacets: [{ value: 'BrandA', size: 3, ... }, { value: 'BrandB', size: 1, ... }]
```

A facet ignores the list's own condition on the faceted attribute, so `Clothing` keeps its count while the list shows only Electronics. Conditions on other attributes (here `price`) do reduce the counts.

**Aggregations** - Calculate statistics

```typescript
const stats = productList.catalog.fn.aggregate({
  attribute: 'price',
  types: ['min', 'max', 'avg', 'sum', 'count'],
});
// { attribute: 'price', types: [...], min: 149, max: 999, avg: 511.5, sum: 2046, count: 4 }
```

**Sorting and Pagination** - Order and limit results

```typescript
// Page 1, 20 items per page, cheapest first
productList.catalog.query.applyOptions({
  sort: { attributes: ['price'], order: ['ASC'] },
  start: 0,
  count: 20,
});
```

## Complete flow example

Here is the full data flow from source to query result:

```mermaid
sequenceDiagram
  participant Source as Data Source
  participant Loader as DcuplAppLoader
  participant Store as In-Memory Store
  participant List as DcuplList
  participant App as Your Application

  Source->>Loader: JSON/CSV data
  Loader->>Loader: Parse & transform
  Loader->>Store: Stage data
  Store->>Store: init() processes data, builds indices
  App->>List: Create list
  App->>List: Add conditions
  List->>Store: Execute query
  Store->>List: Filtered items
  List->>App: Results
```

```typescript
import { Dcupl } from '@dcupl/core';
import { DcuplAppLoader } from '@dcupl/loader';

// 1. Initialize
const dcupl = new Dcupl({ config: { projectId: 'demo', apiKey: 'demo-api-key' } });
const loader = new DcuplAppLoader();
dcupl.loaders.add(loader);

// 2. Load (Source -> Parse -> Store)
await loader.config.fetch();
await loader.process();
await dcupl.init();

// 3. Query
const list = dcupl.lists.create({ modelKey: 'Product' });

list.catalog.query.addCondition({
  attribute: 'category',
  operator: 'eq',
  value: 'Electronics',
});

// 4. Results
const items = list.catalog.query.items();
const brandFacets = list.catalog.fn.facets({ attribute: 'brand' });
```

## Performance considerations

### Loading

- Load only needed data using resource tags
- Use CDN for faster delivery
- Consider data size: what fits depends on your data and the device (see [Data size and limits](/introduction#data-size-and-limits)); data too large for the browser belongs on a server-hosted REST API (RAPI) instance

### Storage

- References and `index: true` properties are indexed at load time, other properties on demand (e.g. for facets)
- Memory usage scales with data size
- Reference resolution adds overhead for deeply nested queries

### Querying

- `eq` and `in` conditions on references and `index: true` properties are fast index lookups
- Complex queries with many conditions may scan more items
- Facet calculations touch all matching items

## Next steps

- [Glossary](/learn/concepts/glossary) — Understand dcupl terminology
- [Core concepts](/sdk/core-concepts) — Deep dive into SDK concepts
- [Queries](/sdk/core-concepts/query-overview) — Learn query operators