# Architecture

dcupl follows a modular architecture where you can use as much or as little as you need. At its simplest, you use the SDK alone. For more complex needs, you add the Console platform.

## High-level overview

```mermaid
flowchart TB
  subgraph free["Free on npm"]
    SDK["@dcupl/core<br/>Query Engine"]
    LOADER["@dcupl/loader<br/>Data Loading"]
    CONNECT["@dcupl/connect<br/>Debugging"]
    CLI["@dcupl/cli<br/>Scaffold, Sync, Local Apps"]
  end

  subgraph saas["Console Platform (Optional)"]
    CONSOLE["Console UI<br/>Model & Workflow Editor"]
    RUNNER["Runner (workflow-runner)<br/>Data Transformation"]
    RAPI["Runner (app-runner)<br/>REST API (RAPI) instances"]
  end

  subgraph storage["Storage"]
    CDN["CDN<br/>JSON Files"]
    SOURCES["Data Sources<br/>CSV, APIs, etc."]
  end

  SOURCES --> RUNNER
  RUNNER --> CDN
  CDN --> LOADER
  LOADER --> SDK
  CONSOLE --> RUNNER
  CONSOLE --> RAPI
  SDK --> APP[Your Application]
  CONNECT -.-> CONSOLE
  CLI -.->|"files push / pull"| CDN
```

## SDK packages

The SDK is a set of free npm packages. Every app installs `@dcupl/core`; add the others when you need them, e.g. `npm install @dcupl/core @dcupl/loader`. `@dcupl/loader` and `@dcupl/connect` build on `@dcupl/core`, and the packages require each other in exactly their own version, so install every `@dcupl/*` package in the same version.

### @dcupl/common

The types and helpers you import in your own code. Add it next to `@dcupl/core` when you import from it (`npm install @dcupl/common`):

- **Types** - `ModelDefinition`, `DcuplQuery`, `DcuplQueryGroup`, `DcuplListOptions` and the other types the SDK takes and returns
- **DcuplQueryBuilder** - Builds and edits query groups
- **logger** - The shared logger that `@dcupl/loader` logs through (see [Logging Configuration](/sdk/guides/error-handling#logging-configuration))

### @dcupl/core

The main query engine and entry point:

- **`Dcupl`** - The instance you create: you register models and data, call `init()`, create lists, and run one-off queries and functions on a model (`dcupl.query`, `dcupl.fn`)
- **`DcuplList`** - A filtered view of one model (`dcupl.lists.create()`)
- **Catalog** - The query interface on a list (`list.catalog`): filtering, sorting, facets and aggregations

### @dcupl/loader

Data ingestion and configuration loading:

- **DcuplAppLoader** - Loads a loader configuration (`dcupl.lc.json`) and its resources from the dcupl CDN or any URL
- **DcuplCsvParser** - CSV parsing with type coercion
- Variable interpolation for environment-specific URLs

### @dcupl/connect

Development debugging tool:

- Real-time SDK inspection in Console UI
- Forwards dcupl, list and loader events (such as query changes) to the Console window that opened your app
- State visualization during development

## Execution contexts

The same SDK code runs in multiple environments:

```mermaid
flowchart LR
  subgraph browser["Browser"]
    B_SDK["dcupl SDK"]
    B_DATA["Data in Memory"]
    B_SDK --> B_DATA
  end

  subgraph node["Node.js"]
    N_SDK["dcupl SDK"]
    N_DATA["Data in Memory"]
    N_SDK --> N_DATA
  end

  subgraph rapi["REST API instance on a runner"]
    R_SDK["dcupl SDK"]
    R_DATA["Data in Memory"]
    R_SDK --> R_DATA
  end

  USER[User] --> browser
  SSR[SSR Framework] --> node
  API[API Request] --> rapi
```

### Browser (client-side)

The most common use case. Data loads into the browser, and all queries execute locally.

**Benefits:**

- Zero server cost for queries
- Instant user interactions
- Works offline after initial load

**Best for:** Interactive filtering interfaces on data that fits in the browser's memory (see [Data size and limits](/introduction#data-size-and-limits))

### Node.js (server-side)

The same SDK runs on the server for SSR or batch processing.

**Benefits:**

- SEO-friendly initial renders
- Can handle larger datasets with more memory
- Server-to-server data processing

**Best for:** Initial page renders, large dataset processing, API endpoints

### Hybrid pattern

Combine both for optimal results:

```mermaid
sequenceDiagram
  participant Browser
  participant Server
  participant Data

  Server->>Data: Load data
  Server->>Server: Query for initial render
  Server->>Browser: HTML with data
  Browser->>Data: Load the same data
  Browser->>Browser: dcupl.init()
  Browser->>Browser: All subsequent queries (local)
```

The server renders the initial page for SEO. The browser then loads the same data into its own dcupl instance (the SDK does not transfer its state from the server) and handles all further user interactions locally.

## The Console platform

The Console (console.dcupl.com) is an optional SaaS platform on top of the SDK. In it you edit models and workflows, keep versioned project files, and deploy workflows and REST API instances to runners. Your project files, and the output a workflow writes into them, are served from the dcupl CDN; workflows can also write to S3, Azure or Git. The [High-Level Overview](#high-level-overview) shows how these parts connect, and the sections below describe each one.

### Console UI

The web application at console.dcupl.com, for:

- Visual model editor
- Workflow designer
- Data exploration (the Data Explorer)
- Project management

### Console API

The HTTP API of the Console, at `https://api.dcupl.com`. The CLI calls it for `dcupl files` push and pull and for the `dcupl workflow` commands, authenticated with a CLI API key (see [API Keys](/console/api-keys)). For a self-hosted Console, point the CLI at yours with `dcupl config set --console-api-url <url>`. Scripts and CI jobs can also call it directly with an `api-key` header, for example the [Reporting API endpoints](/console/workflows/reporting#reporting-api-endpoints).

### Workflow runner

Executes data transformation workflows:

- Fetches data from external sources
- Applies transformations
- Can validate the data against your models (in a dcupl instance node)
- Stores processed JSON

### Runner (REST API instances)

A runner of type `app-runner` hosts deployed REST API instances:

- Exposes SDK queries as REST APIs
- Keeps the data in server memory, for datasets too large for the browser
- Checks the API key set in the template (`auth`) on every request; an instance without one is open
- Can also serve an MCP endpoint per instance, so AI agents can query it. You switch it on per REST API template; MCP is part of a paid plan that dcupl turns on for your project, so ask your dcupl contact (see [MCP endpoint](/console/rapi/mcp#turning-it-on)).

### Files and CDN

The Console stores versioned project files (models, data and the loader configuration), and the dcupl CDN (`cdn.dcupl.com`) serves them to the SDK's loader. The CLI syncs your local files with the Console (`dcupl files push` / `pull`).

### CLI

The `@dcupl/cli` package works on the same files from your terminal: it scaffolds projects, validates models (`dcupl validate`), syncs files with the Console, deploys workflows, and runs local data apps (`dcupl app`) with the SDK in memory, which you can query from the terminal. See [How the CLI works](/cli/how-it-works).

## Data storage

dcupl uses a JSON-first architecture for portability:

```mermaid
flowchart LR
  subgraph config["Configuration"]
    LC["dcupl.lc.json<br/>LoaderConfiguration"]
    MODELS["*.dcupl.json<br/>Model Definitions"]
  end

  subgraph data["Data"]
    JSON["*.json<br/>Data Files"]
  end

  LC --> MODELS
  LC --> JSON
  MODELS --> SDK
  JSON --> SDK
```

### LoaderConfiguration

The central configuration file (`dcupl.lc.json`) defines:

- **Resources** - Model, data, script, operator and transformer file URLs with tags
- **Applications** - Which resources to load for each app
- **Environments** - Environment-specific variables
- **Variables** - URL interpolation values (per environment or application)

### Model files

JSON files defining data structure:

```json
{
  "key": "Product",
  "properties": [
    { "key": "name", "type": "string" },
    { "key": "price", "type": "int" }
  ]
}
```

### Data files

JSON arrays of data matching model structure:

```json
[
  { "key": "p1", "name": "Laptop", "price": 999 },
  { "key": "p2", "name": "Mouse", "price": 29 }
]
```

## Integration patterns

### Pattern 1: SDK only

Simplest setup for static or developer-managed data.

```typescript
import { Dcupl } from '@dcupl/core';

const dcupl = new Dcupl();
dcupl.models.set(productModel);
dcupl.data.set(productData, { model: 'Product' });
await dcupl.init();
```

### Pattern 2: SDK + loader

Load configuration and data from URLs (the dcupl CDN or your own server).

```typescript
import { Dcupl } from '@dcupl/core';
import { DcuplAppLoader } from '@dcupl/loader';

const dcupl = new Dcupl({
  config: { projectId: 'your-project', apiKey: 'your-api-key' },
});
const loader = new DcuplAppLoader();
dcupl.loaders.add(loader);

await loader.config.fetch();
await loader.process();
await dcupl.init();
```

For files you host yourself, pass their location instead: `await loader.config.fetch({ baseUrl: 'https://example.com/dcupl' })` loads `https://example.com/dcupl/dcupl.lc.json` and fills the `${baseUrl}` variable in the resource URLs. `projectId` and `apiKey` are only needed for the dcupl CDN.

### Pattern 3: SDK + Console + REST API

Full platform with server-side queries for large datasets.

```mermaid
flowchart LR
  CLIENT[Browser SDK] --> |"Small datasets"| LOCAL[Local Query]
  CLIENT --> |"Large datasets"| RAPI[REST API instance]
  RAPI --> SERVER[Server SDK]
```

## Key takeaways

1. **Modular design** - Use only what you need
2. **Universal SDK** - Same code runs everywhere
3. **JSON-first** - All configuration is portable
4. **Layered architecture** - SDK alone, with Loader, or full Console

## Next steps

- [Data flow](/learn/concepts/data-flow) — See how data moves through the system
- [Glossary](/learn/concepts/glossary) — Understand dcupl terminology
- [SDK Quick Start](/sdk/quick-start) — Build your first application