Quality & validation

dcupl provides a comprehensive quality and validation system to ensure data integrity, enforce business rules, and catch errors early. Configure validation rules at both the model and property level.

What is quality & validation?

Quality configuration enables:

• Required fields - Ensure critical data is present
• Nullable control - Allow or prevent null values
• Type enforcement - Strict or loose data type checking
• Validators - Built-in email, min/max, pattern, enum, unique and more
• Error tracking - Collect and report validation errors
• Error summary - Error counts per model, attribute and error type

Enabling quality validation

Quality validation has two levels, the instance and the model. Both are enabled by default, and checks run only when both are enabled:

import { Dcupl } from '@dcupl/core';
import type { ModelDefinition } from '@dcupl/common';

const dcupl = new Dcupl({
  quality: {
    enabled: true, // Default: true. false turns off all quality checks
    maxErrorsPerGroup: 1000, // Default: 1000. Stored error records per (model, attribute, error type)
  },
});

const userModel: ModelDefinition = {
  key: 'User',
  quality: { enabled: true }, // Default: true. false skips all checks for this model
  properties: [
    // ... properties with validation rules
  ],
};

The older errorTracking: { enabled } option is deprecated and can't turn checks off: checks run when either option is enabled, and quality.enabled defaults to true. Use quality: { enabled: false } instead.

Property quality configuration

Quality options

Instance options go into new Dcupl({ quality }):

Model options go into the model definition's quality:

Attribute options go into a property's (or reference's) quality:

What happens to the value when a check fails:

Model-wide defaults

Attributes are required by default: a declared property or reference that is missing on an item is reported as UndefinedValue. Mark optional fields with required: false, or change the default for a whole model with quality.attributes. It accepts required, nullable, forceStrictDataType and validatorHandling; validators are set per attribute only.

const productModel: ModelDefinition = {
  key: 'Product',
  quality: {
    attributes: { required: false }, // every attribute optional unless it says otherwise
  },
  properties: [
    { key: 'name', type: 'string', quality: { required: true } },
    { key: 'description', type: 'string' }, // optional
  ],
};

Required fields

A required field without a value is reported as UndefinedValue:

const userModel: ModelDefinition = {
  key: 'User',
  properties: [
    { key: 'email', type: 'string', quality: { required: true } },
    { key: 'bio', type: 'string', quality: { required: false } }, // optional
  ],
  data: [
    { key: 'u1', email: 'alice@example.com' }, // ✅ Valid
    { key: 'u2' }, // ❌ Missing required email
  ],
};

dcupl.models.set(userModel);
await dcupl.init();

const errors = dcupl.query.execute({
  modelKey: '$DcuplErrorTrackingErrors',
  queries: [{ attribute: 'model', operator: 'eq', value: 'User' }],
});
console.log(errors.map((e) => `${e.itemKey}.${e.attribute}: ${e.errorType.key}`));
// [ 'u2.email: UndefinedValue' ]

Nullable fields

A null on a non-nullable field is reported as NullValue; the value itself is kept on the item.

const productModel: ModelDefinition = {
  key: 'Product',
  properties: [
    { key: 'name', type: 'string', quality: { nullable: false } }, // the default
    { key: 'description', type: 'string', quality: { required: false, nullable: true } },
  ],
  data: [
    { key: 'p1', name: 'Laptop', description: null }, // ✅ Valid
    { key: 'p2', name: null }, // ❌ name cannot be null (NullValue)
  ],
};

Strict data type checking

By default, values are coerced to the declared type ('5' becomes 5). Values that can't be coerced are dropped and reported as WrongDataType, and so are lossy coercions (5.5 becomes 5 on an int, the coerced value is kept). This happens without any strict setting. A string property accepts any value (42 becomes '42'), and a boolean property accepts the strings 'true'/'false'/'1'/'0' (any case) and the numbers 1/0; any other string, number, object or array is reported as WrongDataType and the value stays undefined. CSV cells follow the same rule, except that an empty cell is no value, not a WrongDataType.

forceStrictDataType: true turns coercion off: every value that isn't already of the declared type is reported as WrongDataType and dropped. date properties ignore the setting.

const orderModel: ModelDefinition = {
  key: 'Order',
  properties: [{ key: 'quantity', type: 'int', quality: { forceStrictDataType: true } }],
  data: [
    { key: 'o1', quantity: 5 }, // ✅ Valid int
    { key: 'o2', quantity: '5' }, // ❌ String not allowed
    { key: 'o3', quantity: 5.5 }, // ❌ Float not allowed
  ],
};

References

References are required by default too, so an item without a value for a reference is reported as UndefinedValue (error group ReferenceDataError). An absent or empty ('') value counts as missing. An explicit null is kept as null on the item (SDK 2.0; eq null matches it, typeof 'undefined' does not) and is reported as NullValue unless nullable is set, as for a property. Of the other attribute options, forceStrictDataType and validators have no effect on references.

validity is set on the reference itself ({ key: 'category', type: 'singleValued', model: 'Category', validity: 'strict' }). A referenced model with supportsAutoCreation creates the missing item instead of reporting it. See the reference options in the Models API.

Validators

A failed validator is reported as InvalidValidator. With the default validatorHandling: 'loose' the value is kept; with 'strict' it is dropped from the item. Validators take { value }; email and unique take {}. The available validators are email, min, max, minLength, maxLength, pattern, enum, startsWith, endsWith, includes and unique.

Email validator

Validate email format:

const contactModel: ModelDefinition = {
  key: 'Contact',
  properties: [
    {
      key: 'email',
      type: 'string',
      quality: {
        required: true,
        validators: {
          email: {}, // Email format validation
        },
      },
    },
  ],
  data: [
    { key: 'c1', email: 'alice@example.com' }, // ✅ Valid
    { key: 'c2', email: 'invalid-email' }, // ❌ Invalid format
    { key: 'c3', email: 'bob@domain' }, // ❌ Invalid format
  ],
};

dcupl.models.set(contactModel);
await dcupl.init();

const errors = dcupl.query.execute({
  modelKey: '$DcuplErrorTrackingErrors',
  queries: [
    { attribute: 'model', operator: 'eq', value: 'Contact' },
    { attribute: 'errorType', operator: 'eq', value: 'InvalidValidator' },
  ],
});
console.log(errors.map((e) => e.description));
// [
//   'The value "invalid-email" is not a valid email address',
//   'The value "bob@domain" is not a valid email address'
// ]

Min/max validators

Validate numeric ranges on int and float properties:

const productModel: ModelDefinition = {
  key: 'Product',
  properties: [
    {
      key: 'rating',
      type: 'float',
      quality: { validators: { min: { value: 1 }, max: { value: 5 } } },
    },
  ],
  data: [
    { key: 'p1', rating: 4.5 }, // ✅ Valid
    { key: 'p2', rating: 0 }, // ❌ 'The value "0" is smaller than the minimum value of "1"'
    { key: 'p3', rating: 6 }, // ❌ 'The value "6" is larger than the maximum value of "5"'
  ],
};

Unique validator

Ensure unique values across all items of the model:

const userModel: ModelDefinition = {
  key: 'User',
  properties: [{ key: 'email', type: 'string', quality: { validators: { unique: {} } } }],
  data: [
    { key: 'u1', email: 'alice@example.com' }, // ✅ Valid
    { key: 'u2', email: 'alice@example.com' }, // ❌ Duplicate email
  ],
};

A duplicate is reported as InvalidValidator (title "Validator Error - Unique") on the second and later items with that value. In SDK 2.0, items without a value count as duplicates of each other too (The value "undefined" is not unique). NonUniqueKey is a different error: two items with the same key.

Common validations

The other validators work like min and max: one { value } each, and each failure is its own InvalidValidator record with the description shown. Several validators on one attribute go into the same validators object ({ email: {}, unique: {}, maxLength: { value: 254 } }) and are checked independently.

Attributes are required by default, so required: true is only needed where the model sets quality: { attributes: { required: false } }.

Error handling

Getting errors

dcupl.quality.errors.summary() returns one entry per (model, attribute, error type) group with its count. The individual error records live in the built-in model $DcuplErrorTrackingErrors, so you query them like any other model. With quality: { enabled: false } these models are not created, and summary() lists only loader errors.

Using the User model from Required Fields:

await dcupl.init();

// One entry per (model, attribute, errorType) group
const summary = dcupl.quality.errors.summary();
console.log(summary);
// [
//   {
//     key: 'User::email::UndefinedValue',
//     model: 'User',
//     attribute: 'email',
//     errorGroup: 'PropertyDataError',
//     errorType: 'UndefinedValue',
//     type: 'model',
//     totalCount: 1,
//     storedCount: 1,
//     truncated: false
//   }
// ]

// Individual error records for one model
const userErrors = dcupl.query.execute({
  modelKey: '$DcuplErrorTrackingErrors',
  queries: [{ attribute: 'model', operator: 'eq', value: 'User' }],
});
console.log(userErrors);
// [
//   {
//     key: '4YfyXvsPa2',
//     title: 'Undefined Value',
//     itemKey: 'u2',
//     attribute: 'email',
//     description: 'The required string property "email" in the "User" model is undefined (no value)',
//     meta: { rawValue: undefined, declaredType: 'string', origin: 'inline' },
//     groupTotal: 1,
//     truncated: false,
//     model: { key: 'User' },
//     errorGroup: { key: 'PropertyDataError' },
//     errorType: { key: 'UndefinedValue' }
//   }
// ]

In error records, model, errorGroup and errorType are references: filter them with a plain string (value: 'User') and read them as error.errorType.key. The record key is generated.

Error types

Model errors (type: 'model'):

• UndefinedValue - A required field has no value on an item
• NullValue - A non-nullable field is null on an item
• WrongDataType - A value can't be coerced to the declared type, or the coercion is lossy. Reported without any strict setting
• InvalidValidator - A validator failed (including unique)
• UndefinedAttribute - A required attribute has no column or field at all in the loaded data (one record without an itemKey)
• UnknownColumn - A column in a loaded CSV file is not declared on the model; its values are dropped. Only reported for CSV resources, not for JSON or inline data
• NonUniqueKey - Two rows loaded together (with data.set(), a model's data or a loader resource) share the same key. The last row wins as a whole, at the first row's position; no values are merged from the replaced rows, and the replaced rows record no further errors. One NonUniqueKey is recorded per replaced row, also when the rows arrive in separate containers (data.set() after init(), model data plus a staged upsert). upsert and update rows with the same key still merge without an error
• MissingKey - An item has no key: no key, a custom keyProperty whose value is undefined/null, or a composite key with every part missing (a single missing part keeps its 'undefined'/'null' part). The item is dropped. On a model with derived, grouped, resolved or query references, or derived or expression properties, a keyless row currently also makes init(), or a dcupl.update() that rebuilds (for example after data.set()), fail: it logs "Initialization failed" or "Update failed" and resolves false. This also happens with quality checks disabled. Remove such rows before loading
• RemoteReferenceKeyNotFound - A reference value points to a key that doesn't exist in the referenced model
• MissingModel - A reference points to a model that isn't registered
• MissingProperty - A derived property's remoteProperty doesn't exist on the remote model
• MissingReference - A derived, resolved or grouped attribute names a reference that doesn't exist (a derived property's localReference, a derived reference's localReference or remoteReference, resolve.reference, groupBy.reference)
• DuplicatePropertyKey - A model declares the same property key twice
• UnknownExpressionVariable - An expression uses a variable that is neither a property nor a reference
• InvalidModelDefinition - The model definition is invalid

Loader errors (type: 'loader', recorded even when quality checks are disabled):

• InvalidResource - A resource could not be fetched or parsed
• TransformerError - A transformer threw while processing a resource
• ScriptingDisabled - A script, operator or transformer resource was skipped because the instance runs with scripting: { enabled: false }
• InvalidConfig - Typed, but never recorded: a loader configuration that can't be fetched or parsed makes loader.config.fetch() throw instead
• EndpointScriptMissing, EndpointDuplicate - Reported by REST API instances, not by the SDK: an endpoint's scriptKey names a script that isn't registered, or the same method and path is configured twice

Errors after updates

There is no public method to reset errors. Errors follow the data instead: when you fix the data and apply the update, the errors for those items are removed.

dcupl.data.update([{ key: 'u2', email: 'bob@example.com' }], { model: 'User' });
await dcupl.update();

console.log(dcupl.quality.errors.summary());
// []

Quality reports

$DcuplErrorTrackingErrors and $DcuplErrorTrackingSummary are built the first time something reads them (query.one/many/execute, fn.* or a list on them) rather than during every init() and update(); they still describe the error set of the last init() or update(), and every read returns the same records. The first read pays for the build. Once a list over an error-tracking model exists (lists.create until it is destroyed; a query.execute or fn.* call on one keeps such a list internally), every init() and update() builds them again right away. Errors whose description does not depend on the row (UndefinedValue, NullValue, rows without a key) share one description string per (model, attribute, error type) group, which keeps the heap lower on error-heavy datasets. Measured on a 23,754-row customer catalog with about 344,000 recorded errors in 887 groups, default options: building them at the first read and sharing the descriptions took the retained heap after init() from 285 MB to 110 MB, and init() from 2763 ms to 2030 ms (2026-10-16).

Error summary

Each summary() entry describes one (model, attribute, error type) group:

The same totals are available as the built-in model $DcuplErrorTrackingSummary.

Limiting stored errors

quality.maxErrorsPerGroup (default 1000) caps how many error records are stored per group. The totals keep counting:

const dcupl = new Dcupl({
  quality: { enabled: true, maxErrorsPerGroup: 2 },
});

dcupl.models.set({
  key: 'Product',
  properties: [{ key: 'price', type: 'float', quality: { validators: { min: { value: 0 } } } }],
  data: [1, 2, 3, 4, 5].map((i) => ({ key: `p${i}`, price: -i })),
});
await dcupl.init();

const [group] = dcupl.quality.errors.summary();
console.log(group.totalCount, group.storedCount, group.truncated);
// 5 2 true

The stored records of a capped group carry truncated: true and the group's groupTotal.

Failed type coercions are the exception. They are consolidated into one WrongDataType record per attribute, without an itemKey, with the number of affected rows in meta.rowCount and one sample value in meta.rawValue. When more values of one attribute fail than maxErrorsPerGroup, the counts around that consolidation aren't exact in SDK 2.0 (for inline data the consolidated record can even be missing), so keep maxErrorsPerGroup above the number of failures you need to count.

Browsing errors in a list

Errors are a model, so lists, queries and facets work on them, for example to count errors by type:

dcupl.models.set({
  key: 'User',
  properties: [
    { key: 'email', type: 'string', quality: { validators: { email: {} } } },
    { key: 'age', type: 'int' },
  ],
  data: [
    { key: 'u1', email: 'not-an-email', age: 'old' },
    { key: 'u2', age: 30 },
    { key: 'u3', email: 'also-wrong', age: 41 },
  ],
});
await dcupl.init();

const errorList = dcupl.lists.create({ modelKey: '$DcuplErrorTrackingErrors' });

const byType = errorList.catalog.fn.facets({ attribute: 'errorType' });
console.log(byType.map((facet) => `${facet.key}: ${facet.size}`));
// [ 'InvalidValidator: 2', 'UndefinedValue: 1', 'WrongDataType: 1' ]

errorList.catalog.query.addCondition({ attribute: 'errorType', operator: 'eq', value: 'InvalidValidator' });
console.log(errorList.catalog.query.items().map((e) => e.itemKey));
// [ 'u1', 'u3' ]

In SDK 2.0 the errorType facet leaves out UnknownColumn, MissingKey, DuplicatePropertyKey, MissingProperty, MissingReference, UnknownExpressionVariable, ScriptingDisabled and TransformerError. Filtering by these types works; for complete counts use summary().

Data import validation

Check rows before you accept them, and report every problem at once:

import { Dcupl } from '@dcupl/core';

async function importProducts(rows: Record<string, unknown>[]) {
  const dcupl = new Dcupl();

  dcupl.models.set({
    key: 'Product',
    properties: [
      {
        key: 'sku',
        type: 'string',
        quality: {
          validators: {
            unique: {},
            pattern: { value: '^[A-Z]{3}-[0-9]{4}$' },
          },
        },
      },
      { key: 'price', type: 'float', quality: { validators: { min: { value: 0 } } } },
      { key: 'description', type: 'string', quality: { required: false } },
    ],
    data: rows,
  });

  await dcupl.init();

  const summary = dcupl.quality.errors.summary();
  const records = dcupl.query.execute({
    modelKey: '$DcuplErrorTrackingErrors',
    queries: [{ attribute: 'model', operator: 'eq', value: 'Product' }],
  });
  return { success: summary.length === 0, summary, records };
}

const result = await importProducts([
  { key: 'p1', sku: 'LPT-0001', price: 999 },
  { key: 'p2', sku: 'LPT-0001', price: 899 }, // duplicate sku
  { key: 'p3', sku: 'mouse', price: -5 }, // wrong pattern, negative price
  { key: 'p4', price: 'free' }, // no sku, price not a number
]);

console.log(result.success); // false
console.log(result.summary.map((s) => `${s.attribute}: ${s.errorType} (${s.totalCount})`));
// [
//   'sku: InvalidValidator (2)',
//   'price: InvalidValidator (1)',
//   'sku: UndefinedValue (1)',
//   'price: WrongDataType (1)'
// ]
console.log(result.records.map((e) => `${e.itemKey ?? '-'}.${e.attribute}: ${e.description}`));
// [
//   'p3.sku: The value "mouse" does not match the pattern "^[A-Z]{3}-[0-9]{4}$"',
//   'p3.price: The value "-5" is smaller than the minimum value of "0"',
//   'p4.sku: The required string property "sku" in the "Product" model is undefined (no value)',
//   '-.price: 1 value(s) in column "price" of model "Product" failed to coerce to the declared type "float" (e.g. "free")',
//   'p2.sku: The value "LPT-0001" is not unique'
// ]

The failed coercion is one consolidated WrongDataType record per attribute, without an itemKey. These rows are inline data: the CSV-only checks (UnknownColumn, parsing of true / false cells) apply only to CSV files loaded through the loader.

Form validation integration

Validate a single form entry with a model that describes the form:

import { Dcupl } from '@dcupl/core';

type FieldError = { key: string; description: string };

async function validateUser(formData: Record<string, string>): Promise<FieldError[]> {
  // A fresh instance per check: its error records are complete after init()
  const dcupl = new Dcupl();
  dcupl.models.set({
    key: 'User',
    properties: [{ key: 'email', type: 'string', quality: { validators: { email: {} } } }],
    data: [{ key: 'form', ...formData }], // every item needs a key
  });
  await dcupl.init();

  return dcupl.query.execute({
    modelKey: '$DcuplErrorTrackingErrors',
    queries: [{ attribute: 'model', operator: 'eq', value: 'User' }],
  });
}

Show the returned descriptions next to the field and submit only when the list is empty. Key the messages by the error record's key: one field can fail several validators. With a single item, a field that is missing altogether is reported as UndefinedAttribute (The Attribute "email" is not present in the dataset for model "User"), not as UndefinedValue, so map it to your own message.

Troubleshooting

Validation not running

Problem: No validation errors despite invalid data

Solution: Both levels are enabled by default, so check that neither new Dcupl({ quality: { enabled: false } }) nor a model's quality: { enabled: false } switches them off. Validators never run on boolean, json and date properties, and don't run on int and float properties that use forceStrictDataType: true (see Strict Data Type Checking).

Too many errors

Problem: Overwhelming number of validation errors

Solution: Look at the summary first: one entry per group instead of one record per item. Many UndefinedValue errors usually mean optional fields that aren't marked required: false (attributes are required by default). maxErrorsPerGroup limits how many records are stored.

console.table(dcupl.quality.errors.summary());

// Mark optional fields, or make all attributes of a model optional
{ key: 'bio', type: 'string', quality: { required: false } }
{ key: 'LogEntry', quality: { attributes: { required: false } }, properties: [...] }

// Validate critical fields first, add more validation gradually
{ key: 'email', quality: { required: true, validators: { email: {} } } }