Data Quality

Data quality checks and validation are core principles in dcupl. As a developer, you're often working with external data sources — APIs, services, or customer inputs — and need to ensure your application functions correctly even when these sources change. dcupl helps maintain data integrity by integrating quality checks directly into model definitions.

info

You can check every (potential) error in the Quality Analyzer Tab in the dcupl Console.

Setup

First of all - quality checks are optional. You can toggle this feature globally for your application, enable/disable checks for specific models and inherit logic down to the attribute level.

const dcupl = new Dcupl({
  quality: {
    enabled: true, // true | false
  },
});

{
  "key": "myModel",
  "quality": {
    "enabled": true // true | false
  }
}

Model-Level Quality Checks

In dcupl, models are defined in JSON. Quality checks can be applied at the model level to ensure overall data integrity. For example:

{
  "key": "myModel",
  "quality": {
    "attributes": {
      "required": true,
      "nullable": false,
      "forceStrictDataType": false,
      "validatorHandling": "loose" // "strict" | "loose"
    }
  },
  "properties": []
}

setting	default	description
required	true	the attribute value is requred or can be undefined
nullable	false	allows the attribute value to be null
forceStrictDataType	false	checks if strings, booleans, integers, floats, ... are transmitted correctly since for example a string "1" may be casted to valid integer 1
validatorHandling	loose	defines what should happen with the data entry if the validation fails. "strict": don't apply the data to the model. "loose": register the error but still apply the data

Attribute Level Quality Checks

dcupl comes with predefined validators for common use cases, allowing you to quickly implement data quality checks without writing custom logic. These validators cover typical scenarios like checking for non-empty fields, enforcing specific formats (e.g., email addresses), and validating numerical ranges. By embedding these checks directly in your model definitions, dcupl helps ensure your application consistently processes reliable and accurate data.

{
  "properties": [
    {
      "key": "email",
      "type": "string",
      "quality": {
        "validators": {
          "email": {}
        }
      }
    },
    {
      "key": "price_int",
      "type": "int",
      "quality": {
        // "nullable": true,
        // "required": true,
        // "forceStrictDataType": true,
        // "validatorHandling": "strict", // "strict" | "loose"
        "validators": {
          "min": {
            "value": 20
          },
          "max": {
            "value": 100
          }
        }
      }
    },
    {
      "key": "weight",
      "type": "string",
      "quality": {
        "validators": {
          "pattern": {
            "value": "^\\d+(\\.\\d+)?\\s+lbs$" // matches 1.1 lbs
          }
        }
      }
    }
  ]
}

Validator	value
custom	{key: 'my-validator', value: any}
email	-
max	number
min	number
maxLength	number
minLength	number
startsWith	string
endsWith	string
includes	string
enum	string[]
unique	-
pattern	Regex Pattern as string (remember to escape your backslashes)