Transformers

In the development process, it's common to encounter imperfect data that might not be well-suited for your application. This is especially true during early stages when resources are not optimized, and data might be missing, incorrectly mapped, or inaccurate.

tip

You can find code related to this topic in our IMDB demo repository

In frontend applications, developers often handle these issues by writing code to map and correct the data specific to their application. However, it's important to note that these fixes are only applicable to that particular application.

To address this and make such fixes more widely applicable, we've introduced a feature called Transformers. With Transformers, you can implement custom data transformation logic that can be used at a global level, making it easier to handle varying data sources and ensuring consistent data processing across different applications.

Transformers enable you to write custom code that can be loaded in the dcupl Console

To enhance data processing before it is loaded or indexed by the dcupl SDK, Transformers will be integrated into the dcupl App Loader. These Transformers will provide custom functionality to manipulate the data according to your specific needs.

The flexibility of Transformers allows them to be configured for loading from various sources, and they can be selectively applied to individual applications or resources using tags. This enables you to tailor data transformations to suit the requirements of different parts of your application or specific datasets.

info

Caution: When utilizing Transformers, it's important to be mindful of their potential impact on your application's startup time. The size of your data and the complexity of the transformer function can significantly slow down the process.

To optimize performance, we recommend considering a preprocessing step for data transformation. By moving the transformer logic away from the client's device, you can enhance the overall efficiency of your application.

Transformer code example

A transformer is a JavaScript function two parameters: the relevant resource and the items. Remember, any data returned from this function will be applied directly to the connected model. This gives you the flexibility to customize and adapt the data based on your specific requirements before it is integrated into the model.

// dcupl/transformers/data_transformer.js

return (dataTransformer = (resource, items) => {
  return items.map((item) => {
    if (item.Episodes && typeof item.Episodes === 'string') {
      const episodes = item.Episodes.split(' ');
      if (episodes.length > 1) {
        item.EpisodesAsNumber = parseInt(episodes[0]);
      }
    }
    return item;
  });
});

loader configuration

To include your transformer function in your application simply add it as a new resource in your loader configuration. The applyTo lists all resource tags affected by the transformer.

{
  "applications": [
    {
      "key": "my_app",
      "resourceTags": ["data", "my_transformer"]
    }
  ],
  "resources": [
    ...
    {
      "url": "${baseUrl}/transformers/data_transformer.js",
      "type": "transformer",
      "tags": ["my_transformer"],
      "applyTo": ["data"]
      // "transformerType": "parsedFileTransformer" | "rawFileTransformer",
    },
    {
      "url": "${baseUrl}/data/IMDB.csv",
      "type": "data",
      "model": "show",
      "tags": ["data", "transform"]
    }
  ]
}

What to do with transformers

Transformers are a easy and sustainable way to fix data issues right away without waiting for new data from customers or other collegueas. But always try to move transformer logic - if possible - to a earlier preprocessing step to reduce processing time on the client.

Most usecases regarding transformers are:

• Clean up data
• Map/Add translations
• Temporary fix errors