Transformer

Owned by Dana Negrescu
Last updated: Jul 24, 2023
56 min read

In order to assist with the Data Integration (DI) efforts with 3rd party systems (ex: Prismic, GraphCMS, Strapi, etc), Boxalino is presenting the Data Transformer service https://boxalino-di-transformer-krceabfwya-ew.a.run.app/ . This is an ETL-like service.

Use this service to:

  1. Identify the required request BODY for content integration Boxalino Transformer

  2. Identify the required request BODY for API request https://boxalino-di-transformer-krceabfwya-ew.a.run.app/api/view

  3. Identify if your project`s connector (headless CMS/BQ/etc) is supported. If not, please provide a service request.

This is service is currently expanding. We value our integrator`s feedback.
For further service requests and integration reviews, please get in touch with Boxalino.

 

INTEGRATION BENEFITS

When exporting other content to Boxalino ecosystem your project is opening up to further features:

  1. Dynamic content ( top brands/categories/blogs, etc) as part of your API integration/layout

     

  2. Mixed API response for search/listing (ex: blogs, promotions, campaigns alongside products)

     

  3. Dynamic pages for your project content (ex: blog pages driven by the personalization/AI engine)

     

  4. More personalized content to keep your customer`s engaged (as part of our omnichannel approach)

     

  5. Relevant statistics for each of your content reach (performance impact)

  6. A/B testing as to identify what content your customers have a higher response towards

 

INTEGRATION FLOW

  1. Review the Transformer services:

    1. is your connector type is not yet supported, please get in touch with Boxalino.

  2. Use the Boxalino Transformer service to identify the body content requested for the API request.

  3. Identify the content types (blog, banner, categories, etc) that must be integrated from your 3rd party source.

  4. Prepare the connector options Transformer | CONNECTOR with content from your 3rd party system.

  5. Do a test connector request Transformer | TEST REQUEST

  6. Prepare the mapping Transformer | MAPPING for each content type you want to export.

  7. Do a test sample request Transformer | REQUEST DEFINITION

8. Repeat step 6-7 until all desired content is part of the response.

8. Do the sync request Transformer | REQUEST DEFINITION

REQUEST DEFINITION

As an integrator, please create the bellow request to the provided endpoint.

1

Endpoint

full data sync

https://boxalino-di-transformer-krceabfwya-ew.a.run.app

2

 

test / stage

https://boxalino-di-transformer-stage-krceabfwya-ew.a.run.app

3

Action

/sync

4

Method

POST

5

Body

use the public endpoint
https://boxalino-di-transformer-krceabfwya-ew.a.run.app/transformer/view
to check the required BODY structure for your integration use-case (it depends on the content type and connector type)

6

Headers

Authorization

Basic base64<<DATASYNC API key : DATASYNC API Secret>>

note: use the API credentials from your Boxalino account that have the ADMIN role assigned

7

 

Content-Type

application/json

8

 

client

account name

9

 

mode

data sync mode: F for full, D for delta

10

 

type

product, user, content, user_content, order.

if left empty - it will check for all tables with the given tm

11

 

tm

(optional) time , in format: YmdHis

technical: used to identify the documents version

12

 

ts

(optional) timestamp, must be millisecond based in UNIX timestamp

13

 

dev

(optional) use this to mark the content for the dev data index

 

BODY ELEMENTS

The API SYNC request BODY structure can be accessed in the public service Boxalino Transformer

[ { "connector": { "type": "prismic", "options": { "query": [], "method": "GET", "access_token": "", "url": "", "page": 1 } }, "document": { "type": "content" }, "mapping": { "id": "", "type": "", "creation": "", "last_update": "", "persona_type": "", "persona_id": "", "parent_content_ids": [], "products": [], "contents": [], "customers": [], "stores": [], "title": [], "description": [], "short_description": [], "images": [], "link": [], "tags": [], "labels": [], "status": "", "periods": [], "string_attributes": {}, "localized_string_attributes": {}, "numeric_attributes": {}, "localized_numeric_attributes": {}, "datetime_attributes": {}, "localized_datetime_attributes": {} }, "languages": {}, "options": {}, "di": {} } ]

CONNECTOR

The connector properties are:

  1. type (prismic, graphcms, strapi, gcs, plentymarket, sftp)

  2. options (gets generated based on the selected connector type)

For example, bellow is a sample of the connector properties:

Prismic Connector Properties

"connector": { "type": "prismic", "options": { "query": [], "access_token": "", "url": "" } }

The query content must be a list of filters and it should be escaped (copy-paste each [] filter into an escape/unescape online tool)

Review this sample from Boxalino`s public Prismic repository:

"connector": { "type": "prismic", "options": { "url": "https://rtux-integration.cdn.prismic.io/api/v2/", "access_token": "MC5ZSkU4SnhBQUFDSUFXSXZR.77-9QO-_vWR577-977-9HO-_ve-_ve-_vXPvv73vv71LeO-_vUNi77-9Ii3vv73vv73vv71sC2Xvv71A77-9IA", "query": ["[at(document.type, \"blog\")]"] } }

GraphCMS Connector Properties

The fields and filters must be escaped (copy-paste the graphQL section into an escape/unescape online tool)


Review this sample from Boxalino`s public GraphCMS repository:

The filters are optional.
We recommend to push the available content types without filters and use the API request to set filters.

Strapi Connector Properties


Review this sample from Strapi`s live demo repository:

The query is optional. The query/filters must be escaped (copy-paste the query section into an escape/unescape online tool).

We recommend to sync each collection type (no query/filters) and use the API request to set filters.

 

MAPPING

The mapping elements depend on the document that requires to be synchronized in the Boxalino ecosystem. We expect our clients to work with headless CMS / 3rd party environments for their content sync.

The mapping exposes the Boxalino Data Structure for the requested document (ex: doc_content ).

For each property, as an integrator, you have to map the connector`s output path (wildcards [*] allowed):

 

a) for the properties returned as “string” or that have a single value: only one mapping is allowed (ex: description, link, images, title, short_description, etc)

(ex: "persona_id": "author.data.id", "persona_type": "author.data.attributes.email", "images": "image.data.attributes.url")

 

b) (if you intend to change the name of some attributes in the export) for the properties returned as "array" ([]) or as "objects" {}, set a list with index property name (output)=> mapping path (input) :

  1. the output is the explicit type (ex: for tags, the tagging source can be tags or default) or exposed property name

  2. the input is the mapping logic (ex: all product IDs from data.product_ids[*].id are set as product_ids in localized_numeric_attributes)

(ex: "numeric_attributes": { "place_id":"place.data.id"} , "localized_string_attributes" : {"seo_meta_title": "seo.metaTitle","img_thumbnail":"image.data.attributes.formats.thumbnail.url"})

 

c) (if you do not want to change the property name in *_attributes mapping) set a list (as array [] ) with mapping path (input)

(ex: "localized_string_attributes":{"menu_title":"data.menu_title"} can be declared as "localized_string_attributes":["data.menu_title"] )

 

d) there are a series of default mappings supported which do not have to be declared in your request

  1. do a test/sample request without any mapping declaration to identify the default mappings

    1. In the case of Prismic integrations, the following mappings are added by default:

b. In the case of GraphCMS integrations, the following mappings are added by default:

c. In the case of Strapi integrations, the following mappings are added by default:

 

 

e) export of properties as raw JSON - use the raw node in the mapping

  1. properties will be available as bxraw_<property name>

  2. this is useful for exporting headless CMS specific content types that would require specific rendering in your template (ex: slices, sections, etc)

  3. it does not allow wildcards/paths (export the API response node as is)

 

f) export linked-content : either using the contents property from the Data Structure or by exporting it as raw JSON

  1. the mapping for contents is done in the following way: "contents":{"<source for input>":{"data-structure field":"CMS field"}}"

(ex: "contents": { "data.author_link": { "content_type": "type", "content_id": "id" }, "data.categories[*].category": {"content_type": "type", "content_id": "id" }} will access the property type & id from the data.author_link & from the data.categories[*].category to create the Referenced Schema Types | CONTENT contents data structure)

 

g) in order to establish in which languages the content is available, a “language status” property is added dynamically ( value 0/1 based on language)

  1. it will take one of the following name _bx_locale_status

 

Check bellow a sample for various integrations following the designed blog logic in our public repository:

Prismic Mapping Sample (doc_content)

GraphCMS Mapping Sample (doc_content)

Strapi Mapping Sample (doc_content)

 


The connector response returns the requested content (results) in the following:

 

LANGUAGES

This must be declared in order to map the exported language (ex: “de”) with your connector defined value (ex:”en-de”).

 

AVAILABLE SERVICES

In order to allow our integrators to validate that the existing data available in the 3rd party systems is properly exported, a few helper services are available:

 

TEST CONNECTOR REQUEST /test/connector

Use a simple TEST request in order to validate that the connection to your configured endpoint (connector property) is valid.

 

In the sample above, the following BODY is used:

TEST SAMPLE REQUEST /test/sample

You can also use the SAMPLE trigger in order to access one element of the requested types.

The JSON response will be the JSONL structure as it is exported to Boxalino GCS.

 

In the sample above, the following BODY is used: