In order to assist with the Data Integration (DI) efforts with 3rd party systems (ex: graphQL, Prismic, GraphCMS, etc), Boxalino is presenting the Data Transformer service.
REQUEST DEFINITION
Endpoint | full data sync | ||
|---|---|---|---|
| 1 |
| delta data sync | TBC |
| 2 |
| instant data sync | TBC |
| 3 | Action | /sync | |
| 4 | Method | POST | |
| 5 | Body | use the public endpoint | |
| 6 | Headers | Authorization | Basic base64<<DATASYNC API key : DATASYNC API Secret>> |
| 7 |
| Content-Type | application/json |
| 8 |
| client | account name |
| 9 |
| dev | (optional) use this to mark the content for the dev data index |
| 10 |
| mode | data sync mode: F for full |
| 11 |
| type | product, user, content, user_content, order. if left empty - it will check for all tables with the given tm |
| 12 |
| tm | (optional) time , in format: YmdHis technical: used to identify the documents version |
| 13 |
| ts | (optional) timestamp, must be millisecond based in UNIX timestamp |
BODY ELEMENTS
The API SYNC request BODY structure can be accessed in the public service https://boxalino-di-transformer-krceabfwya-ew.a.run.app/transformer/view
CONNECTOR
The connector properties are:
type (select from the available options)
options (gets generated based on the selected connector type)
For example, bellow is a sample of the connector properties.
"connector": {
"type": "prismic",
"options": {
"query": [],
"method": "GET",
"access_token": "",
"url": ""
}
}
MAPPING
The mapping elements depend on the document that requires to be synchronized in the Boxalino ecosystem. We expect our clients to work with headless CMS / 3rd party environments for their content sync.
The mapping exposes the Boxalino Data Structure for the requested document (ex: https://boxalino.atlassian.net/wiki/spaces/BPKB/pages/252280968/doc%2Bcontent ).
For each property, as an integrator, you have to map the connector`s output path (wildcards [*] allowed):
a) for the properties returned as “string”/single value: only one mapping is allowed (ex: id, type, creation, description, etc)
b) for the properties returned as “array” ([]) or “objects”({}), set a list with index property name (output)=> mapping path (input) :
the
outputis the explicit type (ex: for tags, the tagging source can be tags or default) or exposed property namethe
inputis the mapping logic (ex: all product IDs from data.product_ids[*].id are set as product_ids in localized_numeric_attributes)
This is a sample for a prismic integration:
"mapping": {
"id": "id",
"type": "type",
"creation": "first_publication_date",
"last_update": "last_publication_date",
"persona_type": "",
"persona_id": "",
"title": "data.title",
"short_description": "",
"description": "data.content",
"images": "data.header_image.url",
"link": "uid",
"tags": {
"default":"tags",
"category":"data.category.tags"
},
"parent_content_ids": "data.category.id",
"contents": [],
"products": [],
"customers": [],
"stores": "",
"status": "data.status",
"string_attributes": {
"tags": "tags",
"category_uid": "data.category.uid"
},
"localized_string_attributes": {
"slugs": "slugs",
"alternate_language_id": "alternate_languages[*].id",
"meta_title":"data.meta_title",
"content":"data.content"
},
"numeric_attributes": {
"position": "data.position"
},
"localized_numeric_attributes": {
"product_ids":"data.product_ids[*].id"
},
"datetime_attributes": {},
"localized_datetime_attributes": {}
}
At Boxalino, we try to map the way our client expects for a headless integration to work.
For this reason, if your connector integration libraries allows for content to be accessed as HTML or text (ex: data.content) - this is as well possible.
In the above sample:
data.content is a RichText (editor) input
when using the mapping “description”:data.content - it will be exported as TEXT
when adding the mapping for localized_string_attributes - it will be exported as HTML.
The connector response returns the requested content (results) in the following:
LANGUAGES
This must be declared in order to map the exported language (ex: “de”) with your connector export value (ex:”en-de”).
"languages": {
"de": "en-de",
"fr": "en-fr"
}
AVAILABLE SERVICES
In order to allow our integrators to validate that the existing data available in the 3rd party systems is properly exported, a few helper services are available:
TEST REQUEST
Use a simple TEST request in order to validate that the connection to your configured endpoint (connector property) is valid.
curl https://boxalino-di-transformer-krceabfwya-ew.a.run.app/test \
-X POST \
-d "{\r\n \"connector\": {\r\n \"type\": \"prismic\",\r\n \"options\": {\r\n \"url\": \"https:\/\/rtux-integration.cdn.prismic.io\/api\/v2\/\",\r\n \"access_token\": \"MC5ZSkU4SnhBQUFDSUFXSXZR.77-9QO-_vWR577-977-9HO-_ve-_ve-_vXPvv73vv71LeO-_vUNi77-9Ii3vv73vv73vv71sC2Xvv71A77-9IA\",\r\n \"query\": [\"[at(document.type, \\\"blog\\\")]\", \"[at(document.tags, [\\\"haare\\\"])]\"]\r\n }\r\n },\r\n \"document\" : {\r\n \"type\" : \"content\"\r\n },\r\n \"mapping\": {},\r\n \"languages\":{}}" \
-H "Content-Type: application/json" \
-H "mode: F" \
-H "type: content"
In the sample above, the following BODY has been sent:
{
"connector": {
"type": "prismic",
"options": {
"url": "https://rtux-integration.cdn.prismic.io/api/v2/",
"access_token": "MC5ZSkU4SnhBQUFDSUFXSXZR.77-9QO-_vWR577-977-9HO-_ve-_ve-_vXPvv73vv71LeO-_vUNi77-9Ii3vv73vv73vv71sC2Xvv71A77-9IA",
"query": ["[at(document.type, \"blog\")]", "[at(document.tags, [\"haare\"])]"]
}
},
"document" : {
"type" : "content"
},
"mapping": {},
"languages": {}
}
SAMPLE REQUEST
You can also use the SAMPLE trigger in order to access one element of the given type. The JSON response will be the JSONL structure as it is exported to Boxalino GCS. Use this service to validate & check that all your data is exported as desired.
curl https://boxalino-di-transformer-krceabfwya-ew.a.run.app/sample \
-X POST \
-d "{\r\n \"connector\": {\r\n \"type\": \"prismic\",\r\n \"options\": {\r\n \"url\": \"https:\/\/rtux-integration.cdn.prismic.io\/api\/v2\/\",\r\n \"access_token\": \"MC5ZSkU4SnhBQUFDSUFXSXZR.77-9QO-_vWR577-977-9HO-_ve-_ve-_vXPvv73vv71LeO-_vUNi77-9Ii3vv73vv73vv71sC2Xvv71A77-9IA\",\r\n \"query\": [\"[at(document.type, \\\"blog\\\")]\", \"[at(document.tags, [\\\"haare\\\"])]\"]\r\n }\r\n },\r\n \"document\" : {\r\n \"type\" : \"content\"\r\n },\r\n \"mapping\": {\r\n \"id\": \"id\",\r\n \"type\": \"type\",\r\n \"creation\": \"first_publication_date\",\r\n \"last_update\": \"last_publication_date\",\r\n \"persona_type\": \"\",\r\n \"persona_id\": \"\",\r\n \"title\": \"data.title\",\r\n \"short_description\": \"\",\r\n \"description\": \"data.content\",\r\n \"images\": \"data.header_image.url\",\r\n \"link\": \"uid\",\r\n \"tags\": {\r\n \"default\":\"tags\",\r\n \"category\":\"data.category.tags\"\r\n },\r\n \"parent_content_ids\": \"data.category.id\",\r\n \"contents\": [],\r\n \"products\": [],\r\n \"customers\": [],\r\n \"stores\": \"\",\r\n \"status\": \"data.status\",\r\n \"string_attributes\": {\r\n \"tags\": \"tags\",\r\n \"category_uid\": \"data.category.uid\"\r\n },\r\n \"localized_string_attributes\": {\r\n \"slugs\": \"slugs\",\r\n \"alternate_languages\": \"alternate_languages[*].id\",\r\n \"meta_title\":\"data.meta_title\",\r\n \"content\":\"data.content\"\r\n },\r\n \"numeric_attributes\": {\r\n \"position\": \"data.position\"\r\n },\r\n \"localized_numeric_attributes\": {\r\n \"product_ids\":\"data.product_ids[*].id\"\r\n },\r\n \"datetime_attributes\": {},\r\n \"localized_datetime_attributes\": {}\r\n },\r\n \"languages\": {\r\n \"de\": \"en-de\",\r\n \"fr\": \"en-fr\"\r\n }\r\n}" \
-H "Content-Type: application/json" \
-H "mode: F" \
-H "type: content"
In the sample above, the following BODY has been sent:
{
"connector": {
"type": "prismic",
"options": {
"url": "https://rtux-integration.cdn.prismic.io/api/v2/",
"access_token": "MC5ZSkU4SnhBQUFDSUFXSXZR.77-9QO-_vWR577-977-9HO-_ve-_ve-_vXPvv73vv71LeO-_vUNi77-9Ii3vv73vv73vv71sC2Xvv71A77-9IA",
"query": ["[at(document.type, \"blog\")]", "[at(document.tags, [\"haare\"])]"]
}
},
"document" : {
"type" : "content"
},
"mapping": {
"id": "id",
"type": "type",
"creation": "first_publication_date",
"last_update": "last_publication_date",
"persona_type": "",
"persona_id": "",
"title": "data.title",
"short_description": "",
"description": "data.content",
"images": "data.header_image.url",
"link": "uid",
"tags": {
"default":"tags",
"category":"data.category.tags"
},
"parent_content_ids": "data.category.id",
"contents": [],
"products": [],
"customers": [],
"stores": "",
"status": "data.status",
"string_attributes": {
"tags": "tags",
"category_uid": "data.category.uid"
},
"localized_string_attributes": {
"slugs": "slugs",
"alternate_language_id": "alternate_languages[*].id",
"meta_title":"data.meta_title",
"content":"data.content"
},
"numeric_attributes": {
"position": "data.position"
},
"localized_numeric_attributes": {
"product_ids":"data.product_ids[*].id"
},
"datetime_attributes": {},
"localized_datetime_attributes": {}
},
"languages": {
"de": "en-de",
"fr": "en-fr"
}
}
Add Comment