In order to assist with the Data Integration (DI) efforts with 3rd party systems (ex: graphQL, Prismic, GraphCMS, BigQuery, etc), Boxalino is presenting the Data Transformer service https://boxalino-di-transformer-krceabfwya-ew.a.run.app/
Use this service to:
Identify the required request BODY for content integration https://boxalino-di-transformer-krceabfwya-ew.a.run.app/transformer/view
Identify the required request BODY for API request https://boxalino-di-transformer-krceabfwya-ew.a.run.app/api/view
Identify if your project`s connector (headless CMS/BQ/etc) is supported. If not, please provide a service request.
This is service is currently expanding. We value our integrator`s feedback.
For further service requests and integration reviews, please get in touch with Boxalino.
INTEGRATION BENEFITS
When exporting other content to Boxalino ecosystem your project is opening up to further features:
Dynamic content ( top brands/categories/blogs, etc) as part of your API integration/layout
Mixed API response for search/listing (ex: blogs, promotions, campaigns alongside products)
Dynamic pages for your project content (ex: blog pages driven by the personalization/AI engine)
More personalized content to keep your customer`s engaged (as part of our omnichannel approach)
Relevant statistics for each of your content reach (performance impact)
A/B testing as to identify what content your customers have a higher response towards
INTEGRATION FLOW
Review the Transformer services:
is your connector type is not yet supported, please get in touch with Boxalino.
Use the https://boxalino-di-transformer-krceabfwya-ew.a.run.app/transformer/view service to identify the body content requested for the API request.
Identify the content types (blog, banner, categories, etc) that must be integrated from your 3rd party source.
Prepare the connector options https://boxalino.atlassian.net/wiki/spaces/BPKB/pages/662536193/Transformer#CONNECTOR with content from your 3rd party system.
Do a test request https://boxalino.atlassian.net/wiki/spaces/BPKB/pages/662536193/Transformer#TEST-REQUEST
Prepare the mapping https://boxalino.atlassian.net/wiki/spaces/BPKB/pages/662536193/Transformer#MAPPING for each content type you want to export.
Do a sample request https://boxalino.atlassian.net/wiki/spaces/BPKB/pages/662536193/Transformer#REQUEST-DEFINITION
8. Repeat step 6-7 until all desired content is part of the response.
8. Do the sync request https://boxalino.atlassian.net/wiki/spaces/BPKB/pages/662536193/Transformer#REQUEST-DEFINITION
REQUEST DEFINITION
As an integrator, please create the bellow request to the provided endpoint.
There should be a process within your own project that triggers the data sync between a 3rd party source (connector) and Boxalino.
Endpoint | full data sync | ||
|---|---|---|---|
| 1 |
| delta data sync | TBC |
| 2 |
| instant data sync | TBC |
| 3 | Action | /sync | |
| 4 | Method | POST | |
| 5 | Body | use the public endpoint | |
| 6 | Headers | Authorization | Basic base64<<DATASYNC API key : DATASYNC API Secret>>  |
| 7 |
| Content-Type | application/json |
| 8 |
| client | account name |
| 9 |
| mode | data sync mode: F for full |
| 10 |
| type | product, user, content, user_content, order. if left empty - it will check for all tables with the given tm |
| 11 |
| tm | (optional) time , in format: YmdHis technical: used to identify the documents version |
| 12 |
| ts | (optional) timestamp, must be millisecond based in UNIX timestamp |
| 13 | dev | (optional) use this to mark the content for the dev data index | |
BODY ELEMENTS
The API SYNC request BODY structure can be accessed in the public service https://boxalino-di-transformer-krceabfwya-ew.a.run.app/transformer/view
There should be a transformer request element per data type (ex: blog, banner, etc). The body is a list (array) of such transformer request definitions (connector, document, mapping, languages)
CONNECTOR
The connector properties are:
type (select from the available options)
options (gets generated based on the selected connector type)
For example, bellow is a sample of the connector properties.
The options vary based on the connector type.
The connector required options should be properly described in your 3rd party system documentation (it`s a generic remote access)
MAPPING
The mapping elements depend on the document that requires to be synchronized in the Boxalino ecosystem. We expect our clients to work with headless CMS / 3rd party environments for their content sync.
The mapping exposes the Boxalino Data Structure for the requested document (ex: https://boxalino.atlassian.net/wiki/spaces/BPKB/pages/252280968/doc%2Bcontent ).
For each property, as an integrator, you have to map the connector`s output path (wildcards [*] allowed):
a) for the properties returned as “string”/single value: only one mapping is allowed (ex: id, type, creation, description, etc)
b) (if you intend to change the name of some attributes in the export) for the properties returned as "array" ([]) or as "objects" {}, set a list with index property name (output)=> mapping path (input) :
the
outputis the explicit type (ex: for tags, the tagging source can be tags or default) or exposed property namethe
inputis the mapping logic (ex: all product IDs from data.product_ids[*].id are set as product_ids in localized_numeric_attributes)
c) (if you do not want to change the property name in *_attributes mapping) set a list (as array [] ) with mapping path (input)
"localized_string_attributes":{"menu_title":"data.menu_title"}can be declared as"localized_string_attributes":["data.menu_title"]
d) there are a series of default mappings supported which do not have to be declared in your request
do a
test/samplerequest without anymappingdeclaration to identify the default mappingsIn the case of prismic integrations, the following mappings are added by default:
"id": "id", "type": "type", "creation": "first_publication_date", "last_update": "last_publication_date", "link": "uid", "string_attributes" : {"linked_ids":"alternate_languages[*].id"}, "localized_string_attributes" : ["slugs", "href"], "raw" : { "localized_string_attributes" : [ "alternate_languages", "linked_documents" ] }
e) export of properties as raw JSON - use the raw node in the mapping
properties will be available as
bxraw_<repository name>this is helpful for exporting slices and other properties that would require specific rendering in your template
f) export linked-content : either using the contents property from the Data Structure or by exporting it as raw JSON
the mapping for
contentsis done in the following way:"contents":{"<source for input>":{"data-structure field":"prismic field"}}"
g) in order to establish in which languages the content is available, a “language status” property is added dynamically ( value 0/1 based on language)
it will take one of the following name
_bx_locale_status
Check bellow a sample for various integrations following the designed blog logic in our public repository:
At Boxalino, we try to map the way our client expects for a headless integration to work.
For this reason, if your connector integration libraries allows for content to be accessed as HTML or text (ex: data.content) - this is as well possible.
In the above sample:
data.content is a RichText (editor) input
when using the mapping “description”:data.content - it will be exported as TEXT
when adding the mapping for localized_string_attributes - it will be exported as HTML.
when adding the mapping in
rawit will be exported as raw JSON
The connector response returns the requested content (results) in the following:
LANGUAGES
This must be declared in order to map the exported language (ex: “de”) with your connector export value (ex:”en-de”).
"languages": {
"de": "en-de",
"fr": "fr-ch"
}
AVAILABLE SERVICES
In order to allow our integrators to validate that the existing data available in the 3rd party systems is properly exported, a few helper services are available:
TEST CONNECTOR REQUEST /test/connector
Use a simple TEST request in order to validate that the connection to your configured endpoint (connector property) is valid.
curl https://boxalino-di-transformer-krceabfwya-ew.a.run.app/test/connector \
-X POST \
-d "[{\"connector\":{\"type\":\"prismic\",\"options\":{\"url\":\"https:\/\/rtux-integration.cdn.prismic.io\/api\/v2\/\",\"access_token\":\"MC5ZSkU4SnhBQUFDSUFXSXZR.77-9QO-_vWR577-977-9HO-_ve-_ve-_vXPvv73vv71LeO-_vUNi77-9Ii3vv73vv73vv71sC2Xvv71A77-9IA\",\"query\":[\"[at(document.type, \\\"category\\\")]\"]}},\"document\":{},\"mapping\":{},\"languages\":{}},{\"connector\":{\"type\":\"prismic\",\"options\":{\"url\":\"https:\/\/rtux-integration.cdn.prismic.io\/api\/v2\/\",\"access_token\":\"MC5ZSkU4SnhBQUFDSUFXSXZR.77-9QO-_vWR577-977-9HO-_ve-_ve-_vXPvv73vv71LeO-_vUNi77-9Ii3vv73vv73vv71sC2Xvv71A77-9IA\",\"query\":[\"[at(document.type, \\\"blog\\\")]\"]}},\"document\":{},\"mapping\":{},\"languages\":{}},{\r\n \"connector\": {\r\n \"type\": \"graphql\",\r\n \"options\": {\r\n \"endpoint\": \"https:\/\/api-eu-central-1.graphcms.com\/v2\/ckll4a83ni7og01yzfr3k9yc4\/master\",\r\n \"token\": \"eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCIsImtpZCI6ImdjbXMtbWFpbi1wcm9kdWN0aW9uIn0.eyJ2ZXJzaW9uIjozLCJpYXQiOjE2MjI1NjIzMzMsImF1ZCI6WyJodHRwczovL2FwaS1ldS1jZW50cmFsLTEuZ3JhcGhjbXMuY29tL3YyL2NrbGw0YTgzbmk3b2cwMXl6ZnIzazl5YzQvbWFzdGVyIiwiaHR0cHM6Ly9tYW5hZ2VtZW50LW5leHQuZ3JhcGhjbXMuY29tIl0sImlzcyI6Imh0dHBzOi8vbWFuYWdlbWVudC5ncmFwaGNtcy5jb20vIiwic3ViIjoiZjZlMTcyNGYtOTI2NS00OTFjLWEwMzktYzFkNGI2ZDExZDA5IiwianRpIjoiY2twZTdweDVoZGR2bzAxejU1NjNwZjc1cCJ9.MjYP8I7OvwyOkiHiB22LPwOfRzEt_ilvp6jHTVufkQEs6UTxJeuOEDvcojbMAgiEa3lStbc3mOVDezOQhMoLdD-UYoSzGyKmzIBI76vsVv5xbkhKwnLBnmX2ssRHMfOgIuyEpG16SwNiyKuBOZdhC39dapEWwSn2bzWqSi4Y-LvIdsANlHfgdwzVzkJNfNAO-2oOMQC4h-QfBlUVoSEQPYLPmGDOUpcawnHXc60Lyg9rCdxooG9YVzXUeqoi4SIc8feHH4My4s_Eb9mprP6sewezTIb4ldPCVwAlh2xl5BrWYDLLNbhgVMmUVaNZjXSS40egT4xxIWiNPw4axCAKZbgsk78lgHo51t0hz8j6qIfn9G_ZNzMagKQRupx4guuMQXiXg30WjSE09HBv6QjDLQRz9A89eRAKCiGVddKGAvXy_cZrAi2OZ_gtsX993wR-kt6AtNZ7s2U5IZPribvLbiNBMtwh1kRfz_t_bUFLCJs7J332odSDkxL38LqR7oRNStZv6EqAXmelWYqBsUvnwZWI3DYqjysYfjb4JALSTwOa4zfDHDkzlaKOiIZwZkI0GroSed_fmL6cfkpmN63CRY3qNQfwG7cXT0mOuOTJSzRBw1QigfCVnM_M2881eBaBdpdwXw3zF7FlxJL1Y0uMSQ0RZ2l3WC81xH7KqUUlUPw\",\r\n \"type\": \"blogs\",\r\n \"filters\": \"active:true\"\r\n }\r\n },\r\n \"document\" : {\r\n \"type\" : \"content\"\r\n },\r\n \"mapping\": {},\r\n \"languages\":{}\r\n}]" \
-H "Content-Type: application/json" \
-H "mode: F" \
-H "type: content"
In the sample above, the following BODY is used:
TEST SAMPLE REQUEST /test/sample
You can also use the SAMPLE trigger in order to access one element of the given types.
The JSON response will be the JSONL structure as it is exported to Boxalino GCS.
Use this service to validate & check that all your data is exported as desired.
curl https://boxalino-di-transformer-krceabfwya-ew.a.run.app/test/sample \
-X POST \
-d "[{\"connector\":{\"type\":\"prismic\",\"options\":{\"url\":\"https:\/\/rtux-integration.cdn.prismic.io\/api\/v2\/\",\"access_token\":\"MC5ZSkU4SnhBQUFDSUFXSXZR.77-9QO-_vWR577-977-9HO-_ve-_ve-_vXPvv73vv71LeO-_vUNi77-9Ii3vv73vv73vv71sC2Xvv71A77-9IA\",\"query\":[\"[at(document.type, \\\"category\\\")]\"]}},\"document\":{\"type\":\"content\"},\"mapping\":{\"id\":\"id\",\"type\":\"type\",\"creation\":\"first_publication_date\",\"last_update\":\"last_publication_date\",\"persona_type\":\"\",\"persona_id\":\"\",\"title\":\"data.name\",\"short_description\":\"\",\"description\":\"data.description\",\"images\":\"data.header_image.url\",\"link\":\"uid\",\"tags\":{\"default\":\"tags\"},\"parent_content_ids\":\"data.parent_category.id\",\"contents\":[],\"products\":[],\"customers\":[],\"stores\":\"\",\"status\":\"data.status\",\"string_attributes\":{\"default_tags\":\"tags\",\"version\":\"version\"},\"localized_string_attributes\":{\"category_title\":\"data.category_title\",\"slugs\":\"slugs\",\"linked_language_id\":\"alternate_languages[*].id\"},\"numeric_attributes\":{\"position\":\"data.position\",\"show_in_navigation\":\"data.show_in_navigation\"},\"localized_numeric_attributes\":{},\"datetime_attributes\":{},\"localized_datetime_attributes\":{}},\"languages\":{\"de\":\"en-de\",\"fr\":\"fr-ch\"}},{\"connector\":{\"type\":\"prismic\",\"options\":{\"url\":\"https:\/\/rtux-integration.cdn.prismic.io\/api\/v2\/\",\"access_token\":\"MC5ZSkU4SnhBQUFDSUFXSXZR.77-9QO-_vWR577-977-9HO-_ve-_ve-_vXPvv73vv71LeO-_vUNi77-9Ii3vv73vv73vv71sC2Xvv71A77-9IA\",\"query\":[\"[at(document.type, \\\"blog\\\")]\"]}},\"document\":{\"type\":\"content\"},\"mapping\":{\"id\":\"id\",\"type\":\"type\",\"creation\":\"first_publication_date\",\"last_update\":\"last_publication_date\",\"persona_type\":\"\",\"persona_id\":\"\",\"title\":\"data.title\",\"short_description\":\"\",\"description\":\"data.content\",\"images\":\"data.header_image.url\",\"link\":\"uid\",\"tags\":{\"default\":\"tags\",\"category\":\"data.category.tags\"},\"parent_content_ids\":\"data.category.id\",\"contents\":[],\"products\":[],\"customers\":[],\"stores\":\"\",\"status\":\"data.status\",\"string_attributes\":{\"tags\":\"tags\",\"category_uid\":\"data.category.uid\"},\"localized_string_attributes\":{\"slugs\":\"slugs\",\"alternate_languages\":\"alternate_languages[*].id\",\"meta_title\":\"data.meta_title\",\"content\":\"data.content\"},\"numeric_attributes\":{\"position\":\"data.position\"},\"localized_numeric_attributes\":{\"product_ids\":\"data.product_ids[*].id\"},\"datetime_attributes\":{},\"localized_datetime_attributes\":{}},\"languages\":{\"de\":\"en-de\",\"fr\":\"fr-ch\"}},{\"connector\":{\"type\":\"graphql\",\"options\":{\"endpoint\":\"https:\/\/api-eu-central-1.graphcms.com\/v2\/ckll4a83ni7og01yzfr3k9yc4\/master\",\"token\":\"eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCIsImtpZCI6ImdjbXMtbWFpbi1wcm9kdWN0aW9uIn0.eyJ2ZXJzaW9uIjozLCJpYXQiOjE2MjI1NjIzMzMsImF1ZCI6WyJodHRwczovL2FwaS1ldS1jZW50cmFsLTEuZ3JhcGhjbXMuY29tL3YyL2NrbGw0YTgzbmk3b2cwMXl6ZnIzazl5YzQvbWFzdGVyIiwiaHR0cHM6Ly9tYW5hZ2VtZW50LW5leHQuZ3JhcGhjbXMuY29tIl0sImlzcyI6Imh0dHBzOi8vbWFuYWdlbWVudC5ncmFwaGNtcy5jb20vIiwic3ViIjoiZjZlMTcyNGYtOTI2NS00OTFjLWEwMzktYzFkNGI2ZDExZDA5IiwianRpIjoiY2twZTdweDVoZGR2bzAxejU1NjNwZjc1cCJ9.MjYP8I7OvwyOkiHiB22LPwOfRzEt_ilvp6jHTVufkQEs6UTxJeuOEDvcojbMAgiEa3lStbc3mOVDezOQhMoLdD-UYoSzGyKmzIBI76vsVv5xbkhKwnLBnmX2ssRHMfOgIuyEpG16SwNiyKuBOZdhC39dapEWwSn2bzWqSi4Y-LvIdsANlHfgdwzVzkJNfNAO-2oOMQC4h-QfBlUVoSEQPYLPmGDOUpcawnHXc60Lyg9rCdxooG9YVzXUeqoi4SIc8feHH4My4s_Eb9mprP6sewezTIb4ldPCVwAlh2xl5BrWYDLLNbhgVMmUVaNZjXSS40egT4xxIWiNPw4axCAKZbgsk78lgHo51t0hz8j6qIfn9G_ZNzMagKQRupx4guuMQXiXg30WjSE09HBv6QjDLQRz9A89eRAKCiGVddKGAvXy_cZrAi2OZ_gtsX993wR-kt6AtNZ7s2U5IZPribvLbiNBMtwh1kRfz_t_bUFLCJs7J332odSDkxL38LqR7oRNStZv6EqAXmelWYqBsUvnwZWI3DYqjysYfjb4JALSTwOa4zfDHDkzlaKOiIZwZkI0GroSed_fmL6cfkpmN63CRY3qNQfwG7cXT0mOuOTJSzRBw1QigfCVnM_M2881eBaBdpdwXw3zF7FlxJL1Y0uMSQ0RZ2l3WC81xH7KqUUlUPw\",\"type\":\"blogs\",\"fields\":\"{ title __typename locale createdAt active id metaDescription metaTitle publishedAt seoLink updatedAt content { html text } headerImage { url id } stage background { css hex } brands { id name } createdBy { id name } history { createdAt } publishedBy { id name } updatedBy { id name } }\",\"filters\":\"active:true\",\"locales\":[\"de_CH\",\"fr_CH\"]}},\"document\":{\"type\":\"content\"},\"mapping\":{\"id\":\"id\",\"type\":\"__typename\",\"creation\":\"createdAt\",\"last_update\":\"updatedAt\",\"persona_type\":\"createdBy.name\",\"persona_id\":\"createdBy.id\",\"title\":\"title\",\"short_description\":\"\",\"description\":\"content.text\",\"images\":\"headerImage.url\",\"link\":\"seoLink\",\"tags\":{\"brands\":\"brands[*].name\"},\"parent_content_ids\":\"\",\"contents\":[],\"products\":[],\"customers\":[],\"stores\":\"\",\"status\":\"state\",\"string_attributes\":{\"background\":\"background.hex\",\"background_css\":\"background.css\",\"brand_ids\":\"brands[*].id\"},\"localized_string_attributes\":{\"meta_title\":\"metaTitle\",\"meta_description\":\"metaDescription\",\"content\":\"content.html\",\"slugs\":\"seoLink\"},\"numeric_attributes\":{},\"localized_numeric_attributes\":{\"display\":\"active\"},\"datetime_attributes\":{\"published_at\":\"publishedAt\"},\"localized_datetime_attributes\":{}},\"languages\":{\"de\":\"de_CH\",\"fr\":\"fr_CH\"}}]" \
-H "Content-Type: application/json" \
-H "mode: F" \
-H "type: content"
In the sample above, the following BODY is used:
Add Comment