In order to assist with the Data Integration (DI) efforts with 3rd party systems (ex: Prismic, GraphCMS, BigQuery, PlentyMarkets, GCSStrapi, etc), Boxalino is presenting the Data Transformer service https://boxalino-di-transformer-krceabfwya-ew.a.run.app/ . This is an ETL-like service.
Use this service to:
Identify the required request BODY for content integration https://boxalino-di-transformer-krceabfwya-ew.a.run.app/transformer/view
Identify the required request BODY for API request https://boxalino-di-transformer-krceabfwya-ew.a.run.app/api/view
Identify if your project`s connector (headless CMS/BQ/etc) is supported. If not, please provide a service request.
...
Endpoint | full data sync | ||
---|---|---|---|
1 | test / stage | https://boxalino-di-transformer-stage-krceabfwya-ew.a.run.app | |
2 | Action | /sync | |
23 | Method | POST | |
34 | Body | use the public endpoint | |
45 | Headers | Authorization | Basic base64<<DATASYNC API key : DATASYNC API Secret>> |
56 |
| Content-Type | application/json |
67 |
| client | account name |
78 |
| mode | data sync mode: F for full, D for delta |
89 |
| type | product, user, content, user_content, order. if left empty - it will check for all tables with the given tm |
910 |
| tm | (optional) time , in format: YmdHis technical: used to identify the documents version |
1011 |
| ts | (optional) timestamp, must be millisecond based in UNIX timestamp |
1112 | dev | (optional) use this to mark the content for the dev data index |
...
Strapi Connector Properties
Expand | ||||||||
---|---|---|---|---|---|---|---|---|
| ||||||||
The query is optional. The query/filters must be escaped (copy-paste the query section into an escape/unescape online tool). Additional, to the url are added the following default parameters: We recommend to sync each collection type (no query/filters) and use the API request to set filters.
|
...
Strapi Mapping Sample (doc_content)
Expand | ||
---|---|---|
| ||
|
...
At Boxalino, we try to map the way our client expects for a headless integration to work.
...
data.content is a RichText (editor) input
when using the mapping “description”:data.content - it will be exported as TEXT
when adding the mapping for localized_string_attributes - it will be exported as HTML.
when adding the mapping in
raw
it will be exported as raw JSON
...
Expand | ||
---|---|---|
| ||
Code Block | ||
Expand | ||
| ||
|
Info |
---|
At Boxalino, we try to map the way our client expects for a headless integration to work. For this reason, if your connector integration libraries allows for content to be accessed as HTML or text (ex: data.content) - this is as well possible.
|
The connector response returns the requested content (results) in the following:
Expand | ||
---|---|---|
| ||
|
...
Code Block |
---|
curl https://boxalino-di-transformer-stage-krceabfwya-ew.a.run.app/test/connector \
-X POST \
-d "[{\"connector\":{\"type\":\"prismic\",\"options\":{\"url\":\"https:\/\/rtux-integration.cdn.prismic.io\/api\/v2\/\",\"access_token\":\"MC5ZSkU4SnhBQUFDSUFXSXZR.77-9QO-_vWR577-977-9HO-_ve-_ve-_vXPvv73vv71LeO-_vUNi77-9Ii3vv73vv73vv71sC2Xvv71A77-9IA\",\"query\":[\"[at(document.type, \\\"category\\\")]\"]}},\"document\":{},\"mapping\":{},\"languages\":{}},{\"connector\":{\"type\":\"prismic\",\"options\":{\"url\":\"https:\/\/rtux-integration.cdn.prismic.io\/api\/v2\/\",\"access_token\":\"MC5ZSkU4SnhBQUFDSUFXSXZR.77-9QO-_vWR577-977-9HO-_ve-_ve-_vXPvv73vv71LeO-_vUNi77-9Ii3vv73vv73vv71sC2Xvv71A77-9IA\",\"query\":[\"[at(document.type, \\\"blog\\\")]\"]}},\"document\":{},\"mapping\":{},\"languages\":{}},{\r\n \"connector\": {\r\n \"type\": \"graphql\",\r\n \"options\": {\r\n \"endpoint\": \"https:\/\/api-eu-central-1.graphcms.com\/v2\/ckll4a83ni7og01yzfr3k9yc4\/master\",\r\n \"token\": \"eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCIsImtpZCI6ImdjbXMtbWFpbi1wcm9kdWN0aW9uIn0.eyJ2ZXJzaW9uIjozLCJpYXQiOjE2MjI1NjIzMzMsImF1ZCI6WyJodHRwczovL2FwaS1ldS1jZW50cmFsLTEuZ3JhcGhjbXMuY29tL3YyL2NrbGw0YTgzbmk3b2cwMXl6ZnIzazl5YzQvbWFzdGVyIiwiaHR0cHM6Ly9tYW5hZ2VtZW50LW5leHQuZ3JhcGhjbXMuY29tIl0sImlzcyI6Imh0dHBzOi8vbWFuYWdlbWVudC5ncmFwaGNtcy5jb20vIiwic3ViIjoiZjZlMTcyNGYtOTI2NS00OTFjLWEwMzktYzFkNGI2ZDExZDA5IiwianRpIjoiY2twZTdweDVoZGR2bzAxejU1NjNwZjc1cCJ9.MjYP8I7OvwyOkiHiB22LPwOfRzEt_ilvp6jHTVufkQEs6UTxJeuOEDvcojbMAgiEa3lStbc3mOVDezOQhMoLdD-UYoSzGyKmzIBI76vsVv5xbkhKwnLBnmX2ssRHMfOgIuyEpG16SwNiyKuBOZdhC39dapEWwSn2bzWqSi4Y-LvIdsANlHfgdwzVzkJNfNAO-2oOMQC4h-QfBlUVoSEQPYLPmGDOUpcawnHXc60Lyg9rCdxooG9YVzXUeqoi4SIc8feHH4My4s_Eb9mprP6sewezTIb4ldPCVwAlh2xl5BrWYDLLNbhgVMmUVaNZjXSS40egT4xxIWiNPw4axCAKZbgsk78lgHo51t0hz8j6qIfn9G_ZNzMagKQRupx4guuMQXiXg30WjSE09HBv6QjDLQRz9A89eRAKCiGVddKGAvXy_cZrAi2OZ_gtsX993wR-kt6AtNZ7s2U5IZPribvLbiNBMtwh1kRfz_t_bUFLCJs7J332odSDkxL38LqR7oRNStZv6EqAXmelWYqBsUvnwZWI3DYqjysYfjb4JALSTwOa4zfDHDkzlaKOiIZwZkI0GroSed_fmL6cfkpmN63CRY3qNQfwG7cXT0mOuOTJSzRBw1QigfCVnM_M2881eBaBdpdwXw3zF7FlxJL1Y0uMSQ0RZ2l3WC81xH7KqUUlUPw\",\r\n \"type\": \"blogs\",\r\n \"filters\": \"active:true\"\r\n }\r\n },\r\n \"document\" : {\r\n \"type\" : \"content\"\r\n },\r\n \"mapping\": {},\r\n \"languages\":{}\r\n}]" \
-H "Content-Type: application/json" \
-H "mode: F" \
-H "type: content" |
...
Expand | |||||
---|---|---|---|---|---|
| |||||
|
TEST SAMPLE REQUEST /test/sample
You can also use the SAMPLE trigger in order to access one element of the requested types.
The JSON response will be the JSONL structure as it is exported to Boxalino GCS.
Tip |
---|
Use this service to validate & check that all your data is exported as desired. |
Code Block | |
---|---|
curl https://boxalino-di-transformer-krceabfwya-ew.a.run.app/test/sample \
-X POST \
-d "[{\"connector\":{\"type\":\"prismic\",\"options\":{\"url\":\"https:\/\/rtux-integration.cdn.prismic.io\/api\/v2\/\",\"access_token\":\"MC5ZSkU4SnhBQUFDSUFXSXZR.77-9QO-_vWR577-977-9HO-_ve-_ve-_vXPvv73vv71LeO-_vUNi77-9Ii3vv73vv73vv71sC2Xvv71A77-9IA\",\"query\":[\"[at(document.type, \\\"category\\\")]\"]}},\"document\":{\"type\":\"content\"},\"mapping\":{\"id\":\"id\",\"type\":\"type\",\"creation\":\"first_publication_date\",\"last_update\":\"last_publication_date\",\"persona_type\":\"\",\"persona_id\":\"\",\"title\":\"data.name\",\"short_description\":\"\",\"description\":\"data.description\",\"images\":\"data.header_image.url\",\"link\":\"uid\",\"tags\":{\"default\":\"tags\"},\"parent_content_ids\":\"data.parent_category.id\",\"contents\":[],\"products\":[],\"customers\":[],\"stores\":\"\",\"status\":\"data.status\",\"string_attributes\":{\"default_tags\":\"tags\",\"version\":\"version\"},\"localized_string_attributes\":{\"category_title\":\"data.category_title\",\"slugs\":\"slugs\",\"linked_language_id\":\"alternate_languages[*].id\"},\"numeric_attributes\":{\"position\":\"data.position\",\"show_in_navigation\":\"data.show_in_navigation\"},\"localized_numeric_attributes\":{},\"datetime_attributes\":{},\"localized_datetime_attributes\":{}},\"languages\":{\"de\":\"en-de\",\"fr\":\"fr-ch\"}},{\"connector\":{\"type\":\"prismic\",\"options\":{\"url\":\"https:\/\/rtux-integration.cdn.prismic.io\/api\/v2\/\",\"access_token\":\"MC5ZSkU4SnhBQUFDSUFXSXZR.77-9QO-_vWR577-977-9HO-_ve-_ve-_vXPvv73vv71LeO-_vUNi77-9Ii3vv73vv73vv71sC2Xvv71A77-9IA\",\"query\":[\"[at(document.type, \\\"blog\\\")]\"]}},\"document\":{\"type\":\"content\"},\"mapping\":{\"id\":\"id\",\"type\":\"type\",\"creation\":\"first_publication_date\",\"last_update\":\"last_publication_date\",\"persona_type\":\"\",\"persona_id\":\"\",\"title\":\"data.title\",\"short_description\":\"\",\"description\":\"data.content\",\"images\":\"data.header_image.url\",\"link\":\"uid\",\"tags\":{\"default\":\"tags\",\"category\":\"data.category.tags\"},\"parent_content_ids\":\"data.category.id\",\"contents\":[],\"products\":[],\"customers\":[],\"stores\":\"\",\"status\":\"data.status\",\"string_attributes\":{\"tags\":\"tags\",\"category_uid\":\"data.category.uid\"},\"localized_string_attributes\":{\"slugs\":\"slugs\",\"alternate_languages\":\"alternate_languages[*].id\",\"meta_title\":\"data.meta_title\",\"content
|
Warning |
---|
The sample CURL request does not include the STRAPI connector details because the STRAPI connector is accessing a demo setup. |
TEST SAMPLE REQUEST /test/sample
You can also use the SAMPLE trigger in order to access one element of the requested types.
The JSON response will be the JSONL structure as it is exported to Boxalino GCS.
Tip |
---|
Use this service to validate & check that all your data is exported as desired. |
Code Block |
---|
curl https://boxalino-di-transformer-stage-krceabfwya-ew.a.run.app/test/sample \ -X POST \ -d "[{\"connector\":{\"type\":\"prismic\",\"options\":{\"url\":\"https:\/\/rtux-integration.cdn.prismic.io\/api\/v2\/\",\"access_token\":\"MC5ZSkU4SnhBQUFDSUFXSXZR.77-9QO-_vWR577-977-9HO-_ve-_ve-_vXPvv73vv71LeO-_vUNi77-9Ii3vv73vv73vv71sC2Xvv71A77-9IA\",\"query\":[\"[at(document.type, \\\"category\\\")]\"]}},\"document\":{\"type\":\"content\"},\"mapping\":{\"id\":\"id\",\"type\":\"type\",\"creation\":\"first_publication_date\",\"last_update\":\"last_publication_date\",\"persona_type\":\"\",\"persona_id\":\"\",\"title\":\"data.name\",\"short_description\":\"\",\"description\":\"data.description\",\"images\":\"data.header_image.url\",\"link\":\"uid\",\"tags\":{\"default\":\"tags\"},\"parent_content_ids\":\"data.parent_category.id\",\"contents\":[],\"products\":[],\"customers\":[],\"stores\":\"\",\"status\":\"data.status\",\"string_attributes\":{\"default_tags\":\"tags\",\"version\":\"version\"},\"localized_string_attributes\":{\"category_title\":\"data.category_title\",\"slugs\":\"slugs\",\"linked_language_id\":\"alternate_languages[*].id\"},\"numeric_attributes\":{\"position\":\"data.position\",\"show_in_navigation\":\"data.show_in_navigation\"},\"localized_numeric_attributes\":{},\"datetime_attributes\":{},\"localized_datetime_attributes\":{}},\"languages\":{\"de\":\"en-de\",\"fr\":\"fr-ch\"}},{\"connector\":{\"type\":\"prismic\",\"options\":{\"url\":\"https:\/\/rtux-integration.cdn.prismic.io\/api\/v2\/\",\"access_token\":\"MC5ZSkU4SnhBQUFDSUFXSXZR.77-9QO-_vWR577-977-9HO-_ve-_ve-_vXPvv73vv71LeO-_vUNi77-9Ii3vv73vv73vv71sC2Xvv71A77-9IA\",\"query\":[\"[at(document.type, \\\"blog\\\")]\"]}},\"document\":{\"type\":\"content\"},\"mapping\":{\"id\":\"id\",\"type\":\"type\",\"creation\":\"first_publication_date\",\"last_update\":\"last_publication_date\",\"persona_type\":\"\",\"persona_id\":\"\",\"title\":\"data.title\",\"short_description\":\"\",\"description\":\"data.content\"},\"numeric_attributesimages\":{\"positiondata.header_image.url\",\"link\":\"data.positionuid\"},\"localized_numeric_attributestags\":{\"product\"default\":\"tags\",\"category\":\"data.category.tags\"},\"parent_content_ids\":\"data.product_ids[*]category.id\"},\"datetime_attributescontents\":{}[],\"localized_datetime_attributesproducts\":{}}[],\"languagescustomers\":{[],\"destores\":\"en-de\",\"frstatus\":\"fr-chdata.status\"}},{\"connectorstring_attributes\":{\"typetags\":\"graphqltags\",\"optionscategory_uid\":{\"data.category.uid\"endpoint},\":localized_string_attributes\"https:{\/\/api-eu-central-1.graphcms.com\/v2\/ckll4a83ni7og01yzfr3k9yc4\/master\"slugs\":\"slugs\",\"tokenalternate_languages\":\"eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCIsImtpZCI6ImdjbXMtbWFpbi1wcm9kdWN0aW9uIn0.eyJ2ZXJzaW9uIjozLCJpYXQiOjE2MjI1NjIzMzMsImF1ZCI6WyJodHRwczovL2FwaS1ldS1jZW50cmFsLTEuZ3JhcGhjbXMuY29tL3YyL2NrbGw0YTgzbmk3b2cwMXl6ZnIzazl5YzQvbWFzdGVyIiwiaHR0cHM6Ly9tYW5hZ2VtZW50LW5leHQuZ3JhcGhjbXMuY29tIl0sImlzcyI6Imh0dHBzOi8vbWFuYWdlbWVudC5ncmFwaGNtcy5jb20vIiwic3ViIjoiZjZlMTcyNGYtOTI2NS00OTFjLWEwMzktYzFkNGI2ZDExZDA5IiwianRpIjoiY2twZTdweDVoZGR2bzAxejU1NjNwZjc1cCJ9.MjYP8I7OvwyOkiHiB22LPwOfRzEt_ilvp6jHTVufkQEs6UTxJeuOEDvcojbMAgiEa3lStbc3mOVDezOQhMoLdD-UYoSzGyKmzIBI76vsVv5xbkhKwnLBnmX2ssRHMfOgIuyEpG16SwNiyKuBOZdhC39dapEWwSn2bzWqSi4Y-LvIdsANlHfgdwzVzkJNfNAO-2oOMQC4h-QfBlUVoSEQPYLPmGDOUpcawnHXc60Lyg9rCdxooG9YVzXUeqoi4SIc8feHH4My4s_Eb9mprP6sewezTIb4ldPCVwAlh2xl5BrWYDLLNbhgVMmUVaNZjXSS40egT4xxIWiNPw4axCAKZbgsk78lgHo51t0hz8j6qIfn9G_ZNzMagKQRupx4guuMQXiXg30WjSE09HBv6QjDLQRz9A89eRAKCiGVddKGAvXy_cZrAi2OZ_gtsX993wR-kt6AtNZ7s2U5IZPribvLbiNBMtwh1kRfz_t_bUFLCJs7J332odSDkxL38LqR7oRNStZv6EqAXmelWYqBsUvnwZWI3DYqjysYfjb4JALSTwOa4zfDHDkzlaKOiIZwZkI0GroSed_fmL6cfkpmN63CRY3qNQfwG7cXT0mOuOTJSzRBw1QigfCVnM_M2881eBaBdpdwXw3zF7FlxJL1Y0uMSQ0RZ2l3WC81xH7KqUUlUPwalternate_languages[*].id\",\"typemeta_title\":\"blogsdata.meta_title\",\"fieldscontent\":\"{ title __typename locale createdAt active id metaDescription metaTitle publishedAt seoLink updatedAt content { html text } headerImage { url id } stage background { css hex } brands { id name } createdBy { id name } history { createdAt } publishedBy { id name } updatedBy { id name } }\",\"filters\":\"active:true\",\"locales\":[\"de_CH\",\"fr_CH\"]}},\"document\":{\"type\":\"content\"},\"mappingdata.content\"},\"numeric_attributes\":{\"position\":\"data.position\"},\"localized_numeric_attributes\":{\"product_ids\":\"data.product_ids[*].id\"},\"datetime_attributes\":{},\"localized_datetime_attributes\":{}},\"languages\":{\"idde\":\"iden-de\",\"typefr\":\"__typenamefr-ch\"}},{\"creationconnector\":{\"createdAt\",\"last_updatetype\":\"updatedAtgraphql\",\"persona_typeoptions\":{\"createdBy.name\",endpoint\"persona_id\":\"createdBy.id\",\"title\"https:\"title\",\"short_description\":\"/\/api-eu-central-1.graphcms.com\/v2\/ckll4a83ni7og01yzfr3k9yc4\/master\",\"descriptiontoken\":\"content.texteyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCIsImtpZCI6ImdjbXMtbWFpbi1wcm9kdWN0aW9uIn0.eyJ2ZXJzaW9uIjozLCJpYXQiOjE2MjI1NjIzMzMsImF1ZCI6WyJodHRwczovL2FwaS1ldS1jZW50cmFsLTEuZ3JhcGhjbXMuY29tL3YyL2NrbGw0YTgzbmk3b2cwMXl6ZnIzazl5YzQvbWFzdGVyIiwiaHR0cHM6Ly9tYW5hZ2VtZW50LW5leHQuZ3JhcGhjbXMuY29tIl0sImlzcyI6Imh0dHBzOi8vbWFuYWdlbWVudC5ncmFwaGNtcy5jb20vIiwic3ViIjoiZjZlMTcyNGYtOTI2NS00OTFjLWEwMzktYzFkNGI2ZDExZDA5IiwianRpIjoiY2twZTdweDVoZGR2bzAxejU1NjNwZjc1cCJ9.MjYP8I7OvwyOkiHiB22LPwOfRzEt_ilvp6jHTVufkQEs6UTxJeuOEDvcojbMAgiEa3lStbc3mOVDezOQhMoLdD-UYoSzGyKmzIBI76vsVv5xbkhKwnLBnmX2ssRHMfOgIuyEpG16SwNiyKuBOZdhC39dapEWwSn2bzWqSi4Y-LvIdsANlHfgdwzVzkJNfNAO-2oOMQC4h-QfBlUVoSEQPYLPmGDOUpcawnHXc60Lyg9rCdxooG9YVzXUeqoi4SIc8feHH4My4s_Eb9mprP6sewezTIb4ldPCVwAlh2xl5BrWYDLLNbhgVMmUVaNZjXSS40egT4xxIWiNPw4axCAKZbgsk78lgHo51t0hz8j6qIfn9G_ZNzMagKQRupx4guuMQXiXg30WjSE09HBv6QjDLQRz9A89eRAKCiGVddKGAvXy_cZrAi2OZ_gtsX993wR-kt6AtNZ7s2U5IZPribvLbiNBMtwh1kRfz_t_bUFLCJs7J332odSDkxL38LqR7oRNStZv6EqAXmelWYqBsUvnwZWI3DYqjysYfjb4JALSTwOa4zfDHDkzlaKOiIZwZkI0GroSed_fmL6cfkpmN63CRY3qNQfwG7cXT0mOuOTJSzRBw1QigfCVnM_M2881eBaBdpdwXw3zF7FlxJL1Y0uMSQ0RZ2l3WC81xH7KqUUlUPw\",\"imagestype\":\"headerImage.urlblogs\",\"linkfields\":\"seoLink\",\"tags\":{\"brands\":\"brands[*].name\"},\"parent_content_ids{ title __typename locale createdAt active id metaDescription metaTitle publishedAt seoLink updatedAt content { html text } headerImage { url id } stage background { css hex } brands { id name } createdBy { id name } history { createdAt } publishedBy { id name } updatedBy { id name } }\",\"filters\":\"active:true\",\"contentslocales\":[],\"productsde_CH\":[],\"customersfr_CH\":[]}},\"storesdocument\":{\"\",\"statustype\":\"statecontent\"},\"string_attributesmapping\":{\"backgroundid\":\"background.hexid\",\"background_csstype\":\"background.css__typename\",\"creation\":\"createdAt\",\"last_update\":\"updatedAt\",\"persona_type\":\"createdBy.name\",\"persona_id\":\"createdBy.id\",\"title\":\"title\",\"brand_idsshort_description\":\"\",\"description\":\"content.text\",\"images\":\"headerImage.url\",\"link\":\"seoLink\",\"tags\":{\"brands\":\"brands[*].id\"},\"localized_string_attributes\":{\"meta_title\":\"metaTitle\",\"meta_description\":\"metaDescription\",\"content\":\"content.html\",\"slugs\":\"seoLink\"},\"numeric_attributes\":{},\"localized_numeric_attributes\":{\"display\":\"active\"},\"datetime_attributes\":{\"published_at\":\"publishedAt\"},\"localized_datetime_attributes\":{}},\"languages\":{\"de\":\"de_CH\",\"fr\":\"fr_CH\"}}]" \ -H "Content-Type: application/json" \ -H "mode: F" \ -H "type: content" |
In the sample above, the following BODY is used:
Expand | ||
---|---|---|
| ||
Code Block | | |
|
In the sample above, the following BODY is used:
Expand | |||||
---|---|---|---|---|---|
| |||||
|
Warning |
---|
The sample CURL request does not include the STRAPI connector details because the STRAPI connector is accessing a demo setup. As an integrator, you can create your own demo setup (https://strapi.io/demo ) in order to check their available data. |