Load Request

At this point, the following steps should already be achieved by the Data Integration (DI) team:

Create the JSONL files https://boxalino.atlassian.net/wiki/spaces/BPKB/pages/252149803/Data+Integration#1.-Export-JSONL-files

The load request is done for every document JSONL (following the required Data Structure) required for your Data Integration process, once they are generated.

For a product data sync, make sure the following tables have been prepared and are provided: doc_product, doc_attributes, doc_attribute_values and doc_languages

1 /load
2 Request Definition
3 Integration Strategy
- 3.1 Load in Batches / data > 32 MB
  - 3.1.1 1. Make a request for public upload link
  - 3.1.2 2. Upload the content on the public link
  - 3.1.3 3. Load data to BigQuery
- 3.2 Fallback / retry policies
4 Using private GCP resources for DI
- 4.1 BigQuery Datasets
  - 4.1.1 1. Naming
  - 4.1.2 2. Location
  - 4.1.3 3. Permissions
- 4.2 Storage Bucket
  - 4.2.1 1. Naming
  - 4.2.2 2. Location
  - 4.2.3 3. Permissions

/load

The load request will do the following things:

It saves your JSONL (for every data type required for the data sync type) in a Google Cloud Storage (GCS) Bucket from the Boxalino project
1. the files are available for 30 days in the clients` GCS bucket
It loads every JSONL in a BigQuery table in Boxalino project
1. for Instant Updates : the tables have a retention policy of 1 day
2. for Delta Updates: the tables have a retention policy of 7 days
3. for Full Updates: the tables have a retention policy of 15 days

Boxalino will provide access to the data Storage Buckets and BigQuery datasets that store your data.

The service response is a JSON payload as documented here https://boxalino.atlassian.net/wiki/spaces/BPKB/pages/747208705

Request Definition

1	Endpoint	full data sync	https://boxalino-di-full-krceabfwya-ew.a.run.app
2		delta data sync	https://boxalino-di-delta-krceabfwya-ew.a.run.app
3		instant-update data sync	https://boxalino-di-instant-update-krceabfwya-ew.a.run.app
4		stage / testing	https://boxalino-di-stage-krceabfwya-ew.a.run.app
5	Action	/load
6	Method	POST
7	Body	the document JSONL
8	Headers	Authorization	Basic base64<<DATASYNC API key : DATASYNC API Secret>> Note: only the API credentials with role=ADMIN are valid. https://boxalino.atlassian.net/wiki/spaces/BPKB/pages/713785345/API+Credentials#Roles
9		Content-Type	application/json
10		doc	the document type (as described in https://boxalino.atlassian.net/wiki/spaces/BPKB/pages/252280881 )
11		client	your Boxalino account name
12		dev	only add it if the dev data index must be updated
13		mode	D for delta , I for instant update, F for full technical: the constants from the GcpClientInterface https://github.com/boxalino/data-integration-doc-php/blob/3.0.0/src/Service/GcpRequestInterface.php#L18
14		type	integration type (product, order, etc) https://github.com/boxalino/data-integration-doc-php/blob/3.0.0/src/Service/GcpRequestInterface.php#L22
15		tm	time, in format: YmdHis requirement: the same tm value must be used from the begging of the DI process until the end, for all files. technical: used to identify the version of the documents and create the content.
16		ts	(optional) timestamp, must be millisecond based in UNIX timestamp technical: calculated from the time the data has been exported to BQ; the update requests will be applied in version ts order; https://github.com/boxalino/data-integration-doc-php/blob/3.0.0/src/Service/Flow/DiRequestTrait.php#L140 if it is not set, our system sets it
17		chunk	(optional) for loading content by chunks see https://boxalino.atlassian.net/wiki/spaces/BPKB/pages/415432770/Load+Request#Load-By-Chunks
18		dataset	(optional) the dataset in which the doc_X tables must be stored; if not provided, the service will check the <index>_<mode> dataset in the Boxalino project, to which you will have access
19		project	(optional) the project name where the documents are to be stored;
20		bucket	(optional) the storage bucket where the doc_X will be loaded; if not provided, the service will use the Boxalino project.

A LOAD REQUEST code-sample is available in the data-integration-doc-php library: https://github.com/boxalino/data-integration-doc-php/blob/3.0.0/src/Service/Flow/LoadTrait.php

Optionally, if you are to use an advanced rest client , you can configure it as such:

Advanced REST Client SYNC request structure

Integration Strategy

doc_attribute, doc_attribute_value, doc_language can be synchronized with a call to the load endpoint
- because GCP has a size limit of 32MB for POST requests, do not use the /load service for big data exports
- if the data is too big, the service returns a 413 ENTITY TOO LARGE response. In this case, switch to the batch load
doc_product / doc_order / doc_user / doc_content / etc and other content bigger than 32 MB must be loaded in batches (max size of 32MB) or via GCS Public Signed URL /load/chunk endpoint
- NOTE: when using /load/chunk service to generate a Google Cloud Storage Public URL, there is no size limit! It is possible to load all data at once using one GCS Public URL link.

Load in Batches / data > 32 MB

For content over 32MB, we provide an endpoint to access a Signed GCS Url that would put all your streamed content into a file (currently there is no defined file size limit in GCS)

1. Make a request for public upload link

This is the generic POST request:

curl --connect-timeout 60 --max-time 300 "https://boxalino-di-<mode>-krceabfwya-ew.a.run.app/load/chunk" \
  -X POST \
  -H "Content-Type: application/json" \
  -H "client: <account>" \
  -H "dev: true|false" \
  -H "tm: YYYYmmddHHiiss" \
  -H "type: product|content|order|user|communication_history|communication_planning|user_generated_content" \
  -H "mode: <mode>" \
  -H "chunk: <id>" \
  -H "doc: doc_product|doc_order|doc_user|doc_content|.." \
  -H "Authorization: Basic <encode of the account>"

The response will be an upload link that can only be used in order to create the file in the clients` GCS bucket. The link is valid for 30 minutes.

2. Upload the content on the public link

A code sample is available in our generic PHP library

curl --connect-timeout 60 --timeout 0 <GCS-signed-url> \
    -X PUT \
    -H "Content-Type: application/octet-stream" \
    -d "<YOUR DOCUMENT JSONL CONTENT (STREAM)>"

3. Load data to BigQuery

Make an HTTP POST call to load/bq endpoint. A code sample is available in our generic PHP library

This is the generic POST request:

curl --connect-timeout 60 --max-time 300 "https://boxalino-di-<mode>-krceabfwya-ew.a.run.app/load/bq" \
  -X POST \
  -H "Content-Type: application/json" \
  -H "client: <account>" \
  -H "dev: true|false" \
  -H "tm: YYYYmmddHHiiss" \
  -H "type: product|content|order|user|communication_history|communication_planning|user_generated_content" \
  -H "mode: <mode>" \
  -H "doc: doc_product|doc_order|doc_user|doc_content|.." \
  -H "Authorization: Basic <encode of the account>"

Fallback / retry policies

We do our best to ensure that our services are continuously available to receive requests. The endpoints have a timeout of 3600s. This means that your POST request will receive a response when the service finished your request (either error or succes) or when the timeout is reached.

The /load , /load/chunk , /load/bq have a succesfull response on average in 10sec (so the wait time is minimal). Howether, we recommend for your fallback/retry policy, to use a timeout configuration of 3-5 min.

Based on your system, the timeout is represented by different parameters for the HTTP request.

--max-time is the timeout: wait 10s for a response from the server or stop connection
--connect-timeout is the connection timeout: wait 30s to make a connection to the service or stop connection

Instead of stop connection, we recommend to integrate a fallback policy with a retry.

Using private GCP resources for DI

For the case when the client wants to use their own GCP resources for the Data Integration with Boxalino, the following requirements must be complied with:

BigQuery Datasets

1. Naming

The BigQuery dataset in which your account documents are loaded can be named after the data index and process it is meant to synchronize to:

if the request is for the dev data index - <client>_dev_<mode>
if the request is for the production data index - <client>_<mode>

, where:

<client> is the account name provided by Boxalino.
<mode> is the process F (for full), D (for delta), I (for instant)

Example, for our an account boxalino_client, the following datasets must exist (upon your integration use-cases):

for dev index: boxalino_client_dev_I, boxalino_client_dev_F, boxalino_client_dev_D
for production data index: boxalino_client_I, boxalino_client_F, boxalino_client_D

The above datasets must exist in your project.

2. Location

Upon the creation of your dataset, please use the Data Location: EU

3. Permissions

In order to have read & write access to your private dataset, please provide the following permissions to the Data Integration Service Account (DISA) 55483703770-compute@developer.gserviceaccount.com:

BigQuery Data Viewer
BigQuery Metadata Viewer
BigQuery Data Editor / Owner
BigQuery Job User

or BigQuery Admin to the created datasets datasets.

Storage Bucket

1. Naming

In your custom project must be available the storage buckets in which #1 step can be done (loading your generated JSONL document).

Follow the Google documentation on how to create storage buckets.

The storage buckets have the requirement to be unique within the scope of the integration. Please use the following naming formula:

<your-custom-project>_<your-boxalino_account>_dev
<your-custom-project>_<your-boxalino-account>

2. Location

For content storage, please use either Multi-region (eu) or Region (europe-west1).

3. Permissions

In order to have load & read access to your private Google Cloud Storage buckets, please provide the following permissions to the Data Integration Service Account (DISA) 55483703770-compute@developer.gserviceaccount.com:

Storage Object Creator
Storage Object Admin

The above permissions can be replaced with the Storage Admin role.