Data Integration

Owned by Sylvain Paillard
Last updated: Jan 22, 2021
2 min read

Overview

To integrate your Data with Boxalino, you can either follow these 4 steps (where you load your data in your own BigQuery Dataset) or only step 1 and 4 (where you send the JSONL files directly to the Boxalino Process):

1. Export JSONL files

Export your data in JSONL for each selected Data Type of the Data Structure.

JSONL = Newline delimited JSON ( https://en.wikipedia.org/wiki/JSON_streaming )

Make sure to name each file with the name starting with the Data Type (e.g.: “doc_product”), with the mode in the middle ('F': full, ‘D': delta) and finishing with the Datetime (YYYYMMDDHHMMSS) of when the export has started (it’s important to have it when it started, as if you have then new delta starting later but finishing before, they will need to be replayed based on their Datetime).
So as a result, file names should be like: “doc_product_ F_YYYYMMDDHHMMSS.json”

2. Load files to Google Storage

This step is optional, you can also decide to push the files directly to the Boxalino Proces

Load your files into a Google Cloud Storage bucket you need to set-up in your GCP environment.

More information about how set-up your GCP environment and how to load files into Google Cloud Storage can be found here: https://boxalino.atlassian.net/wiki/spaces/BPKB/pages/34439169

You can put an automated expiration for the files so they don’t accumulate over time

3. Load files to BigQuery tables

Create new tables with the Field Schema provided at the end of each Data Structure (e.g.: here for the doc_product table) for each of the files with exactly the same name in a BigQuery DataSet you need to set-up in your GCP environment.

More information about how set-up your GCP environment and how to load files from Google Cloud Storage to BigQuery can be found here: https://boxalino.atlassian.net/wiki/spaces/BPKB/pages/34439169

4. Trigger Boxalino Process

If you only did the first step and want to send your files directly, then call first the https://boxalino.atlassian.net/wiki/spaces/BPKB/pages/415432770 for each of your files.

Whether you called the Load Process first or not before, you need then to call the (once for all the files).