Index update processes

Owned by Sylvain Paillard
Last updated: Feb 05, 2021
2 min read

In this document we describe the update processes of the Boxalino real-time indexes.

This page document the indexing process of Boxalino. It is typically not needed as an information for our clients and partner as it is an internal process.

Product Full

  1. Zip account main configuration files and push them to SolrSync (type=config)

  2. Compute the fields (from doc_product and doc_attributes) and push them to SolrSync (type=fields)

  3. Compute the products data and push them to SolrSync (type=datafull&gzip=true)

  4. Zip account corpus configuration files and push them to SolrSync (type=corpus_config)

  5. Compute the corpus data and push them to SolrSync (type=corpus)

Product Delta

  1. Compute the fields (from doc_product and doc_attributes) (but no push)

  2. Compute the products data and push them to SolrSync (type=datadelta&gzip=true)

Product Instant

  1. Compute the products instant data and push them to SolrSync (type=datadelta&gzip=true)
    (different SQL process than delta and full)

Autocomplete (initial setup)

  1. Zip account autocomplete configuration files and push them to SolrSync (type=ac_config)

  2. Compute the autocomplete light data and push them to SolrSync (type=ac)
    (different process than in the daily from Airflow, will only do something if the autocomplete index was empty before, to update / add data, push data in account_lab.autocomplete directly or empty the table and rerun the process)

Daily (Airflow)

  1. Zip account generic configuration files (type=generic_config&name=properties) and push them to SolrSync

  2. Compute the fields (type=generic_fields&name=properties) and push them to SolrSync

  3. Compute the generic properties data and push them to SolrSync (type=generic_datafull&name=properties&gzip=true)

  4. Zip account autocomplete configuration files (type=ac_config) and push them to SolrSync

  5. Compute the autocomplete data and push them to SolrSync (type=ac)

Data Push (general_lab_load)

  1. Compute the campaigns and topics data and push them to SolrSync (type=datadelta&gzip=true)