Skip to main content

Engines

An engine controls the indexing logic of your retrieval systems.

Creating an engine​

You can create an engine in two ways:

  1. Uploading a YAML file via CLI or the Shaped console (recommended)
  2. Making a POST request to the /engines endpoint with your engine config

We recommend starting with a simple engine - with only data and index config - and then build up complexity from there.

Configuration​

Shaped provides an interface to declare your retrieval engines. There are four components to each engine:

  • data - Defines what data the engine should ingest and how it is indexed
  • index - Defines how to create indexes on your data - which columns to use and what embeddings
  • training - Defines additional model training and evaluation
  • deployment - Configures additional parameters for how the engine is deployed and served
  • queries - Defines a set of 'saved queries' that can be used at runtime

In the following sections, we will go through each configuration section and some examples.

Data​

The data config defines what tables and connectors the engine will use.

Engines must have a source item_table that holds candidate items to be looked up (search and retrieval).

You can also add a user_table with user attributes and an interaction_table with clicks or transactions. Your engine can use these profiles and interaction history to do personalized retrieval.

Connect tables​

You can connect your engine to tables in your Shaped account by its name. The following example connects to three tables with no additional options.

data:
  item_table:
    name: apparel_catalog
  user_table:
    name: customers
  interaction_table: 
    name: transactions

Change ID columns to item_id and user_id​

You can use the query option to run an SQL query when fetching items. You can use this to change column names, add a rating column and more. For more complex SQL operations such as joins, use an SQL transform.

data:
  item_table:
    name: apparel_catalog
    query: |
      select product_id as item_id, color, name, department, tags, updated_at, created_at 
      from apparel_catalog
  user_table:
    name: customers
    query: |
      select customer_id as user_id, name, gender, most_recent_interaction, joined_at
      from customers
  interaction_table: 
    name: transactions
    query: |
      select article_id as item_id, customer_id as user_id, transaction_type, price as label
      from transactions 
      where created_at < NOW() - INTERVAL '6 months'
note

Shaped looks up items and users by the item_id and user_id columns, respectively. Your engine will weigh interactions using the label column. Use the query option or an SQL transform to include these columns.

Specify how column features should be handled​

The schema_override option allows you to specify how specific features in your datasets should be handled. For example, you can specify that movie genres should be interpreted as a sequence of categories rather than a string literal.

data:
  item_table:
    name: movies
  schema_override: 
    item:
      id: item_id
      features: # list of columns to use as features
        - name: genre
          type: Sequence[TextCategory]
        - name: poster_url
          type: Image
        - name: movie_title
          type: Text
        - name: movie_age
          type: Numerical
        - name: primary_genre
          type: TextCategory
      created_at: created_at # which column to use as the created_at feature/deduplication

index​

The index config defines how the engine will use search and embeddings. You can use a pre-trained embedding model from huggingface, or train your own embedding model.

Use a pre-trained model​

The hugging_face encoder type supports any pre-trained encoder from Huggingface's Sentence Transformers or CLIP model library. The following example uses a Qwen3 embedding model.

index:
  embeddings:
    - name: text_embedding
      encoder:
        type: hugging_face
        model_name: Qwen/Qwen3-Embedding-8B
        item_fields:
          - title
          - url_captions

Create embeddings for text and image features​

embeddings is a list and can take multiple options. Each embedding can be specified in the same way as shown in the example above.

index:
  embeddings:
    - name: name_embedding
      encoder:
        type: hugging_face
        model_name: sentence-transformers/modernbert
        item_fields:
          - title
          - url_captions
    - name: image_embedding
      encoder:
        type: hugging_face
        model_name: openai/clip-vit-base-patch32
        item_fields:
          - image_url

The lexical_search field configures the fields to use for BM25 search.

index:
  lexical_search:
    item_fields:
      - description
      - movie_title
      - interests
      - directors
      - writers

training​

This block defines how to train scoring models on your data.

Engines can have a single model to base scoring on, or run multiple model policies at the same time. You can also combine scores using a score_ensemble policy.

Each scoring policy trains on your data on a given schedule.

training: 
  models: 
    - name: popularity_model
      policy_type: popular
      time_window_in_days: 90

Train a model to score item similarity​

training: 
  models: 
    - name: my_elsa_score_model
      policy_type: elsa
      strategy: early_stopping

Train multiple models and combine their scores​

The score-ensemble policy trains two models (ELSA and LightGBM) and then adds their scores together.

training: 
  models: 
    - name: add_scores_together
      policy_type: score-ensemble
      policies: 
        - name: elsa_score
          policy_type: elsa
          strategy: early_stopping
        - name: lgbm_score
          policy_type: lightgbm
      value_model: elsa_score + lgbm_score

Deployment​

The deployment config allows you to configure how the engine is deployed and served. This includes settings for compute resources, scaling, and runtime parameters.

Queries​

The queries key contains a list of saved queries. See the Query Reference to learn how to make queries.