Skip to main content

Query basics

To retrieve data from a Shaped engine, you use a query.

Every query is a POST to the /query endpoint. The request body can be JSON or a ShapedQL query.

The Shaped query interface maps onto the four-stage recommender architecture:

  1. Retrieve: Fetches a set of candidates based on a filter expression, similarity search, or a ranked column (eg _derived_popular_rank).
  2. Filter: Removes candidates from the result set based on rules you set. For example, remove items that a user has viewed.
  3. Score: Assign a score to each candidate based on a value model expression. For example, you can use expressions for distance between embedding vectors (eg cosine similarity) or combine the output of two scoring models. Learn more about value modeling in the Value modeling section below.
  4. Reorder: Diversify the final result set so that all queries don't return the exact same results.

This framework is shared across text search and user recommendations: the retrieve stage can use text_search(...), similarity(...), ids(...), and other retrievers, and the score stage applies the same expression language in all cases.

Many standard text reranking patterns map directly to the score stage:

  • Text rerankers (e.g., colbert_v2(...), cross_encoder(...)).
  • Rank fusion (e.g., reciprocal rank fusion using retrieval.get_rank(...)).
  • Linear interpolation (e.g., combining retrieval scores with weights).

See the reference documentation in ShapedQL.

Make a query​

Every query should be a POST request to the /v2/engines/{engine_name}/query endpoint.

The following query retreives similar items using an ALS embedding model.

fetch("https://api.shaped.ai/v2/engines/movie_engine/query", {
  method: "POST",
  headers: {
    "Content-Type": "application/json",
    "x-api-key": "YOUR_API_KEY"
  },
  body: JSON.stringify({
    query: "SELECT * FROM similarity(embedding_ref='als', limit=50, encoder='item_attribute_pooling', input_item_id=$item_id) LIMIT 20",
    parameters: {
      item_id: "db1234"
    }
  })
})

Query examples​

The /query endpoint accepts ad-hoc queries that you define at runtime. You can use it to test different retrieval strategies without modifying your engine configuration.

The following section provides quick examples of common query strategies. For more comprehensive guides with prerequisites, use cases, and advanced patterns, see the dedicated use-case pages in the Querying Data section.

For lexical search, use the text_search function with mode='lexical':

SELECT *
FROM text_search(mode='lexical', input_text_query='Blue shirt', limit=50)
LIMIT 20

For more details on lexical search, including prerequisites and advanced patterns, see the Search guide.

Vector search retriever​

Vector search also uses text_search, but with mode='vector'. Use this for semantic search or RAG.

These queries require an engine with an embedding policy in the index block.

SELECT *
FROM text_search(mode='vector', text_embedding_ref='name_embedding', input_text_query='Blue shirt', limit=50)
LIMIT 20

For more details on semantic/vector search, including prerequisites and hybrid search patterns, see the Search guide.

Similarity - Retrieve similar items to a user's recent interactions​

This retriever is like vector search, but more wide ranging: it retrieves the items most similar to the user's recent interaction set.

SELECT *
FROM similarity(embedding_ref='item_content_embedding', encoder='interaction_round_robin', input_user_id=$user_id, limit=50)
LIMIT 20

For more details on finding similar items, including content-based and collaborative similarity, see the Similar Items guide.

Use column_order() with _derived_popular_rank to retrieve popular items, or _derived_chronological_rank for chronological ordering. These columns are automatically generated when you have an interaction_table. See Derived columns for details.

SELECT *
FROM column_order(columns='_derived_popular_rank ASC', limit=50)
LIMIT 20

For more details on retrieving popular and trending items, see the Feeds guide.

Similarly, this query returns popular items but in a specific category (by index_group_name).

SELECT *
FROM column_order(columns='_derived_popular_rank ASC', limit=50)
WHERE index_group_name = 'Baby/Children'
LIMIT 20

Retrieve and score - Get popular items, ranked based on my recent interactions​

The previous examples only included a retrieval step. This example has a retrieval and scoring step. It gets 1000 most popular items and then scores them based on the click_through_rate and conversion_rate score for a given user.

SELECT *
FROM column_order(columns='_derived_popular_rank ASC', limit=1000)
ORDER BY score(expression='0.3 * click_through_rate + 0.7 * conversion_rate', input_user_id='$user_id', input_interactions_item_ids='$interaction_item_ids')
LIMIT 20

Reordering with diversity - making recommendations feel more natural​

A diversity reorder step still returns items from the candidate set, but may return items that would otherwise be lower in the result set because of their score.

SELECT *
FROM column_order(columns='_derived_popular_rank ASC', limit=1000)
ORDER BY score(expression='0.3 * click_through_rate + 0.7 * conversion_rate', input_user_id='$user_id', input_interactions_item_ids='$interaction_item_ids')
REORDER BY diversity(diversity_lookback_window=50)
LIMIT 20

Reordering - adding exploration and breaking filter bubbles​

If you rely only on derived features like popularity, your result set may become repetitive or result in a filter bubble. To avoid this, you can add exploration to your query. It runs another retrieve step at the end of the rank pipeline to add variety from outside the candidate set.

Exploration is also the primary way to boost content. By including a retriever that contains items you want to boost (e.g., a specific collection or list of IDs) alongside your main retrievers, you can use REORDER BY exploration() to inject those items into the final results.

SELECT *
FROM column_order(columns='_derived_popular_rank ASC', limit=1000)
ORDER BY score(expression='0.3 * click_through_rate + 0.7 * conversion_rate', input_user_id='$user_id', input_interactions_item_ids='$interaction_item_ids')
REORDER BY exploration(diversity_lookback_window=50)
LIMIT 20

For more comprehensive examples and use cases, see the dedicated pages in the Querying Data section: Search, Feeds, Similar Items, Reranking, and more.

Use a saved query​

Saved queries are declared in your engine configuration, enabling multiple clients (e.g., mobile app and website) to execute the same queries for consistency across your application.

Defining saved queries​

Saved queries are defined in the queries section of your engine configuration using ShapedQL:

version: v2
name: my_engine

data:
  item_table:
    name: movies
    type: table
  # ...your other config here
queries:
  get_recommended_items:
    params:
      item_id:
        type: string
        required: true
    query: |
      SELECT *
      FROM similarity(embedding_ref='als', limit=50, encoder='item_attribute_pooling', input_item_id=$item_id)
      LIMIT 20

Executing saved queries​

Call the query endpoint with the query name and pass input parameters in the request body.

fetch("https://api.shaped.ai/v2/engines/movie_engine/queries/get_recommended_items", {
  method: "POST",
  headers: {
    "Content-Type": "application/json",
    "x-api-key": "YOUR_API_KEY"
  },
  body: JSON.stringify({
    parameters: {
      item_id: "db1234"
    }
  })
})

Value modeling​

Value models are Python-like mathematical expressions that compute a score for each user-item pair. They enable you to combine outputs from multiple trained models, incorporate user and item attributes into scoring, use text encodings for semantic similarity, access retrieval scores from named retrievers, and apply custom business logic.

How it works:

Value models are evaluated pair-wise (one score per user-item combination), enabling personalized scoring. Text encodings are pre-computed and cached; retrieval scores are only materialized when referenced.

Use value models when:

  • Combining multiple trained models (ensembles)
  • Incorporating user/item attributes into scoring
  • Blending retrieval scores with model predictions
  • Applying custom business logic (e.g., boost by price, location, etc.)

Use default scoring when:

  • Retriever scores are sufficient
  • You don't need personalization
  • Performance is critical and you want minimal overhead

The score stage can use either:

  • PassthroughScore (default): Uses the retriever's default score without additional computation
  • ScoreEnsemble: Uses a value model expression to compute scores (e.g., 0.6 * click_through_rate + 0.4 * conversion_rate)

For detailed syntax and available functions, see the Value Model Expressions section in the ShapedQL reference. You'll also find value modeling examples throughout the use-case pages below.