Skip to main content

The basics

Shaped is a real-time retrieval engine that lets you build state-of-the-art retrieval systems. You can:

  • Import from 20+ connectors
  • Transform data with SQL or AI into materialized views for training
  • Define retrieval engines, models, and ranking pipelines as YAML files
  • Test different models, ranking strategies, and configurations in parallel
  • Index on new data automatically and in real-time
  • Scale up or down automatically based on usage

Shaped has three layers: data, intelligence, and query.

Architecture overview​

Shaped's architecture follows a layered data flow where each layer builds upon the previous one:

  1. Data layer (Tables and Views) forms the foundation, connecting to your existing data sources via connectors. Tables store raw data with defined schemas, while Views transform and enrich this data through SQL queries or AI-powered enrichment. This layer serves as the data warehouse that feeds into the intelligence layer.

  2. Intelligence layer (Engines) consumes data from tables and views to build retrieval systems. Engines define how data is indexed (through embeddings for vector search or lexical search for BM25), which scoring models are trained (from simple popularity-based models to complex machine learning models), and how the system is deployed and scaled. This layer handles all the model training, embedding generation, and indexing logic.

  3. Query layer executes retrieval operations against your engines through a REST API. Each query follows a four-stage pipeline: retrieve candidates, filter based on business rules, score using trained models, and reorder for diversity. This layer provides the real-time interface for retrieving and ranking results.

Data flows through Shaped in a pipeline: your production data sources connect to Tables via connectors. Views transform this raw data into the shape needed for your use case. Engines then ingest data from tables and views, train models, and build indexes. Finally, Queries execute against engines to retrieve and rank results in real-time.

Data layer​

This is where your data lives.

Tables​

You can import data from an external source using one of our 20+ connectors. If you need a custom table, you can declare a custom table and add rows to it via API.

Once your data is imported, it exists in Shaped as tables. Depending on the connector you use, new data may be updated in real-time (streams) or every 15 minutes (batch).

Table views​

If your tables are not in the right structure to be trained on, you can use views to fix or enrich them into new views.

SQL views let you:

  • Convert data types, fix null values, or rename columns
  • Join multiple tables into one massive interactions table.
  • Combine multiple sources into one table - eg user events from your product with billing transactions)

AI views let you:

  • Add new semantic data from existing columns
  • Create new columns like clothing_color from an image of clothes

Each view is materialized, so you can use the outputs to train a model (or transform in a different way).

Intelligence layer​

The intelligence layer is where indexing, embedding generation, and model training happens. You will declare an engine which contains the scoring and ranking logic for your retrieval system.

Engines​

When you configure an engine, you can define multiple retrieval components that will be trained together. You will also describe what data the engine is connected to and deployment configuration (such as number of pods, data tier, etc).

Your engine does not have to be limited to a single embedding or scoring model. You can configure multiple models, plus embeddings for semantic search, plus a custom lexical/bm25 search engine, all in one engine. Each component has an API to fine-tune it accordingly.

You can also run multiple engines in parallel to test the best one.

Query layer​

The query layer is how you retrieve data using your engine. It is a REST API with a simple configuration to handle multiple different retrieval strategies with varying complexity.

At the simple end, you can do standard BM25 lexical search with a text query.

At the more complex end, you can run complex 4-stage ranking pipelines with retrieval, filtering, scoring, and ordering steps.

tip

If you want to learn more about the four-stage recommender model, check out our series: Anatomy of Modern Ranking Architectures