Skip to main content

Tables

Tables are the core component of the data layer in Shaped.

Each table is populated via a connector to a source, such as a MySQL database or Amplitude account.

Connectors​

Connectors establish a connection to where your data already lives. You can use a custom connector to stream data directly into Shaped using the Shaped API or Shaped CLI.

Built-in Connectors​

Shaped supports out-of-the-box connectors to a range of systems, including:

  • Data Warehouses: Such as BigQuery, Snowflake, and Redshift.
  • Databases: Such as PostgreSQL and MySQL.
  • Blob Storage: Such as S3 and Google Cloud Storage (GCS).
  • Analytics Applications: Such as Segment and Amplitude.

Custom connectors​

If your datasource is not supported, you can declare a custom table.

Use the "Create table" endpoint to declare a new table and its schema.

Then use a /table/insert endpoint to add records to the table.

Table schemas​

Every table has a schema that describes the structure of your data.

Built-in connectors infer the column structure from the datasource you connect.

Custom datasets require you to upload a schema. It should include:

  • columns: The names of the fields in your data.
  • unique_keys: A list of columns that uniquely identify each row in the dataset (e.g., ["id"]).

Append-only tables​

Some tables are created from event streams and are append-only. When you add rows to such tables, they are added to the bottom.

To ensure deduplication happens correctly, make sure that each record contains the primary_key and replication_key columns defined in your schema.