Custom
If you don't see your data source listed in our supported integrations, you can still stream data into Shaped using our custom data source support. This feature allows you to write data directly into a Shaped dataset using the Shaped API or the Shaped CLI.
Setting up a Shaped Dataset to receive events
Before you can stream data into Shaped, you need to have provisioned a Shaped Dataset to receive the data. If you don't already have a Shaped Dataset, you can create one by calling the Create Dataset API endpoint, or with the Shaped CLI.
For example, let's create a dataset to stream custom events:
name: my_custom_dataset
schema_type: CUSTOM
unique_keys: [id]
column_schema:
event_name: String
event_time: DateTime
user_id: String
event_weight: Float
event_value: Int32
event_tags: Array(String)
json_payload: String
unique_keys deduplicates rows in the dataset based on the provided key. It is common to use an id column which results in a unique row for each id value representing its latest version.
CLI
shaped create-dataset --file custom_dataset.yaml
Supported Data Types
| Data Type | Description |
|---|---|
| Bytes | A sequence of bytes. |
| String | A sequence of characters. |
| Array(String) | An array of strings. |
| Array(Int32) | An array of 32-bit integers. |
| Array(Int64) | An array of 64-bit integers. |
| Int32 | A 32-bit integer. |
| Int64 | A 64-bit integer. |
| Float | A single-precision floating point number. |
| Double | A double-precision floating point number. |
| Array(Float64) | An array of double-precision floating point numbers. |
| Bool | A boolean value (true or false). |
| DateTime | A date and time value. |
If you want to send JSON, send it as a String type. You'll need to stringify the JSON first. Then you can use the ->> operator in the model queries to manipulate the JSON String.
Sending data to a Shaped Dataset
CLI
For experimentation, you can use the Shaped CLI to insert data into a dataset. Supported formats for the CLI include JSONL, CSV, TSV and Parquet.
shaped dataset-insert --dataset-name my_custom_dataset --file my_data.jsonl
API
See the API documentation for more information on how to insert data into a dataset.
curl -X GET "https://api.shaped.ai/v1/datasets/my_custom_dataset/dataset_insert" \
-H "x-api-key: <API_KEY>" \
-d @my_data.jsonl