Custom
If you don't see your data source listed in our supported integrations, you can still stream data into Shaped using our custom data source support. This feature allows you to write data directly into a Shaped dataset using the Shaped API or the Shaped CLI.
Setting up a Shaped Dataset to receive events
Before you can stream data into Shaped, you need to have provisioned a Shaped Dataset to receive the data. If you don't already have a Shaped Dataset, you can create one by calling the Create Dataset API endpoint, or with the Shaped CLI.
For example, let's create a dataset to stream custom events:
name: my_custom_dataset
schema_type: CUSTOM
column_schema:
- event_name: String
- event_time: DateTime
- user_id: String
- event_weight: Float
- event_value: Int32
- event_tags: ArrayOfString
CLI
shaped create-dataset --file custom_dataset.yaml
Supported Data Types
| Data Type | Description |
|---|---|
| Bytes | A sequence of bytes. |
| String | A sequence of characters. |
| ArrayOfString | An array of strings. |
| ArrayOfInteger | An array of 64-bit integers. |
| Int32 | A 32-bit integer. |
| Int64 | A 64-bit integer. |
| Float | A single-precision floating point number. |
| Double | A double-precision floating point number. |
| ArrayOfDouble | An array of double-precision floating point numbers. |
| Bool | A boolean value (true or false). |
| DateTime | A date and time value. |
| Json | A JSON object. |
Sending data to a Shaped Dataset
CLI
For experimentation, you can use the Shaped CLI to insert data into a dataset. Supported formats for the CLI include JSONL, CSV, TSV and Parquet.
shaped dataset-insert --dataset-name my_custom_dataset --file my_data.jsonl
API
See the API documentation for more information on how to insert data into a dataset.
curl -X GET "https://api.shaped.ai/v1/datasets/my_custom_dataset/dataset_insert" \
-H "x-api-key: <API_KEY>" \
-d @my_data.jsonl