Loaded Data Sources API

Loaded data sources are tables that accept direct row insertions via HTTP API. They provide a simple way to programmatically populate data tables for analysis and reconciliation workflows.

Creating a Loaded Data Source

Endpoint: POST https://api4.bitwave.io/v3/orgs/{orgId}/data-sources

Request Body

1 {
2 "data": {
3 "name": "Customer Transactions",
4 "description": "Daily transaction data from payment processor",
5 "type": "loaded",
6 "schema": [
7 {
8 "name": "transaction_id",
9 "type": "stringType"
10 },
11 {
12 "name": "customer_id",
13 "type": "stringType"
14 },
15 {
16 "name": "amount",
17 "type": "numberType"
18 },
19 {
20 "name": "currency",
21 "type": "stringType"
22 },
23 {
24 "name": "processed_at",
25 "type": "timestamp"
26 },
27 {
28 "name": "is_refunded",
29 "type": "boolean"
30 },
31 {
32 "name": "fee_amount",
33 "type": "floatType"
34 }
35 ]
36 }
37 }

Response

1 {
2 "id": "jgH5MtiE9KmHY8yc4AUV",
3 "name": "Customer Transactions",
4 "type": "loaded",
5 "feature": "data-fusion",
6 "region": "us-central1",
7 "projectId": "bitwave-prod",
8 "dataSetId": "614577b68201ba7649d0",
9 "table": "83e3932b-2688-4571-acef-29f6c3f52eb4",
10 "description": "Daily transaction data from payment processor"
11 }

Adding Rows to a Loaded Data Source

Endpoint: POST https://api4.bitwave.io/v3/orgs/{orgId}/data-sources/{dataSourceId}/add-rows

Request Body

1 {
2 "rows": [
3 {
4 "transaction_id": "txn_001",
5 "customer_id": "cust_12345",
6 "amount": 99.99,
7 "currency": "USD",
8 "processed_at": "2024-01-15T10:30:00Z",
9 "is_refunded": false,
10 "fee_amount": 2.99
11 },
12 {
13 "transaction_id": "txn_002",
14 "customer_id": "cust_67890",
15 "amount": 249.50,
16 "currency": "USD",
17 "processed_at": "2024-01-15T11:45:00Z",
18 "is_refunded": true,
19 "fee_amount": 7.49
20 },
21 {
22 "transaction_id": "txn_003",
23 "customer_id": "cust_11111",
24 "amount": 1500.00,
25 "currency": "EUR",
26 "processed_at": "2024-01-15T14:20:00Z",
27 "is_refunded": false,
28 "fee_amount": 45.00
29 }
30 ]
31 }

Response

1 {
2 "success": true,
3 "message": "Successfully added 3 rows to datasource",
4 "rowCount": 3
5 }

Supported Field Types

Error Responses

Schema Validation Errors

1 {
2 "error": "invalid field type 'string' for field 'amount'. Field type must be one of: stringType, numberType,
3 bigNumber, floatType, boolean, enum, date, datetime, timestamp, ticker, array, object, range",
4 "context": {
5 "field": "schema",
6 "traceId": "abc123"
7 }
8 }

Row Validation Errors

1 {
2 "error": "Row 2 has column 'status' which is not present in the first row",
3 "context": {
4 "field": "rows",
5 "traceId": "def456"
6 }
7 }

Best Practices

Schema Design: Define all required fields upfront. Schema changes require creating a new data source.
Data Types: Use appropriate types for your data:
1. numberType for integers (IDs, counts)
2. floatType for decimals (prices, percentages)
3. timestamp for precise time data
4. stringType for text and identifiers
Batch Inserts: Send multiple rows in a single request for better performance. Recommended batch size is 250 rows for typical row lengths.
Consistent Structure: All rows must have identical column names and structure.
Timestamp Format: Use ISO 8601 format (YYYY-MM-DDTHH:MM:SSZ) for timestamp fields.

Integration Example

1 # Create data sources
2 curl -X POST "https://api4.bitwave.io/v3/orgs/614577b68201ba7649d0/data-sources"
3 -H "Content-Type: application/json"
4 -H "Authorization: Bearer $API_TOKEN"
5 -d '{
6 "data": {
7 "name": "Transaction Data",
8 "description": "Daily transaction records",
9 "type": "loaded",
10 "schema": [
11 {"name": "transaction_id", "type": "stringType"},
12 {"name": "amount", "type": "floatType"},
13 {"name": "timestamp", "type": "timestamp"}
14 ]
15 }
16 }'
17
18 # Add rows using returned data source ID
19 curl -X POST "https://api4.bitwave.io/v3/orgs/614577b68201ba7649d0/data-sources/jgH5MtiE9KmHY8yc4AUV/add-rows"
20 -H "Content-Type: application/json"
21 -H "Authorization: Bearer $API_TOKEN"
22 -d '{
23 "rows": [
24 {
25 "transaction_id": "txn_001",
26 "amount": 150.75,
27 "timestamp": "2024-01-15T10:30:00Z"
28 },
29 {
30 "transaction_id": "txn_002",
31 "amount": 89.50,
32 "timestamp": "2024-01-15T11:15:00Z"
33 }
34 ]
35 }'