Failures, retries, and batching

The batch is not atomic

Each row has its own database transaction. The server stops at the first failing row, rolls that row back, and does not attempt any later rows. Rows before it remain committed.

row 0  committed
row 1  committed
row 2  failed and rolled back
row 3  not attempted
row 4  not attempted

The failure preserves the original gRPC status code and adds context in this form:

Bulk insert failed at row 2 after inserting 2 rows: <reason>

Row indexes are zero-based. In this example, the source row at index 2 is the third row.

Do not retry the whole request blindly

Bulk import does not use the idempotency metadata supported by single-row posting. Retrying the original request can insert successful prefix rows again.

Recovery after a failed request

When a request fails:

1. Record the exact source chunk, zero-based failed index, and inserted-prefix count.
2. Inspect and correct the failed source row.
3. Reconcile the committed prefix against the target table.
4. Resume only from the first row that was not committed.

If the connection is lost before a response arrives, the client may not know how much of the request committed. Reconcile using a stable source identifier or another business key designed into the table. The bulk endpoint itself does not provide a batch ID, import ledger, status lookup, rollback operation, or duplicate detection.

Do not infer success from an increased row count when concurrent writers exist. For critical imports, preserve a source manifest that can be matched deterministically to destination rows.

Successful response

When every row succeeds, the response contains one result per input row in the same order. Each result includes the inserted row ID and committed row revision.

Preserve this mapping:

source row → request index → inserted ID → row revision

It is needed when later tables link to imported rows and is useful for audit and reconciliation. When the RPC fails, there is no normal response containing the successful prefix’s individual IDs; only the failure message reports how many rows committed.

Common failure causes

A row may fail because of:

• an unknown profile, table, or column alias;
• a missing required value or an invalid validation rule value;
• an incorrectly typed integer, decimal, money, boolean, or temporal value;
• a link to a row that does not exist;
• a missing or mismatched Steel-computed target;
• an attempt to write a quantity-ledger or accounting-transfer source column;
• a missing or incompatible account on an accounting-enabled table;
• accounting activity in a closed journal or protected period;
• a database constraint or unavailable physical table;
• an internal script, database, or indexing-outbox failure.

The server automatically retries a row transaction for recognized serialization and deadlock failures, up to ten attempts with short backoff. Other errors are returned immediately and stop the request.

Choosing a chunk size

The server accepts up to 10,000 rows, but 10,000 is a ceiling rather than a recommended default. Smaller chunks:

• reduce how much must be reconciled after an uncertain response;
• return validation failures sooner;
• stay farther below the 128 MiB decoding limit;
• reduce the duration of one RPC and the amount of result data held in memory.

Larger chunks reduce request overhead but increase the committed prefix that may need reconciliation. Choose a size using representative rows and script/accounting cost, then measure it in the real deployment.

The server processes rows sequentially, so increasing the chunk size does not make their database work parallel. Run concurrent bulk requests only after considering table dependencies, aggregate-based Steel scripts, account and quantity-ledger locks, database capacity, and partial-failure recovery.

Validate before the production import

The server has no dry-run bulk operation. Client-side checks can catch file and mapping mistakes, but only a real server insert exercises current table validation, authoritative Steel execution, link constraints, accounting rules, and database state.

Before a large or irreversible import:

• verify the target profile, table, and current aliases;
• validate and normalize the entire source file client-side;
• calculate all Steel targets using current dependencies;
• ensure parent rows and accounts already exist;
• test representative chunks in an appropriate non-production environment;
• take and verify a database backup when the recovery policy requires it;
• define how committed rows will be identified and removed or corrected if necessary;
• retain the original source, transformed values, request boundaries, and response mapping.

Bulk import only creates rows. It does not provide a matching bulk update, bulk delete, import rollback, or compensating transaction.

Server logs

The server logs the start and completion of a bulk request and progress after each 1,000 successful rows. It records elapsed time, throughput, the slowest row, and accumulated time spent in major phases such as lookups, validations, scripts, inserts, and index notification.

Rows taking at least 250 ms produce a slow-row warning. A failed request logs its zero-based row index, committed count, elapsed time, phase timings, and the server error. These logs are operational diagnostics; they are not a persistent import history available through the API.

Return to Bulk import for value preparation and per-row behavior.