Performance
These tests were ran by Perspectium bulk sharing 100,000 incidents from ServiceNow to a Snowflake database.
NOTE: Batch processing is intended only when a queue has one table type in it. Since the Agent is configured so each <task> configuration reads from one queue (whether the <task> has multiple parallel threads reading from that queue with the attribute instances or just one thread), if the queue has records from different tables in it, this can lead to data discrepancies and batch processing is not recommended to be used.
Insert Only
| Number of tasks | Batch insert size | Messages/second |
|---|---|---|
| 1 | --- | 1/s |
| 5 | --- | 3/s |
| 5 | 20 | 20/s |
| 10 | --- | 5/s |
| 10 | 20 | 40/s |
Upsert
| Number of tasks | Batch insert size | Messages/second |
|---|---|---|
| 1 | --- | 0.8/s |
| 5 | --- | 3/s |
| 5 | 20 | 10/s |
| 10 | --- | 5/s |
| 10 | 20 | 20/s |
