IO Round Trip
The following IO round trip times are sampled by using a program with no compute on the Vollo accelerator from the Vollo runtime.
More specifically this Vollo accelerator program waits for the last input byte to arrive before it sends the first output byte back. This method takes into account some of the overheads (such as copying to the DMA buffer in the Vollo runtime) associated with IO and this test is set up to see how it scales with difference sizes of inputs and output values.
The following tables shows the round trip times in μs on the IA420F
board (similar times were observed on IA840F), each value is a bfloat16 (2 bytes),
using fewer than 32 values gives the same times as 32 values.
To reproduce these values on your own hardware run the provided benchmark
script with environment variable RUN_IO_TEST=1.
User buffers
This includes copying data to/from DMA buffers.
mean:
| out\in | 32 | 64 | 128 | 256 | 512 | 1024 | 2048 | 4096 | 8192 |
|---|---|---|---|---|---|---|---|---|---|
| 32 | 1.8 | 1.8 | 1.9 | 1.9 | 1.9 | 2.1 | 2.7 | 3.4 | 4.6 |
| 64 | 1.8 | 1.9 | 1.9 | 1.9 | 2.0 | 2.2 | 2.8 | 3.3 | 4.7 |
| 128 | 1.9 | 1.9 | 1.9 | 2.0 | 2.0 | 2.2 | 2.7 | 3.3 | 4.7 |
| 256 | 1.9 | 1.9 | 1.9 | 2.0 | 2.0 | 2.2 | 2.7 | 3.4 | 5.0 |
| 512 | 1.9 | 1.9 | 2.0 | 2.0 | 2.1 | 2.2 | 2.8 | 3.4 | 4.8 |
| 1024 | 2.2 | 2.2 | 2.2 | 2.1 | 2.3 | 2.3 | 2.7 | 3.6 | 4.9 |
| 2048 | 2.4 | 2.4 | 2.5 | 2.5 | 2.5 | 2.6 | 2.8 | 3.8 | 5.0 |
| 4096 | 2.9 | 2.9 | 2.9 | 3.0 | 3.0 | 3.3 | 3.6 | 3.6 | 5.2 |
| 8192 | 3.8 | 3.8 | 3.9 | 3.9 | 3.9 | 4.1 | 4.7 | 4.5 | 5.1 |
p99:
| out\in | 32 | 64 | 128 | 256 | 512 | 1024 | 2048 | 4096 | 8192 |
|---|---|---|---|---|---|---|---|---|---|
| 32 | 2.0 | 2.0 | 2.1 | 2.0 | 2.1 | 2.4 | 3.0 | 3.7 | 4.9 |
| 64 | 2.0 | 2.0 | 2.1 | 2.1 | 2.1 | 2.4 | 3.0 | 3.6 | 5.0 |
| 128 | 2.0 | 2.0 | 2.1 | 2.1 | 2.2 | 2.4 | 2.9 | 3.6 | 5.0 |
| 256 | 2.0 | 2.0 | 2.0 | 2.1 | 2.2 | 2.5 | 3.1 | 3.6 | 5.3 |
| 512 | 2.0 | 2.1 | 2.2 | 2.1 | 2.3 | 2.4 | 3.0 | 3.7 | 5.1 |
| 1024 | 2.4 | 2.4 | 2.4 | 2.4 | 2.4 | 2.5 | 3.1 | 3.8 | 5.4 |
| 2048 | 2.6 | 2.6 | 2.7 | 2.7 | 2.6 | 2.8 | 3.1 | 4.2 | 5.4 |
| 4096 | 3.2 | 3.2 | 3.2 | 3.3 | 3.3 | 3.6 | 3.9 | 3.8 | 5.5 |
| 8192 | 4.1 | 4.1 | 4.1 | 4.1 | 4.2 | 4.4 | 5.0 | 4.7 | 5.3 |
Raw DMA buffers
This is using buffers allocated with vollo_rt_get_raw_buffer which lets the runtime skip IO copy.
mean:
| out\in | 32 | 64 | 128 | 256 | 512 | 1024 | 2048 | 4096 | 8192 |
|---|---|---|---|---|---|---|---|---|---|
| 32 | 1.7 | 1.8 | 1.8 | 1.8 | 1.9 | 1.9 | 2.0 | 2.2 | 2.6 |
| 64 | 1.7 | 1.8 | 1.8 | 1.9 | 1.9 | 1.9 | 2.0 | 2.2 | 2.7 |
| 128 | 1.8 | 1.8 | 1.8 | 1.8 | 1.9 | 1.9 | 2.0 | 2.2 | 2.6 |
| 256 | 1.8 | 1.8 | 1.8 | 1.9 | 1.8 | 1.9 | 2.1 | 2.2 | 2.6 |
| 512 | 1.8 | 1.8 | 1.8 | 1.8 | 1.9 | 1.9 | 2.1 | 2.3 | 2.6 |
| 1024 | 1.9 | 1.9 | 1.9 | 1.9 | 1.9 | 2.0 | 2.1 | 2.3 | 2.7 |
| 2048 | 1.9 | 1.9 | 2.0 | 2.0 | 2.0 | 2.1 | 2.2 | 2.4 | 2.8 |
| 4096 | 2.2 | 2.2 | 2.2 | 2.2 | 2.2 | 2.3 | 2.4 | 2.6 | 3.0 |
| 8192 | 2.5 | 2.5 | 2.6 | 2.6 | 2.6 | 2.7 | 2.8 | 3.0 | 3.4 |
p99:
| out\in | 32 | 64 | 128 | 256 | 512 | 1024 | 2048 | 4096 | 8192 |
|---|---|---|---|---|---|---|---|---|---|
| 32 | 1.9 | 1.9 | 1.9 | 1.9 | 2.1 | 2.1 | 2.2 | 2.4 | 2.9 |
| 64 | 1.9 | 1.9 | 2.0 | 1.9 | 2.0 | 2.1 | 2.2 | 2.4 | 2.9 |
| 128 | 1.9 | 1.9 | 1.9 | 1.9 | 2.1 | 2.0 | 2.2 | 2.4 | 2.9 |
| 256 | 1.9 | 2.0 | 2.0 | 1.9 | 2.0 | 2.1 | 2.2 | 2.4 | 2.8 |
| 512 | 1.9 | 2.0 | 2.0 | 2.0 | 2.0 | 2.1 | 2.2 | 2.5 | 2.9 |
| 1024 | 2.1 | 2.0 | 2.0 | 2.1 | 2.1 | 2.1 | 2.3 | 2.6 | 2.9 |
| 2048 | 2.1 | 2.1 | 2.1 | 2.2 | 2.2 | 2.2 | 2.3 | 2.6 | 3.0 |
| 4096 | 2.3 | 2.3 | 2.3 | 2.4 | 2.4 | 2.5 | 2.6 | 2.9 | 3.4 |
| 8192 | 2.8 | 2.7 | 2.7 | 2.7 | 2.8 | 2.8 | 3.0 | 3.2 | 3.6 |
MMIO
This is skipping DMA for the input (raw DMA buffers are used for the output).
It is configured via the VOLLO_MMIO_MAX_SIZE environment variable.
Typically MMIO would only be used for smaller inputs, but this table shows it used for the entire input.
mean:
| out\in | 32 | 64 | 128 | 256 | 512 | 1024 | 2048 | 4096 | 8192 |
|---|---|---|---|---|---|---|---|---|---|
| 32 | 1.0 | 1.3 | 1.5 | 2.0 | 3.1 | 5.4 | 9.6 | 18.5 | 35.9 |
| 64 | 1.1 | 1.2 | 1.5 | 2.0 | 3.1 | 5.3 | 9.7 | 18.4 | 34.8 |
| 128 | 1.1 | 1.2 | 1.5 | 2.0 | 3.2 | 5.2 | 9.5 | 18.4 | 35.8 |
| 256 | 1.1 | 1.2 | 1.5 | 2.0 | 3.1 | 5.2 | 9.7 | 18.4 | 35.9 |
| 512 | 1.1 | 1.2 | 1.5 | 2.1 | 3.1 | 5.3 | 9.7 | 18.4 | 35.8 |
| 1024 | 1.2 | 1.3 | 1.6 | 2.1 | 3.2 | 5.3 | 9.8 | 18.5 | 36.0 |
| 2048 | 1.2 | 1.4 | 1.7 | 2.2 | 3.2 | 5.5 | 9.9 | 18.5 | 35.9 |
| 4096 | 1.5 | 1.6 | 1.9 | 2.4 | 3.6 | 5.7 | 10.1 | 18.8 | 36.1 |
| 8192 | 1.9 | 2.0 | 2.2 | 2.8 | 3.9 | 6.1 | 10.4 | 19.2 | 36.7 |
p99:
| out\in | 32 | 64 | 128 | 256 | 512 | 1024 | 2048 | 4096 | 8192 |
|---|---|---|---|---|---|---|---|---|---|
| 32 | 1.2 | 1.5 | 1.6 | 2.1 | 3.2 | 5.5 | 9.9 | 19.1 | 37.0 |
| 64 | 1.2 | 1.4 | 1.6 | 2.2 | 3.2 | 5.5 | 9.8 | 19.0 | 35.9 |
| 128 | 1.2 | 1.3 | 1.6 | 2.1 | 3.4 | 5.5 | 10.0 | 19.1 | 36.9 |
| 256 | 1.2 | 1.4 | 1.6 | 2.2 | 3.2 | 5.3 | 9.8 | 19.1 | 36.8 |
| 512 | 1.3 | 1.4 | 1.6 | 2.2 | 3.3 | 5.5 | 9.9 | 19.1 | 36.8 |
| 1024 | 1.5 | 1.4 | 1.7 | 2.2 | 3.3 | 5.5 | 9.8 | 19.2 | 37.0 |
| 2048 | 1.3 | 1.4 | 1.8 | 2.4 | 3.4 | 5.5 | 10.1 | 19.2 | 36.9 |
| 4096 | 1.6 | 1.8 | 2.0 | 2.5 | 3.6 | 5.9 | 10.2 | 19.4 | 37.1 |
| 8192 | 2.0 | 2.1 | 2.3 | 2.9 | 4.0 | 6.3 | 10.6 | 19.8 | 37.6 |