IO Round Trip

The following IO round trip times are sampled by using a program with no compute on the Vollo accelerator from the Vollo runtime.

More specifically this Vollo accelerator program waits for the last input byte to arrive before it sends the first output byte back. This method takes into account some of the overheads (such as copying between the application's buffers and the runtime's DMA buffers) associated with IO and this test is set up to see how it scales with difference sizes of inputs and outputs.

The following tables show the round trip times in μs on the V80 board, for different input and output sizes, with the runtime's defaults. Using fewer than 64 bytes gives the same times as 64 bytes.

To reproduce these values on your own hardware run the provided benchmark script with environment variable RUN_IO_TEST=1.

By default the runtime currently sends an inference's input over MMIO, written straight from the application's buffer, when the model's inputs total at most 8 KiB, and over DMA otherwise. MMIO also needs no DMA input in flight and room in the card's input ring, so with several inferences in flight a small input can still go over DMA. These tables run one inference at a time, so every input up to 8192 B goes over MMIO and only the 16384 B column goes over DMA. The VOLLO_MMIO_MAX_SIZE environment variable sets the threshold in bytes, and 0 sends every input over DMA. On an SR-IOV VF, vollo-tool setup-vfs chooses instead, for the whole card (see MMIO and DMA inputs on VFs). Outputs always come back over DMA.

User buffers

The application passes its own buffers. The runtime copies each output from its DMA buffer into the application's buffer. An input is only copied into a DMA buffer when it goes over DMA (the 16384 B column); up to 8 KiB it is written over MMIO, with no copy.

mean:

out\in64 B128 B256 B512 B1024 B2048 B4096 B8192 B16384 B
64 B0.860.870.870.890.941.021.211.562.82
128 B0.920.920.930.950.991.081.261.622.90
256 B0.960.970.950.991.031.121.281.652.91
512 B1.001.021.031.031.101.191.371.712.96
1024 B1.071.051.061.101.151.241.411.773.03
2048 B1.141.141.181.211.231.301.461.893.10
4096 B1.381.361.351.381.441.511.742.053.35
8192 B1.761.901.801.821.881.932.152.513.88
16384 B2.712.722.712.712.792.943.053.464.72

p99:

out\in64 B128 B256 B512 B1024 B2048 B4096 B8192 B16384 B
64 B0.880.880.890.910.971.051.231.593.04
128 B1.231.231.241.261.311.391.571.933.15
256 B1.261.261.271.301.341.431.601.963.15
512 B1.311.321.321.331.341.491.652.003.27
1024 B1.351.341.351.361.431.511.682.063.32
2048 B1.431.431.441.471.511.591.762.123.38
4096 B1.661.661.671.691.741.812.012.353.63
8192 B2.042.352.062.072.132.442.602.774.07
16384 B3.163.163.153.153.223.363.503.905.41

Raw DMA buffers

This is using buffers allocated with vollo_rt_get_raw_buffer, which lets the runtime skip the copies to and from its DMA buffers.

mean:

out\in64 B128 B256 B512 B1024 B2048 B4096 B8192 B16384 B
64 B0.850.840.850.870.921.011.191.542.77
128 B0.850.860.860.880.931.021.201.552.77
256 B0.880.880.880.900.941.041.221.572.80
512 B0.910.910.920.940.991.071.251.612.84
1024 B0.940.940.950.971.021.101.271.642.91
2048 B1.011.011.021.051.091.181.341.702.91
4096 B1.151.151.161.181.221.311.491.853.07
8192 B1.421.421.431.451.491.581.772.123.34
16384 B1.971.971.982.012.042.142.322.683.89

p99:

out\in64 B128 B256 B512 B1024 B2048 B4096 B8192 B16384 B
64 B0.860.860.870.890.941.041.211.572.96
128 B0.870.870.870.890.951.041.221.572.94
256 B0.890.890.900.920.971.061.241.603.10
512 B0.920.930.940.951.021.101.271.633.17
1024 B0.960.960.970.991.041.121.291.663.09
2048 B1.021.031.041.061.111.201.371.732.96
4096 B1.161.171.181.201.231.331.511.873.36
8192 B1.441.431.441.461.511.601.792.153.60
16384 B1.991.992.002.032.062.162.352.713.91