The speed of using embedded models is much slower compared to xinference

I use the BGE-M3 model and send the same request, especially with xinference taking about 10 seconds and ollama taking about 200 seconds. I'm sure they all use GPUs. I found that xinference allocates more video memory, while ollama's video memory usage remains basically unchanged. Perhaps this is the reason for the speed difference?