Different outputs for differents numbers of threads (same seed)

Hello,

I simply wanted to bring up the point that the output can vary based on the number of threads selected, even if the seed stays constant.

I have an intel core i7 10700K that has 16 threads.

For this example I'm using the 13B model (./models/13B/ggml-model-q4_0.bin)

When I put -t 14 (make -j && ./main -m ./models/13B/ggml-model-q4_0.bin -p "Building a website can be done in 10 simple steps:" -t 14 -n 50 --seed 1678486056), I got this result:

When I put -t 15 (make -j && ./main -m ./models/13B/ggml-model-q4_0.bin -p "Building a website can be done in 10 simple steps:" -t 15 -n 50 --seed 1678486056), I got this result:

I have zero knowledge in machine learning, perhaps this is a normal behavior.

Looking forward for your reactions!

That isn't surprising, as each thread may be getting its own random seed. Changing the number of threads would then change the random seed initialisation, thus generating different output.

It's not the random seed. The random seed is only one in the main thread.

The problem is more complex - the tensor operations are multi-threaded and based on the number of threads, they process different sizes of the input data. Since these are floating point numbers and they are often accumulated, the finite precision of floating-point operations causes the final results to be slightly different for different number of threads. This effect is amplified by the auto-regressive nature of the transformer.

There are ways to fix this, but a major refactor in ggml is required.

@BadisG

@BadisG
I think I have fixed this behaviour on latest master. If you can give it a try and let me know if the results for different number of threads is now the same