Skip to content

faster performance on older machines #18

Closed as not planned
Closed as not planned
faster performance on older machines#18
Labels
questionFurther information is requested

Description

@sibeliu
sibeliu
opened on Mar 11, 2023

On machines with smaller memory and slower processors, it can be useful to reduce the overall number of threads running. For instance on my MacBook Pro Intel i5 16Gb machine, 4 threads is much faster than 8. Try:

make -j && ./main -m ./models/7B/ggml-model-q4_0.bin -p "Author-contribution statements and acknowledgements in research papers should state clearly and specifically whether, and to what extent, the authors used AI technologies such as ChatGPT " -t 4 -n 512

Activity

  1. prusnak commented on Mar 11, 2023

    @prusnak
    prusnak
    on Mar 11, 2023
    Contributor

    @sibeliu what does getconf _NPROCESSORS_ONLN say on your machine?

  2. loretoparisi commented on Mar 11, 2023

    @loretoparisi
    loretoparisi
    on Mar 11, 2023

    Not sure why, but on my Mac M1 Pro / 16GB using 4 threads works far better than 8 threads:

    (base) musixmatch@Loretos-MBP llama.cpp % ./main -m ./models/7B/ggml-model-q4_0.bin -p "Building a website can be done in 10 simple steps:" -t 4 -n 512
main: seed = 1678565187
llama_model_load: loading model from './models/7B/ggml-model-q4_0.bin' - please wait ...
llama_model_load: n_vocab = 32000
llama_model_load: n_ctx   = 512
llama_model_load: n_embd  = 4096
llama_model_load: n_mult  = 256
llama_model_load: n_head  = 32
llama_model_load: n_layer = 32
llama_model_load: n_rot   = 128
llama_model_load: f16     = 2
llama_model_load: n_ff    = 11008
llama_model_load: n_parts = 1
llama_model_load: ggml ctx size = 4529.34 MB
llama_model_load: memory_size =   512.00 MB, n_mem = 16384
llama_model_load: loading model part 1/1 from './models/7B/ggml-model-q4_0.bin'
llama_model_load: .................................... done
llama_model_load: model size =  4017.27 MB / num tensors = 291

main: prompt: 'Building a website can be done in 10 simple steps:'
main: number of tokens in prompt = 15
     1 -> ''
  8893 -> 'Build'
   292 -> 'ing'
   263 -> ' a'
  4700 -> ' website'
   508 -> ' can'
   367 -> ' be'
  2309 -> ' done'
   297 -> ' in'
 29871 -> ' '
 29896 -> '1'
 29900 -> '0'
  2560 -> ' simple'
  6576 -> ' steps'
 29901 -> ':'

sampling parameters: temp = 0.800000, top_k = 40, top_p = 0.950000


Building a website can be done in 10 simple steps:
1. Get a server
2. Set up a database
3. Get a name server
4. Get a web server
5. Get a control panel
6. Set up a phpMyAdmin interface for MySQL
7. Set up a database user
8. Get a root password for your server
9. Upload your files
10. Test your files
After that you need to make sure that your website is indexed in Google. You can do that with Google Webmaster Tools.
I have only listed the most important and core steps to create a website. The rest of the process depends on your level of experience and on what you want to accomplish with your website.
In this article I will focus on the first 10 steps. I hope that this article will give you a good understanding of how to create a website from scratch.
My goal is to create a website that I can use to make a living online. My vision is to make a website that can generate more than $100 per day, that’s $3,000 per month.
I don’t think that it’s possible to have a website that will generate money while you sleep. There will always be work involved, so I don’t want to start off with a website that I need to work on all day long.
I want to have a website that can generate at least $100 per day, that’s $3,000 per month. It’s not going to be easy, but I know that it’s possible.
Some people create websites to make money, others do it to learn new skills. Creating a website is a good exercise in itself, but the real benefit is to get more traffic to your website.
The more traffic you get, the bigger your website will be. The bigger your website is, the more income it will generate.
In this article I will guide you through the process of creating a website step by step.
A website doesn’t have to be something fancy, but it needs to be professional looking. You don’t need to go for the best looking website out there, but make sure that your website has a good overall impression.
If you want to create a website to make money online, then make sure that your website is easy to navigate. A website that is easy to navigate will be a lot easier to

main: mem per token = 14368644 bytes
main:     load time =  1301.80 ms
main:   sample time =  1098.24 ms
main:  predict time = 59317.72 ms / 116.08 ms per token
main:    total time = 62136.24 ms

  3. prusnak commented on Mar 11, 2023

    @prusnak
    prusnak
    on Mar 11, 2023
    Contributor

    Here's my benchmark on Apple M1 16 GB:

    threads 1 2 4 8
    7B 4-bit 316.52 164.76 98.56 277.92
    13B 4-bit 628.48 376.14 224.21 538.87

    (ms per token)

    I think it's good that the default value for -t is 4.

  4. wizzard0 commented on Mar 11, 2023

    @wizzard0
    wizzard0
    on Mar 11, 2023
    Contributor

    @prusnak that's because there's 4 performance cores on M1

  5. loretoparisi commented on Mar 11, 2023

    @loretoparisi
    loretoparisi
    on Mar 11, 2023

    @prusnak that's because there's 4 performance cores on M1

    but on my M1 Pro I have 8 cores...

  6. sibeliu commented on Mar 11, 2023

    @sibeliu
    sibeliu
    on Mar 11, 2023
    Author

    8, which would be nice to use. With the current setup I'm only using 4

  7. ggerganov
    added
    questionFurther information is requested
    on Mar 12, 2023

  8. loretoparisi commented on Mar 12, 2023

    @loretoparisi
    loretoparisi
    on Mar 12, 2023

    8, which would be nice to use. With the current setup I'm only using 4

    It should, but it seems that there is some bottleneck on the M1 Pro that prevents to have better perfs with the 8 threads, resulting in a slower inference when specifyng n=4, not sure why. I will do a better test.

  9. plhosk commented on Mar 12, 2023

    @plhosk
    plhosk
    on Mar 12, 2023

    I'm getting the same reuslts on a 4c/8t i7 skylake on linux (7B model, 4-bit). -t 4 is several times faster than -t 8

  10. prusnak commented on Mar 12, 2023

    @prusnak
    prusnak
    on Mar 12, 2023
    Contributor

    I guess this is because hyperthreading does not help with running the model? So the number of virtual cores is not important only number of physical cores?

  11. plhosk commented on Mar 12, 2023

    @plhosk
    plhosk
    on Mar 12, 2023

    Upon further testing it seems like if I have anything else using the CPU e.g. having Firefox open and watching a video, -t 8 slows to a crawl while -t 4 is relatively unaffected, but after closing all CPU-consuming programs -t 8 becomes faster than -t 4.

  12. sibeliu commented on Mar 12, 2023

    @sibeliu
    sibeliu
    on Mar 12, 2023
    Author

    I guess this is because hyperthreading does not help with running the model? So the number of virtual cores is not important only number of physical cores?

    That looks like the cause. Even though getconf _NPROCESSORS_ONLN says 8, there are only 4 physical cores on my processor. But it is still odd that both -t 4 and 8 utilize only 50% of my available processor. If I launch other apps it goes over 50%.

    BTW can you think of any way to make the GPU help out? It isn't doing anything at the moment

  13. plhosk commented on Mar 12, 2023

    @plhosk
    plhosk
    on Mar 12, 2023

    BTW can you think of any way to make the GPU help out? It isn't doing anything at the moment

    This project is CPU only however there's a different one that runs on the GPU. Keep in mind the weights are not compatible between the two projects.

    https://github.com/oobabooga/text-generation-webui

  14. sibeliu commented on Mar 12, 2023

    @sibeliu
    sibeliu
    on Mar 12, 2023
    Author

    This project is CPU only however there's a different one that runs on the GPU. Keep in mind the weights are not compatible between the two projects.

    https://github.com/oobabooga/text-generation-webui

    Thank you! I'll take a look. I just have the GPU in my macbook, wish I had an A100 or something...

  15. l29ah commented on Mar 13, 2023

    @l29ah
    l29ah
    on Mar 13, 2023
    Contributor

    Interestingly, -t4 works much faster than -t8 on my 4-cores 8-threads i7-8550U too.

  16. 33 remaining items

  17. oobabooga
    added a commit that references this issue on Jun 1, 2026

    ci: build and publish our own macOS llama.cpp prebuilts daily (ggml-o…

    a495487
  18. AaronStGeorge
    added a commit that references this issue on Jun 5, 2026

    Extend Split-k flash attention GQA kernel support to cases used in ll…

    c1af878
  19. platima
    added a commit that references this issue on Jul 9, 2026

    fix(server): fix SMT multimodal crash on hybrid-recurrent models + dr…

    a7b3dc1
  20. MrLordCat
    added a commit that references this issue on Jul 16, 2026

    ggml: backend-agnostic tensor parallelism (experimental) (#19378)

    d850df3
  21. zommiommy
    added a commit that references this issue on Aug 18, 2026

    ggml: backend-agnostic tensor parallelism (experimental) (ggml-org#19378

    4d40025
  22. wdenejko
    added 2 commits that reference this issue on Aug 30, 2026

    vulkan: tile-epilog fusion phase B - fused trailing MUL-by-tensor

    80d62a3

    qwen4exp: gathered sparse prefill behind Q4X_QSA_GP (ggml-org#18)

    d102226
  23. kainlan
    added a commit that references this issue on Sep 12, 2026

    fix(sycl): address all 19 spec-review findings on the DIRECT reuse po…

    c881bf0
  24. KokerZhou
    added a commit that references this issue on Oct 14, 2026

    cannge handoff notes: note-30..35 (910B3 bring-up: smoke coherent, gg…

    6c73fd6
  25. github-actions
    added a commit that references this issue on Oct 26, 2026

    Merge pull request ggml-org#18 from thecodacus/fable5/decision-on-perf

    906247a
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    questionFurther information is requested

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions