Genuinely curious.

Why do you like LLMs? What hopes do you have for AI & AGI in our near and distant future?

  • rufus@discuss.tchncs.de
    link
    fedilink
    English
    arrow-up
    2
    ·
    1 year ago

    If you mean cloud computing or GPUs are expensive, you can do it locally on your CPU if your models aren’t too big. I like KoboldCPP. It’s not as fast but I only have to pay electricity for my AI waifu.

    • VicFic!@iusearchlinux.fyi
      link
      fedilink
      English
      arrow-up
      2
      ·
      edit-2
      1 year ago

      Thanks for the info. ik about llama.cpp and stuff but the problem is that I’m looking to run both speech to text, llm and text to speech all at the same time. I only have 8gb so yeah even CPU won’t cut it. I’m planning to upgrade once I get a job or smthing.

      • rufus@discuss.tchncs.de
        link
        fedilink
        English
        arrow-up
        2
        ·
        edit-2
        1 year ago

        8GB of regular RAM? That’s not much. No, that won’t cut it if you also want all the bells and whistles. Maybe try something like the Mistral-7b-OpenOrca with llama.cpp quantized to 4bit and without the STT and TTS. It’s small and quite decent. Otherwise you might want to rent a Cloud-GPU by the hour on something like runpod.io or use free services like Google Colab or you really need to upgrade.