• 0 Posts
  • 16 Comments
Joined 2 years ago
cake
Cake day: June 15th, 2023

help-circle






  • DNS over HTTPS. It allows encrypted DNS lookup with a URL, which allows for url-based customizations not possible with traditional DNS lookups (e.g. the server could have /ads or /trackers endpoints so you can choose what to block).

    DNS Over TLS (DoT) is similar, but it doesn’t use URLs, just IP addresses like generic DNS. Both are encrypted.




  • I’m not entirely sure how I need to effectively use these models, I guess. I tried some basic coding prompts, and the results were very bad. Using R1 Distill Qwen 32B, 4-bit quant.

    The first answer had incorrect, non-runnable syntax. I was able to get it to fix that after multiple followup prompts, but I was NOT able to get it to fix the bugs. It took several minutes of thinking time for each prompt, and gave me worse answers than the stock Qwen model.

    For comparison, GPT 4o and Claude Sonnet 3.5 gave me code that would at least run on the first shot. 4o’s was even functional in one shot (Sonnet’s was close but had bugs). And that took just a few seconds instead of 10+ minutes.

    Looking over its chain of thought, it seems to get caught in circles, just stating the same points again and again.

    Not sure exactly what the use case is for this. For coding, it seems worse than useless.


  • But any 50 watt chip will get absolutely destroyed by a 500 watt gpu

    If you are memory-bound (and since OP’s talking about 192GB, it’s pretty safe to assume they are), then it’s hard to make a direct comparison here.

    You’d need 8 high-end consumer GPUs to get 192GB. Not only is that insanely expensive to buy and run, but you won’t even be able to support it on a standard residential electrical circuit, or any consumer-level motherboard. Even 4 GPUs (which would be great for 70B models) would cost more than a Mac.

    The speed advantage you get from discrete GPUs rapidly disappears as your memory requirements exceed VRAM capacity. Partial offloading to GPU is better than nothing, but if we’re talking about standard PC hardware, it’s not going to be as fast as Apple Silicon for anything that requires a lot of memory.

    This might change in the near future as AMD and Intel catch up to Apple Silicon in terms of memory bandwidth and integrated NPU performance. Then you can sidestep the Apple tax, and perhaps you will be able to pair a discrete GPU and get a meaningful performance boost even with larger models.


  • If you’re running a consumer level GPU, you’ll be operating with 24GB of VRAM max (RTX 4090, RTX 3090, or Radeon 7900XTX).

    90b model = 90GB at 8-bit quantization (plus some extra based on your context size and general overhead, but as a ballpark estimate, just going by the model size is good enough). You would need to drop down to 2-bit quantization to have any hope to fit it in a single consumer GPU. At that point you’d probably be better off using a smaller model will less aggressive quantization, like a 32b model at 4-bit quantization.

    So forget about consumer GPUs for that size of model. Instead, you can look at systems with integrated memory, like a Mac with 96-128GB of memory, or something similar. HP has announced a mini PC that might be good, and Nvidia has announced a dedicated AI box as well. Neither of those are available for purchase yet, though.

    You could also consider using multiple consumer GPUs. You might be able to get multiple RTX 3090s for cheaper than a Mac with the same amount of memory. But then you’ll be using several times more power to run it, so keep that in mind.





  • The ideal amount of storage is enough that I literally never need to think about it, never need to delete anything, and never need to use cloud services for things that could realistically be local.

    It’s hard to say what that would be because I’ve never had a phone that even came close.

    The largest phone I’ve owned was 256GB. That was “fine”, but it was NOT big enough that I could fundamentally change my habits. For example, I don’t carry my entire music collection on my phone. I don’t even do that on my laptop anymore since the advent of SSDs.

    I have a 128GB phone now and it sucks. I’ve set up a one-way copy to my home desktop with Syncthing so I can safely delete photos, videos, and screen recordings from my phone. I need to do this frequently.

    With the standard price-gouging in the industry, I will probably settle for 256 with my next phone. If prices were reasonable, I’d go for 1TB at least.

    I miss SD cards but there are no viable options with slots anymore.