Save text as JSON as comments but the file itself is a PNG so that you can use it as an image (like previewing it) as they would ignore the comments. However, the OP’s editor can load the file back, parse the comments, and get the original data and continue to edit. Just one file to maintain. Quite clever actually.
I totally agree with this. I wanted to make it really easy for non-technical users with an app that hid all the complexities. I basically just wanted to embed the engine without making users open their terminal, let alone make them configure. I started with llama.cpp amd almost gave up on the idea before I stumbled upon Ollama, which made the app happen[1]
There are many flaws in Ollama but it makes many things much easier esp. if you don’t want to bother building and configuring. They do take a long time to merge any PRs though. One of my PRs has been waiting for 8 months and there was this another PR about KV cache quantization that took them 6 months to merge.
> They do take a long time to merge any PRs though.
I guess you have a point there, seeing as after many months of waiting we finally have a comment on this PR from someone with real involvement in Ollama - see https://github.com/ollama/ollama/pull/5059#issuecomment-2628... . Of course this is very welcome news.
It's not really welcome news, he is just saying they're putting it on the long finger because they think other stuff is more important. He's the same guy that kept ignoring the KV cache quant merge.
And the actual patch is tiny..
I think it's about time for a bleeding-edge fork of ollama. These guys are too static and that is not what AI development is all about.
He specifically says that they're reworking the Ollama server implementation in order to better support other kinds of models, and that such work has priority and is going to be a roadblock for this patch. This is not even news to those who were following the project, and it seems reasonable in many ways - users will want Vulkan to work across the board if it's made available at all, not for it to be limited to the kinds of models that exist today.
If anyone is looking for a one click solution without having to have a Docker running, try Msty - something that I have been working on for almost a year. Has RAG and Web Search built in among others and can connect to your Obsidian vaults as well.
Interesting that you show Mistral Large in your screenshots. That model is only licensed for "research" use. While it's not terribly clear how "research" is defined, selling a product with it featured is clearly not "research".
I would try this but I want to be able to use it on mobile too which requires hosting. I don’t see how valuable an LLM interface can be if I can only use it on desktop
I wanted to present it as - “Msty As x Alternative”. But I think maybe I like Compare better. I will think about changing it. Thank you for the suggestion.
I went with the LG CX model based on what I read on rtings.com
That’s a previous-generation model. I think all of the LG TVs are good.
There are / were technical caveats. I believe all of them are solved by M3 macs that have HDMI 2.1 ports. (M3 or M3 Pro or something? The ones advertised as 8K capable.) Out of the box, those will do 4K 120Hz HDR with variable refresh rate and full 444 color. This is what you want.
It is possible to get that going on older machines, except for VRR which is more of a nice-to-have anyway.
I have a 2018 Macbook Pro 15”. Disclaimer!: My setup was a “complexity pet”, a tinkering project; There are simpler ways to connect a 120Hz 4K HDR HDMI 2.1 display to a non-HDMI-2-1 mac. And! My tinkering project wasn’t only about getting the display working correctly. It was more about messing with eGPUs and virtualization and stuff. Definitely a long way round.
On my Intel mac, I use an AMD Radeon 6800 XT eGPU with Club3D or CableMatters DisplayPort-to-HDMI 2.1 adapters. Plus some EDID hacking which is easy to do.
EDID is how the display identifies itself to the OS. The EDID payload can be overridden on the OS side. Mostly it’s about copying the display’s EDID and deleting the entry that says the display can accept 4:2:0 color. Only then does macOS switch to 4:4:4 color. I also created a custom “modeline” with tighter timing to get 120Hz going fully.
—Please be assured that this was way more complex than it needed to be. It was for fun!
There are much easier ways to do this. Lots of forum posts on it. On the MacRumors forums iirc? User joevt is The Man.
And even then, what I wrote above is actually easy to do once you know it’s possible.
Mostly though you really want an M3 Mac that just has HDMI 2.1 and is ready to go.
There are/were also OLED gaming monitors available, such as from Alienware. Those have DisplayPort inputs and are ready to go with almost any older Mac. Might be able to find one for a price equivalent to a TV, idk.
I am working on making using LLMs easy for everyone. Focus is to make users productive rather than having them configuring their environment to death. Wrote this app for my wife who, even as a smart software engineer, was getting overwhelmed with AI and LLMs.
I've recently started using Arc mostly out of FOMO and there are parts that I like and parts I don't. I have been a Firefox user for a long time before that so glad to see something similar that's one of the major priorities is the aesthetics and I very much appreciate that.
I tried to use it on my macOS Apple Silicone and got an error about it being broken and macOS suggested to trash it. Not sure if it is a bug issue. Will come back and try it again though to give it a second chance :)
I'll suggest that bypassing Gatekeeper for an unknwon app from an unknown developer is a bad idea. I'll wait until they implement official code signing from Apple.
thats pretty unfair to the devs. You need a Developer ID through Apple, a mac, and you need to pay. Thats kind of a ridiculous expectation to begin with. Two of those things can be solved with money (a significant amount of money mind you, for the purpose of releasing a single application) and Apple can arbitrarily make the Dev ID process arduous and painful.
This super unhelpful error is sometimes the result of trying to run an unsigned or developer signed binary on Apple Silicon. Try `xattr -d com.apple.quarantine program.app`, then open it by right clicking on the app, and selecting 'Open' while holding option + command.
Given the audience on HN, I think we can presume any readers of my comment are not trying to execute completely random untrusted binaries. There are legitimate cases when you need to do this to run a binary you trust, but the system doesn't.
I always run everything through virustotal first. (I should probably add that this is step two in my process: step one is to be very cautious and not download most things in the first place.)
Wow! The benchmarks are truly impressive, showing significant improvements across almost all categories. It's fascinating to see how rapidly this field is evolving. If someone had told me last year that Meta would be leading the charge in open-source models, I probably wouldn't have believed them. Yet here we are, witnessing Meta's substantial contributions to AI research and democratization.
On a related note, for those interested in experimenting with large language models locally, I've been working on an app called Msty [1]. It allows you to run models like this with just one click and features a clean, functional interface. Just added support for both 8B and 70B. Still in development, but I'd appreciate any feedback.
Tried using msty today and it refused to open and demanded an upgrade from 0.9 - remotely breaking a local app that had been working is unacceptable. Good luck retaining users.
This is a great release! If you are looking to try it locally with a great interface, I am working on an app [1] and I just pushed an update to support Gemma2.
Wow, msty looks really cool. I've bookmarked it to look into more later as a replacement for how I use a locally-hosted instance of LibreChat. It'd be a huge improvement to use local models rather than remote ones, for much of my queries.
That said, do you have a reason for keeping msty closed source rather than open? I read your FAQ for "why should I trust msty" and it feels lacking.
> We are a small team of developers who are passionate about AI and privacy. We have worked on projects before that have been used by thousands of people such as this (I've never heard of Cleavr). There are real faces (real faces = Twitter account link?) behind the product. And come chat with us on our Discord server to know us better.
This is much, much better than having no attribution, but it's miles away from being able to verify trust by reading the code. Would love to hear what your reasons against this are.
Looks cool even though closed source makes me wary.
Trying to save Anthropic API key on Arch Linux doesn't do anything and there's a message "If you're experiencing problems saving API keys especially on Linux, contact Discord", if it's so common problem maybe you should have a link with possible fixes? Adding another Discord server and searching for answers for a question that clearly has been asked often enough feels like quite a hurdle for testing it out.
[1]: https://msty.ai