A developer posting on dev.to under the handle icegaming has published a comparison arguing that GitHub stars are a poor proxy for tool quality, using their own project as the test case. The post, titled "Stars Aren't Benchmarks: How a 10-Star Memory Tool Out-Scored the 60k-Star One," sets palace-rs, which the author describes as having about ten stars, against MemPalace, which the author says has well over 60,000 stars and whose README calls it the best-benchmarked open-source AI memory system. The author states plainly that the comparison rests on each project's own published or self-run numbers rather than a third-party audit, and invites readers to rerun either side.

The central figure is a retrieval score on LongMemEval, a memory benchmark the post links to. The author reports that on the standard 470-question set — the 500-question s_cleaned split minus 30 abstention questions that the benchmark is designed to exclude from retrieval scoring — palace-rs reaches R@5 of 0.987, or 98.7%. That is placed against MemPalace's 98.4% hybrid number. The author also reports an all-500 figure of 98.6% R@5 for palace-rs against a 96.6% raw figure, explicitly so that excluding the abstention questions cannot be said to flatter the result.

The post is careful about the size of the gap. On 500 questions, the author notes, two points amounts to ten questions, and they decline to call it a breakthrough. The narrower claim is that a project with about ten stars published the higher number and that nobody would have learned this from the star count. That framing matters for how the piece should be read: it is an argument about discovery signals, illustrated with a small measured difference, not a claim of dramatic superiority.

The author gives MemPalace credit for how it handled its own tuning. According to the post, MemPalace's hybrid mode adds targeted boosts for quoted phrases, person names and nostalgia-style phrasing, and those fixes were developed by examining three specific questions out of the 500. The author calls that a normal way to improve a retriever but says fixing failures visible in the test set stops being a clean measurement, and notes that MemPalace publishes the held-out 450-question result as its honest number — which the author describes as the right and transparent call.

The author says, by contrast, that palace-rs reports all 500 questions, and that no retrieval parameters were fitted on the test set: weights, chunking and scoring were set independently of those questions. According to the description, the palace-rs run used hybrid search with BM25 weighted 0.35 and cosine 0.65, a MiniLM-family local embedder, and its usual chunking of exchange pairs, and the retrieval path contains no generative LLM at any point. The author argues that tuning for three particular questions does not amount to having a better baseline.

Beyond benchmark numbers, the post contrasts what each tool requires in order to run. The MemPalace default is said to be ChromaDB, which the author characterizes as a vector database dependency that brings a service, a client library, a storage format and associated failure modes. palace-rs is described as one Rust binary and one SQLite file, with embeddings computed locally, nothing to keep in sync, no extra process and no server to start — memory that can be copied or backed up as a single file. The author's framing is that on published numbers, the option that is simpler to run is not worse at retrieval.

The post then widens into a critique of how open-source tools get discovered. A star costs one click, the author writes, while a benchmark run costs days or weeks, yet star count decides what developers try first, which in turn decides which projects receive issues, contributors, tutorials and trust. The author lists consequences: careful new tools starve for attention; strong claims attract stars, which make the claims look more credible, without any measurement entering the loop; users inherit the cost when the popular option is heavier or less accurate; and unchecked superlatives stop needing to be true.

The author is also clear that none of this requires bad faith, noting that an honest claim made by a popular project can simply become outdated, or that a launch can go viral for reasons unrelated to engineering. That concession keeps the argument from being an accusation about any specific project's motives, and it is worth preserving: the post's target is the incentive structure around stars, not MemPalace's integrity.

The post proposes habits for making quality claims cheaper to check: publish reproducible commands, publish per-question outputs rather than only aggregates, separate tuned from held-out numbers, state whether a generative model is in the loop, report on the full standard question set, and attach a measurement and a date to any superlative. These are the author's recommendations, not findings from a study, and should be read as such.

The limitations section of the post is unusually direct. The author states that each project ran its own runner, so while both numbers are on the same 500 LongMemEval questions and the comparison is like for like on the data, anyone wanting to rule out runner differences would need to rerun both. LongMemEval S is one benchmark and says nothing about every kind of memory workload; the comparison covers the split both projects report, not a win everywhere. The two-point gap is described as enough to disprove that nobody is ahead, but not enough to call anyone dramatically better.

The author also flags that MemPalace reports a 99%-or-higher figure with an LLM rerank step, which is a different system with a generative model in the loop, and that the comparison here is retrieval without one. With the command and per-question outputs in the repository, the palace-rs run is said to cover all 500 questions, and on the same 500 is MemPalace's 96.6%. Readers are invited, in closing, to rerun either and report different numbers, and to check the public figures rather than trust the stars or the author.

For working developers choosing a memory layer for an agent or application, the practical takeaway is procedural rather than a product recommendation. The post's own checklist — what exactly was measured, what must be run to reproduce it, and whether the answer is on the page — is a reasonable filter to apply to any README carrying a benchmark claim, including this one. The operational contrast it draws, a single binary and SQLite file versus a vector database service, is the kind of setup cost that rarely appears in a star count but shows up in every deployment.

The tradeoffs cut both ways. A held-out number, as the author concedes, is the more honest measurement even when it is lower, so a project that publishes a tuned figure alongside a clean split is being more transparent than one that publishes only its best result. Conversely, a small project reporting all 500 questions with per-question outputs is offering auditability that a larger project may not. Neither practice is proof of quality on its own; both simply make the claim checkable.

What remains unknown is substantial. There is no independent replication in the supplied material, no third-party evaluation of either tool, and no evidence about performance on memory workloads outside LongMemEval S. The post does not address pricing, licensing, maintenance activity or long-term support for either project, and the star counts are the author's characterization rather than a verified snapshot. Readers should treat the 98.7% and 98.4% figures as claims by the respective projects, relayed by an interested party.

The broader point survives those caveats even if the specific numbers are later revised. If a two-point difference on a public benchmark can separate a ten-star repository from a sixty-thousand-star one, then star count is at best a noisy proxy, and the cost of checking a claim — a command, a raw output file, a stated question count — is low enough that developers can afford to check before adopting. The author's own instruction, not to trust the stars and not to trust the author either, is the correct way to read the post.