{
  "version": "2",
  "id": "https://freelancenews.online/news/author-of-a-small-memory-tool-says-it-edged-a-60-000-star-rival-on-70cccb94",
  "title": "Author of a Small Memory Tool Says It Edged a 60,000-Star Rival on One Benchmark",
  "summary": "In a dev.to post, the developer behind palace-rs reports 98.7% R@5 on the standard 470-question LongMemEval set against MemPalace's published 98.4% hybrid figure, and argues star counts mislead tool selection. Both numbers come from each project's own runner and have not been independently verified.",
  "body": "A developer posting on dev.to under the handle icegaming has published a comparison arguing that GitHub stars are a poor proxy for tool quality, using their own project as the test case. The post, titled \"Stars Aren't Benchmarks: How a 10-Star Memory Tool Out-Scored the 60k-Star One,\" sets palace-rs, which the author describes as having about ten stars, against MemPalace, which the author says has well over 60,000 stars and whose README calls it the best-benchmarked open-source AI memory system. The author states plainly that the comparison rests on each project's own published or self-run numbers rather than a third-party audit, and invites readers to rerun either side.\n\nThe central figure is a retrieval score on LongMemEval, a memory benchmark the post links to. The author reports that on the standard 470-question set — the 500-question s_cleaned split minus 30 abstention questions that the benchmark is designed to exclude from retrieval scoring — palace-rs reaches R@5 of 0.987, or 98.7%. That is placed against MemPalace's 98.4% hybrid number. The author also reports an all-500 figure of 98.6% R@5 for palace-rs against a 96.6% raw figure, explicitly so that excluding the abstention questions cannot be said to flatter the result.\n\nThe post is careful about the size of the gap. On 500 questions, the author notes, two points amounts to ten questions, and they decline to call it a breakthrough. The narrower claim is that a project with about ten stars published the higher number and that nobody would have learned this from the star count. That framing matters for how the piece should be read: it is an argument about discovery signals, illustrated with a small measured difference, not a claim of dramatic superiority.\n\nThe author gives MemPalace credit for how it handled its own tuning. According to the post, MemPalace's hybrid mode adds targeted boosts for quoted phrases, person names and nostalgia-style phrasing, and those fixes were developed by examining three specific questions out of the 500. The author calls that a normal way to improve a retriever but says fixing failures visible in the test set stops being a clean measurement, and notes that MemPalace publishes the held-out 450-question result as its honest number — which the author describes as the right and transparent call.\n\nThe author says, by contrast, that palace-rs reports all 500 questions, and that no retrieval parameters were fitted on the test set: weights, chunking and scoring were set independently of those questions. According to the description, the palace-rs run used hybrid search with BM25 weighted 0.35 and cosine 0.65, a MiniLM-family local embedder, and its usual chunking of exchange pairs, and the retrieval path contains no generative LLM at any point. The author argues that tuning for three particular questions does not amount to having a better baseline.\n\nBeyond benchmark numbers, the post contrasts what each tool requires in order to run. The MemPalace default is said to be ChromaDB, which the author characterizes as a vector database dependency that brings a service, a client library, a storage format and associated failure modes. palace-rs is described as one Rust binary and one SQLite file, with embeddings computed locally, nothing to keep in sync, no extra process and no server to start — memory that can be copied or backed up as a single file. The author's framing is that on published numbers, the option that is simpler to run is not worse at retrieval.\n\nThe post then widens into a critique of how open-source tools get discovered. A star costs one click, the author writes, while a benchmark run costs days or weeks, yet star count decides what developers try first, which in turn decides which projects receive issues, contributors, tutorials and trust. The author lists consequences: careful new tools starve for attention; strong claims attract stars, which make the claims look more credible, without any measurement entering the loop; users inherit the cost when the popular option is heavier or less accurate; and unchecked superlatives stop needing to be true.\n\nThe author is also clear that none of this requires bad faith, noting that an honest claim made by a popular project can simply become outdated, or that a launch can go viral for reasons unrelated to engineering. That concession keeps the argument from being an accusation about any specific project's motives, and it is worth preserving: the post's target is the incentive structure around stars, not MemPalace's integrity.\n\nThe post proposes habits for making quality claims cheaper to check: publish reproducible commands, publish per-question outputs rather than only aggregates, separate tuned from held-out numbers, state whether a generative model is in the loop, report on the full standard question set, and attach a measurement and a date to any superlative. These are the author's recommendations, not findings from a study, and should be read as such.\n\nThe limitations section of the post is unusually direct. The author states that each project ran its own runner, so while both numbers are on the same 500 LongMemEval questions and the comparison is like for like on the data, anyone wanting to rule out runner differences would need to rerun both. LongMemEval S is one benchmark and says nothing about every kind of memory workload; the comparison covers the split both projects report, not a win everywhere. The two-point gap is described as enough to disprove that nobody is ahead, but not enough to call anyone dramatically better.\n\nThe author also flags that MemPalace reports a 99%-or-higher figure with an LLM rerank step, which is a different system with a generative model in the loop, and that the comparison here is retrieval without one. With the command and per-question outputs in the repository, the palace-rs run is said to cover all 500 questions, and on the same 500 is MemPalace's 96.6%. Readers are invited, in closing, to rerun either and report different numbers, and to check the public figures rather than trust the stars or the author.\n\nFor working developers choosing a memory layer for an agent or application, the practical takeaway is procedural rather than a product recommendation. The post's own checklist — what exactly was measured, what must be run to reproduce it, and whether the answer is on the page — is a reasonable filter to apply to any README carrying a benchmark claim, including this one. The operational contrast it draws, a single binary and SQLite file versus a vector database service, is the kind of setup cost that rarely appears in a star count but shows up in every deployment.\n\nThe tradeoffs cut both ways. A held-out number, as the author concedes, is the more honest measurement even when it is lower, so a project that publishes a tuned figure alongside a clean split is being more transparent than one that publishes only its best result. Conversely, a small project reporting all 500 questions with per-question outputs is offering auditability that a larger project may not. Neither practice is proof of quality on its own; both simply make the claim checkable.\n\nWhat remains unknown is substantial. There is no independent replication in the supplied material, no third-party evaluation of either tool, and no evidence about performance on memory workloads outside LongMemEval S. The post does not address pricing, licensing, maintenance activity or long-term support for either project, and the star counts are the author's characterization rather than a verified snapshot. Readers should treat the 98.7% and 98.4% figures as claims by the respective projects, relayed by an interested party.\n\nThe broader point survives those caveats even if the specific numbers are later revised. If a two-point difference on a public benchmark can separate a ten-star repository from a sixty-thousand-star one, then star count is at best a noisy proxy, and the cost of checking a claim — a command, a raw output file, a stated question count — is low enough that developers can afford to check before adopting. The author's own instruction, not to trust the stars and not to trust the author either, is the correct way to read the post.",
  "category": "dev",
  "language": "en",
  "datePublished": "2026-11-09T12:18:43.252Z",
  "dateModified": "2026-11-09T12:18:43.252Z",
  "eventDate": null,
  "sourcePublicationDate": "2026-11-09T12:00:37.000Z",
  "source": {
    "name": "dev.to",
    "url": "https://dev.to/icegaming/stars-arent-benchmarks-how-a-10-star-memory-tool-out-scored-the-60k-star-one-42mk",
    "kind": "community"
  },
  "practicalImpact": "Editorial interpretation: developers evaluating memory tools should ask for the exact question set, the run command and per-question outputs before trusting a README benchmark or a star count, and should weigh setup cost — a single binary and file versus a vector database service — since that affects every deployment. The figures here are self-reported by an interested party and have not been independently replicated.",
  "limitations": "Both benchmark numbers come from each project's own runner, as the author states; no independent replication or third-party evaluation is in the supplied evidence. LongMemEval S is one benchmark and does not cover all memory workloads. The two-point gap is small. MemPalace's 99%-plus figure with an LLM rerank is a different system and is not compared here. Star counts are the author's characterization, not a verified snapshot. No pricing, licensing or maintenance information is provided for either tool.",
  "keyPoints": [
    "The author of palace-rs reports 98.7% R@5 on the standard 470-question LongMemEval set, against MemPalace's published 98.4% hybrid figure, and 98.6% on all 500 questions against a 96.6% raw figure.",
    "The author says MemPalace's hybrid boosts were developed by looking at three specific questions and credits it for publishing a held-out 450-question number as its honest result.",
    "palace-rs is described as one Rust binary and one SQLite file with local embeddings and no generative LLM in retrieval, versus MemPalace's default ChromaDB dependency.",
    "The author concedes each project ran its own runner, that LongMemEval S is a single benchmark, and that a two-point gap is small.",
    "The post recommends reproducible commands, per-question outputs, separated tuned and held-out numbers, and dated measurements behind superlatives."
  ],
  "review": {
    "status": "source-reviewed",
    "checkedAt": "2026-11-09T12:18:43.252Z",
    "method": "Automated comparison against retrieved source text; not independent fact-checking.",
    "correctionNote": null
  },
  "sources": [
    {
      "id": 1,
      "url": "https://dev.to/icegaming/stars-arent-benchmarks-how-a-10-star-memory-tool-out-scored-the-60k-star-one-42mk",
      "publisher": "dev.to",
      "title": "Stars Aren't Benchmarks: How a 10-Star Memory Tool Out-Scored the 60k-Star One",
      "publishedAt": 1794225637000,
      "fetchedAt": 1794226678179,
      "hash": "043e36d6253398795d8057ea0dca5e6928e95b5cbddcd474cd695d164689c95f",
      "kind": "community"
    }
  ],
  "claims": [
    {
      "claim": "The author reports palace-rs scoring R@5 0.987 on the standard 470-question LongMemEval set, above MemPalace's 98.4% hybrid figure.",
      "source": 1,
      "id": "claim-1",
      "url": "https://freelancenews.online/news/author-of-a-small-memory-tool-says-it-edged-a-60-000-star-rival-on-70cccb94#claim-1"
    },
    {
      "claim": "The author states MemPalace's hybrid boosts were developed by examining three specific questions and that it publishes the held-out 450 as its honest number.",
      "source": 1,
      "id": "claim-2",
      "url": "https://freelancenews.online/news/author-of-a-small-memory-tool-says-it-edged-a-60-000-star-rival-on-70cccb94#claim-2"
    },
    {
      "claim": "palace-rs is described as one Rust binary and one SQLite file with local embeddings, while MemPalace defaults to ChromaDB.",
      "source": 1,
      "id": "claim-3",
      "url": "https://freelancenews.online/news/author-of-a-small-memory-tool-says-it-edged-a-60-000-star-rival-on-70cccb94#claim-3"
    },
    {
      "claim": "The author concedes each project ran its own runner and that LongMemEval S is a single benchmark.",
      "source": 1,
      "id": "claim-4",
      "url": "https://freelancenews.online/news/author-of-a-small-memory-tool-says-it-edged-a-60-000-star-rival-on-70cccb94#claim-4"
    },
    {
      "claim": "The author notes MemPalace reports a 99%-or-higher figure with an LLM rerank step, a different system not compared here.",
      "source": 1,
      "id": "claim-5",
      "url": "https://freelancenews.online/news/author-of-a-small-memory-tool-says-it-edged-a-60-000-star-rival-on-70cccb94#claim-5"
    },
    {
      "claim": "The post recommends publishing reproducible commands and per-question outputs and separating tuned from held-out numbers.",
      "source": 1,
      "id": "claim-6",
      "url": "https://freelancenews.online/news/author-of-a-small-memory-tool-says-it-edged-a-60-000-star-rival-on-70cccb94#claim-6"
    }
  ],
  "formats": {
    "html": "https://freelancenews.online/news/author-of-a-small-memory-tool-says-it-edged-a-60-000-star-rival-on-70cccb94",
    "markdown": "https://freelancenews.online/news/author-of-a-small-memory-tool-says-it-edged-a-60-000-star-rival-on-70cccb94.md",
    "json": "https://freelancenews.online/news/author-of-a-small-memory-tool-says-it-edged-a-60-000-star-rival-on-70cccb94.json"
  }
}