Show HN: Haystack – Review the PRs that need human attention

hckrnws

Show HN: Haystack – Review the PRs that need human attention

by akshaysg

Hey HN! We're building Haystack (https://haystackeditor.com/) to help teams deal with the explosion in the number of pull requests that need to be reviewed due to the rise of coding agents.

Haystack replaces the GitHub PR review system with a queue that triages each PR before a human has to read any diffs. It looks at the diffs, the codebase, and the coding-agent conversation that produced the PR. Haystack then routes it into one of three buckets:

1. Safe to merge. This means the PR has enough evidence behind it that the team can merge it without another human's review.

Some examples:

-- A small UI copy change that includes a screenshot showing the final state

-- A backend change where the author clearly tested the important paths and ran the changes in a real environment

2. Needs fixes. This means that the PR has bugs or violates a rule in your codebase and therefore the PR needs to be fixed by the author.

Some examples:

-- The agent was asked to make loading a large table faster by adding pagination, but the PR still loads every result at once and "implements" pagination in the UI

-- The PR silently catches an error instead of logging, surfacing, or handling it. This violates the team's "no silent error swallowing" rule

3. Needs human review. This means that the PR could not be sufficiently verified by the author or is touching a sensitive part of the codebase (determined by user-input guidelines) and thus requires human review.

Some examples:

-- The PR changes a significant amount of logic in billing

-- The PR changes an important user flow like onboarding, but the author only ran unit tests and never opened the app to check the flow end-to-end. That violates the team's rule that high-impact user-facing changes need manual verification.

Instead of starting with line-by-line diffs, Haystack immediately tells the reviewer the goal behind the PR, what design decisions the author made (informed by their coding-agent conversation), and how much the author did to verify that the pull request works (e.g. run scripts, checked the frontend, etc.).

In this way, review shifts from "what changed?" to "is this the right behavior and is there evidence that it works?".

Here's a quick demo: https://www.tella.tv/video/streamlining-code-reviews-with-ha...

We previously launched Haystack as a tool for understanding large PRs (https://news.ycombinator.com/item?id=45201703). As many of you can probably relate to, the release of Opus 4.5 completely shattered our conception of how fast an engineer could craft a PR.

And as coding agents got even better from 4.5, we realized that pull requests did not scale along with our coding velocity. With each member of our team being able to pump out more than 20 pull requests a day, code review quickly became cognitively exhausting and less helpful.

After talking with other folks, we learned many feel similarly, and currently face the binary option of either not doing review at all or trying to keep up with a fire hose of pull requests.

Haystack is our attempt at a third path. We still believe in code review, but as coding agents produce more code, human reviewer attention becomes more valuable and more expensive.

Haystack helps teams spend that attention on the PRs where a human can meaningfully change the outcome of that PR. And for such PRs, Haystack shows the reviewer what the PR intended to do, whether the author showed that it works, and what design decisions need a second pair of eyes.

We're still quite early and are figuring out whether Haystack truly makes code review better. We would love any and all feedback!

oersted
64d
Man, it's such a shame that you pivoted away from the canvas-based editor concept, it was such a pleasure to use, it's so much better than tabs.
https://github.com/haystackeditor/haystack-editor
It's still probably the best tool to navigate, visualize and understand complex codebases, which is particularly important now with AI coded repos. I keep looking for alternatives but they are all notably worse.
About a month ago I spent a few frustrating hours building it from source for my system and making it work, and I've enjoyed using it as my main IDE since.
I wish I had the time to make a fork and bring in a newer version of VSCode. If anyone takes it up I might help at least.
1. akshaysg • OP
  34d
  Yeah I agree it's a shame. Unfortunately coding has changed fast and I was not confident that the editor was in the correct direction with AI coding becoming so prevalent.
  I think there is a lot of value with "reconnecting" with your codebase, so I do have some plans to bring the core concept of Haystack back in one form or another.
  1. wsve
    23d
    I am very confident that editor is the direction I want coding to go. I really encourage you to consider picking it up again. Debugging complex graphs of function calls is not going away quite yet.
    akshaysg • OP
    13d
    If you see my reply, I would absolutely love it if you could email akshaysg@haystackeditor.com with some details about your desired use cases!
2. visekr
  14d
  100% - tried forking VSCode too, scratched it. Way too much work to restyle, and I started wondering if IDEs should even be code-focused anymore. Built https://getmesa.dev instead, a canvas-based IDE for Claude Code & other CLI Agents. Kinda stuck on direction though - lmk if you've got ideas or wanna help out : )
  1. tikimcfee
    04d
    I'd certainly love to help! I've also been working on a canvas for building agentic flows and workspaces, and I'm actually at an intersection to what you're doing with that application you're showing off. I have a simpler node based stream system for flight control of arbitrary agent actions, meant to be operated by one or more orchestrating agents. Eg, the agent understands the surface it can create nodes with, generate dynamic tools and functions within the node specs, and then write graphs of them together so it's live and observable. Extremely intersectional, haha.
    Happy to chat whenever you'd like, I can stalk a contact for ya or vice versa!
eddy-sekorti
13d
Congratulations, this is a nice concept, this is actually usefull for large corporates with distibuted teams. The only issue is it is very hard these days to sell developer tools to enterprises, every tool any corporate is using these days are trying to inflate their offering with AI stuff. But if you can close them, definately this will be usefull. One more thing, your pricing is quite aggressive at 30/seat.
1. akshaysg • OP
  03d
  Thank you for the kind words!
  RE Pricing: Because of the high volume of our customers, this is quite necessary unfortunately. We can definitely shift down costs as existing intelligence becomes cheaper though.
luplex
04d
Haystack is a popular Framework for building AI agents: https://haystack.deepset.ai/
vforno
03d
Yes, I think that having the code completely checked by AI is not a good idea, but an AI that says, "Check these," because they are noteworthy. This is my idea of a future where I hope AI is like the movie "Limitless." That is, it supports you in improving yourself and giving you greater capabilities, not in replacing you entirely.
ramon156
14d
I like this idea. To be blunt, would this have more features than hooking up Claude/Gemini/Codex and saying "If - at any point - you're unsure, step back and let a human review"
1. akshaysg • OP
  04d
  I think Haystack offers:
  1. A centralized review mechanism for a team or org that operates on coding agent conversations in addition to diffs (and the codebase). It evaluates multiple different variables (e.g. how sensitive are the changes, how much did the author do to derisk, and what did the author's coding agent gloss over) and helps enforce your team's guidelines moreso than just an individual's prompt
  2. Adversarial review that operates in addition to other AI review agents (e.g. BugBot, or Greptile) and filters any comments to only the things the author cares about. This helps cut down on the "AI reviewer battleground" that is present in pull requests
  3. A review interface that allows human reviewers to quickly understand what the author did to verify their changes and focus on the author's design decisions
  We actually jury-rigged all of this together before building Haystack, but found that it doesn't scale to the team level (since every individual has their own ideas/opinions of what constitutes a human review).
  We also found that reviewing through purely Claude Code/Codex was slow and difficult because stuff like author traces are not pre-processed and you have to get your agent to specifically explore/understand them.
isaisabella
23d
Let AI review the code written by AI? interesting...
1. akshaysg • OP
  03d
  I think that AI coding is only going to grow in usage. I also think that AI more-or-less writes working code that is faithful to the user's prompt.
  At the same time, AI does not write code that's easy for humans to review, as it writes very verbosely and with the goal of getting the job done vs producing readable code.
  I really don't think humans should be reviewing the absolute mountains of LLM-produced code; doing so would only exhaust someone's cognitive budget.
  Therefore, I think that bug finding should be left to AI to review, while architectural decisions should be reviewed by humans.
softwaredoug
04d
Just to say great idea. But the name "Haystack" is used by several dozen things FWIW :)