Prefer research agents. Do not love when they cite a search result that never existed, quote a paragraph that was never on the page, or invent a competitor feature list that sounds right until you click.
The failure mode is always the same. The model is asked to “research” without being forced to ground on fetched text. So it improvises. Confidently. With headings.
Scrapers plus LLMs work when the scraper is the source of truth and the model is the analyst. Flip that and you get fan fiction about the SERP.
The pipeline you trust
- Query plan from a human brief (what we need to learn, not “go learn everything”)
- Fetch search results from a real search API or a controlled scrape
- Fetch the chosen pages
- Extract main content cleanly
- Store raw text with URL, timestamp, and hash
- Ask the model to summarize *only from that store*
- Require citations that map to stored chunks
- Human spot-check before anything becomes strategy
If step 6 can see the open web only through its training vibes, you already lost.
Force citations or throw the answer away
My research outputs need links. Not decorative links. Links that match the URLs we fetched in this run.
If the model cites a URL that is not in the corpus, the answer is invalid. Retry or fail. Do not shrug and publish.
Same for quotes. If it puts text in quotes, that text should appear in the fetched content. Fuzzy paraphrase is fine when labeled as paraphrase. Fake quotes are not.
Separate “what the page says” from “what we conclude”
Analysts blur these. Models blur them harder.
Ask for two blocks:
- Evidence: bullets with citations
- Interpretation: what it might mean for us, clearly labeled as ours
That keeps a competitor’s marketing claim from becoming “fact” in your doc because the model repeated it smoothly.
Scraping ethics and practicality
Respect robots rules and terms. Prefer official APIs when they exist. Rate-limit. Cache. Do not hammer small sites like a villain.
Also accept that some pages will block you. A blocked fetch is not proof the page is empty. It is proof you need another path: API, manual save, or skip.
For SEO research, I care more about a smaller set of high-quality SERP captures than a giant pile of partial HTML and guesswork.
Eval the liar behaviors
Build a tiny suite of known pages. Ask questions with known answers. Ask questions the page does not answer and make sure the agent says it does not know.
Reward “insufficient evidence.” Punish confident invention.
This is the same mindset as agent evals for customer bots. Research agents just lie in footnotes instead of phone calls.
Where this pays off
- competitive page outlines grounded in what rivals actually published
- content gap lists tied to real ranking pages
- weekly change reports (“this URL’s H1 and primary CTA changed”)
- briefing memos for humans who still make the call
It does not replace judgment. It replaces twenty browser tabs and a messy notes doc.
Keep humans on the strategy call
you will let an agent draft the competitive matrix. Do not let it decide we should chase a keyword that would attract the wrong patient mix or a margin we cannot serve.
Research agents accelerate homework. Operators still pick the fight. If a memo cannot show the URLs it used, it does not enter the planning meeting.
Timestamp everything
SERPs change. Pages change. A memo without fetch timestamps becomes folklore by next quarter. Keep the run id, query list, and capture time with every research packet so we can replay what the agent actually saw.
Bottom line
A research agent that cannot show its homework will eventually invent homework.
Fetch first. Analyze second. Cite always. Separate evidence from opinion. Spot-check before the memo becomes gospel.
Do that and scrapers plus LLMs become a real research assistant. Skip it and you have a storyteller with a search costume.
