RAG Web Browser - Google Search to Markdown for AI

Eimantas V · @amethyst

Web search and fetch for AI agents and RAG: searches Google and returns the top results as clean Markdown with token counts, or reads any URL. No browser.

AIDeveloper toolsSEO tools

About

RAG Web Browser searches Google and returns the top results' pages as clean Markdown, ready for an LLM, a RAG pipeline or an AI agent. Give it a search, a list of searches, or any URL, and get one row per page with its content, metadata and the Google result it came from. No login, no API key, no browser, one flat monthly price.

# What is retrieval augmented generation (RAG)?

Published 21 October 2024

Updated 02 October 2026

By [Ivan Belcic](https://www.ibm.com/think/author/ivan-belcic) and

## What is retrieval augmented generation (RAG)?

Retrieval augmented generation, or RAG, is an architecture for optimizing the performance of an [artificial intelligence (AI)](https://www.ibm.com/think/topics/artificial-intelligence) model by connect...

That is the markdown of Google's result #2 for "what is retrieval augmented generation" (IBM), with tokenCount 3,987 and wordCount 2,309 beside it.

What does RAG Web Browser do?

It gives AI agents and LLM apps the two things a web browsing tool needs: search and read. A search goes to Google (any country, any language, any time range) and the top organic results are fetched in parallel, cleaned down to their main content and converted to Markdown. A URL is read directly. One call returns everything, so an agent spends one tool call instead of a search plus N page fetches.

  • Search or read: a Google search (with operators such as site:, -site:, "exact phrase", after:2026-01-01), a page URL, a bare domain like docs.apify.com, or a pasted Google search URL. One or thousands per run.
  • Main content only: menus, sidebars, footers, cookie banners and comment threads are removed, so the tokens you pay your LLM for are the article, the docs or the post. Switch to "Whole page" when you want everything.
  • Markdown, plain text, cleaned HTML and links: any combination, at the same price. Tables, code blocks with their language, nested lists and absolute links are kept.
  • Made for LLM context windows: every page comes with tokenCount and wordCount, and an optional character cap cuts each page on a paragraph boundary.
  • Always the number of pages you asked for: when a result cannot be read (a site that blocks automated visitors, a PDF, a dead link), the next Google result is read in its place.
  • Clear errors. Failed pages, skipped sites and empty searches return a row that says why.
  • Built for agents and automation: callable as a tool through the Bytewells MCP server, the Bytewells API and schedules.

Why it is fast and cheap

There is no browser. Google's lightweight results page is read over plain HTTP, and each result page is fetched with a real browser's network fingerprint, then parsed and converted in the same process. That is why one search with its pages takes about 3 seconds of run time, why the Actor runs on 1 GB instead of 8 GB, and why a page uses a fraction of the compute a browser needs. A site that does not answer within 4 seconds gets a second request from another IP address, and the first answer wins.

What data can you extract from the web?

FieldExampleWhat it is
markdown# What is RAG?\n\nRetrieval augmented...The page's content as Markdown (also text, html, links when chosen)
metadata.urlhttps://www.ibm.com/think/topics/retrieval-augmented-generationThe page's address after redirects
metadata.titleWhat is RAG (Retrieval Augmented Generation)? | IBMThe page's title
metadata.descriptionRetrieval augmented generation (RAG) is an architecture...Meta description
metadata.publishedAt2024-10-31T00:00:00.000ZPublication date (meta tags or JSON-LD), ISO 8601
metadata.author, siteName, languageCode, canonicalUrl, imageUrl, openGraphen, https://...The rest of the page's metadata
searchResult.position2The page's rank on Google
searchResult.title, description, displayedUrl, dateWhat is RAG...? - IBMThe Google result itself, with its snippet
tokenCount3987Estimated LLM tokens of the Markdown
wordCount2309Words in the content
contentScopemainmain when narrowed to the main content, page for the whole page
contentTruncatedfalseTrue when your character cap cut the content
crawl.httpStatusCode, loadTimeMs, contentType200, 1288, text/htmlHow the page loaded
inputwhat is retrieval augmented generationThe search or URL exactly as you entered it

How to search Google and get pages as Markdown

  1. Type a search or a URL into Search query or URL (more go under More queries or URLs, one per line).
  2. Set Results per search (3-5 is typical for an AI agent) and pick the Output formats.
  3. Click Start. One search with 3 pages takes a few seconds.
  4. Open the Output tab, or read the dataset as JSON.

How much does it cost?

One flat monthly rental, shown on this page, with a free trial. There is no per-page or per-search fee: read as many pages as you need.

Runs use your credits for compute and any proxy traffic, as every Actor does. Pages load from the platform's own address; Google's results page usually needs the residential proxy, about 10-40 KB a search. When Google refuses every address for a while, the Actor rests and retries the refused searches up to four times, 1-4 minutes apart, so such a run takes longer but uses little more than its idle compute.

Input

{
    "query": "best open source vector database",
    "maxResults": 5,
    "outputFormats": ["markdown"],
    "countryCode": "",
    "timeRange": "y",
    "maxCharsPerPage": 8000
}
FieldDefaultNotes
Search query or URL (query)what is retrieval augmented generationA Google search or a page URL; several lines are several queries
More queries or URLs (queries)noneAny number, one per line; duplicates are skipped
Start URLs (startUrls)nonePages in the URL-list shape other Actors output, for chaining
Results per search (maxResults)31-100; Google's further result pages are fetched as needed
Replace pages that fail (replaceFailedPages)onReads the next result in place of one that cannot be read
Output formats (outputFormats)MarkdownMarkdown, plain text, HTML (cleaned) and links, any combination
Content to keep (htmlTransformer)Main content onlyreadableText (main content) or none (whole page)
Max characters per page (maxCharsPerPage)0 (no cap)Cuts Markdown and text on a paragraph boundary
Remove HTML elements (removeElementsCssSelector)nav, footer, script, ...CSS selector of extra elements to drop
Search from country (countryCode)United StatesGoogle's gl: results as a person there sees them
Interface language (languageCode)the country'sGoogle's hl
Results from (timeRange)any timePast hour, 24 hours, week, month or year
Max pages per run (maxItems)1000Total across all queries
Time per query (requestTimeoutSecs)40 sPages still loading then become TIMEOUT rows, so an agent gets its answer in time
Retries per page (maxRequestRetries)1Each retry comes from a fresh IP address
Queries at a time (maxConcurrency)5Pages of one query are always fetched in parallel
Proxy for pages (proxyConfiguration)offOff: the Actor's own IP, with refused pages retried through the platform's datacenter proxy automatically. On: every page through datacenter proxy or your own proxy URLs
Debug log (debugMode)offLogs every request with its route, status and time

The input names (query, maxResults, outputFormats, htmlTransformer, removeElementsCssSelector, requestTimeoutSecs, maxRequestRetries, debugMode) and the output fields (searchResult, metadata, markdown, text, html, links, crawl) follow the common RAG web browser format, so an existing integration works unchanged.

Output

Each page read is one row. The Overview view shows query, rank, URL, title, size and Markdown; Search results shows the Google results; Page metadata shows dates and authors; Errors lists what could not be read. Every export includes all fields.

{
    "recordType": "item",
    "input": "what is retrieval augmented generation",
    "searchResult": {
        "position": 2,
        "title": "What is RAG (Retrieval Augmented Generation)? - IBM",
        "description": "Retrieval augmented generation (RAG) is an architecture for optimizing the performance of an artificial intelligence (AI) model by ...",
        "url": "https://www.ibm.com/think/topics/retrieval-augmented-generation",
        "displayedUrl": "www.ibm.com › think › topics › retrieval-augmented-generation",
        "date": "2024-10-21T00:00:00.000Z",
        "resultType": "ORGANIC"
    },
    "metadata": {
        "url": "https://www.ibm.com/think/topics/retrieval-augmented-generation",
        "title": "What is RAG (Retrieval Augmented Generation)? | IBM",
        "description": "Retrieval augmented generation (RAG) is an architecture for optimizing the performance of an artificial intelligence (AI) model by connecting it with external knowledge bases.",
        "languageCode": "en",
        "canonicalUrl": "https://www.ibm.com/think/topics/retrieval-augmented-generation",
        "author": null,
        "publishedAt": "2024-10-31T00:00:00.000Z",
        "modifiedAt": "2026-10-02T00:00:00.000Z",
        "siteName": null,
        "imageUrl": "https://www.ibm.com/content/dam/worldwide-content/stock-assets/adb-stk/ul/g/ce/5b/adobestock_451765072.jpeg/_jcr_content/renditions/cq5dam.web.1280.1280.jpeg",
        "openGraph": [{ "property": "og:type", "content": "website" }]
    },
    "markdown": "# What is retrieval augmented generation (RAG)?\n\nPublished 21 October 2024\n\nUpdated 02 October 2026\n\nBy [Ivan Belcic](https://www.ibm.com/think/author/ivan-belcic) ...",
    "wordCount": 2309,
    "tokenCount": 3987,
    "contentTruncated": false,
    "contentScope": "main",
    "crawl": {
        "httpStatusCode": 200,
        "httpStatusMessage": "OK",
        "loadedAt": "2026-10-29T13:42:45.777Z",
        "requestStatus": "handled",
        "loadTimeMs": 1288,
        "contentType": "text/html"
    },
    "scrapedAt": "2026-10-29T13:42:44.186Z"
}

A page or search that cannot be read becomes a row with "recordType": "error", the page's url, its searchResult and an errorCode, instead of failing the run:

errorCodeMeaning
BLOCKEDThe site refused automated visitors (HTTP 401/403/429 or a bot check) from every address tried, or is a site known to refuse them (then skipped without a request)
TIMEOUTThe site did not answer within the time per query
NOT_FOUNDThe page is gone (HTTP 404/410)
UNSUPPORTED_CONTENTNot a web page: a PDF, image, video, archive or office file
HTTP_ERROR / NETWORK / INVALID_URLThe site answered with an error, the connection failed, or the address is malformed
NO_RESULTSGoogle returned no web results for the search
RATE_LIMITEDGoogle refused the search from every address this run tried (retried up to four times at the end of the run)
PARSE_FAILEDThe page loaded but could not be read (please report it)

Tips and limits

  • Agents: keep context small. Set Max characters per page (8,000 characters is about 2,000 tokens) and 3-5 results per search. Use site: in the query to stay on documentation you trust.
  • No JavaScript is run. Pages that build their text in the browser (some single-page apps) return little content. Most news, docs, blogs, wikis and product pages are served as HTML and read fully. Plain Markdown and text files (llms.txt, README files) are returned as they are.
  • Some sites refuse automated visitors from cloud addresses: Reddit, Medium and Stack Overflow always do, so their results are skipped and replaced by the next result. Others (Cloudflare-protected sites) sometimes do; those pages are retried from another IP, then reported as BLOCKED.
  • PDFs and office files are not read. They are listed as UNSUPPORTED_CONTENT.
  • Web results only. Google's maps, news carousels and AI Overview are not part of the pages read; for full results-page data use the Google Search Results Scraper.
  • Google sometimes refuses searches for a few minutes. Refused searches are retried up to four times at the end of the run, 1-4 minutes apart, within the run's timeout; a search that is still refused gets a RATE_LIMITED row: run it again later. URLs you give directly do not depend on Google.

FAQ

Is it legal to scrape Google and websites?

This Actor reads only pages that are publicly available without logging in, the same pages a visitor's browser loads. Pages can still contain personal data, which laws such as the GDPR protect, and some sites' terms restrict automated access. Use the data for a legitimate purpose, respect copyright when you reuse content, and ask a lawyer if you are unsure. Google is a trademark of its owner; this Actor is not affiliated with or endorsed by Google.

Can AI agents use it?

Yes, that is what it is built for. Agents connected to the Bytewells MCP server can add it as a tool: one call searches, reads the pages and returns Markdown with token counts. The input takes a single query, so an agent needs no other parameters.

Can I use it through the API?

Yes. Every run can be started through the Bytewells API (the run-sync endpoint returns the pages in the same call) and its dataset read as JSON. The API is Apify-compatible, so apify-client works with the Bytewells base URL. Runs can also be scheduled and can trigger webhooks.

Can I switch from another RAG web browser?

Yes. The input fields query, maxResults, outputFormats, htmlTransformer, removeElementsCssSelector, requestTimeoutSecs and maxRequestRetries, and the output fields searchResult, metadata, markdown, text, html, links and crawl, use the common format. Additions are on top: batch queries, country, language and time range, token counts, the character cap and page replacement.

Why did I get fewer pages than I asked for?

Google had fewer results than you asked for, every remaining result failed (the error rows say why), the time per query ran out, or Google kept refusing the search through every retry. The log says which, and nothing collected before that is lost.

Related scrapers

Support

A page that comes back empty or wrong, or a site that should work? Report it with the input you used, and it will be looked at quickly.

FAQ

What does RAG Web Browser return?
For a search: the top Google results, each fetched and cleaned down to its main content, as Markdown, plain text, cleaned HTML or links, with title, description, URL, token count and word count. A URL or bare domain is read directly.
Can AI agents use it?
Yes, it is built for them. Agents connected to the Bytewells MCP server (bytewells.com/docs/mcp) can add it as a tool: one call searches, reads the pages and returns Markdown. The input takes a single query, so an agent needs no other parameters.
How do I keep the context small for my LLM?
Set Max characters per page (8,000 characters is about 2,000 tokens) and ask for 3-5 results per search. Use site: in the query to stay on sources you trust. Every page comes with its token count.
Does it run JavaScript?
No. Pages that build their text in the browser return little content; most news, docs, blogs, wikis and product pages are served as HTML and read fully. PDFs and office files are not read.
Why are some results skipped?
Some sites refuse automated visitors from cloud addresses: Reddit, Medium and Stack Overflow always do. Those results are replaced by the next Google result, so you still get the number of pages you asked for.
Can I switch from another RAG web browser?
Yes. The input fields query, maxResults, outputFormats, htmlTransformer, removeElementsCssSelector, requestTimeoutSecs and maxRequestRetries, and the output fields searchResult, metadata, markdown, text, html and links use the common format.
Can I use it through the API?
Yes. Start runs through the Bytewells API (bytewells.com/docs/api) and read the pages as JSON; schedules and webhooks work too. The API is Apify-compatible, so apify-client works with the Bytewells base URL.

Related

More Actors from the same publisher.

View all from @amethyst