Search that shows its work

Deep search: find, read and DIGEST AN ANSWER. The fifth tool of the module and the only one that answers a question rather than handing back material. WHEN TO CALL. The question requires several sources to be brought together: what is happening with something, how one thing differs from another, what the figures of a specific organisation are. The tool composes the queries itself, reads the pages and writes an answer with references to the sources. WHEN NOT TO CALL. You need a list of links — web_search is tens of times cheaper. You need the text of a known page — web_read. This tool spends a model and minutes; call it on a question, not on a query. WHAT IT RETURNS. answer — the digested answer with [1]-style references; sources[] — the pages that were read; markers — the features used to check that the pages are about THE SUBJECT ASKED ABOUT; timing_ms — where the time went (searching, reading, the model); usage.by_model — tokens per model, with money left to whoever holds the price registry. HOW TO READ THE ANSWER — five things. 1. THE MAIN FIELD IS `outcome`, NOT `answer`. Five values: found — the markers met on a page; ambiguous — the sources hold SEVERAL DIFFERENT subjects under this name, and they are listed in ambiguity.variants; off_target — material was found but about ANOTHER subject (a namesake, a different city); not_found — we looked and no source came back; not_attempted — the search never ran (no model configured, or an empty question), which is NOT the same as finding nothing; unknown — there were no markers, so there was nothing to check with. On off_target the answer looks convincing and is about the wrong thing. On ambiguous the answer applies to THE LARGEST GROUP and not to all of them: the other variants are real, and if one of them is wanted, ask the person or refine the question rather than choosing yourself. 2. summarised_from_on_target says whether the answer was digested from verified pages or from whatever was found. False means read the answer as a draft. 3. stopped_because and waves_done show HOW MUCH work was done. A full answer and a short one look alike; this is the only place they can be told apart. 4. THREE NUMBERS ABOUT SOURCES, AND THEY ARE DIFFERENT. sources_total — how many were found; sources_with_content — how many could be read (a block returns zero characters and stays in the list); sources_on_target — on how many the markers met. The answer stands on the third number and sounds weighty because of the first. 5. sources_confirmed_2plus and confirmed_by_engines count INDEPENDENCE, not correctness: how many different engines found the same link. On an ambiguous name the most corroboration goes to the best-indexed namesake. If the answer looks confident while the question admits several different subjects under one name, look at `outcome` first: `ambiguous` means the tool composed exactly that answer — the variants are in ambiguity.variants, and the digest applies to the largest group only. Corroboration counts do NOT decide between them. 6. plan_trouble is about US AND THE MODEL, not about the search: empty means the plan parsed, non-empty means a wave ran on a fallback plan (the question itself) because the model's answer could not be read. Invisible in the sources, so it is said here.

Image search through our own metasearch layer. The third tool of the module: web_search finds pages, web_read extracts their content, this one finds IMAGES. WHEN TO CALL. You need a picture of an object, a product, a building, a person, a diagram. You need the address of the image file itself rather than of a page about it. WHEN NOT TO CALL. You need text about the object — that is web_search. You need the content of a specific page — web_read. This tool does NOT look at the pictures and does not describe them: it finds addresses, and whoever can see looks at them. WHAT IT RETURNS, by default nine fields: results[], engines_asked, engines_answered, all_engines_on_topic, trouble, contract, ok, error, cause. `cause` is empty on success and names whose fault a refusal is: `caller` (fix the request, repeating it will not help) or `upstream` (the metasearch failed; retrying later may work). Each result carries image_url (the file itself), page_url (the page it was found on), domain (the site the image is SOURCED from), page_domain, thumbnail, title, author, published, via. HOW TO READ THE ANSWER — four things. 1. AN IMAGE HAS TWO ADDRESSES and they must not be confused: image_url is the file, page_url is the page. Showing the page instead of the image is a mistake only a human notices. 2. AN EMPTY LIST IS A SUCCESS, not a failure: we looked and found nothing. A failure arrives with ok=false and a reason. 3. TROUBLE IS THE FIELD TO BRANCH ON, and it is empty when nothing went wrong. Inside it: engines_irrelevant names engines whose results were discarded ENTIRELY as not being about the query — with images this is common, an engine returns its own catalogue regardless of the query and looks excellent by result count; unresponsive_engines, asked and silent; search_aborted with engines_unasked; pool_unmeasured, meaning the image pool was never computed from observation; arguments_adjusted. 4. all_engines_on_topic IS NOT the trust label of web search. There are no references for the image category yet, so no engine here carries one; this flag says only that nothing answered off topic. Set verbose=true for the per-engine accounting: count, query, page, engines_skipped, pool_source.

Read a web page by address and return its text. HOW IT DIFFERS FROM web_search, WHICH ALSO READS. That one reads the top three links of its own results at 6000 characters each — enough for an answer. This one takes the addresses YOU name, up to five at a time, reads them in full with a cursor over a long document, can demand the browser and can check for a marker. If you need an answer, search is enough; if you need to work with a document, come here. WHEN TO CALL. You need the text of a specific page whose address is already known — from web_search results or from the user. You need facts from an article rather than a snippet about it. Read a long page in parts: the same call with the offset named at the end of the truncated text. WHEN NOT TO CALL. There is no address yet — use web_search first. You need an office document (DOCX, XLSX) — the tool does not parse those and will say so plainly; PDF, however, IS read. A search-engine result page must not be read: it merges neighbouring results into one text and hands you facts about a namesake. WHAT IT RETURNS. results[] per address: content — the page text, status — what became of it, title, published, lang, final_url (where a redirect led), stub_check — whether this is a block; text_source — HOW the text was obtained. HOW TO READ THE ANSWER — four things that are easy to get wrong. 1. EMPTY CONTENT IS A SUCCESS, not a failure: the page opened and has no text in it. Repeating is pointless, take another source. On a refusal or a failure to open, repeating does make sense. 2. `ok` is about the TOOL, not about the pages: it stays true even if not one page was read. Look at count and failed, and at the status of each address: `read`, `empty`, `stub`, `refused`, `unreachable`, `forbidden`, `not_reached`. They mean different things and call for different next steps — `empty` is not worth repeating, `refused` and `unreachable` are; `not_reached` is news about US (out of time, the per-domain rate limit, the browser busy — its reason says so, retry later — or beyond the batch ceiling) and says nothing about the page. 3. text_source IS REQUIRED READING when it says the text was recognised. That is a scanned PDF with no text layer: the pages were rendered and read by a vision model, and such text MUST NOT be quoted as exact — a measurement recovered 94% of the reference numbers. The details are in the `recognition` field, including the dpi and whether the model's answer was cut off. A text layer means "copied out of the file" and is quotable verbatim. 4. A stub status means we met an anti-bot shield or a paywall: text arrived, but it is not from the page. Do not retell it as the content. A stub_check of "not checked" is NOT "clean". PDF. It is read; `pages` and `pages_read` say how many pages the document has and how many were parsed. An empty result on a PDF means "there are pages and no text" — that is a scan, cured by recognition rather than by repeating. And remember: in a PDF the characters can be extracted correctly while the reading order falls apart, so labels come away from their values. Do not assemble "property: value" pairs out of adjacent PDF lines without checking that they really are adjacent.

A PNG screenshot of a web page. The fourth tool of the module. WHEN TO CALL. You need to SHOW a page to a person — the layout, the design, what the text does not carry. And you need to CROSS-CHECK: the shot and the text are obtained in one browser visit but by different routes — the pixels are drawn by the layout engine, the text comes from the DOM. A disagreement between them catches what neither route sees alone. WHEN NOT TO CALL. You need the text of the page — that is web_read, many times cheaper. A screenshot costs a browser launch. WHAT IT RETURNS. png_base64 — the shot itself; bytes — its size; page_text — the text of THE SAME visit, up to max_chars; page_text_chars — the length of the whole text, which may be greater; page_text_truncated; browser_version. HOW TO READ THE ANSWER — two things, and they are DIFFERENT. 1. shot_taken — THE SHOT WAS TAKEN: the browser is alive, the page loaded. 2. expected_found — WHAT WE EXPECTED IS ON THE PAGE (when `expect` was given). A shot can be taken flawlessly and show the wrong thing: a stub, a captcha, an error page. Do not confuse these two fields — a page that honestly failed a check and a browser that never opened are different events. A signal worth seeing: there is a shot and page_text_chars is near zero — the page drew, and has nothing to say.

Web search through our own metasearch layer over several independent search engines. Returns links together with information about WHO found them and how far that source can be trusted. WHEN TO CALL. You need fresh information from the web; you need to find an organisation or a person by name. For the second and third page of results, use the same call with page=1, 2 and so on; pages are numbered from zero. TO CHECK A FACT AGAINST INDEPENDENT SOURCES, ASK FOR IT. By default the sweep STOPS at the first engine that gave enough links — that is the cheap path, and one engine is one witness. `min_engines: 3` keeps asking, and only then do corroborated_by_url and corroborated_by_domain count anything. It costs several times the outbound requests. IT READS BY DEFAULT, AND THAT IS THE COST OF THE CALL. The top three links are fetched and their text arrives in `content`, so no second call is needed — measured on a live door: about 12 400 characters and 4.8-10.5 seconds. With read=false: about 2 800 characters and 0.7 seconds. CHOOSE BEFORE YOU CALL, not after the bill. Need the text of the top results — leave it as is. Need to see WHAT EXISTS on a question, or working under a narrow context ceiling — read=false, and fetch what you actually want with web_read. `read_top` sets how many pages are read (0 means "no preference", not "none": for none, use read=false). WHEN NOT TO CALL. You need the text of a KNOWN page (you already have the address) — that is web_read, which reads up to five addresses and offers a cursor over a long document. You need a finished ANSWER across several sources rather than material — that is web_deep_search. Not suitable for searching inside a known document or repository. HOW IT DIFFERS FROM web_deep_search. Here there is ONE pass: what was found is what was read, and you compose the answer yourself from `content`. There the tool composes the queries itself, goes in waves and returns a DIGESTED ANSWER together with what it failed to find. This one hands you material, that one hands you a judgement. WHAT IT RETURNS, by default ten fields: results[], engines_asked, engines_answered, engines_trust, all_engines_clean, trouble, contract, ok, error, cause. WHEN IT REFUSES, `cause` SAYS WHAT TO DO NEXT, and it is empty on success. `caller` — fix the request: an engine name that does not exist, a query that is not text; repeating it fails the same way. `upstream` — the metasearch behind us failed; retrying later may work. `busy` — WE are at capacity (every engine cooling or inside our pacing); wait `retry_after_s` seconds, then retry — that field appears only then. On the plain door the same three are 400, 502 and 429 with a Retry-After header. Each result carries title, url, snippet, domain, content (the page text for the ones that were read), chars, read_status, stub_check, text_source, via — which engines found this particular link — query_words_here — WHICH WORDS OF YOUR QUERY this particular result actually contains, evidence about THIS answer rather than about the engine or the instance — and corroborated_by_url / corroborated_by_domain — ABSENT unless at least two engines found results, because with one witness the number would mean "nobody else was asked" rather than anything measured. TROUBLE IS THE FIELD TO BRANCH ON. It is always present and EMPTY when nothing went wrong, so `if not trouble` is the whole of the good case. When it is not empty it holds only news that makes the answer less than it looks: search_aborted (the metasearch stopped answering MID-SWEEP) with engines_unasked beside it; unresponsive_engines (asked, stayed silent); engines_irrelevant (answered about something else, results already discarded); relevance_not_checked (that check could not run at all — your query has no word of four characters or more, so an empty engines_irrelevant beside it means nothing); pool_unmeasured (the engine pool was never computed from observation); arguments_adjusted (an argument was unusable, so we answered a slightly different question). THE ACCOUNTING IS NOT SENT UNLESS ASKED. Set verbose=true to add timing_ms, pages_read / pages_empty / pages_failed, count, query, page, read, read_top, engines_skipped, engines_used, tiers_used, pool_source and pool_reason. They explain the call rather than change what you do with it, and on a narrow context ceiling they crowd out the answer. HOW TO READ THE ANSWER — six things that are easy to get wrong. 1. AN EMPTY LIST IS A SUCCESS, not a failure: we looked and found nothing. A failure arrives separately, with ok=false and a reason. Do not repeat the query because the list was empty — repeat it rephrased. 2. engines_skipped means "not asked" (the rate limit applied), NOT "asked and stayed silent". The silent ones are in unresponsive_engines, the ones that answered a different question are in engines_irrelevant. 3. CORROBORATION IS NOT CORRECTNESS, and by default there is nothing to corroborate with: the sweep stops at the first engine that gives enough links, so the two fields are absent. min_engines=3 buys width and brings them back. corroborated_by_url means "this many independent engines found this same link" and does NOT mean "this answer is truer". On an ambiguous query the most corroboration goes to the best-indexed namesake rather than to the subject asked about: one name can belong to a retailer, an investigations platform, a maker of enclosures and a maker of portable power supplies at once, and all of them are real. If the sources describe DIFFERENT subjects under one name, the correct answer is "there are several, here they are", not a choice by the number of corroborations. 4. engines_trust labels each engine: clean — checked against references and does not substitute the subject; substitutes — has answered about a namesake; not checked or unavailable — there is no information. If all_engines_clean is false, the results are worth verifying against features of the subject sought: an engine may have answered about a different company of the same name. The label `candidate` means "no misses yet, but fewer than three observations" — not "clean". 5. A RESULT WITH text_source="not_recognised (scan)" IS A DOCUMENT WE DID NOT READ, not a page without text. It is a scanned PDF: search does not pay for a vision model on a link it merely found. If you need it, call web_read on that address — reading recognises it. Where this server requires a token, only a token that holds the paid scope (`paid` or `*`) gets a scan recognised; without it web_read answers not_recognised (scan) too, and names the missing scope.