Answerly Tracker — a platform that asks AI assistants about a brand on a schedule and shows every metric with its sample size

01 Project passport

Verified Unmarked fields are confirmed by the product’s code, database and cron as of 7 Oct 2026.

02 The task before development

On a schedule, with the sample size next to every figure, the platform had to answer four questions:

1. Does the assistant name the brand, and does it cite the site?
2. Which page does it cite, and which page did it find and pass over?
3. Who is cited instead, and from which kind of source?
4. What do the answers claim about the brand, and is it true?

The second condition: this is a product for agencies. Several organisations work in one system, each with its own projects, and an agency’s client sees the results but not the internals: prices, notes, raw lists.

03 Scope

04 How it is built

Two halves meet in the database: the cabinet writes settings and reads aggregates, and the collectors on a VPS query the engines on a schedule and write the answers.

Three rules hold across the code:

• A check is a sample, not a verdict. Every share carries its n and a 95% margin of error: 1.96·√(p(1−p)/n), where n is the number of answers in the latest check. A change no larger than that margin is labelled noise. For the average citation position the bound comes from a Welch t-test; with fewer than 5 citations in a check, the position is not compared and is marked “small sample”.
• No made-up numbers. A score component with no data is left out and the share of covered weight is shown; a gap is not counted as zero. An AI Overview that a query does not have stays empty.
• Method changes are marked. When a source changes, the chart gets a break mark, and no comparison is made across it.

05 The hardest decision: measuring the real ChatGPT, not the API

Problem. At first the collector asked the model through the OpenAI API with web search. On 5 Oct 2026 we ran the same 72 prompts through the API and through the ChatGPT app, once each, with location Ukraine and Ukrainian language. One pair returned an error. In 11 of the remaining 71, the app’s answer had no sources at all. Of the 60 pairs where both had sources, 28 shared no domain; the median was 1 shared domain, a Jaccard index of 0.08. The app searches differently and cites other sources.

Decision. Since 5 Oct 2026 the collector takes the app’s own answers through a data-collection provider. With each answer it stores the assistant’s own web searches and the pages it opened. The switch is marked as a method break: the dashboard and the report do not compare values across the date. The API path stays as a fallback behind one setting.

Trade-off. History before 5 Oct cannot be compared with the new data, the sample per check is smaller and noisier, and there is one more external dependency. The gain: the platform shows the page the assistant found and did not cite. If all you need is a trend on a large sample, the API is cheaper and steadier.

The second hard part — honest statistics. A share carries a 95% margin of error; the average citation position carries a 95% confidence interval for the difference (Welch t-test). On the reference project a 0.6 shift in position against a ±2.3 bound is labelled noise, not movement. The price is fewer “findings” on screen.

06 Who uses it, and how

07 Modules in the current version

• Prompts and personas. Prompts by direction; personas imported from the agency’s .xlsx template; visibility per persona.
• Discovery. New queries every Monday from Search Console and Serpstat.
• Pages. Our URLs by prompt and engine: cited, found and passed over, with the assistant’s own searches; how AI crawlers see the page.
• Brand. Tone, roles and claims from every answer that names the brand, checked against a card of true facts, in the project’s language.
• Placements. Domains the engines cite instead of us, by kind of source: ratings, listicles, media, communities.
• Links. Domains that link to two or more competitors but not to us, next to the sources of AI answers; a daily check of links already placed.
• Cocoon and action plan. The site architecture from the agency’s brief builder next to what the engines do with each page; an action plan grounded in search results. The cocoon is visible to the team only.
• Report and digests. A report behind a signed link; after each check, a Telegram digest with a significance test on shared prompt × engine pairs.

Placements · where the answers come from

Kinds of sources the engines cited on the agency’s prompts, and what to do with each. Example domains are hidden in the screenshot.

08 Quality and stability

A partial check means some answers did not arrive and the rest were stored; no credits are charged for the missing answers.

09 Security and privacy

• Passwordless sign-in: an email link, signed and valid for 10 minutes; a 7-day session in a protected cookie.
• Provider keys are read only by the server and never reach the browser; the owner replaces them in the admin panel. In the database they are currently stored unencrypted — a deliberate trade-off, with encryption planned. The Google token for Search Console and GA4 is encrypted (AES-GCM). Organisation API keys are stored only as a hash.
• Isolation: in 38 automated scenarios no user reached another organisation’s data through the cabinet or the API. Isolation is currently enforced by the application’s server layer; database-level policies are still being introduced.
• Rate limits on sign-up and chat.
• The client view shows no cost, notes or prices, in the cabinet or in exports.

10 Release and links

11 Result

Product

On 5–6 Oct 2026 we took five queries about choosing an online-store developer and got three ChatGPT app answers to each: a pilot on 5 Oct and two tracker checks on 5 and 6 Oct. Ukraine, Ukrainian language, 15 answers. “Named” means the agency’s name appears in the answer text; “cited” means its domain is among the answer’s sources. The agency was named in 7 answers and cited in 4. One competitor was cited in 12, including all four answers that cited us. The assistant opened the agency’s service page and cited others. That finding became the brief for rewriting the page; we will measure the result on the same queries after at least three checks.

Technical

Business

On 7 Oct 2026 the platform had 5 organisations and 8 projects: the agency’s own, pilots and trial accounts.

12 Limitations and what we would do differently

ChatGPT app answers are collected through an outside data provider. That is slower and noisier than an API, and the provider can change without notice.

One check cannot reliably tell a real change from random variation. The platform suggests a week between checks and does not block an earlier one; it only asks for confirmation.

Isolation lives in the application, not the database. The cabinet reaches the database with a service key, so protection is a layer of checks in the application plus a 38-scenario test. Per-user database policies are planned.

A database shared with other agency services. Reads are already separated; a full split is planned.

Not done yet: failure monitoring for the collectors, automatic test runs in CI, Google verification so Search Console connects without a warning.

13 For a client who wants a similar platform

Why this stack

Server-rendered Astro on Cloudflare Pages gives a cabinet without a separate API server. Supabase gives Postgres with an HTTP interface the cabinet and the collectors share. The collectors stayed in Python, where the providers’ libraries are strongest.

What task the platform solved

A brand’s team sees on a schedule whether AI assistants name it and cite its pages on its customers’ questions, and gets the exact page the assistant found and passed over. Every figure comes with its sample size and a date, so two checks can be compared without guessing.

What Chyzh Agency did

• product design and the data model;
• collectors for five engines with credit metering, a schedule and a queue;
• a cabinet in three languages, a read-only API, exports;
• the link-analysis module and the connection to the agency’s brief builder;
• digests and a test that checks one client cannot see another’s data.

What to plan for in a similar project

Budget for third-party answer data: it is the main variable cost. Agree on a rule for method changes before the first client sees a trend. Keep a card of true brand facts from day one, or the claim check has nothing to compare against.