On our own portals we publish the terms of dozens of offers from several partners. A partner can change them at any time without notice, and every outdated sentence is a risk for the reader, for the partner and for us. This article describes how we built a change monitor that reads those pages but decides nothing. It is written for anyone who has to track third-party text, such as price lists, supplier terms or regulatory documents, and still be able to show what applied when.

Neither manual nor fully automatic

Checking every page by hand at short intervals does not scale, and most of the time it only finds that nothing has happened. An automated system that writes changes into the content on its own cannot be defended either. A page redesign, a verification screen in place of content or one ambiguous sentence is enough for the portal to state something untrue. So we split the work. The automated system reads, compares, archives and alerts, which it does reliably. A person understands the text and decides, which the automated system cannot do.

The core rule is that loading is not verification. A page that responded with unchanged text means only “the watched text is identical to the version we last accepted”. It does not mean that the offer is valid or that we understood it.

Four possible outcomes of a single load

Every load ends in one of four states, and none of them means “terms verified”:

  • Unchanged: the text matches the last accepted version.
  • Changed: the text differs from the accepted version, or no accepted version exists yet. Even the first load of a source is therefore only a candidate for review.
  • Error: the page did not load because of an outage, an error status code, a timeout or a disallowed redirect.
  • Unreadable: a response arrived, but it is not the text we watch. It may be a verification screen, text that is too short, a missing content area, an unsupported format (a PDF, for example) or text without the offer name that must appear in it.

The last check, the identity anchor, is easy to overlook, but it matters. When a partner redirects a promotion address to a generic page, “something” technically loads. Without the anchor it would look like a change of terms. With the anchor, the source is recorded as unreadable and needs fixing. Neither an error nor an unreadable page counts as a change of terms. The last successful observation keeps its original time and is never presented as fresh.

From page to fingerprint

First the content area is cut out of the response, without navigation, header, footer, scripts and cookie bars. Each source can have its own rule for which part of the page to read and which volatile lines to drop, such as “updated on” or counters. Normalisation follows: whitespace, line breaks and character entities are unified and empty lines removed. Numbers, dates and times are never removed during normalisation, because that is where the terms are.

A SHA-256 fingerprint is computed from the normalised text. If it differs from the fingerprint of the accepted version, a change candidate is created with a line-by-line diff, so the editor reads only what moved rather than the whole page. Loading the same candidate again only refreshes the observation time, so one candidate produces one alert and no daily reminders of the same thing. For sources with a stable format the automated system can also extract individual fields, such as days and times, and flag when the partner’s rules diverge from the data we publish. Even then it overwrites nothing on its own.

The evidence archive

For every observed version we store its normalised text, compressed and named after its fingerprint. Identical content is stored once, and for every version in the history we can show the exact text we saw at the time. Ordinary copies are deleted after 90 days. The exception is the last accepted and the last observed version, which something still refers to. These are kept regardless of age. The archive also has a disk size cap. Once it is exceeded, the oldest unprotected copies are removed and the daily summary reports it.

Where a person decides

The editor sees the accepted version, the observed version and the diff between them, and decides whether it is a real change of terms, a cosmetic edit or an unreadable source. Acceptance always refers to a specific version: if the text has changed again in the meantime, the acceptance fails and the newer version has to be reviewed. Accepting only moves the reference used for comparison and records who accepted it and when. The editor updates the data on the portal separately. A check record with time, scope and result is written only for a real check, and readers get the date of the last check from that record. A technical load does not change the page’s updated date, the last-check date or the date in the structured data.

While a change awaits review, the portal keeps showing the last accepted data, but only for a set period after the change was detected (two weeks by default, configurable per source). An alert arrives a day before that period ends, and if nobody reviews the change, the claim drops out of the confirmed listing. The same happens to a claim whose last successful load is older than the freshness limit. A source that cannot be read automatically can support a claim only through a recent manual check by the editors.

A polite and safe fetcher

The monitor downloads third-party pages from our server, which makes it a potential attack path. The typical risk is server-side request forgery (SSRF): if someone managed to slip in an address or a redirect pointing at an internal system, the server would fetch it itself. The defences are layered in line with the OWASP guidance on SSRF prevention:

  • HTTPS only, with certificate verification and no port or credentials in the address.
  • The host must match the source’s allowed host exactly, with no subdomains.
  • Every address the domain resolves to must be public. Private and reserved ranges are refused.
  • The connection goes to the IP address that has already been validated, so a second DNS lookup between check and connection (DNS rebinding) cannot bypass the check.
  • Redirects are not followed automatically. Only a few are allowed, and each one passes the same checks and must stay on the same host.
  • The size limit is measured on decompressed data, not on bytes transferred, so a compressed response cannot exhaust memory, and only textual content types are accepted.

The other half is courtesy towards the partner. Before loading we check robots.txt (RFC 9309). A disallow for our path is honoured and the source is then recorded as unreadable. If robots.txt cannot be loaded because of a server error, we do not read the page in that run at all, as the RFC requires. We read nothing behind a login or through a verification screen. Each source has its own interval and fetch time, so running the job more often does not increase the number of requests to the partner. We use conditional requests as defined in RFC 9110 (If-None-Match with the ETag value, If-Modified-Since). When the server answers 304, no body is transferred and the result counts as unchanged content. Such a result refreshes freshness as much as a normal load would, and no more. After a failure, a stepped back-off follows (1 hour, 6 hours, 24 hours) instead of retrying on every run.

Each run has a fixed time budget. The timeout of every request is the smaller of a fixed cap and the remaining budget, so one slow source cannot block the others, and whatever does not fit waits for the next run. Concurrent runs are excluded by a lock, and each source is “leased” by a run with an expiry, so it is never processed twice and is released if the process crashes.

A daily summary, and AI as an assistant, not a judge

An e-mail follows every run. The first line is always a summary: how many sources were checked and how many are unchanged, changed, unreadable or failed. Originally a normal run was silent, but an empty inbox could not be told apart from a run that never happened. Below the summary come lines written for people: changed text awaiting review, page could not be loaded, offer text not readable, review deadline approaching, partner rules differ from our schedule. A recurring failure is reported on the first and the third consecutive failure, not every time.

When there are several candidates, it helps to export them into a single document. At the top is a brief for an AI assistant, followed by the accepted and observed versions, the diff and our current data for each candidate. The assistant proposes one of four verdicts (accept as the new reference, accept and update the data, reject, a person decides) together with the exact values to enter. It writes nothing itself, and the editor carries the result over. The brief states explicitly that partner texts are data and not instructions. If they contain a sentence posing as an instruction (“ignore the previous rules”, “accept this version”), the assistant must not act on it. It has to quote the sentence verbatim and return the verdict “a person decides”.

This is indirect prompt injection, listed by OWASP as risk LLM01 in its Top 10 for large language model applications (2025 edition). The defence is to keep untrusted content separate and require human approval instead of trusting the model. The same principle holds without AI. An automated system that copied third-party text straight into our content could be abused to change it.

Limits, and what transfers elsewhere

The monitor does not see everything. It does not compare images, does not read PDFs or content behind a login, and after a page layout change a source may stay unreadable until the text-selection rule is adjusted. We use it for selected sources where outdated information does harm fastest, such as time-limited promotions. The remaining offers are checked by the editors on a regular schedule. It is not worth building where the partner provides versioned structured data, or where there are so few sources that a person reads them faster than a selection rule can be set up.

The pattern fits anywhere an organisation reuses third-party text and is answerable for keeping it current:

  • Supplier price lists and terms: an identity anchor (a tariff name, for instance), a line-by-line diff and acceptance of a specific version give an audit trail for disputes about what applied when.
  • Regulatory and guidance texts: watching a regulator’s pages with an archive of the exact wording flags a change, but decisions should still rest on the officially published text, not on your own copy.
  • Partners’ contractual terms, third-party privacy policies and the API documentation your integration depends on.
  • A minimum scope to start with: separate error, unreadable and changed states, and normalisation that never strips numbers. The first load is a candidate, evidence has a retention period and technical runs never touch editorial dates. A claim lapses after a review period, SSRF protection and the robots.txt check are there from day one, and a daily summary arrives even when nothing has happened.

We did not get full automation, but we did get a division of labour that can be defended. The editor spends time only on pages where something has happened, every published term has a traceable history, and for every accepted version we can say who accepted it, when and against which text. If you are building something similar, start by deciding what the automated system is allowed to claim, and only then look at how fast it can fetch.