Network equipment and a clock for incident timing
Network equipment and a clock for incident timing
In this guide
  1. Separate a traffic change from a serving incident
  2. Align the time range and host
  3. Move from report patterns to server evidence
  4. Example: compare a release with a failure window
  5. Distinguish recovery from later search processing
  6. Choose additional monitoring only when needed

Separate a traffic change from a serving incident

The deliverable is an incident record covering the suspected period, affected URL groups, evidence and corrective action. Fewer search clicks do not establish that Google failed to retrieve pages. First check public availability and recent releases or configuration changes.

Google describes Crawl Stats as an advanced report rather than a routine concern for small sites. Without an availability signal, individual URL checks and content work may be more useful. The objective here is explaining a suspected serving failure, not increasing request counts for their own sake or treating every traffic movement as a technical incident.

Align the time range and host

Record dates, time zone, hostname, releases, incident notifications and response categories. Identify whether the investigation concerns the apex host, a www host or a separate resource host. Requests to resources on another domain are not part of the selected host’s evidence.

Do not treat example URLs in the report as a complete inventory of every affected request. Use them to form a question for server-side records. Align the timestamps before comparing patterns, especially where the hosting dashboard and a downloaded log use different time zones or reporting intervals.

Move from report patterns to server evidence

Choose the period with a notable change in failures or response time and compare it with deployments, access restrictions, authentication changes and database incidents. In server evidence, examine timestamp, requested path, response code and processing duration. Do not identify a crawler as genuine from its user-agent label alone.

Inspect a small set of representative URLs through URL Inspection and ordinary public access. Distinguish blocked or temporarily unavailable content from a URL that correctly does not exist. Turning every legitimate not-found response into a success response is not a recovery strategy.

Records to reconcile
Records to reconcile
CheckRecordAcceptance
TimeWindow, time zone and releaseComparable timeline
ResponseURL, code and durationFailure separated from valid 404
RecoveryRepresentative URLs and resourcesExternal checks complete
An incident timeline beside network cables
An incident timeline beside network cables

Example: compare a release with a failure window

In a fictional incident, a release at 10:00 increases failures for one product template and a correction is deployed at 10:20. Record responses around the release, the affected template and the correction. A daily chart alone cannot establish the exact boundaries of a twenty-minute incident.

Match server timestamps and representative URLs, then check whether other page types were affected. After the correction, repeat checks against the same URL groups. A successful response and the presence of the expected main content are separate acceptance checks. This is a diagnostic example, not a claimed production incident.

Distinguish recovery from later search processing

After a fix, retrieve representative pages externally and check response, main content and required resources. A recovered site does not imply simultaneous changes in search reports or indexed state. Record the date and observation and avoid immediately layering on unrelated large changes.

For recurring failures, give the hosting team the relevant URLs, time window and release history. Do not assume that a larger server will necessarily solve an unidentified problem. Establish whether capacity, configuration, an upstream dependency or a specific template actually caused the failure before changing the hosting plan.

From checks to action
From checks to actionEvidence of serving failure? → Match logs and release history (Confirmed); Evidence of serving failure? → Check URLs and search intent (Incomplete); Match logs and release history → Recheck released version (Accept); Check URLs and search intent → Evidence of serving failure? (After correction)ConfirmedIncompleteAcceptEvidence of serving failure?Match logs and releasehistoryCheck URLs and searchintentRecheck released versionFrom checks to actionEvidence of serving failure? → Match logs and release history (Confirmed); Evidence of serving failure? → Check URLs and search intent (Incomplete); Match logs and release history → Recheck released version (Accept); Check URLs and search intent → Evidence of serving failure? (After correction)ConfirmedIncompleteAcceptEvidence of servingfailure?Match logsand releasehistoryCheck URLsand searchintentRecheck releasedversion
  • After correction：Check URLs and search intent → Evidence of serving failure?

Choose additional monitoring only when needed

Search Console and existing hosting logs may be sufficient. If using Screaming Frog for a follow-up, define scope, authentication and an appropriate request rate so an audit does not worsen an ongoing incident. A new paid monitoring contract should address an identified evidence gap.

Separate log retention, monitoring, investigation time and infrastructure costs. The investigation is complete when the cause and action can be explained, important URLs can be retrieved correctly and a follow-up owner is assigned. Reproducing an old chart shape is not the acceptance criterion.

Resources for the next step

Publishing and site improvement

Organise article planning, images, launch checks and refresh priorities as your publication grows.

Official sources

Pricing, availability and feature limits can change. Check the linked official pages before subscribing.

How we compareAdvertising disclosure