Preserve the failed observation
A page that opens now may still have failed when a crawler requested it. Start with the exact affected URL, recorded response code and observation time, including the timezone. Keep the original report rather than replacing it with a screenshot of today's successful load. An intermittent problem needs a timeline, not a single verdict.
Separate an HTTP server response from a connection timeout or DNS failure. If a tool reports only that fetching failed, retain that wording until logs identify the failure. Do not assign a 500 status that was never observed. Note whether the affected request was for the page itself or a resource required by it.
Correlate the request across layers
Ask the operator for a narrow log window around the event. Compare the public-facing edge response with the upstream or application response using request identifiers where available. Normalize timezones before joining records. A gateway error at the edge and a successful application request several minutes later do not describe the same event.
Record the host, path, status, response duration and relevant deployment or infrastructure event. Limit shared excerpts to what is needed and remove credentials or personal query values. Treat a Googlebot user-agent string as a claimed identity, not sufficient verification by itself; keep crawler attribution separate from the observable server failure.
Look for a pattern before choosing a fix
Group failures by route and time window. Compare whether they coincide with a release, a scheduled job, upstream timeouts or a particular backend instance. These are hypotheses to check against logs, not reasons to change every timeout or purchase more capacity immediately. Include successful requests in the same window to understand the scope.
Google documents that 5xx responses and 429 can slow crawling, with persistent server failures affecting continued indexing. That makes recovery important, but it does not identify the cause of this incident. Avoid changing error responses to 200 while leaving the error page in place: the response should accurately represent what the server delivered.
Worked example: a short deployment window
In an illustrative incident, an article request receives 502 at 10:02 UTC, while a browser request succeeds at 10:08. The deployment log records a backend restart at 10:01, but that timing alone is not proof. Matching request records show that the gateway could not reach the intended backend during the failed request.
The team investigates readiness and traffic handover for that backend rather than rewriting the article. It tests the revised deployment behavior in a controlled environment and checks the public route after release. If the request logs instead showed a healthy backend and an edge-generated block, the investigation would follow a different branch. The example provides a reasoning method, not a universal diagnosis.
Verify recovery over a relevant interval
Choose a modest check frequency and a window covering the suspected trigger, such as the next scheduled job or deployment. Avoid aggressive probing that adds load to an already struggling service. Record response codes and expected content, and review server logs for failures between your samples. A few successful probes cannot establish that every request succeeded.
Save the incident timeline, supported cause, targeted change and remaining uncertainty in your website-audit action list. Keep subsequent Google observations separate from local recovery checks. Report that the tested routes recovered during the observed interval; do not claim that crawling frequency or indexing has recovered until there is evidence for those outcomes.