Why Personal-Site 404 Traffic Spiked

Over the past few days, I’ve been watching the XBSTACK backend, and what’s most glaring isn’t low traffic—it’s the rising number of 404 hits.

At first, I thought I’d messed up the Astro routing or that a recent deployment had dropped some pages. But after digging into Cloudflare logs, I realized many of these requests weren’t coming from internal site clicks. Instead, they were arriving from Zhihu, Juejin, search engine caches, and historical external links, all carrying broken paths.

404 is insidious.

It doesn’t crash as obviously as 500, nor does it make the homepage inaccessible so you can spot it immediately. It acts more like a leak: users arrive, hit a wall, and leave; search engines arrive, hit a wall, and lose trust; I only see overall traffic numbers but have no idea where the traffic is leaking out.

So this time, instead of writing another AI Agent tutorial, I stopped to investigate the site itself.

404 isn’t a minor technical glitch; it’s a traffic funnel problem

A few 404 hits are normal. Any public website gets hammered by crawlers, scanners, old links, and typos pointing to non-existent paths.

But if 404 keeps growing, you can’t just dismiss it as “normal internet noise.” Especially for personal sites with limited traffic, every valid visit is expensive. A reader clicking through from Zhihu might have finished an article and then casually navigated to the AI Development Hub or used the site search. If their first impression is a 404, they’ll likely close the tab immediately.

Search engines are affected too.

They won’t penalize your entire site over a random junk path, but if certain old paths are consistently pointed to by sitemaps, external links, and internal links yet keep returning 404, it indicates a dirty discovery path for the site. For a content-heavy site like XBSTACK with over a hundred articles and multiple sections, 404 isn’t a single-point failure; it’s a crack in the content asset system.

I previously wrote a post titled XBSTACK Content Quality Audit Build Log, which mainly covered cover images, internal links, duplicate topics, Pagefind, sitemaps, and robots.txt. That audit proved there were no obvious high-risk issues with the internal structure. However, the increase in online 404 reminded me: a clean internal structure doesn’t guarantee clean external paths.

I ruled out a “total site routing collapse” first

When troubleshooting 404, the first step wasn’t to immediately write redirects or blame the framework.

I started by looking at a few basic facts:

: healthy
AI: healthy
LangGraph / MCP / Workflow: healthy
: healthy
sitemap-index.xml:
sitemap-0.xml:
Pagefind: healthy

This indicates that the issue is not a site-wide build failure, nor is it a complete breakdown of Astro dynamic routing.

If the entire routing system had collapsed, the symptoms would be far more obvious: numerous normal articles would fail to load, sitemap paths would go offline in bulk, and search pages might become inaccessible. But that is not what happened here. The problem is isolated to specific paths, which often contain old slugs, typos, truncated links, or traces of historical external links.

In other words, this isn’t a case of “the site is down,” but rather “some entry points are pointing to the wrong place.”

The sources of 404 are often not just one

I categorize these personal websites, 404, into five types.

The first type is outdated internal links. For example, a previous article might have referenced a specific path, but the slug was later changed without updating the body text. When readers click through from that older post, they encounter a 404 error.

The second type is broken links from external platforms. When articles are reposted on Zhihu, Juejin, CSDN, Xiaohongshu, or WeChat Official Accounts, copying the URL with even a single extra character or a missing slash can permanently create 404. Unlike internal links, external platforms do not automatically update when I fix the link on my own site.

The third scenario is search engine caching of old paths. If an old page was previously crawled and you later migrated categories or changed slugs, search engines will continue to access the old addresses for a short period.

The fourth scenario is historical remnants in the sitemap. Theoretically, sitemaps should only contain valid URLs, but if the generation script or deployment cache isn’t refreshed promptly, it may continue to instruct search engines to crawl outdated paths.

The fifth scenario is spam scanning. For example, someone might scan /wp-admin/, /phpmyadmin/, /.env, /admin/login. This kind of 404 doesn’t require fixing; retaining 404 is actually the correct outcome.

The key point is: Do not treat these five scenarios with a single approach.

If you redirect all spam scans 301 to the homepage, it may appear that 404 has decreased, but you are actually polluting your logs. Later, when reviewing the backend, you won’t be able to distinguish between genuine users hitting a wall and scripts scanning randomly.

What to look for in Cloudflare logs

The greatest value of edge-layer logs from providers like Cloudflare isn’t just telling you “there’s a 404,” but rather showing you “who hit which path from where.”

I review them using four fields.

path
referer
user-agent
count

If a path’s Referer originates from Zhihu or Juejin and appears repeatedly, prioritize handling it. This indicates genuine external links driving traffic rather than spam scanning.

If the User-Agent is Googlebot, Bingbot, or Baiduspider, and the path resembles an old article slug, check the sitemap and legacy backlinks. Search engines are not the enemy; they simply followed an old trail to get there.

If the path matches /.env, /wp-login.php, or /phpmyadmin, there is no need to worry. Retain 404 for these requests directly, applying rate limiting only at the Cloudflare WAF level if necessary.

The scenario I fear most is another type: paths that appear to be normal articles but actually do not exist.

/ai/langgraph-observability/
/ai/n8n-webhook-hardening/
/ai/xbstack-content-audit/
/tools/compound-calculator/

This path isn’t discovered by hackers; it’s replicated by humans or platforms. It’s only a hair’s breadth away from the correct path, yet that tiny gap is enough to make users hit a wall.

The Most Common Path Issues in Astro Sites

Astro static sites are inherently stable, but path governance relies on human diligence.

I currently focus on these categories of issues:

First, inconsistent trailing slashes.

Some links end with /ai/foo, while others use /ai/foo/. If the site lacks a unified configuration standard, external platforms may apply their own truncation or encoding, resulting in divergent paths.

Second, missing redirects after slug changes.

The most common scenario for content sites is this: early titles were written hastily, and later they were revised for SEO purposes, including changes to the title and slug. However, old slugs were not redirected via 301. The site looks clean internally, but all external legacy links are broken.

Third, truncation of Chinese titles when converted to slugs.

When some platforms copy links, they automatically process Chinese characters, spaces, punctuation, or trailing slashes. If an article’s slug is already too long, or if special characters have slipped into the path, the probability of errors increases significantly.

Fourth, category migrations.

XBSTACK had some early content migrated between archive, lens, ai, and horizon. The migration itself was fine, but old paths must be preserved as fallbacks afterward. Otherwise, search engines and legacy external links will continue to follow the old routes.

Fifth, the sitemap is out of sync with actual pages.

If the sitemap still contains old URLs, it’s essentially inviting search engines to crawl non-existent pages. This issue is more critical than broken external links because it represents an active error signal emitted by the site itself.

Which 404 Issues Must Be Fixed, and Which Can Be Ignored

I’m no longer aiming to reduce 404 to zero. That’s unrealistic and unnecessary.

I only address three categories.

The first category involves 404 with genuine external link sources. For example, when the Referer comes from Zhihu, Juejin, CSDN, or WeChat Official Account redirect pages, and the path clearly corresponds to an existing article on the site. These should either be fixed in the original external source or handled internally via 301.

The second category consists of old paths showing signs of search engine access. Specifically, paths frequently crawled by Googlebot, Bingbot, or Baiduspider should be redirected if they correspond to a currently existing article.

The third category is 404 caused by outdated internal links. This must be fixed in the content, not just handled with redirects as a fallback. Internal links are an asset I have full control over; they cannot be allowed to remain broken for long.

The rest can be ignored.

/wp-admin/
/wp-login.php
/phpmyadmin/
/.env
/server-status
/random-test-path

These aren’t traffic; they’re noise. Trying to fix noise into an entry point is the real mistake.

My 404 Repair Checklist

I’ll handle this in the following order.

1. from Cloudflare 7 404
2. count,
3. referer,,,
4. YesNo
5.: 301
6.:
7.: 404,
8. sitemap checkYesNo
9. check
10. 404, AI, Tools, Yes Not Found

Item 10 is particularly important.

The 404 page should not be a tombstone; it should serve as a rescue entry point. When users hit a non-existent path, they should at least be able to continue searching through All Articles, Site Search, AI Engineering, or Tools.

If the 404 page merely displays the cold, unhelpful message Not Found, it effectively drives users away.

Don’t Overdo Redirect Rules

Many people fix 404 by swinging to the other extreme: redirecting any non-existent path straight to the homepage.

That’s incorrect.

Redirects should be as precise as possible.

/old-ai-agent-url/  -> /ai/ai-agent-architecture/
/old-n8n-webhook/   -> /ai/n8n-webhook-production-hardening/
/old-content-audit/ -> /ai/xbstack-content-quality-audit-builder-log/

If you can’t find a clear corresponding page, don’t force a redirect to the homepage. Users are looking for answers to specific questions, not the homepage. Search engines also detect these irrelevant redirects; over time, this weakens your page signals.

My rule is simple: if there’s a one-to-one match, use 301; if not, let the 404 page handle navigation; and if it’s clearly junk, continue with 404.

What This Taught Me

Once a personal site reaches a certain stage, the real work isn’t just “what am I publishing today.”

It also involves periodically reviewing which articles have been cited externally, which paths have changed, which old entry points are still being accessed by search engines, and which pages exist but leave users with no next step once they arrive.

This is why I increasingly emphasize content inventory management.

XBSTACK already has topic hubs like LangGraph, MCP, and Workflow, as well as tool pages like Compound Interest Calculator, AI Earnings Report Assistant, and Lunest. As the number of articles grows, paths become part of the asset itself.

Titles can be updated, structures optimized, and pages refactored, but old paths shouldn’t be casually discarded.

Because search engines remember them, external platforms remember them, and reader bookmarks remember them.

Pre-Publish Checklist

From now on, every time I publish or change a slug, I’ll take a few extra steps.

- slug YesNo,,
- sitemap YesNo URL
- YesNo
- YesNo
- YesNo URL
- YesNo slug,
- slug YesNo 301
- Cloudflare YesNoFound
- 404 YesNo

These actions aren’t complex, but they determine whether a personal website is just a pile of files or a true asset.

Continue Reading

404 The scariest part isn’t that pages don’t exist.

It’s thinking no one visits your site when they actually do—they just hit a wall and leave.

For a personal website, troubleshooting 404 isn’t about ops perfectionism; it’s about systematically recovering every bit of lost traffic.

Xiaobai

Why Personal-Site 404 Traffic Spiked: External Links, Cloudflare, and Astro Routes