NordVPN Promotion

Home / Blogs

The Web Has Become a Content Delivery Network

The public Web used to employ a predictable model: operators registered domain names, set up the records, opened ports 80 and 443, and allowed the machine to answer requests from the Internet. The DNS name identified the service, the IP address showed where it could be reached, and the hosting provider supplied the machine on which it ran.

That model is still technically available, but a newly reachable server can now expect almost immediate automated vulnerability probes, password attacks, scraping, spam, denial-of-service traffic, and requests from crawlers that have no meaningful relationship with the website’s human audience. The marginal cost of performing these scans has become so low that local associations, small-scale services, and personal projects are exposed to much of the same background hostility as major commercial websites.

Imperva’s 2025 report estimated that automated systems generated 51% of the Web traffic it measured in 2024, with malicious bots accounting for 37%. (IMPERVA, 2025) Wordfence reported blocking more than 55 billion password-related attacks against WordPress installations during the same year. (WORDFENCE, 2025) Wikimedia reported a 50% rise in multimedia bandwidth since January 2024, driven largely by automated scraping associated with AI model development. (WIKIMEDIA FOUNDATION, 2025)

The 2025 Web Almanac found CDN adoption among approximately 70% of the 10,000 most popular mobile sites, with 35% among the top ten million. (HTTP ARCHIVE, 2025) These figures already understate the operational shift taking place among smaller operators seeking protection rather than performance. For many of them, the question is no longer whether global caching would improve availability. It is whether the origin server should be exposed to the public Internet at all.

As the person who currently leads ICANNWiki, I oversaw our own transition to Cloudflare, not primarily to improve performance or global availability, but because of the relentless volume of bot crawling we had faced for approximately two years. Put bluntly, this was less an architectural preference and more a defensive necessity. For an independent project operated on behalf of a community, it no longer made sense to continuously filter, block, and manage an ever-growing volume of non-human traffic.

ICANNWiki did not primarily need content delivery. It needed someone else to operate its public-facing network boundary.

From delivery network to defensive perimeter

The term “content delivery network” reflects the technology’s original value proposition. CDNs placed cached copies of content closer to users, reduced latency, and prevented distant origin servers from answering every request. Those functions remain important, but they no longer explain why many websites adopt these services.

The modern CDN commonly operates as a reverse proxy. (FIELDING; NOTTINGHAM; RESCHKE, 2022) Instead of connecting directly to the website’s server, the user connects first to infrastructure controlled by the CDN provider. That intermediary can terminate the connections, rate-limit clients, classify bots, absorb denial-of-service traffic, and decide whether the origin server should be contacted at all. (CLOUDFLARE, 2026a)

For the user, the CDN effectively presents itself as the website. For the operator, it becomes a delegated defensive perimeter. This arrangement does not violate the Internet protocols, nor does it mean that the origin server has disappeared. The important change is one of control. The first endpoint encountered by the public is increasingly operated by a third party with authority to shape traffic before the website itself sees the request.

LLM/AI crawlers have made this role particularly visible. The Robots Exclusion Protocol allows website operators to publish instructions in a robots.txt file, but compliance has always been voluntary. The protocol itself is not an access-control mechanism. (KOSTER et al., 2022) When crawler operators ignore those instructions, meaningful enforcement must occur elsewhere.

That enforcement is increasingly performed at the CDN layer. Providers can identify suspected LLM crawlers and apply different policies according to crawler category, or experiment with payment and licensing models. (CLOUDFLARE, 2025a; CLOUDFLARE, 2026b) A conflict between publishers and LLM companies over access to information is therefore implemented not only through law, contract, or technical standards, but through privately controlled edge-network rules.

When the DNS answer is no longer the host

This changes what familiar Internet resources communicate. In a conventional configuration, a domain name resolves to an address associated with the server hosting the website. That relationship was never perfectly direct, particularly because of shared hosting and load balancing, but it remained conceptually useful.

The DNS answer indicated where the service could be reached, and the IP address helped identify the network responsible for delivering it. When a website operates behind a reverse proxy, the DNS normally returns an address belonging to the CDN. The origin server’s address is intentionally concealed and may be configured to reject any traffic that does not arrive through the intermediary. (CLOUDFLARE, 2026c)

The DNS now identifies the network authorized to receive traffic on the website’s behalf. The IP address becomes correspondingly less informative. A single anycast address may represent a large number of unrelated websites and may route users toward different physical facilities depending on their location and the provider’s internal traffic-management decisions. (CLOUDFLARE, 2026d) From the outside, it may be difficult to determine where the origin is hosted, where a request was processed, or which jurisdiction is operationally relevant. (KUMAR, 2026)

This has practical consequences. IP-based blocking may affect many unrelated services at once. IP geolocation becomes unreliable as a method of identifying where the website’s origin is operated. Network measurements often observe the CDN’s infrastructure rather than the website’s actual host. Law enforcement, researchers, regulators, and abuse investigators now all find themselves interacting with a layer of control only. The hosting provider also becomes much less visible, as it is no longer the infrastructure directly interacting with the public.

This creates an important distinction. A CDN may correctly state that it does not host particular content and therefore cannot remove that content from its origin. At the same time, withdrawing its security or proxy services may expose the origin to attack or make the website operationally unsustainable. The CDN may not be the host, but it can still become essential to whether the service remains reachable.

An infrastructure layer without governance

Internet governance has traditionally organized responsibility around identifiable resources and actors. ICANN coordinates domain names and contracts with registries and registrars. Regional Internet registries administer number resources. Network operators route packets. Hosting providers maintain servers. Platforms manage applications and user-generated content.

CDNs cross several of these categories without fitting neatly into any of them.

A single provider may operate authoritative DNS, return its own IP addresses, terminate TLS, inspect application traffic, conceal the origin, classify automated users, steer requests among countries, and execute code at the edge. (CLOUDFLARE, 2026a) Its operational role can be more visible and consequential than that of the host, even though it may not be treated as either a telecommunications provider, a hosting company, or a platform.

Existing policy structures frequently remain actor-specific. ICANN’s contractual framework for DNS Abuse, for example, focuses on what registries and registrars can reasonably investigate and mitigate within their roles. (ICANN, 2024) That boundary is legitimate and extremely necessary. The problem arises when the analysis stops with the domain name, as though the registrar, registry, and origin host were the only important control points.

The CDN sits between surfaces. This intermediary role also creates concentration risk. The infrastructure required to absorb large attacks, operate global anycast networks, classify bots, and maintain sophisticated security systems is massive and expensive. A comparatively small number of providers can deliver these capabilities at scale, (PHOKEER, 2025) while smaller website operators increasingly depend on free tiers, standardized configurations, and bundled hosting arrangements.

The same services that democratize access to security therefore centralize policy decisions. A configuration error, outage, policy change, or service termination at one intermediary can affect large numbers of otherwise unrelated websites. Cloudflare’s November 2025 outage, triggered by a malformed Bot Management configuration file that propagated across its network, offered a direct example of this shared failure domain. (CLOUDFLARE, 2025b)

Looking beyond AI governance

Recognizing CDNs as an Internet governance vector does not mean placing them under ICANN or creating a single global regulator. It means acknowledging that they have become infrastructure actors whose decisions can affect access, resilience, competition, jurisdiction, and the practical operation of Internet resources.

Several policy questions immediately emerge from this new reality. If CDNs are increasingly deciding where traffic is processed, how much transparency should there be around those decisions? What happens when automated security systems block legitimate users? There are also very practical questions around dependency: how easy is it to move firewall rules, bot policies, logs, and security configurations from one provider to another? For a small operator, is maintaining more than one CDN even realistic, or does adopting this infrastructure effectively mean becoming dependent on a single protective intermediary? In this scenario, service termination starts to look like something more consequential than simply cancelling a vendor contract.

These questions are currently divided among cybersecurity, cloud computing, competition policy, telecommunications, intermediary liability, and content governance. No single institution needs to control the whole field, but Internet governance forums should recognize the common infrastructure layer connecting them.

LLM/AI crawling is an important part of this transformation, but it did not create it. The shift toward CDN dependence was already being driven by the vertiginous increase in cyber attacks. There is a risk that the current emphasis on AI governance becomes the frame through which every technological development is interpreted.

Evidently, AI deserves sustained attention, but other structural changes are occurring at the same time. The growing dependence on CDNs alters the meaning of addresses, obscures hosting relationships, turns DNS resolution into proprietary traffic steering, and places private rule engines between users and origin servers.

The Internet has not literally become a content delivery network. The public Web, however, is increasingly composed of origin servers reached through a relatively small number of privately operated edge networks. The domain name still identifies the service, the IP layer still routes the packet, and the host still stores or generates the content. Increasingly, however, the CDN determines where the request goes, whether it reaches the host, and under what conditions it will be answered.

Unless CDNs are treated as a major subject of policy development, the Internet governance community risks continuing to govern an architectural map that no longer matches the territory. We may spend years debating the future of AI while failing to notice that the front door of the Internet has already moved.

NORDVPN DISCOUNT - CircleID x NordVPN
Get NordVPN  [74% +3 extra months, from $2.99/month]
By Mark Datysgeld, Project Lead Developer at ICANNWiki, CTO at DNS AXE

Filed Under

Comments

Comment Title:

  Notify me of follow-up comments

We encourage you to post comments and engage in discussions that advance this post through relevant opinion, anecdotes, links and data. If you see a comment that you believe is irrelevant or inappropriate, you can report it using the link at the end of each comment. Views expressed in the comments do not represent those of CircleID. For more information on our comment policy, see Codes of Conduct.

CircleID Newsletter The Weekly Wrap

More and more professionals are choosing to publish critical posts on CircleID from all corners of the Internet industry. If you find it hard to keep up daily, consider subscribing to our weekly digest. We will provide you a convenient summary report once a week sent directly to your inbox. It's a quick and easy read.

Related

Topics

Domain Names

Sponsored byVerisign

DNS

Sponsored byDNIB.com

Brand Protection

Sponsored byCSC

New TLDs

Sponsored byRadix

IPv4 Markets

Sponsored byIPv4.Global

DNS Security

Sponsored byWhoisXML API

Cybersecurity

Sponsored byVerisign

NordVPN Promotion