The State of Modern AI Discovery Formats in Australia (2026): Empirical Data, Enterprise Adoption, and the Dual-Plane Architecture

For thirty years, web architecture developed around a single client model: a human sitting in front of a graphical web browser. Web developers constructed sites out of document object models (DOM), rendered cascading style sheets, bundled client-side JavaScript, and layered analytics trackers, cookie consent banners, and interactive carousels. Traditional search crawlers like Googlebot adapted by running headless Chromium instances to render JavaScript and parse visual layout trees.

That architectural model is no longer sufficient.

In modern web development, websites serve two distinct classes of automated, non-human consumers:

1. Reasoning Engines and Synthetic Browsing Agents: Large language model (LLM) powered systems (such as Perplexity, Claude-Web, OpenAI Operator, Browser-Use, and GPTBot) that crawl, ingest, and synthesise content to answer natural language queries or navigate workflows on behalf of users.
2. Autonomous Programmatic Agents: Machine-to-machine agents operating over protocols like Agent-to-Agent (A2A), Agent Protocol, or Model Context Protocol (MCP) seeking callable API tools, deterministic JSON schemas, and structured communication channels to execute actions directly.

To understand how Australian enterprises, financial institutions, tech scaleups, and digital agencies are adapting to this architectural shift, we conducted an empirical probe across 175 leading Australian domains across 8 commercial sectors.

The data reveals a striking market asymmetry: while Australian technology leaders and Big Four banks are rapidly standardising on passive LLM context files (llms.txt), virtually the entire Australian enterprise sector remains blind to programmatic agent execution (agent.json, agents.md, and mcp.json).

This report provides the empirical findings of our Australian market scan, analyses live edge telemetry from production deployments, and presents a complete architectural blueprint and glossary for implementing modern AI discovery.

Empirical Findings: The Australian AI Discovery Scan

In September 2026, we scanned 175 high-authority Australian domains across banking, fintech, enterprise SaaS, marketplaces, retail, energy, telecommunications, universities, government, media, and digital agencies.

Each domain was tested against six modern AI discovery endpoints, as well as dynamic HTTP Markdown Content Negotiation (Accept: text/markdown), with strict content-type verification to filter out soft-404 HTML fallback responses.

Summary of Adoption Across 175 Australian Domains

• Overall Adoption Rate: 43 out of 175 domains (24.6%) have deployed at least one modern AI discovery format.
• `llms.txt` Adoption: 41 out of 175 domains (23.4%) serve a live, valid /llms.txt file.
• `llms-full.txt` (Two-Tier Context): 5 out of 175 domains (2.9%) maintain both /llms.txt and /llms-full.txt bundles (Commonwealth Bank of Australia, Atlassian, Dovetail, Appliances Online, and rahulsengupta.digital).
• HTTP Markdown Content Negotiation: 8 out of 175 domains (4.6%) dynamically return Content-Type: text/markdown when queried with Accept: text/markdown (Optimising, Safari Digital, Online Marketing Gurus, Reload Media, In Marketing We Trust, Linktree, Secure Code Warrior, and rahulsengupta.digital).
• Public `agents.md` Directives: 1 out of 175 domains (0.6%) has deployed root-level /agents.md directives (rahulsengupta.digital).
• Public `agent.json` / `mcp.json` Action Manifests: 2 out of 175 domains (1.1%) expose machine-callable tool manifests (AGL Energy API integration and rahulsengupta.digital).

Sector-by-Sector Breakdown

Verified Active Deployments in Australia

Key Analytical Takeaways from the Data

1. Big Four Banking Leadership: Three of Australia's Big Four banks (Commonwealth Bank, National Australia Bank, and Westpac) have adopted /llms.txt. CBA has gone a step further by shipping /llms-full.txt, providing an index of financial products, interest rates, and compliance documentation formatted for AI context ingestion.
2. Tech Unicorn Consensus: Australian technology leaders (Atlassian, Airwallex, CultureAmp, Employment Hero, Rokt, Buildkite, Dovetail, Linktree, and Megaport) have treated /llms.txt as a standard production deployment alongside robots.txt.
3. Agency Content Negotiation Experimentation: Digital agencies like Optimising, Safari Digital, Online Marketing Gurus, Reload Media, and In Marketing We Trust are the first Australian cohort to deploy dynamic Cloudflare Markdown for Agents, allowing crawlers to request plain markdown instead of HTML.
4. The Critical Enterprise Blindspot: While 23.4% of top Australian sites feed text to AI readers, only 1.1% of commercial sites have implemented machine action layers (agent.json, mcp.json). Australian brands currently treat AI as a passive audience rather than an active channel for transactions and customer conversion.

The Great Asymmetry: Passive Context vs. Autonomous Execution

The Australian market scan highlights a fundamental divide in how engineering teams conceptualise AI traffic.

Currently, most implementations focus entirely on Passive Ingestion:

• Delivering concise Markdown summaries so ChatGPT, Claude, and Perplexity do not hallucinate product features or company facts.
• Stripping HTML navigational boilerplate to save token costs during retrieval augmented generation (RAG).

While essential, passive ingestion solves only half the problem.

                                  [ Inbound Web Traffic ]
                                             │
                      ┌──────────────────────┴──────────────────────┐
                      ▼                                             ▼
        [ Natural Language Consumers ]               [ Programmatic Execution Agents ]
        • Perplexity Search                          • Model Context Protocol (MCP)
        • Claude-Web / Artifacts                     • Agent-to-Agent (A2A) Protocols
        • OpenAI Search & Operator                   • Custom LangChain / LlamaIndex Tools
        • Synthetic Crawlers (GPTBot)                • Enterprise Workflow Automations
                      │                                             │
                      ▼                                             ▼
             Needs: Context & Truth                        Needs: Deterministic Action
        • Token-efficient text                       • Typed JSON Schemas
        • Semantic entity boundaries                 • REST / RPC endpoints
        • Editorial & scope directives               • Auth & rate-limit specs
        • Clean Markdown structures                  • Structured output contracts

When an autonomous AI agent (such as OpenAI Operator or a corporate procurement agent) visits an enterprise website, it does not just want to read an article. It wants to take action:

• Book a consultation or discovery call.
• Check real-time inventory and pricing tiers.
• Submit a structured inquiry or Request for Proposal (RFP).
• Retrieve verified case studies for a specific industry vertical.

If the website only offers a graphical HTML form, the agent must parse DOM trees, guess input selectors, handle client-side validation errors, and navigate reCAPTCHA challenges. This process is fragile and fails frequently.

By providing a structured JSON Manifest (`agent.json` or `mcp.json`), the website exposes direct, typed API endpoints that agents can execute deterministically in milliseconds.

The Dual-Plane Architecture: Markdown vs. JSON

To serve both classes of automated traffic effectively, modern web applications require a dual-plane discovery architecture.

1. The Markdown Layer: Context & Truth

The Markdown layer exists to eliminate prompt overhead and enforce entity truth.

Standard web pages deliver hundreds of kilobytes of layout markup, SVG icons, inline CSS, script tags, and third-party tracking pixels. When an LLM crawler ingests raw HTML, it wastes context window tokens processing layout noise. Furthermore, navigation menus and sidebars often pollute the model's semantic understanding of the core content.

The Markdown layer resolves this through three mechanisms:

• `agents.md`: Establishes behavioural rules, defines the official source of truth, specifies brand boundaries, and instructs browsing agents on how to navigate the property.
• `llms.txt`: Provides an organised, token-efficient index of core documentation, services, and authoritative links.
• Dynamic Content Negotiation (`Accept: text/markdown`): Automatically converts HTML responses into clean GitHub-Flavored Markdown (GFM) at the edge before serving them to AI bots.

2. The JSON Layer: Deterministic Action

The JSON layer converts the website from a static publication into an executable platform for AI agents.

By exposing a standardised manifest at /.well-known/agent.json or /.well-known/mcp.json, the website defines:

• Callable Tools: Specific actions the agent can perform (e.g., book_consultation, request_pricing, query_knowledge_base).
• JSON Schemas: Strict data types, required fields, and parameter descriptions for each tool.
• Authentication & Rate Limits: Clear rules for message quotas and security tokens.
• Asynchronous Inboxes: Dedicated webhook or polling endpoints for agent communication.

In addition, /sitemap-agentic.json delivers a typed hierarchical knowledge graph that maps every URL to its entity type (schemaType, pillar, leadMetric, lastModified), eliminating the need for AI systems to reconstruct page relationships from flat XML sitemaps.

Real-World Edge Telemetry: What AI Bots Hit in Production

To understand how AI crawlers behave when exposed to this dual-plane architecture, we analysed live production telemetry logged via Cloudflare Workers and Cloudflare D1 on rahulsengupta.digital.

Our edge middleware captures every automated request, recording timestamp, requested path, discovery category, bot family, raw User-Agent, Accept headers, origin country, ASN, AS Organization, and Cloudflare edge data centre.

Crawler Profiling: Three Distinct Behaviors

Analysis of edge logs reveals three distinct operational profiles among automated visitors:

1. Generative Search Engines (GPTBot, ClaudeBot, PerplexityBot): These systems demonstrate sophisticated discovery patterns. They systematically request /llms.txt, read /agents.md, and utilise HTTP Content Negotiation (Accept: text/markdown) to retrieve clean Markdown versions of blog posts and case studies.
2. Autonomous Browsing Agents (Browser-Use, Claude-Web, Custom Agent Runtimes): These clients target behavioural directives and tool manifests first. They probe /agents.md for site rules, check /.well-known/agent.json for callable tools, and ingest /sitemap-agentic.json to understand site structure before initiating deeper tasks.
3. High-Volume Brute Scrapers (Bytespider, Scraping Utilities): These bots generate significant request volume across arbitrary paths without evaluating discovery protocols. Edge filtering and Markdown delivery are essential here to protect server resources and prevent excessive origin CPU consumption.

Edge Performance Benchmarks

Serving clean Markdown and JSON manifests directly from Cloudflare Workers produces substantial performance improvements over traditional HTML rendering:

• Edge CPU Execution Time: Dropped from an average of 42.6 ms for full HTML page rendering to 3.5 ms for Markdown delivery (a 91.8% reduction).
• Data Transfer Payload: Reduced from 184.2 KB (HTML + hydration payloads + script bundles) to 9.5 KB (structured Markdown), representing a 94.8% payload decrease.
• Edge Time to First Byte (TTFB): Maintained consistently below 20 ms globally via Cloudflare D1 and edge caching.

Discovery & Orchestration: Wiring the Full Protocol Stack

Deploying discovery files is only effective if AI clients know where to find them. Discovery must be orchestrated across three interconnected layers:

[ Domain Entry Point ]
         │
         ├──► 1. robots.txt (Crawler Directives & Explicit Manifest Links)
         │         ├── Agent-Directives: https://domain/agents.md
         │         ├── Agent-Manifest: https://domain/.well-known/agent.json
         │         ├── Agent-MCP: https://domain/.well-known/mcp.json
         │         ├── LLMS-Index: https://domain/llms.txt
         │         └── Sitemap: https://domain/sitemap_agentic_discovery.xml
         │
         ├──► 2. HTML Document <head> (DOM Link Relations for Browsing Agents)
         │         ├── <link rel="sitemap" type="application/json" href="/sitemap-agentic.json" />
         │         ├── <link rel="agent-directives" type="text/markdown" href="/agents.md" />
         │         ├── <link rel="agent-manifest" type="application/json" href="/.well-known/agent.json" />
         │         ├── <link rel="mcp-manifest" type="application/json" href="/.well-known/mcp.json" />
         │         └── <link rel="describedby" type="text/markdown" href="/llms.txt" />
         │
         └──► 3. Discovery XML Sitemap (Search Engine Discovery Engine)
                   └── Plain urlset exposing all AI-facing resources to standard indexers

Modern AI File Formats Glossary

Canonical specs for the formats below: llms.txt is documented at llmstxt.org, the agents.md convention lives at agents.md, and the Model Context Protocol is at modelcontextprotocol.io. This site runs the full stack, so you can also inspect a live llms.txt, an agent.json action surface, an OKF knowledge bundle and an agentic sitemap instead of taking my word for any of it.

To provide technical clarity for engineering teams and digital leaders, this glossary defines the nine core file formats, manifests, and transport mechanisms comprising modern AI discovery.

1. llms.txt

A standardised, plain text Markdown document placed at the root directory (/llms.txt) designed to provide LLMs with a curated, token-efficient index of a website. Proposed by Jeremy Howard and Answer.ai, it summarises site purpose, core offerings, and authoritative URLs while stripping out layout noise. It serves as the primary orientation map for generative answer engines like Perplexity, ChatGPT Search, and Claude during web retrieval.

2. llms-full.txt

A comprehensive, concatenated Markdown document located at /llms-full.txt that bundles the complete, unabridged text of all critical documentation, product specifications, or service pages into a single file. While llms.txt functions as an index, llms-full.txt provides the full source context in one network request, allowing LLMs with large context windows to ingest an entire knowledge base without performing dozens of sequential HTTP sub-requests.

3. agents.md (and webagents.md)

A root-level Markdown document (/agents.md, /agent.md, or /webagents.md) that delivers explicit operational directives, entity definitions, and behavioural boundaries directly to autonomous browsing agents. Popularised by agent frameworks such as Browser-Use, Atlan, and PinmeTo, it instructs AI agents on official sources of truth, acceptable navigation paths, entity facts, and anti-hallucination guardrails, preventing automated agents from making incorrect assumptions during workflow execution.

4. /.well-known/agent.json

A machine-readable JSON Schema document located at /.well-known/agent.json that implements the Agent Protocol and Agent-to-Agent (A2A) specifications. It explicitly defines callable programmatic tools, input parameters, response structures, rate limits, authentication mechanisms, and inbox endpoints. This format allows autonomous AI software agents to execute transactional operations (such as booking meetings or submitting briefs) directly via APIs without interacting with graphical user interface elements.

5. /.well-known/mcp.json

A standardised JSON discovery manifest located at /.well-known/mcp.json that exposes Model Context Protocol (MCP) server endpoints and capabilities for an application or domain. Created to integrate with Anthropic's Model Context Protocol architecture, it allows AI coding environments, desktop assistants, and remote agent runtimes to auto-discover callable tools, context prompts, and real-time resources over Server-Sent Events (SSE) or HTTP transports.

6. /sitemap-agentic.json

A structured JSON sitemap that represents a website as a typed, hierarchical knowledge graph rather than a flat list of URLs. Unlike legacy XML sitemaps that only provide location and last-modified dates, sitemap-agentic.json decorates every resource with rich metadata, including Schema.org types, thematic pillar categories, parent-child relationships, and primary metrics. This enables AI systems to instantly parse site architecture and semantic context.

7. /sitemap_agentic_discovery.xml

A specialised XML sitemap built according to standard Sitemaps.org protocols that specifically indexes all AI discovery files, manifests, and machine-readable resources on a domain. By referencing this sitemap in robots.txt, website owners ensure that traditional search engine bots, specialised AI crawlers, and SEO indexers discover llms.txt, agents.md, agent.json, and agentic JSON sitemaps systematically without heuristic guessing.

8. /okf/index.md (Open Knowledge Format)

An atomic Markdown knowledge repository structure (/okf/index.md) that organises domain knowledge into modular, interlinked concept files with YAML frontmatter. Based on the Open Knowledge Format standard, it splits complex business entities, service offerings, and credentials into distinct, referenceable Markdown notes. This structure is specifically optimised for retrieval augmented generation (RAG) chunking, vector embeddings, and local knowledge graphs.

9. Accept: text/markdown (Content Negotiation)

An HTTP content negotiation mechanism where web servers inspect the inbound Accept header and dynamically return clean GitHub-Flavored Markdown (text/markdown) instead of graphical HTML (text/html). Supported natively by Cloudflare Markdown for Agents and modern edge middleware, this approach allows human visitors to view fully styled web pages while delivering lightweight, token-efficient Markdown directly to AI crawlers on the exact same canonical URLs.

The 2026 Action Plan for Australian Technical Teams

To capitalise on the first-mover advantage identified in our market research, Australian engineering and technical SEO teams should implement the following four-phase roadmap:

Step 1: Deploy /llms.txt and /llms-full.txt

Audit your core product documentation, service offerings, and brand positioning. Create a curated /llms.txt index following Answer.ai standards and deploy it to your root domain. For technical or financial properties, provide a concatenated /llms-full.txt file to streamline single-request RAG ingestion.

Step 2: Establish Entity Boundaries in /agents.md

Create a root /agents.md file specifying exactly who your company is, what your products are, where your official factsheet resides (e.g., /ai-info/), and what information AI models must not hallucinate. Add aliases for /agent.md and /webagents.md.

Step 3: Implement Machine Actions via agent.json and mcp.json

Identify your primary conversion pathways (e.g., consultation bookings, pricing quotes, lead qualification). Define these actions as typed JSON Schemas in /.well-known/agent.json and connect them to an asynchronous inbox endpoint. If applicable, expose remote MCP endpoints in /.well-known/mcp.json.

Step 4: Configure Edge Content Negotiation and Telemetry Logging

Deploy edge middleware (using Cloudflare Workers or serverless functions) to transform HTML responses to Markdown when clients send Accept: text/markdown. Ensure all discovery paths and AI user agents are logged to a serverless SQL database like Cloudflare D1, providing your team with visibility into automated search engine access patterns.

Related reading: I laid out the strategy behind this shift in Fragmentation of Search: Optimising for the Non-Human Visitor, verified what Google's I/O search announcements actually shipped in Google I/O AI Search: Predictions vs. Reality, and measured whether page speed changes how often AI engines cite a page in Does Page Speed Impact AI Search Visibility, Parsing and Citation Frequency?.

Conclusion

The shift from graphical DOM parsing to structured AI discovery is the most significant evolution in web architecture since the advent of mobile responsiveness.

While leading Australian enterprises like Commonwealth Bank and Atlassian have taken the first step by adopting llms.txt, the broader market remains unprepared for autonomous action agents. By implementing a comprehensive dual-plane architecture covering both Markdown context and JSON execution manifests, engineering leaders can ensure their digital properties remain visible, authoritative, and usable in the era of generative AI search.