DOC #074 03_DEVELOPER_IMPLEMENTATION

SUBKUZ INDEXING, MULTILINGUAL, SITEMAP, CRAWLER AND AI DISCOVERY SPECIFICATION

Subkuz Global Governance Framework | Issued by Subkuz Media Networks Private Limited (CIN: U72900JH2020PTC014521)

Document #074 | Verified Master Version

Search Integrity and Discoverability for Google, Google News, AI Search and Other Public Crawlers

1. Objective

Subkuz must be discoverable because its pages are useful, attributable, technically accessible and internally coherent, not because it creates duplicated location pages, hidden text, inflated dates or paid-link structures. Search and AI discovery controls implement the public editorial architecture. They do not replace original reporting, accountable authorship, evidence, correction or reader value.

2. Indexation classes

Class Directive Examples
Current public authority page index,follow Policies, About, Newsroom, Ownership, Funding, Edition Directory, Transparency Reports, Registers and public standards
Article and authorised edition page index,follow Original or materially useful journalism and active edition landing pages
Operational public form noindex,follow All fourteen forms and Cookie Settings interface
Superseded policy archive noindex,follow Historical version linked from Change Log
Internal search, empty filter and preview noindex,follow or restricted access Search queries, previews, staging and test pages
Duplicate print or application webview canonical to the public page or noindex Non-primary presentation of the same content
Unsafe or permanently removed material 410 or controlled removal notice No irrelevant redirect
Commercial page case-specific Index only where it serves readers directly and complies with commercial and search-integrity standards

3. Multilingual implementation

Each complete language version has its own URL and one dominant page language. Navigation and boilerplate should not claim one language while the substantive content remains another. Every hreflang set is reciprocal and contains only valid, indexable equivalents. Same-language regional variants use language-region values where the content genuinely differs for the region. The x-default target is the global master or a neutral selector, not an automatic redirect based on inferred location.

A machine translation is not published merely to multiply pages. Translation is reviewed under the Edition, Geography and Language Accountability Standard. Pages with only translated navigation and unchanged body content are treated as duplicates and must not be presented as separate language editions.

4. City and edition integrity

A city URL is indexable only when it has authorised editorial purpose, useful current or enduring content, a place in the Edition Directory, a responsible desk or edition, and an experience materially different from a doorway page. Hundreds of city pages that repeat the same national feed with only a changed location name are not acceptable. Inactive, test and empty city pages are removed from sitemaps and noindexed or returned as an appropriate status.

The city selector links to real URLs and does not create an unlimited parameter space. Pagination, tags, dates and filters are controlled so that crawlers cannot generate millions of low-value combinations.

5. Sitemap architecture

Sitemap Contents Rule
sitemap_index.xml Index of current public sitemap files Referenced in robots.txt
news-sitemap.xml News articles published in the preceding two days Original publication time and visible article title; maximum one thousand news entries per file
articles-sitemap-N.xml Current indexable article URLs Split at operationally manageable size
pages-sitemap.xml Current public institutional and policy pages Exclude forms and archived policies
authors-sitemap.xml Current public author and desk profiles Only verified active profiles
editions-sitemap.xml Authorised country, city, regional and language landing pages Exclude test, planned and inactive pages
images-sitemap.xml Material crawlable images where useful Respect rights, privacy and media labels
video-sitemap.xml Eligible video pages Include accessible public landing pages

6. News sitemap rules

  • Include only article URLs first published within the last two days.
  • Use the original first-publication timestamp, never the sitemap generation timestamp.
  • Use the title as displayed on the article and omit author, publication name and date from the news title field.
  • Use the actual publication language code.
  • Remove news-specific metadata after two days while retaining the URL in the ordinary article sitemap where appropriate.
  • Regenerate promptly when an article is published, removed or materially corrected.

7. Canonical and status-code discipline

Redirects, rel=canonical and sitemap inclusion should reinforce the same preferred URL. Canonical is a strong hint, not a licence to maintain uncontrolled duplicates. A removed page does not return 200 with a thin apology or redirect to the homepage. Server errors are not disguised as 200. Preview, parameter and uppercase variants resolve consistently.

8. Robots and crawler policy

The recommended production robots.txt allows ordinary search crawling of public content, identifies the current sitemap index and news sitemap, and expressly allows OAI-SearchBot and GPTBot under the current publication choice. Search discovery and model-training permission remain separate governance decisions. Any later restriction or change affecting an artificial-intelligence crawler requires authorised copyright, editorial, privacy and product review and must be reflected consistently in the live robots file and public licensing position.

Robots.txt is not used to protect private information. Private, source, grievance, applicant, administration and staging systems require authentication and access control. Noindex pages remain crawlable so the directive can be observed. A crawler blocked by robots.txt cannot read a page-level noindex instruction.

9. AI search visibility

Google AI features and ChatGPT search rely on crawlable, indexable, high-quality public pages. Subkuz should not add unsupported AI-only metadata, invisible answer blocks or artificial mention-building text. Clear definitions, accountable authors, stable citations, direct sources, current dates, original reporting and coherent internal links improve both human and machine understanding. The public Network Facts page, author profiles, policy architecture and article trust block provide strong entity and provenance signals without making a guarantee of inclusion or citation.

10. Paid content and site reputation protection

Third-party, sponsored, advertorial and paid-publication content is accepted only when it serves Subkuz readers directly, fits the publication, receives genuine compliance and editorial-quality review, and is not placed mainly to exploit Subkuz ranking signals. Irrelevant high-risk niches, duplicated agency articles distributed across many domains, white-label sections, rented authority, doorway pages, hidden sponsorship and guaranteed ranking packages are prohibited.

Paid outbound links use rel=sponsored. User-generated links use rel=ugc where such content is enabled. Noindex may be applied to a lawful commercial page when its search value is insufficient or the commercial relationship creates a site-reputation risk. No client contract overrides the search-integrity decision.

11. Homepage and component duplication

The repeated horoscope output currently visible on the homepage must be corrected at the data, query, loop or component level before launch. The same test must cover sports widgets, live tickers, web stories, related articles, advertisements and city selectors. Duplicate blocks can create poor usability, excessive page weight, repeated structured data and misleading content volume. The release is no-go if a runaway component remains on a major page.

12. Search validation

  • Verify domain ownership in Google Search Console and other relevant official webmaster tools.
  • Submit the sitemap index and news sitemap after production validation.
  • Inspect representative article, policy, author, edition and form URLs.
  • Monitor indexing exclusions, duplicate canonicals, soft 404s, structured-data errors and manual actions.
  • Track old institutional URLs until redirects and indexing have stabilised.
  • Review search queries and referral pages for unexpected commercial or duplicate-page exposure.
  • Do not declare completion merely because a sitemap was submitted.

13. Current official standards basis

Area Official guidance applied
Article and author markup Google Search Central Article and ProfilePage structured-data guidance
Dates Google Search Central byline-date guidance
Languages and regions Google Search Central multilingual and multi-regional guidance
Sitemaps Google Search Central sitemap and news-sitemap guidance
Canonicals and redirects Google Search Central canonicalisation and redirect guidance
Paid links and commercial pages Google Search Central outbound-link and spam policies
AI discovery Google Search Central AI-features guidance and OpenAI publisher crawler guidance
DOCUMENT CONTROL DETAILS
Issued by Subkuz Media Networks Private Limited
Corporate Identification Number CIN U72900JH2020PTC014521
Document classification Developer implementation instrument
Effective date 6 August 2026
Last reviewed 7 August 2026
Previous legal name during statutory disclosure period Subkuz Media Networks Private Limited