Tor Hidden Service Search Engine Indexing

Ahmia (ahmia.fi) is operated by Juha Nurmi as an open-source project and is the most comprehensive public .onion search engine. Ahmia crawls submitted .onion addresses, indexes their content, and provides a search interface accessible via both clearnet (ahmia.fi) and a .onion address. The crawler accesses .onion services via Tor, follows links, indexes text content, and excludes sites that explicitly opt out via robots.txt. Ahmia maintains a safe-to-use policy: it attempts to exclude sites that facilitate serious harm. Ahmia's search index enables users to discover .onion services by keyword search, providing the primary organic discovery mechanism for .onion services (equivalent to Google for clearnet). For a .onion service that wants organic traffic: Ahmia indexing is the most important discovery mechanism.

Submit your .onion service to Ahmia via the submission form at ahmia.fi/add (accessible via clearnet or Tor). Provide the .onion URL (without specific page path - just the root address). Ahmia's crawler will then discover and index the site's content. After submission: Ahmia's crawler typically visits the site within days to weeks. The site appears in search results after the crawler indexes it. Site data remains in the index as long as the crawler can access it. If the .onion service goes offline for extended periods, Ahmia may remove it from the index. To submit anonymously: access ahmia.fi/add via Tor Browser and submit without providing personal information. The submission form requires only the .onion URL, not any personal details.

robots.txt controls what Ahmia and other crawlers index. A .onion service's robots.txt is at http://youronion.onion/robots.txt. To allow all crawling (default if no robots.txt): no file needed. To block all crawling (private service, no indexing wanted): create robots.txt with User-agent: * and Disallow: /. Ahmia respects robots.txt directives. To allow indexing but exclude specific paths: User-agent: * Disallow: /admin/ (prevents indexing of admin area). The robots.txt itself is a public file - anyone can see which paths are excluded, which may reveal information about the service structure. For services that want to be discoverable: ensure robots.txt allows crawling, create meaningful page titles and content headings (Ahmia uses these for search snippets), and link between pages (Ahmia's crawler follows links to discover all pages).

XML sitemaps tell search engines which URLs to index. Create a sitemap.xml at http://youronion.onion/sitemap.xml. Format: standard XML sitemap with http://youronion.onion/page entries. Note: use http (not https) unless HTTPS is configured. The sitemap lists all important pages the service wants indexed. For dynamically generated .onion sites: generate the sitemap automatically from the application (most frameworks have sitemap generation plugins). Submit the sitemap URL to Ahmia: include it in robots.txt (Sitemap: http://youronion.onion/sitemap.xml line at the top of robots.txt). Ahmia's crawler reads the Sitemap directive from robots.txt and crawls all listed URLs. An up-to-date sitemap ensures all pages are discoverable even if Ahmia's link-following crawler misses some.

Beyond Ahmia: (1) Tor66 (another .onion search engine) accepts submissions and provides an alternative index. (2) The Hidden Wiki (various versions, community-maintained) lists .onion services by category. Submit to relevant categories to reach users browsing the wiki. (3) Community directories: subject-specific .onion directories exist for news sites, software tools, communication services, etc. Find relevant directories and request inclusion. (4) Cross-linking: having other indexed .onion services link to your service is the most organic discovery method - Ahmia follows links and discovers linked sites. (5) Clearnet visibility: if the service operator has a clearnet presence (without deanonymizing themselves), mentioning the .onion address on clearnet sites that Ahmia or Google index provides cross-network discovery. Users searching on clearnet can find the .onion address. (6) .onion list aggregators: maintain updated lists of .onion services for specific categories.