Ni1 is the collective name for the web crawlers used by Ni1 Search to discover, analyze, and index publicly accessible content across the internet.
Ni1 currently operates two crawler types:
- Ni1 Web Crawler — A general-purpose crawler that discovers and indexes web pages, documents, and other text-based content.
- Ni1 Media Crawler — A specialized crawler that discovers publicly accessible images and media resources.
You can identify the crawler subtype by examining the HTTP User-Agent header included with each request. While the crawlers may perform different functions, both respect the same robots.txt directives and website owner preferences.
How Ni1 Accesses Your Website
Ni1 discovers new URLs through links found on previously crawled pages, XML sitemaps, and other publicly available discovery mechanisms.
For most websites, Ni1 is designed to crawl responsibly and efficiently, minimizing server impact while maintaining a comprehensive and current search index. Crawl frequency may vary depending on factors such as website size, update frequency, server responsiveness, and webmaster directives.
Before accessing content, Ni1 checks the website’s robots.txt file and follows applicable instructions. Website owners retain full control over crawler access through standard web protocols.
Public Content Only
Ni1 is designed to index publicly accessible information only. Our crawlers avoid intentionally accessing private, restricted, or protected content and respect website owner directives regarding crawl permissions.
Blocking Ni1 From Your Website
Website owners can control Ni1 crawling through robots.txt.
To prevent Ni1 from accessing your website:
User-agent: Ni1
Disallow: /To allow Ni1 to access your website:
User-agent: Ni1
Allow: /Specific directories may also be controlled individually:
User-agent: Ni1
Disallow: /private/
Disallow: /admin/Crawling and Indexing
Crawling and indexing are separate processes.
Blocking Ni1 from crawling content prevents our crawlers from accessing that content. If content should remain accessible to visitors but not appear in search results, website owners should use appropriate indexing directives or access controls.
For content that should not be accessible to either users or crawlers, authentication and access restrictions should be used.
Our Commitment
Ni1 was built on the principles of transparency, privacy, fairness, and respect for the open web.
Our crawlers:
- Respect robots.txt directives.
- Identify themselves clearly.
- Focus on publicly available content.
- Minimize unnecessary server load.
- Operate without collecting user browsing profiles.
- Support an open and accessible web ecosystem.
Ni1’s goal is to build a high-quality, privacy-focused index of the public web while giving website owners clear and effective control over how their content is accessed.