The New York Times sues Brave, citing its unnamed crawler as evidence of intent

Search engines send programs called crawlers to collect web documents. Site owners look at the name a crawler supplies with each request, known as its user agent, to decide whom to let in and whom to block. On November 2, The New York Times sued Brave, which makes a browser and a search engine, in the U.S. District Court for the Southern District of New York over a crawler that did not identify itself by name.

In its complaint, The Times alleges that Brave runs two businesses using its own index of more than 30 billion web documents. One is Ask Brave, which feeds text from the index into a language model to answer questions directly. The other is the Brave API, which sells the same index to AI companies. The Times says its articles were included in that index without permission.

The complaint also challenges a browser feature. The Brave browser has a "Paywall blocker" setting that gets past subscription paywalls to display articles, and Brave promoted it as "Bypass Paywalls". The Times alleges that this feature violates Section 1201 of the DMCA, a provision of U.S. copyright law that prohibits circumventing access controls.

The crawler's name deserves particular attention. In its search help documentation, Brave explains that it does not give its crawler a distinct name to avoid discrimination by sites that allow only Google. Instead, it says its crawler stays away from pages that Googlebot cannot access. The Times argues that this choice denied site owners a way to block the crawler and cites it as evidence of willful infringement.

That changes the question crawler developers must answer. Until now, identifying a crawler by name has largely been a courtesy to site owners, while hiding the name has offered a practical advantage: fewer blocks. In this lawsuit, however, withholding the name is being read as a deliberate act that deprived site owners of a way to block it. If linking to articles in search results and having AI read them to compose answers operate under the same name, site owners cannot choose to allow just one.

Teams building crawlers or AI answer features therefore need to start with the name. They should publish a separate name for each use and let site owners block those names through robots.txt. Google also provides a robots.txt name, Google-Extended, that lets sites decide separately from search whether their content can be used for Gemini training and as a basis for answers. Google says this name does not affect visibility in search.

The contents of the complaint are allegations by The Times, however, not facts established by a court. Brave has said that it goes only where Googlebot can go, leaving room for it to argue that sites allowing Google have already opened the door.

A crawler's name has become more than a technical choice. It is now part of the record to be interpreted in court. Site operators have a task of their own: check whether robots.txt is a list that allows only Google, then set rules for each crawler name so they can make separate choices about search visibility and AI use.

Sources

1. Complaint in The New York Times Co. v. Brave Software, Inc., U.S. District Court for the Southern District of New York, case number 1:26-cv-09549, filed November 2, 2026.
2. Brave Search Help: Brave Search Crawler, accessed November 3, 2026.
3. Google crawling documentation: Google-Extended, accessed November 3, 2026.
4. The Verge: The New York Times sues Brave web browser for AI scraping and bypassing paywalls, November 2, 2026 (U.S. time).