Content Web Scraper

What is this?

When a page sends us an ad request, we scan its content to determine the context in which those ads were displayed. This contextual data is cached in order to improve the ad serving experience and allow advertisers to show ads alongside content they believe is relevant to their target market.

We believe this benefits the entire ecosystem — including content creators, advertisers, and consumers — by increasing advertiser confidence and providing a more relevant ad experience for consumers.

If you'd like to whitelist our scraping efforts, you can find our list of IPs here:
https://ttd-content.adsrvr.org/ips

Can I block it? And what happens if I do?

Blocking the bot means that some advertisers won’t be able to determine the context of your pages, which could impact the ad bidding on your domain through The Trade Desk platform.

If you’d like to block or temporarily suspend the bot, you can add or update robots.txt on your domain to stop requests from TTD-Content.

Disallow a specific path:

User-agent: TTD-Content

Disallow: /private/

Block TTD-Content from all URLs containing variables:

User-agent: TTD-Content

Disallow: /*?

Block TTD-Content from all URLs:

User-agent: TTD-Content

Disallow: /

How do I report an issue?

Please reach out to [email protected] to report any problems.