Cur8Bot
Cur8Bot is Cur8's web crawler. Cur8 runs discovery sites for beer, coffee, wine and spirits, and Cur8Bot keeps the producers listed on them up to date.
Why it visits
Every producer on Cur8 has a page showing what they make, with current prices and stock and a link to buy from their own shop. Cur8Bot fetches that information from the producer's website, so the page matches what's really on sale.
What it reads
Only what your website already shows the public: products, prices, stock, photos and descriptions. Where your shop publishes a machine-readable copy of its catalogue, Cur8Bot reads that instead of your pages, which is lighter on your site.
What it never does
- Log in or read anything behind a password.
- Fill in forms, add to baskets or place orders.
- Collect anything about your customers.
How often it visits
A few times a day per site, one request at a time. If your robots.txt sets a Crawl-delay, Cur8Bot waits that long between requests (up to 10 seconds).
How to recognise it
Cur8Bot always uses this user agent:
Mozilla/5.0 (compatible; Cur8Bot/1.0; +https://cur8search.com/company/bot)
Anyone can copy a user agent, so every Cur8Bot request is also signed using Web Bot Auth. Cur8Bot runs on cloud servers without fixed IP addresses, so check the signature rather than the IP. Our public keys are here:
https://cur8search.com/.well-known/http-message-signatures-directory
How to control it
Cur8Bot follows robots.txt. It checks yours before visiting and picks up changes within a day. To keep it out of your whole site:
User-agent: Cur8Bot Disallow: /
To keep it out of part of your site, list just those paths. If no rule names Cur8Bot, it follows your rules for User-agent: *.
User-agent: Cur8Bot Disallow: /wholesale/
If your firewall or bot protection asks Cur8Bot to prove it's a browser, it won't try to get past. To let it in, allow requests whose user agent contains Cur8Bot.
Get in touch
Questions, or want Cur8Bot to visit less, stop, or pick up something it's missing? Email us with your website address: