Skip to content

Glintscout's web crawler

GlintscoutBot

Glintscout builds B2B lead lists: businesses with their address, phone number and contact email. GlintscoutBot is its crawler. It reads the public contact pages of business websites to find the business's own contact email address, such as info@ on the business's domain. The websites it visits come from public business listings, such as Google Maps.

How to recognise it

Every request it makes carries this User-Agent header:

Mozilla/5.0 (compatible; GlintscoutBot/2.0; +https://glintscout.com/bot)

Where its requests come from

GlintscoutBot sends its requests from two fixed IP addresses and no others, each for its own job. Each has a reverse DNS name that points back to the same address (forward-confirmed reverse DNS):

  • 167.233.187.89, reverse DNS name crawler.glintscout.com: it reads the websites of the businesses our customers search for
  • 77.42.10.240, reverse DNS name crawl.glintscout.com: it keeps Glintscout's business directory up to date

So you can check a visitor calling itself GlintscoutBot with two lookups:

$ host 167.233.187.89
89.187.233.167.in-addr.arpa domain name pointer crawler.glintscout.com.
$ host crawler.glintscout.com
crawler.glintscout.com has address 167.233.187.89

$ host 77.42.10.240
240.10.42.77.in-addr.arpa domain name pointer crawl.glintscout.com.
$ host crawl.glintscout.com
crawl.glintscout.com has address 77.42.10.240

Not sure about a visit that calls itself GlintscoutBot but comes from another address? Write to us (see Contact below) with the address and the time, and we will tell you whether it was us.

What it reads

  • Your robots.txt first, and then only what it allows.
  • The homepage and at most 3 more pages of the same site: a contact page, an imprint (Impressum) or an about page.
  • Your sitemap (sitemap.xml), only when the homepage links none of those pages.
  • One request at a time and at most 5 requests per site, counting robots.txt, redirects and the sitemap. It stops as soon as it has found the address.
  • Public pages only: it does not sign in, submit forms or run the pages' JavaScript.

It obeys robots.txt

GlintscoutBot reads your robots.txt before anything else and follows the rules for GlintscoutBot, or the rules for all crawlers (User-agent: *) when there are none for it by name. To keep it off your whole site, add:

User-agent: GlintscoutBot
Disallow: /

It reads robots.txt again at the start of every visit, so a change applies from its next visit. You can also block only part of your site, for example Disallow: /private/.

What it never does

  • It never solves CAPTCHAs or bot challenges (such as a "Just a moment" page) and never tries to get around them: a challenge, an "access denied" answer (401 or 403) or "too many requests" (429) ends its visit. When several sites of one hosting provider answer like this, the address that met them slows down for that whole provider, honours its Retry-After and may visit such a site once more later.
  • It never hides: it always names itself in the User-Agent, never poses as a browser and uses no proxies. Each of its two addresses keeps to its own job, and it never switches to the other one to get past a block: if a site blocks one of them, challenges it or disallows it in robots.txt, the other address leaves that site alone too, for six months (two weeks after a "too many requests"). A robots.txt rule for GlintscoutBot applies to both addresses.

What it keeps

Only email addresses on the business's own domain: on the site example.com, an address such as info@example.com. Addresses on other domains and free email addresses (such as Gmail or Outlook) are left out, and the pages it reads are not stored. How Glintscout uses these addresses, and your rights, are described in our Privacy Policy.

How to opt out

  • For your website: block GlintscoutBot in robots.txt as shown above. It then reads nothing more from your site.
  • For an address Glintscout already holds: ask us to delete it on the Remove your data page. It is deleted and kept from being collected again, by GlintscoutBot too.

Contact

Questions about GlintscoutBot, or a visit you did not expect? Write to support@glintscout.com and include your site's address.