Our crawler
If you look after a museum website and have seen EnfiladeBot in your logs — that is us. This page says who we are, what we take and how to shut us out.
EnfiladeBot is the crawler of a museum catalogue. It gives its full name and leaves the address of this page: a robot with no return address is one you are entitled to block, and you would be right.
Opening hours, prices, address, telephone, the ticket link and the list of exhibitions — the same things a visitor reads on your pages. We take no images, keep no whole pages and crawl nothing beyond those sections. What we take is credited to you with the date it was read, and the link leads to your page, not around it.
No more than one request every two seconds to a host; at night rather than during visiting hours; with conditional requests (ETag and If-Modified-Since), so most of the time we are told “not modified” and take nothing at all. Each kind of fact is revisited on its own schedule: exhibitions weekly, opening hours every three months, the address once a year.
A line in robots.txt — we parse it and obey it, and we settle doubtful cases against ourselves: a disallow outweighs an allow of equal length. A rule naming us replaces the general one entirely.
Mark up your opening hours and prices with schema.org — OpeningHoursSpecification, Offer, ExhibitionEvent. Then we take what you declared yourself instead of what we guessed from your markup, and your hours stop breaking every time the site is redesigned. Of sixty-one museum sites exactly one has done this.
Write to us through the contact form — a request to stop crawling is honoured the same day and without argument. Write to us.