MedllyBot
MedllyBot is the web crawler operated by Medlly Supplies. If you found this page in your server logs, here is what it does and how to control it.
What it does
MedllyBot visits the public websites of businesses in the health, wellness, aesthetics and research-supply industries to keep a business directory up to date. It reads only publicly available pages, such as a home page, contact page, about page, services page and product pages, and records business information that the business itself publishes: name, address, business phone and email, services and products offered, and links to its other public profiles.
How it behaves
- It always identifies itself with the user agent
MedllyBot/<version> (+https://crawler.medllysupplies.org). - It obeys
robots.txt, including rules for theMedllyBottoken, and honorsCrawl-delay. - It makes one request at a time to a site, waits between requests, and slows down or stops when a site returns errors or asks it to wait.
- It never logs in, submits forms, solves captchas or tries to get around any block. If your site blocks it, it leaves.
- It only fetches a small number of pages per site and revisits them infrequently.
How to block it
Add this to your site's robots.txt. MedllyBot checks it before crawling and at least once a day.
User-agent: MedllyBot
Disallow: /To block only part of your site, list those paths instead of /.
Remove your business or stop contact
Email crawler@medllysupplies.org with your website address. We will stop crawling your site and remove your contact details from our directory, usually within 2 business days, and keep a record of the request so we do not add them again.
Verify it is really us
Requests from MedllyBot come from the IP addresses listed below. Each address has reverse DNS pointing to a host under medllysupplies.org. A machine-readable list is at /ips.json.
138.197.114.6