Robots.txt signposts critical paths.

Protecting sensitive paths is a critical to maintaining your site security. Just as every house has a weak access point, every endpoint has it's weak points.

Trying to manage bots with Robots.txt is like managing a college bar with no ID checks. You may get away with it for a while, but sooner or later, it’s going to lead to illegal behaviour and a bad headache.

Many of you will be all too familiar with the challenges of robots.txt.

Hackers use your “no follow” instructions as a calling card to go straight to the areas you want to protect. It’s like signposting which flower pot the spare key to your house is hidden under.

We scan your robots.txt and automatically match it against our database of legitimate bot services. We show you the bot behaviour profile by category, and provide a recommendation engine so you can just automatically apply the suggestions. We then enforce your robots.txt rules to ensure the bots don't then access your critical paths and sensitive data. This is all automated for you.

Using our Command & Control centre, shown above, you can instantly select multiple domains and critical paths and simply block access to them.

You can specifically add the login and admin paths you need to protect and create?dynamic rules?which adapt according to the threats on your actual path.

You can now finally enforce Robots.txt from the vast majority of bots that don't obey it - and keep your sensitive paths hidden from the bots and the hackers for good.