DomainStatsBot

IP: 136.243.17.142, ..59.237, ..222.140; 148.251.121.91

These four IPs, exact down to the last digit, have been used interchangeably since 2019.

The first User-Agent showed up very briefly in 2016; when next seen, in 2019, it had settled on the second form.

Thanks to some inattentiveness on my part, this robot was authorized for quite a while before I noticed that it had started requesting pages in one roboted-out directory. Hmph. Once could be a glitch; recurring over more than a year is misbehavior.

LightspeedSystemsCrawler

IP: 207.200.8.180; 52.36.125.174, 52.36.251.200

LightspeedSystemsCrawler Mozilla/5.0 (Windows; U; MSIE 9.0; Windows NT 9.0; en-US)

It is possible this is really two robots. But I don’t like either of them, so it makes little difference. From 207.200 it requests the root followed by all pages linked therefrom. (Or, if you want to be hairsplittingly accurate: all pages linked from the 403 page. They just happen to be the same list.) From 52.36 it requests robots.txt—where it has been Disallowed for years—generally followed by a few requests for the root.

linkdex

Mozilla/5.0 (compatible; linkdexbot/2.2; +http://www.linkdex.com/bots/)

In the past it has been absent for long stretches. Currently it shows its face every month or so.

ltx71

IP: 52.2.10.94, 104.154.58.95

ltx71 - (http://ltx71.com/)

I first became aware of this robot when it showed up requesting new ebooks. I later discovered that it also likes the page linked from my profile at Webmaster World; I just never noticed because it was always blocked for one reason or another. Consi­dering how often this robot asks for robots.txt—on some visits, it’s all it requests—it is suprising that it has not yet got around to reading the thing.

MegaIndex

IP: 176.9.50.244, 176.9.28.137

Earlier they lived at 88.198.48.46. After taking the year 2016 off, they returned with two new addresses in random alternation.

Some robots request robots.txt only after getting the front page; this is one of them. (“Oh! I just assumed I was allowed to crawl everywhere. Gee, I guess I was wrong.”) But, since they proceeded to ask for the entire contents of a roboted-out directory, it becomes pretty academic.

Qwantify

The peril of putting a robot in the Ignore bin is that you don’t notice when it starts misbehaving—in this case by repeatedly crawling directories that are clearly and unambiguously disallowed in robots.txt. In years past, you would have found this robot in the Miscellaneous list. No more.

IP: 91.242.162, 194.187.169-171, 162.19.101

Very rarely I have seen it from other IPs, but they are so rare I can’t be sure they are not fakers. The 162.19 range is a newer one, first seen in mid-2023. You might expect the full 194.187.168-171 (in CIDR terms, 194.187.168.0/22) but I have never seen it from 168.

The minimalist “Qwantify/1.0” and-that’s-all appeared sporadically until March 2021. For the first few years it requested only the favicon. It then took a prolonged vacation from mid-2018 to December 2019. When it returned, it changed to root / followed by favicon. Around April 2021 it was replaced by the full 1.0 version, with the same root-plus-favicon request pattern.

You might reasonably think that the Qwantify/2 versions came after Qwantify/1.0. In fact they are older, going back to 2015. The change from 2.3w to 2.4w happened in the latter part of April 2017, with no overlap. Qwantify/2.4 crawled from 91.242, requesting only robots.txt. Can’t think why it bothered, since it persistently ignored the Disallow it found there. With an isolated exception, I last saw it in 2022.

Do not ask why Qwantify suddenly ventured into (presumably) “development” and “production” many years into its existence. “Qwantify-dev” first showed up in November 2022, “Qwantify-prod” in July 2023. The third UA is just an example. Beginning around December 2023 “dev” came interspersed with “dev188”; from around November 2023, “prod” sometimes appended a number of its own. To date (January 2024) I’ve seen 112, 147, and 524.

Incidentally, the “help.qwant.com” URL leads to an information page listing half a dozen different User-Agents—four of which I don’t think I have ever set eyes on. The page also says that the robot “respects the robots rules standard”. This is, to use an arcane technical term, a lie.

These versions seem to have been short-lived experiments. While they existed, they were used in random alternation. With one exception, I never saw Mermoz before 2018, and haven’t seen it at all since March 2019. It was replaced by Bleriot/1.1—always from the 91.242 IP—which was last seen in March 2020. Bleriot/1.2.1 made a brief, overlapping appearance near the end of 2019.

SemanticScholarBot

IP: 54.70.40.11

Mozilla/5.0 (compatible) SemanticScholarBot (+https://www.semanticscholar.org/crawler)

Initially I thought this was a compliant robot, because it stayed away when it found its name in robots.txt. But after I authorized it, it headed straight for a roboted-out directory. Nice try, but no cigar.

vuhuvBot

IP: 185.93.54.51

This is a fairly recent arrival to the unwanted-robots collection. It requests robots.txt only after its (blocked) page request—always a sure-fire way to continue being blocked.

Blackboard Safeassign

34.231.5.82, 34.202.93.213

Blackboard Safeassign

For several years I gave this robot a free pass because it’s doing more-or-less-legitimate work, checking for plagiarism. On any one visit it might pick up anywhere from one to a dozen pages, most often in the /ebooks/ directory. Requests always came in pairs: first HEAD, then GET.

When I moved to HTTPS, Blackboard Safeassign racked up a disproportionate number of redirects because it came in one day and asked for the same group of pages over, and over, and over again. Did they think the text was going to change every three seconds? When it reached the point of up to two hundred requests for the same page within a couple of minutes, I threw in the towel. So far they haven’t seemed to get the message; blocked robots seldom do.

For editorial comments on plagiarism-checkers in general, see TurnitinBot under “Former Robots”.

PocketParser

PocketParser/2.0 (+https://getpocket.com/pocketparser_ua)

I originally put this with the Targeted Robots, but its requests don’t always fit the pattern. Happily, it has never been frequent; once it was absent for a whole year. And then it returned, darn it.

Seekport Crawler

IP: distributed

Each individual visit—typically six requests—used a single IP, but they varied all over the map.

First seen in August 2019, this robot was mercifully rare until 2020. And then, in June 2022, it must have come under new management, to be replaced with the compliant SeekportBot.

Chrome/50 with referer

IP: assorted OVH ranges, especially 94.23, 158.69.248, 188.165

This pattern may finally be on its way out; I’ve only seen it once since October 2019. Fortunately it is easy to block, so I can ignore it. The referer is always different—not referer spam but various superficially plausible searches, often from reputable sites. In former years, a robot with the same UA but no referer was common.

Contact Six

This pattern showed up towards the end of 2022, and continues to be seen a few times a month. Each visit consists of—on this site—exactly six requests from a single IP and UA:

/boilerplate/contact.html without referer

immediately followed by five requests, all with the abovenamed /contact.html as referer:

/ (root)
/boilerplate/search.html
/boilerplate/contact.html
/boilerplate/legal.html
/boilerplate/about.html

Now and then I see the same pattern on my personal site. Since that site isn’t big enough for a search page, “Six” becomes “Five”.

“Contact” Botnets

At any given time, there will be two or three variations of what I’ve labeled the “contact” botnet. The IP varies; the UA varies; the headers vary. What they all have in common is that the request will include /boilerplate/contact.html at some point. (Clever robots! They have been taught the word “contact”, and know to look for it, no matter where on the site the contact form may happen to be located.) The exact pattern tends to run for a few years before they get bored and change the script.

Four Requests

GET /some-inner-page

(always a different one) giving root as referer, as above

GET / (root) with auto-referer
GET /boilerplate/contact.html giving root as referer
GET / (root) with auto-referer

Like the “one plus three”, this exact pattern has been around for years. Currently they are all blocked—but only thanks to headers. IP alone won’t do it; the “Contact botnet” plagued me for years. A detail of this pattern is that the first referer has a final / slash, but the other three don’t. And, of course, they must have friends inside, because the first request is for some page that exists but is not visible from the front page.

Contact At Home

The original version of the “At Home with the Robots” group of pages was a single page called athome.html. It did not take robots long to find it—“Look! Someone’s talking about me!”—leading to a fixed pair of requests:

GET /fun/athome.html
GET /boilerplate/contact.html giving /fun/athome.html as referer

Before long, they figured out that they didn’t need to make the first request, but could proceed directly to the contact page, claiming that athome.html had sent them.

It has also not taken them long to learn that the old page has a new URL. Within days I was finding the request pattern:

GET /boilerplate/contact.html giving /fun/robots/ as referer

Contact POST

Once you’re on the contact page, the obvious next step is to, well, make contact. Or, better yet, use the page’s built-in POST mechanism to attempt to do something evil. By mid-2018 this led to the pattern:

This variation first showed up in May 2018, though it didn’t really become common until September. Fortunately, the POST request was always blocked because reasons. (I can’t give away everything, after all.) Still more fortunately, the script very quickly settled on a single user-agent:

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36

This is distressingly up-to-date—it is still in occasional use by humans—but where there’s a will there’s a way.

Two Plus One

This pattern showed up near the end of 2019 and shows no sign of going away, though it has been less frequent in recent months. Each visit consists of exactly three requests from a single IP:

/fun/robots/ with / (root) referer
/ with root referer without the final / (slash)

and then, anywhere from a few minutes to—rarely—over an later,

/ without referer, using a different UA

Sometimes the first request is for some individual page in the /robots/ directory—the directory you’re reading right now—but the front page is most common.

For example:

And repeat, anywhere from one to seven times in the course of a day. So far the record is eleven sets—but even one is one too many.

The first two requests are blocked because of the obviously bogus referer; only the third request gets through. So far I haven’t found a unifying feature that would allow me to block them all, though many do come from bad neighborhoods. Others come from seemingly human ISPs, especially in Asia, Africa or Latin America—parts of the world where, for a variety of reasons, people’s personal computers are especially vulnerable.

The UA in that third, not-yet-blocked request has a final quirk. It’s always different from the opening pair, and always plausibly humanoid. But it will then be reused on later visits, up to half a dozen times before yielding to a new one. This points to craftiness on the botrunner’s part: it’s no use blocking a specific UA, because tomorrow it will be replaced by something else.