Categories
Web

What is ClaudeBot/1.0

ClaudeBot is a web crawler used to download training data for training LLMs (Large Language Models). This bot is operated by Anthropic, the company that runs Claude.ai.

ClaudeBot/1.0; +claudebot@anthropic.com
Categories
Security Web

GitHub follow commits as RSS

There might be times when you want to closely follow updates to a public GitHub repository. I had this use case in June 2024 when following the Swiss NCSC for cyber threat information regarding DDoS attacks.

I solved this problem by setting up an API key and regularly querying the main branch commits. However, this process is unnecessarily complex. A much easier way is to follow a repository as an RSS feed.

Categories
Web

Understanding GLCID, WBRAID, and GBRAID

In today’s digital marketing world, tracking how ads perform is crucial. Google has been helping advertisers track ad performance using something called GLCID. Recently, Google introduced two new tools called WBRAID and GBRAID. These changes are due to new privacy rules from Apple, and they help advertisers see how effective their ads are without compromising user privacy.

Categories
Web

Analyze HAR with HTTP Watch Studio

The HTTP Archive format, known as HAR, is a JSON-formatted archive file format used to log a web browser’s interactions with a website.

To save a HAR file in Chrome, open the Developer Console. The HAR file contains everything, including cookies and sessions, which can be considered secrets. Remember to be careful with whom you share this file. This can be useful if a website encounters an error condition. It can be a valuable way to debug rendering issues.

Categories
Web

ipapi.co and PowerShell free lookups

IP intelligence and looking up IP addresses has been a regular task for me for many years. Whether it’s debugging during attacks or monitoring crawler activity on websites, I have utilized various methods. I began using ipapi.co to look up IP addresses because it does not require an API key, making it convenient an it still allows for some bulk processing. This approach was ideal for the volume of lookups I performed in 2021. While this technique is still effective, exploring other options like locally hosted databases may be more appealing if you require a higher quota and performance.

Categories
Azure Sitecore Web

Redis GUI Client for Debugging

Working on Azure, most application directly interact with Redis. However, for debugging or handling incidents, visibility into Redis and its storage can be beneficial. The Azure portal seems to be constantly adding more features in this area. If this is insufficient, a Redis Client can be beneficial. I personally like this client.

https://goanother.com

Categories
Web

BLP_bbot/0.1

The BLP_bbot/0.1 user agent is associated with Bloomberg Financial Market and is referred to as “BLP” within their own internal naming conventions. It’s likely used by Bloomberg for web crawling purposes, potentially related to financial data aggregation and analysis​.

Categories
Web

RIPE public ASN list

I recently had a lot of contact with ASNs and I was looking for a full list of ASNs. There is an official and public list from RIPE.

https://ftp.ripe.net/ripe/asnames/asn.txt

https://ftp.ripe.net/ripe/asnames/asn.txt (2024-04-24)
Categories
Web

Powershell check link list

This PowerShell script can be used to check the status of a plain list of web links that you have copied to your clipboard.

Categories
Web

PHP gethostbyaddr($ip) too slow

Working on a project, I needed to perform reverse DNS lookups of IP addresses in PHP. The native function gethostbyaddr() seems to be very slow. I managed to vastly improve the speed of the lookups by switching to dns_get_record().

gethostbyaddr($ip);