With the rise of LLMs and their need for vast amounts of training data, there is also a need for a way to indicate whether data may be used to train such models. This article proposes two ways for websites to opt-out from being crawled for training purposes.

This is done while being fully aware of the limitations of relying on the developers of AI models to respect this convention. Server-side blocks could archive this goal more effectively, but they also rely on the convention of bots identifying themselves by a user-agent, and it is increasingly difficult to maintain a comprehensive list of user-agents used to crawl training data. Establishing a voluntary self-commitment can at least provide a convenient way of dealing with good-faith actors, while at the same time providing a way of identifying bad-faith actors.

When I got rid of my previous Django-based website and switched to Hugo, the idea was to have a low-maintenance website. I didn’t have much time and all the content was quite old. However, I always had the idea of bringing back some of the old content and even adding new articles at some point.

In the last few months I’ve been writing articles for Lutra Security and perhaps it’s time to revive this site as well. But before I really get to it, this site needs a few more features. Articles are not an end in themselves. Ideally a good article should make you think and maybe even start a discussion. For this, we need an option to comment on articles.