Towards a better Usenet Everybody who has been on Usenet the past few years knows that it has become a cesspool. The old joke "imminent death of Usenet predicted, film at eleven" hasn't been funny for a long time. So let's do something about it. This document describes my preliminary thoughts on the subject. Please send my any comments or suggestions. What I want to do is rebuild the network from the ground up. In the new Usenet, messages will be made up of four parts: 1. The message ID: this is a unique identifier for each message that also identifies the originating host and the date/time when the message was posted. The msgid should be about 8 to 12 bytes long. 2. The headers. These include author, subject and signatures for the message text and the attachment. There is only limited space in the official header part of the message, so only information that should be available _prior_ to downloading the message text is allowed here. This information should be in 7 bit ASCII. 3. The message text, including extra (longer) header info. The message text may be in 7 bit ASCII, ISO Latin, Unicode or any other character set. The text may be in ASCII or (a subset of) HTML. The network will convert from HTML to ASCII if the client requests it. 4. An attachment. There are many ways to bundle multiple files together, so there is no reason to have more than one attachment. The attachment is stored and transmitted as a binary file. These messages live in "realms". Many different realms may exist, each with separate policies for message distribution, language, character set, and many other variables. Each realm has a (fairly small) number of core servers that control the entrance of messages into the realm. For instance, an "nl" realm could run on ten servers in The Netherlands with the ISO Latin character set and Dutch as the language, distributing msgids, headers and message texts, but not attachments, more or less like the nl.* hierarchy in the current Usenet. The "microsoft patches" realm could run on one or two servers in Redmont in English with the Windows-1252 character set distributing msgids, headers, message texts and attachments, so Microsoft can easily distribute software patches to everybody who's interested. Slashdot could run a realm that only distributes msgids and headers. If someone is interested in an article, they'll have to get it from the /. site, but since the headers show up in the Usenet network, so it is no longer necessary to use Slashdots interface to navigate threads: this can be done in a newsreader program. When a client (newsreader program) wants to display a message, it can request the headers, message text and/or attachment from the closest news server. If this server has the information available, it will be transmitted to the client. The news server may also fetch the information on behalf of the client, much the same way a proxy does. In some cases, the news server is unwilling or unable to serve the requested information, so the client has to contact the originating server and get the information directly from there. This would happen with news sites that like to inject their headers into the news system but want to keep control over how the articles are served, like they have now on their web sites. Since attachments can get very large, it is possible to request only part of an attachment. To get the best performance, a client may even request different parts of an attachment from different servers at the same time. When a message is posted, the news server authenticates the user. A header indicating the identity of the user is added. This will probably be something like "user x on server y". Certain statistics about users are kept on their home server: first posting date, number of postings, total number of complaints, number of postings with complaints. Users are free to select a new ID whenever they want but they will be known as "new users" for a while and probably be ignored for a bit until it becomes clear they are not spammers. The originating news server contacts the realm core server (or even realm servers for several realms, messages may be posted to more than one realm), which authenticates the originating news server. Then, the msgid and headers are added to the distribution list. The msgid + headers distribution lists that each server creates at regular intervals are cryptographically signed. Because the headers also include hashes for the text and attachment, each message is thus protected as soon as it enters the realm core, but only minimal crypto overhead is necessary. The distribution lists are sent to all other core servers, either over direct connections or forwarded from one to another or as multicasts. In most cases, also the message text and maybe even the attachment is transmitted to the other realm servers. Each core server keeps timestamps for the last updates from all other core servers, so it is easy to distribute all information across all core servers without duplication or further checking. Non-core servers may also keep timestamps and receive updates from more than one core server, or just take a full feed from one server. Newsgroups no longer exist: in their place there are different kinds of threads: keyword threads, persistent threads and implicit threads. Implicit threads are created by simply replying to previous messages. Keyword threads are created by adding a number of keywords to the message. Persistent threads are a lot like newsgroups. Everybody can become a moderator and inject information about messages into the system. This can be "very interesting message", "spam", "off topic", "frequently asked question" or some other predefined category. Others may subscribe to this moderation info. For instance, a client could request "all messages in realm "comp" keyword "cisco" thread "juniper sux" except messages flagged "spam" by "spamcop"", or "all messages in realm "slashdot" flagged "very interesting" by "newshound"". Also, users may use the system to complain about other users. These complaints will be added to a user's statistics. These statistics can then be used to automatically blacklist certain users in a killfile-like fashion. Obviously, this is a rather large project. I haven't decided if I even want to start. I can't really afford to spend a lot of time on such a project if it is unlikely to generate any income. On the other hand, it is too cool not to do anything with it. Iljitsch van Beijnum, aug 2001. Ideas can't be copyrighted, but please give proper credit. See my website http://www.muada.com/ for contact information.