What is usenet for databases?
Apr. 9th, 2008 07:43 pm![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
jackRecently my "books lent and borrowed" list ballooned by 100% or more, and it occurred to me how useful this would be if it were stored in a simple database, so "Jack, friend, lent/borrowed, book" would automatically contribute to friend's list as well. (You might make a few tweaks on a server large enough that everyone doesn't automatically trust each other such as recording if both people have approved the entry.) And if it's returned, either of us can check it off.
Obviously someone used to web programming could knock up something that works in a simple way in an evening if they put their mind to it (though I'm rather rusty myself).
However, this is obviously limited by server. Someone would write it and host it on chiark, or other server, and invite people to use it, but if someone elsewhere did the same thing, you'd have to have an account on both if you had friends on both. (Which isn't a big deal for _this_, but I'm curious, and would be great for many applications, maybe even social networking ones.)
What is the most obvious way of letting the two (or many) servers cooperate? Tim described this as "usenet for databases". Mobbsy said the nearest thing that sprang to mind was DNS, which do trade information back and forth. You might find the bandwidth wasn't even that high -- eg. many web apps send email alerts anyway, so the extra hit of sending the data to another server mightn't be so much extra.
(I'm more interested in the question about the technology of distributed databases, but I'm interested in the book thing too -- I haven't searched yet, so it probably exists, or can be integrated to librarything or similar, and if you know of it please do tell me, but that wasn't the real reason for the post.)
(Another wrinkle is my envisaged service/applet/facebook-app I may have mentioned before called "not a bank" which does the same thing as book loans but for money and other fungible amounts, where it can be automatically rebalanced so all your debts cancel out and you owe/are owed by just one person you know. It would be emphasised this was an aide memoire for small loans to someone you trust to pay, or don't care if they don't, but you might not remember.)
(Distributed databases have lots of problems with being transaction safe and not getting data desynchronised between servers, etc. These applications seem superficially more suited as: each person can have a home server that might be considered authoritative; or at worst one server has to push a change to another one; and should be low risk if something does go wrong. A big headache (or showstopper) is security, need servers trust each other? Can you arrange it so if some idiot trusts a friend on a bad server, the worst that happens is they individually get spammed, but not actually corrupted?)
Obviously someone used to web programming could knock up something that works in a simple way in an evening if they put their mind to it (though I'm rather rusty myself).
However, this is obviously limited by server. Someone would write it and host it on chiark, or other server, and invite people to use it, but if someone elsewhere did the same thing, you'd have to have an account on both if you had friends on both. (Which isn't a big deal for _this_, but I'm curious, and would be great for many applications, maybe even social networking ones.)
What is the most obvious way of letting the two (or many) servers cooperate? Tim described this as "usenet for databases". Mobbsy said the nearest thing that sprang to mind was DNS, which do trade information back and forth. You might find the bandwidth wasn't even that high -- eg. many web apps send email alerts anyway, so the extra hit of sending the data to another server mightn't be so much extra.
(I'm more interested in the question about the technology of distributed databases, but I'm interested in the book thing too -- I haven't searched yet, so it probably exists, or can be integrated to librarything or similar, and if you know of it please do tell me, but that wasn't the real reason for the post.)
(Another wrinkle is my envisaged service/applet/facebook-app I may have mentioned before called "not a bank" which does the same thing as book loans but for money and other fungible amounts, where it can be automatically rebalanced so all your debts cancel out and you owe/are owed by just one person you know. It would be emphasised this was an aide memoire for small loans to someone you trust to pay, or don't care if they don't, but you might not remember.)
(Distributed databases have lots of problems with being transaction safe and not getting data desynchronised between servers, etc. These applications seem superficially more suited as: each person can have a home server that might be considered authoritative; or at worst one server has to push a change to another one; and should be low risk if something does go wrong. A big headache (or showstopper) is security, need servers trust each other? Can you arrange it so if some idiot trusts a friend on a bad server, the worst that happens is they individually get spammed, but not actually corrupted?)
no subject
Date: 2008-04-10 08:29 pm (UTC)*Waves*
Databases: Sounds like you want some sort of multimaster system.
The system we use at work is based on knowledge-dates. We timestamp every change to the database, enact it locally, then pump out a message to our other servers telling them what happened and when. We also store history for each table.
When server A gets a message from server B saying that change C happened at time T, server A applies the change to the row as it was at time T then reapplies all changes to the row since then (if any).1
This all works given a messaging service that guarantees a message is processed exactly once (possible, even over dodgy IP) and the servers have synchronised clocks (also possible). It copes with high latency, even to the extent of a server dropping off the net for days before coming back online.
None of the data is guaranteed up-to-date, but there is a guarantee that at some point in the future everyone will agree about what happened in the past. I consider that a bretty big win.
[1]: The implementation we use is a little more complicated and a lot more efficient than this sounds.