300 |
never overwrite existing data. When a bug causes an application |
never overwrite existing data. When a bug causes an application |
301 |
to write malformed data, only the changes from one session |
to write malformed data, only the changes from one session |
302 |
will be lost; the previous version of the data will still |
will be lost; the previous version of the data will still |
303 |
be accessible. (Footnote: This makes Storm well suited as a basis |
be accessible. This makes Storm well suited as a basis |
304 |
for implementing experimental projects (such as ours).) |
for implementing experimental projects (such as ours). |
305 |
|
|
306 |
When used in a network environment, Storm ids do not provide |
When used in a network environment, Storm ids do not provide |
307 |
a hint as to where in the network a specific block can be found. |
a hint as to where in the network a specific block can be found. |
308 |
However, current peer-to-peer systems could be used to |
However, current peer-to-peer systems could be used to |
309 |
find blocks efficiently in a distributed fashion; for example, |
find blocks efficiently in a location independent fashion; for example, |
310 |
Freenet [ref], a few recent Gnutella clients (e.g. Shareaza [ref]), |
Freenet [ref], recent Gnutella-based clients (e.g. Shareaza [ref]), |
311 |
Overnet/eDonkey2000 [ref] also use SHA-1-based identifiers |
Overnet/eDonkey2000 [ref] also use SHA-1-based identifiers |
312 |
[e.g. ref: magnet uri]. |
[e.g. ref: magnet uri]. Footnote:However, we have not put a network |
313 |
(Footnote:However, we have not put a network implementation into regular use |
implementation into regular use yet and thus can only describe our |
314 |
yet and thus can only describe our design, not report on |
design, not report on implementation experience. |
|
implementation experience.) |
|
315 |
We discuss peer-to-peer implementations in Section 7, below. |
We discuss peer-to-peer implementations in Section 7, below. |
316 |
|
|
317 |
The immutability of blocks should make caching trivial, since it is |
The immutability of blocks should make caching trivial, since it is |
361 |
that the link connects. Xanalogical links are external and bidirectional. |
that the link connects. Xanalogical links are external and bidirectional. |
362 |
|
|
363 |
.. [#] Xanalogical storage is not limited to text. We speak about |
.. [#] Xanalogical storage is not limited to text. We speak about |
364 |
*characters* because it simplifies the explanation; pixels |
*characters* because it simplifies the explanation; picture's pixels |
365 |
or frames of video could be substituted. |
or frames of video could be substituted. |
366 |
|
|
367 |
In addition to content links, xanalogical storage keeps an index of |
In addition to content links, xanalogical storage keeps an index of |
375 |
queries the index for other documents containing this character, |
queries the index for other documents containing this character, |
376 |
and shows them as transclusions. Resolving links is a multi-step process. |
and shows them as transclusions. Resolving links is a multi-step process. |
377 |
Each link is modeled as two collections of characters: the two |
Each link is modeled as two collections of characters: the two |
378 |
endpoints of the link. To show links to a document, |
endpoints of the link. To show links to a specific document, |
379 |
the system firstly uses the link index to find links |
the system firstly uses the link index to find links |
380 |
to each character in the documment. Secondly, for each link, |
to each character in the document. Secondly, for each link, |
381 |
it looks at the *other* set of characters in the link-- the target |
it looks at the *other* set of characters in the link-- the target |
382 |
of the link, if the original character was the source, and vice versa. |
of the link, if the original character was the source, and vice versa. |
383 |
Thirdly, it looks for documents containing these target characters. |
Thirdly, it looks for documents containing these target characters. |
384 |
This way, even if both the source and target cjaracters of the link |
This way, even if both the source and target characters of the link |
385 |
are moved to a different document, the link stays connected to them. |
are moved to a different document, the link stays connected to them. |
386 |
|
|
387 |
Of course, doing any expensive operation for *every* character |
Of course, doing any expensive operation for *every* character |
396 |
character, and the number of characters in the span. |
character, and the number of characters in the span. |
397 |
This technique was first introduced in [ref ht02 paper]. |
This technique was first introduced in [ref ht02 paper]. |
398 |
|
|
399 |
In Xanadu, characters are written to append-only *scrolls* |
In Xanadu, characters are stored to append-only *scrolls* |
400 |
when they are typed [ref]. Because of this, we call the blocks |
when they are typed [ref]. Because of this, we call the blocks |
401 |
containing the actual characters *scroll blocks*. The documents |
containing the actual characters *scroll blocks*. The documents |
402 |
do not actually contain the characters; instead, they are |
do not actually contain the characters; instead, they are |
414 |
the data is transcluded. |
the data is transcluded. |
415 |
|
|
416 |
Our current implementation shows only links between documents |
Our current implementation shows only links between documents |
417 |
that are in memory at the same time [screenshot of xupdf]. |
that are in memory at the same time [screenshot of xupdf, perhaps ref too |
418 |
In the future, we will implement a global index atop of |
(submitted) ?]. |
419 |
|
In the future, we will implement a global distributed index at top of |
420 |
a distributed hashtable, with the scroll blocks' ids as the keys. |
a distributed hashtable, with the scroll blocks' ids as the keys. |
421 |
To find the transclusions of a span, the system will retrieve |
To find the transclusions of a span, the system will retrieve |
422 |
all transclusions of any span in the scroll block, then |
all transclusions of any span in the scroll block, then |
427 |
will be relatively small (limited by the amount of text |
will be relatively small (limited by the amount of text |
428 |
the user enters between two saves of a document), we hope |
the user enters between two saves of a document), we hope |
429 |
that this will not be a major scalability problem. Otherwise, |
that this will not be a major scalability problem. Otherwise, |
430 |
systems that allow range queries, such as skip graphs [ref], |
systems that allow range queries, such as skip graphs [ref] |
431 |
may prove useful. |
and skipnet [ref], may prove useful. |
432 |
|
|
433 |
|
[This might be relevant: |
434 |
|
http://www.hpl.hp.com/techreports/2002/HPL-2002-209.pdf |
435 |
|
-Hermanni] |
436 |
|
|
437 |
One question raised by xanalogical storage is which links to show |
One question raised by xanalogical storage is which links to show |
438 |
for a popular document that has been linked to by many users. |
for a popular document that has been linked to by many users. |
455 |
implementation on top of a distributed hashtable |
implementation on top of a distributed hashtable |
456 |
will be trivial. |
will be trivial. |
457 |
|
|
458 |
|
[Benja, this might be useful for defining APIs for DHTs etc: |
459 |
|
http://sahara.cs.berkeley.edu/jan2003-retreat/ravenben_api_talk.pdf |
460 |
|
Full paper will appear in IPTPS 2003 -Hermanni] |
461 |
|
|
462 |
In Storm, applications are not allowed to put arbitrary |
In Storm, applications are not allowed to put arbitrary |
463 |
mappings into the index. Instead, applications that want |
mappings into the index. Instead, applications that want |
464 |
to index blocks provide the following callback |
to index blocks provide the following callback |
517 |
are able to scale to the load to store a mapping for each word |
are able to scale to the load to store a mapping for each word |
518 |
occuring in a document. |
occuring in a document. |
519 |
|
|
520 |
|
[There are two refs about keywords in DHTs-- should we ref these ? -Hermanni] |
521 |
|
|
522 |
|
|
523 |
6. Versioning |
6. Versioning |
524 |
============= |
============= |
611 |
In summary, the current pointer system seems promising, but |
In summary, the current pointer system seems promising, but |
612 |
there are a number of unresolved issues with it: |
there are a number of unresolved issues with it: |
613 |
|
|
614 |
- signatures needed online |
- signatures needed online [possible refs: ConChord, SDSI/SPKI ? -Hermanni] |
615 |
- multiple 'current' versions annoying |
- multiple 'current' versions annoying |
616 |
- not suited to Web-like publishing |
- not suited to Web-like publishing |
617 |
|
|
757 |
operations (footnote about 'stable state' ?), while broadcasting can't achieve |
operations (footnote about 'stable state' ?), while broadcasting can't achieve |
758 |
either of these. |
either of these. |
759 |
|
|
760 |
[footnote: It's not clear whether all proposed DHT designs can preserve |
[footnote: It's not clear whether *all* proposed DHT designs can preserve |
761 |
(poly)logarithmic properties when participants are heterogeneous and they |
(poly)logarithmic properties when participants are heterogeneous and they |
762 |
join and leave the system in a dynamic manner. |
join and leave the system in a dynamic manner. |
763 |
|
|