350 |
5. Indexing |
5. Indexing |
351 |
=========== |
=========== |
352 |
|
|
353 |
XXX |
Finding links to and transclusions of a piece of content in |
354 |
|
Xanalogical storage is but one example of *reverse indexing* |
355 |
|
of Storm blocks: finding a block based on its contents. |
356 |
|
(For other examples, see section 6, below.) |
357 |
|
Storm provides a general API for indexing blocks in |
358 |
|
application-specific ways. We have implemented indexing |
359 |
|
on a local machine, but the interface is designed so that |
360 |
|
implementation on top of a distributed hashtable |
361 |
|
will be trivial. |
362 |
|
|
363 |
|
In Storm, applications are not allowed to put arbitrary |
364 |
|
mappings into the index. Instead, applications that want |
365 |
|
to index blocks provide the following callback |
366 |
|
to a Storm pool:: |
367 |
|
|
368 |
|
getMappings(block) -> set of (key, value) pairs |
369 |
|
|
370 |
|
This callback processes a block and returns a set of mappings |
371 |
|
to be placed into the index. The Storm pool, in turn, provides |
372 |
|
the following interface to the application:: |
373 |
|
|
374 |
|
get(key) -> set of (block, value) pairs |
375 |
|
|
376 |
|
This function finds all mappings created by this application |
377 |
|
with a given key, indicating both the application-provided |
378 |
|
value and the block for which the mapping was created. |
379 |
|
|
380 |
|
We use the ``getMappings()`` approach instead of |
381 |
|
allowing applications to put arbitrary mappings into the database |
382 |
|
because binding mappings to blocks makes it easy for pools |
383 |
|
to e.g. remove associated mappings when deleting a block. |
384 |
|
|
385 |
|
As an example, the ``getMappings()`` method of our Xanalogical |
386 |
|
storage implementation will, for a block containing a document, |
387 |
|
collect all the spans in a document, and return mappings |
388 |
|
from their scroll blocks' ids to the spans and their positions |
389 |
|
in the document. When we want to find the transclusions of |
390 |
|
a span, we ``get()`` the mappings for the id of that span's |
391 |
|
scroll block, and load the document blocks referenced |
392 |
|
by the mappings. |
393 |
|
|
394 |
|
In a networked implementation, each peer is responsible |
395 |
|
for indexing the blocks it stores. Since no peer can |
396 |
|
feasibly know all applications that may want indexing, |
397 |
|
there may be blocks available on the network that have |
398 |
|
not been indexed by a particular application. |
399 |
|
We do not see this as a problem-- it's just like a block |
400 |
|
being available only if there's a peer which wants it to be-- |
401 |
|
but applications must be prepared to deal with it. |
402 |
|
|
403 |
|
Locally, on the other hand, it is guaranteed that |
404 |
|
all blocks in a pool are indexed by all applications |
405 |
|
known by the pool, even if the block was added |
406 |
|
to the pool in a previous session in which the pool |
407 |
|
did not know about the indexing application. To ensure this, |
408 |
|
we maintain for each application a table of blocks |
409 |
|
that have already been indexed. When the Storm pool |
410 |
|
implementation is initialized, it compares the list |
411 |
|
of indexed blocks with the list of all available blocks, |
412 |
|
and asks the application for unindexed blocks' mappings. |
413 |
|
|
414 |
|
One indexing application that may seem obvious is keyword-based |
415 |
|
full-text search. However, no research has been done |
416 |
|
in this direction; it is not clear whether the current |
417 |
|
interface is well suited to this, or whether current implementations |
418 |
|
are able to scale to the load to store a mapping for each word |
419 |
|
occuring in a document. |
420 |
|
|
421 |
|
|
422 |
6. Versioning |
6. Versioning |
450 |
A pointer is a globally unique identifier that can refer to |
A pointer is a globally unique identifier that can refer to |
451 |
different blocks over time. |
different blocks over time. |
452 |
|
|
453 |
To assign targets to pointers, a special type of block |
To assign targets to pointers, a special type of block is used. |
454 |
|
If we have a block B and a pointer P (P is simply a random id), |
455 |
|
then we create an additional block whose mere presence tells Storm |
456 |
|
that pointer P points to block B. This is one application of Storm |
457 |
|
indexing (Section 5): Keeping an index of all pointer blocks |
458 |
|
about pointer ``P``, we can quickly find out the current one. |
459 |
|
|
460 |
|
UML to be drawn:: |
461 |
|
|
462 |
|
+---------+ 1 * +---------------+ |
463 |
|
| Pointer |----------| Pointer block | |
464 |
|
+---------+ +---------------+ |
465 |
|
* | |
466 |
|
| |
467 |
|
| |
468 |
|
1 | target |
469 |
|
+--------+ |
470 |
|
| Target | |
471 |
|
+--------+ |
472 |
|
|
473 |
|
In addition to the pointer and the target, pointer blocks contain |
474 |
|
a list of *obsoleted* pointer blocks. These are pointer blocks |
475 |
|
associated with the same pointer which are |
476 |
|
|
477 |
... |
... |
478 |
|
|