/[gzz]/manuscripts/storm/article.rst
ViewVC logotype

Diff of /manuscripts/storm/article.rst

Parent Directory Parent Directory | Revision Log Revision Log | View Patch Patch

revision 1.68 by antont, Fri Jan 31 14:46:46 2003 UTC revision 1.69 by benja, Sat Feb 1 22:24:52 2003 UTC
# Line 350  Xanalogical storage, pioneered by Projec Line 350  Xanalogical storage, pioneered by Projec
350  5. Indexing  5. Indexing
351  ===========  ===========
352    
353  XXX  Finding links to and transclusions of a piece of content in
354    Xanalogical storage is but one example of *reverse indexing*
355    of Storm blocks: finding a block based on its contents.
356    (For other examples, see section 6, below.)
357    Storm provides a general API for indexing blocks in
358    application-specific ways. We have implemented indexing
359    on a local machine, but the interface is designed so that
360    implementation on top of a distributed hashtable
361    will be trivial.
362    
363    In Storm, applications are not allowed to put arbitrary
364    mappings into the index. Instead, applications that want
365    to index blocks provide the following callback
366    to a Storm pool::
367    
368        getMappings(block) -> set of (key, value) pairs
369    
370    This callback processes a block and returns a set of mappings
371    to be placed into the index. The Storm pool, in turn, provides
372    the following interface to the application::
373    
374        get(key) -> set of (block, value) pairs
375    
376    This function finds all mappings created by this application
377    with a given key, indicating both the application-provided
378    value and the block for which the mapping was created.
379    
380    We use the ``getMappings()`` approach instead of
381    allowing applications to put arbitrary mappings into the database
382    because binding mappings to blocks makes it easy for pools
383    to e.g. remove associated mappings when deleting a block.
384    
385    As an example, the ``getMappings()`` method of our Xanalogical
386    storage implementation will, for a block containing a document,
387    collect all the spans in a document, and return mappings
388    from their scroll blocks' ids to the spans and their positions
389    in the document. When we want to find the transclusions of
390    a span, we ``get()`` the mappings for the id of that span's
391    scroll block, and load the document blocks referenced
392    by the mappings.
393    
394    In a networked implementation, each peer is responsible
395    for indexing the blocks it stores. Since no peer can
396    feasibly know all applications that may want indexing,
397    there may be blocks available on the network that have
398    not been indexed by a particular application.
399    We do not see this as a problem-- it's just like a block
400    being available only if there's a peer which wants it to be--
401    but applications must be prepared to deal with it.
402    
403    Locally, on the other hand, it is guaranteed that
404    all blocks in a pool are indexed by all applications
405    known by the pool, even if the block was added
406    to the pool in a previous session in which the pool
407    did not know about the indexing application. To ensure this,
408    we maintain for each application a table of blocks
409    that have already been indexed. When the Storm pool
410    implementation is initialized, it compares the list
411    of indexed blocks with the list of all available blocks,
412    and asks the application for unindexed blocks' mappings.
413    
414    One indexing application that may seem obvious is keyword-based
415    full-text search. However, no research has been done
416    in this direction; it is not clear whether the current
417    interface is well suited to this, or whether current implementations
418    are able to scale to the load to store a mapping for each word
419    occuring in a document.
420    
421    
422  6. Versioning  6. Versioning
# Line 384  In Storm, *pointers* are used to impleme Line 450  In Storm, *pointers* are used to impleme
450  A pointer is a globally unique identifier that can refer to  A pointer is a globally unique identifier that can refer to
451  different blocks over time.  different blocks over time.
452    
453  To assign targets to pointers, a special type of block  To assign targets to pointers, a special type of block is used.
454    If we have a block B and a pointer P (P is simply a random id),
455    then we create an additional block whose mere presence tells Storm
456    that pointer P points to block B. This is one application of Storm
457    indexing (Section 5): Keeping an index of all pointer blocks
458    about pointer ``P``, we can quickly find out the current one.
459    
460    UML to be drawn::
461    
462        +---------+ 1      * +---------------+
463        | Pointer |----------| Pointer block |
464        +---------+          +---------------+
465                                   * |
466                                     |
467                                     |
468                                   1 | target
469                                +--------+
470                                | Target |
471                                +--------+
472    
473    In addition to the pointer and the target, pointer blocks contain
474    a list of *obsoleted* pointer blocks. These are pointer blocks
475    associated with the same pointer which are
476    
477  ...  ...
478    

Legend:
Removed from v.1.68  
changed lines
  Added in v.1.69

savannah-hackers-public@gnu.org
ViewVC Help
Powered by ViewVC 1.1.26