/[gzz]/gzz/Documentation/misc/hemppah-progradu/masterthesis.tex
ViewVC logotype

Diff of /gzz/Documentation/misc/hemppah-progradu/masterthesis.tex

Parent Directory Parent Directory | Revision Log Revision Log | View Patch Patch

revision 1.181 by hemppah, Tue Mar 25 07:51:50 2003 UTC revision 1.182 by hemppah, Tue Mar 25 09:19:25 2003 UTC
# Line 491  TCP/IP-protocols. PeerNet has the same p Line 491  TCP/IP-protocols. PeerNet has the same p
491  overlays, i.e., $O(\log{n})$ space required for maintaining information about other peers in  overlays, i.e., $O(\log{n})$ space required for maintaining information about other peers in
492  the system and $O(\log{n})$ data lookup efficiency.  the system and $O(\log{n})$ data lookup efficiency.
493    
494    Additionally, authors argue in \cite{balakrishnan03semanticfree} that tightly structured systems
495    are suitable for next generation Reference Resolutions Services (RRS)\footnote{
496    Domain Name System (DNS) \cite{rfc1101} is a widely used RRS system in the Internet.}. They present
497    two requirements about the nature of reference resolution. First, there should be a general-purpose
498    and application-indepedent substrate for reference resolution. Second, the references themselves
499    should be unstructured and semantic-free. Authors emphasize that tightly structured systems (specifically
500    DHT-based) provide an elegant platform for RRS. In this context, we define unstructured reference
501    as a reference that it doesn't expose the target in any way and semantic-free reference as a reference
502    that there are no directives in the reference itself which would expose how the reference should be processed.
503    
504    
505  \section{Differences}  \section{Differences}
506    
# Line 1286  idea which was originally introduced by Line 1296  idea which was originally introduced by
1296  Also, query and routing hot spots may be an issue in tightly structured overlays \cite{ratnasamy02routing}.  Also, query and routing hot spots may be an issue in tightly structured overlays \cite{ratnasamy02routing}.
1297  Hot spots happen, when a specific key is being requested extremely often in tightly structured overlays. Recent study  Hot spots happen, when a specific key is being requested extremely often in tightly structured overlays. Recent study
1298  by Freedman et al. tries to reduce hot spots in the system by performing \emph{sloppy} hashing  by Freedman et al. tries to reduce hot spots in the system by performing \emph{sloppy} hashing
1299  \cite{sloppy:iptps03}. Another key feature of their work is that peers self-organize into clusters,  \cite{sloppy:iptps03}. Authors' technique is especially suitable for the DOLR abstraction of tightly structured overlays.
1300  therefore enabling peers to find nearby data without looking up data from distant peers.  With Sloppy hashing, we are able to reduce the generation of query hot spots. Sloppy hashing enables to
1301    locate nearby data without looking up data from distant peers. Moreover, authors'
1302    proposal for self-organizing clusters using network diameters may be useful,
1303    especially within small groups of working people. Thus, with Sloppy hashing
1304    we can provide locality properties the system.
1305    
1306    
1307    
1308  As mentioned before, an implicit assumption of almost every tightly structured system is that there is a random, uniform  As mentioned before, an implicit assumption of almost every tightly structured system is that there is a random, uniform
1309  distribution of peer and key identifiers. Even if participating peers are extremely heterogeneous, e.g., in  distribution of peer and key identifiers. Even if participating peers are extremely heterogeneous, e.g., in
# Line 1761  are location transparent: data lookups h Line 1777  are location transparent: data lookups h
1777  one ''virtual file'' may need obtaining several Storm blocks, which are distributed  one ''virtual file'' may need obtaining several Storm blocks, which are distributed
1778  randomly throughout the overlay. If not efficient, construction of the ''virtual file''  randomly throughout the overlay. If not efficient, construction of the ''virtual file''
1779  may take reasonable amount of time while rendering system very unusable. Third, Peer-to-Peer  may take reasonable amount of time while rendering system very unusable. Third, Peer-to-Peer
1780  infrastructure has to be scalable and fault tolerant against hostile attacks.  infrastructure has to be scalable, fault tolerant against hostile attacks and resilience in
1781    adverse conditions (e.g., a network partition).
1782    
1783  \section{Evaluation of Peer-to-Peer approaches with regard to Fenfire}  \section{Evaluation of Peer-to-Peer approaches with regard to Fenfire}
1784    
1785    In this section we focus on locating the Storm blocks in Peer-to-Peer environment. We don't
1786    respond to fetching of Storm blocks as fetching of Storm block can be performed easily once
1787    Storm block is located.
1788    
1789  In chapter 2, we discussed main differences between the loosely and the tightly structured  In chapter 2, we discussed main differences between the loosely and the tightly structured
1790  approach. As stated, the most significant difference is that the tightly structured  approach. As stated, the most significant difference is that the tightly structured
1791  approach has logarithmical properties in all internal operations, while the loosely  approach has logarithmical properties in all internal operations, while the loosely
# Line 1780  tightly structured approach over the loo Line 1801  tightly structured approach over the loo
1801  structured systems and Fenfire use similar methods for identifying data in the  structured systems and Fenfire use similar methods for identifying data in the
1802  system, i.e., globally unique identifiers.  system, i.e., globally unique identifiers.
1803  Another key feature of tightly structured overlays is that they are able  Another key feature of tightly structured overlays is that they are able
1804  to provide general purpose \emph{interface} for Reference Resolution Services (RRS)\footnote{  to provide general purpose \emph{interface} for Reference Resolution Services (RRS)
 Domain Name System (DNS) \cite{rfc1101} is a widely used RRS system in the Internet.}  
1805   \cite{balakrishnan03semanticfree}. Authors argue that next generation RRS must be   \cite{balakrishnan03semanticfree}. Authors argue that next generation RRS must be
1806  application-independent and references itself should be \emph{unstructured} and  application-independent and references itself should be \emph{unstructured} and
1807  \emph{semantically free}. Finally, as said, with tightly structured systems it is feasible to  \emph{semantically free}.  Thus, we see the tightly structured approach as the best alternative to
1808  perform \emph{global} data lookups in the overlay. To summarize, these aspects may be the most important features  locate data in Peer-to-Peer environment.
 of Peer-to-Peer infrastructure with regard to Fenfire as a distributed, location transparent hypermedia system.  
 Thus, we see the tightly structured approach as the best alternative to locate data in Peer-to-Peer  
 environment.  
   
 Once located, we can use regular TCP/IP-protocols, such as Hypertext Transfer protocol (HTTP)  
 \cite{rfc2068} for \emph{fetching} Storm blocks from the overlay. However, HTTP-protocol may  
 not be a optimal solution when obtaining large amounts of data from a Peer-to-Peer network (e.g.,  
 videos, images or music). In this case, multisource downloads can be very useful  
 for better efficiency \cite{maymounkov03ratelesscodes, bittorrenturl}. Furthermore,  
 multisource downloads can be used for decreasing load of a certain peer, thus avoiding query  
 hot spots in the system \cite{ratnasamy02routing}. Current implementation of Fenfire uses  
 standard single source downloads (HTTP) and SHA-1 \cite{fips-sha-1} cryptographic content  
 hash for verifying the integrity of data by recomputing the content hash  
 for a scroll block. In face of multisource downloads, Fenfire must support  
 tree-based hashes\footnote{With multisource downloads, tree-based hash functions can be used  
 to verify fixed length segments of data. If hash value of data segment is incorrect,  
 we need only to fetch \emph{segment} of data (instead of whole data) from  
 an other source.}, such as \cite{merkle87hashtree, mohr02thex} for reliable and efficient  
 data validation.  
1809    
1810  Again, there are research challenges with tightly structured systems which have to be  Again, there are research challenges with tightly structured systems which have to be
1811  addressed, as described in chapter 3. The main concerns include decreased performance and fault  addressed, as described in chapter 3. The main concerns include decreased performance and fault
# Line 1813  proximity routing, hostile entities and Line 1814  proximity routing, hostile entities and
1814  Additionally, there is only little real world experiments yet with tightly structured systems  Additionally, there is only little real world experiments yet with tightly structured systems
1815  (e.g., \cite{overneturl, edonkey2kurl}). Therefore, we can't say for sure, how well these  (e.g., \cite{overneturl, edonkey2kurl}). Therefore, we can't say for sure, how well these
1816  systems would perform in real Peer-to-Peer environment. However, we believe that these issues are  systems would perform in real Peer-to-Peer environment. However, we believe that these issues are
1817  solved, since there is a strong and wide research community towards to the tightly structured  solved in near future, since there is a strong and wide research community towards to the tightly structured
1818  overlays \cite{projectirisurl}.  overlays \cite{projectirisurl}.
1819    
1820                
# Line 1825  obtaining Fenfire data from a Peer-to-Pe Line 1826  obtaining Fenfire data from a Peer-to-Pe
1826    
1827  \subsection{System proposal}  \subsection{System proposal}
1828    
1829  Currently, we see Kademlia \cite{maymounkov02kademlia} as the best algorithm for  We emphasize that we prefer \emph{abstraction}
1830    level analysis as very recently better and better tightly structured algorithms have been proposed.
1831    Thus, we don't want to bind our system proposal to a specific algorithm definitively as we expect
1832    that this development continues. Currently, we see Kademlia \cite{maymounkov02kademlia} as the best algorithm for
1833  locating data efficiently in the Peer-to-Peer overlay. There are two  locating data efficiently in the Peer-to-Peer overlay. There are two
1834  reasons for this. First, Kademlia's XOR-based distance function is superior  reasons for this. First, Kademlia's XOR-based distance function is superior
1835  over distance functions of other systems (see section 2.4). Second, there exist already    over distance functions of other systems (see section 2.3.2). Secondly, Kademlia
1836  deployed real-life systems using Kademlia (e.g., \cite{overneturl, edonkey2kurl, kashmirurl,  is one of the only tightly structured systems that has been deployed in real life
1837  kato02gisp}), which means that Kademlia's algorithm is simple and easy to implement.  (e.g., \cite{overneturl, edonkey2kurl, kashmirurl,kato02gisp}), which means that
1838    Kademlia's algorithm is simple and easy to implement.
1839    
1840  On top of Kademlia, we propose the usage of Sloppy hashing \cite{sloppy:iptps03} which  On top of Kademlia, we propose the usage of Sloppy hashing \cite{sloppy:iptps03} which
1841  is optimized for the DOLR abstraction of tightly structured overlays. With the Sloppy hashing,  is optimized for the DOLR abstraction of tightly structured overlays. With Sloppy hashing
 we are able to reduce the generation of query hot spots. Sloppy hashing enables to  
 locate nearby data without looking up data from distant peers. Moreover, authors'  
 proposal for self-organizing clusters using network diameters may be useful,  
 especially within small groups of working people. Thus, with Sloppy hashing  
1842  we can provide locality properties for the Fenfire system.  we can provide locality properties for the Fenfire system.
1843    
1844  For better fault tolerance and self-monitoring for Fenfire, we propose techniques  For better fault tolerance and self-monitoring for Fenfire, we propose techniques
# Line 1846  techniques, we can ensure the performanc Line 1847  techniques, we can ensure the performanc
1847  as sudden network partition, or highly dynamic and heterogeneous environment.  as sudden network partition, or highly dynamic and heterogeneous environment.
1848    
1849  Finally, for more efficient data transfer, we can use variable techniques for this purpose.  Finally, for more efficient data transfer, we can use variable techniques for this purpose.
1850  For small amounts of data, HTTP can be used \cite{rfc2068}. For big amounts of data, we can use  For small amounts of data, HTTP can be used \cite{rfc2068}. For large amounts of data, we can use
1851  multisource downloads for better efficiency and reliability. Specifically, the technology based  multisource downloads for better efficiency and reliability. Specifically, the technology based
1852  on rateless erasure codes \cite{maymounkov03ratelesscodes} seems very promising.  on rateless erasure codes \cite{maymounkov03ratelesscodes} seems very promising.
1853    Furthermore, multisource downloads can be used for decreasing load of a certain peer, thus avoiding query
1854    hot spots in the system \cite{ratnasamy02routing}. Current client-server implementation of Fenfire uses
1855    standard single source downloads (HTTP) and SHA-1 \cite{fips-sha-1} cryptographic content
1856    hash for verifying the integrity of data by recomputing the content hash
1857    for block. In face of multisource downloads, Fenfire must support
1858    tree-based hashes\footnote{With multisource downloads, tree-based hash functions can be used
1859    to verify fixed length segments of data. If hash value of data segment is incorrect,
1860    we need only to fetch \emph{segment} of data (instead of whole data) from
1861    an other source.}, such as \cite{merkle87hashtree, mohr02thex} for reliable and efficient
1862    data validation.
1863    
1864  \subsection{Methods}  \subsection{Methods}
1865    
1866  We use the DOLR abstraction of the tightly structured approach, i.e., each participating peer hosts  We use the DOLR abstraction of the tightly structured approach since DOLR systems locate data without
1867  the data and the overlay maintains only the \emph{pointers} to the data. We decided to use the DOLR  specifying a storage policy explicitly \cite{rhea03benchmarks}, i.e., each participating peer hosts
1868  abstraction in our model, since DOLR systems locate data without specifying a storage policy explicitly \cite{rhea03benchmarks}.  the data they are offering and the overlay maintains only the \emph{pointers} to the data.
1869  DHT-based storage systems, such as CFS \cite{dabek01widearea} and PAST \cite{rowstron01storage}, may have  DHT-based storage systems, such as CFS \cite{dabek01widearea} and PAST \cite{rowstron01storage}, may have
1870  severe problems with load balancing in a highly heterogeneous environment. The problem is caused by peers  severe problems with load balancing in a highly heterogeneous environment \cite{rao03loadbalancing}. The problem is caused by peers
1871  which may not be able to store relatively large amount of data with a key-value pair, assigned randomly by  which may not be able to store relatively large blocks, assigned randomly by the mapping function of the overlay.
 the mapping function of the overlay. These systems waste both storage and bandwidth, and  
 are sensitive to certain attacks (e.g., the DDoS attack). Additionally, we emphasize that we prefer \emph{abstraction}  
 level analysis as very recently better and better tightly structured algorithms have been proposed.  
 Thus, we don't want to bind our system proposal to a specific algorithm definitively as we expect  
 that this development continues. In this model, we use Kademlia's \cite{maymounkov02kademlia} algorithm for  
 locating data in the overlay.  
   
 In the following subsections we assume that we know the structure of  
 the enfilade before hand, i.e., when assembling the ''virtual file'' we know all the Storm  
 blocks, which are required to complete the enfilade. Also, we don't  
 respond to the security issues related to Peer-to-Peer systems, since there is no working solution  
 available yet; we either assume that Fenfire has a reliable technique for identifying individual entities, or  
 there are no hostile entities among participating peers.  
1872    
1873  In our method, each peer maintains the following data structures for local operations: a data structure for listing all  In our method, each peer maintains the following data structures for local operations: a data structure for listing all
1874  key-value pairs which peer maintains; a data structure for listing all key-value pairs in the  key-value pairs which peer maintains; a data structure for listing all key-value pairs in the
# Line 1879  as \emph{keys} of the overlay. Every key Line 1877  as \emph{keys} of the overlay. Every key
1877  (pointer blocks), or a hash of block's content (scroll blocks) as a key. The value is always a reference to a hosting  (pointer blocks), or a hash of block's content (scroll blocks) as a key. The value is always a reference to a hosting
1878  peer (e.g., IP address). Finally, we assume that all local operations can be done in a constant time.  peer (e.g., IP address). Finally, we assume that all local operations can be done in a constant time.
1879    
1880    We assume that we have resolved the construction of the ''virtual file'' before locating any Storm blocks, i.e.,
1881    when assembling the ''virtual file'' we know all the Storm blocks, which are required to complete the ''virtual file''.
1882    Also, we don't respond to the security issues related to Peer-to-Peer systems, since there is no working solution
1883    available yet. Thus, we either assume that Fenfire has a reliable technique for identifying individual entities, or
1884    there are no hostile entities among participating peers, i.e.,  Storm blocks can be identified correctly (e.g., when
1885    performing searches). In the next section, we discuss security problems in more detail.
1886    
1887    
1888  \begin{itemize}  \begin{itemize}
1889  \item Data lookup with a given identifier of Storm scroll block.  \item Data lookup with a given identifier of Storm scroll block.
# Line 1919  Figure \ref{fig:storm_query_urn5} illust Line 1924  Figure \ref{fig:storm_query_urn5} illust
1924  in a tightly structured overlay using the DOLR abstraction, where the pointer random string is given.  in a tightly structured overlay using the DOLR abstraction, where the pointer random string is given.
1925    
1926  Each of these algorithms can locate Fenfire blocks in $O(\log{n})$ time at application level overlay:  Each of these algorithms can locate Fenfire blocks in $O(\log{n})$ time at application level overlay:
1927  $O(\log{n})$ time for query routing to pointer peer and constant time for  $O(\log{n})$ time for query routing to pointer peer (Kademlia \cite{maymounkov02kademlia}) and constant time for
1928  locating hosting peer with a given reference link.  locating hosting peer with a given reference link.
1929    
1930    

Legend:
Removed from v.1.181  
changed lines
  Added in v.1.182

savannah-hackers-public@gnu.org
ViewVC Help
Powered by ViewVC 1.1.26