==================================================================================== A free software toolchain for bidirectional linking between UML diagrams and Javadoc ==================================================================================== :Stamp: $Id: SKETCH,v 1.1 2003/02/11 19:06:53 humppake Exp $ 2nd round todo: =============== - UML description - check metaphor - more about stakeholdrer's (antont) - javadoc history - focus+context menus - how our way is placed in 'Hypermedia and cognition: designing for comprehension' - xp 3rd round ========= - diagrams Introduction ============ UML (the Unified Modeling Language), founded mainly by Booch, Jacobson and Rumbaugh during 1900s to ``XXX blah blah`` [#]_, has become the *de facto* industry-standard language for specifying, visualizing, constructing, and documenting the products in software engineering. Recently, various approaches have been developed towards unifying software architecture design and program source code. This is done by generating the source code from the UML-based architectural model [#]_ or by reverse-engineering the documentation from the code [#]_. Nowadays UML seems to be taken in CASE tools even almost as a visual programming language, though it's not primarly designed to be a programming language [#]_. .. [#] ``XXX`` booch-jacobson-rumbaugh-uml-user-guide? .. [#] harrison-barton-raghavachari00uml-to-java .. [#] pierce-tilley02connecting-documentation-rose .. [#] booch-jacobson-rumbaugh-uml-user-guide p.15 booch-jacobson-rumbaugh-uml-reference-manual p.4 Against this trend of using UML to fully unify the architectural documentation and code, we tend still to define the details in the source code and use UML only to plan and document our software architecture on more general level. We keep UML as a common language for intercommunication within our software development team and within project's all stakeholders. In this purpose we prefer better abstracted and more comprehensible human-drawn diagrams to exact spesifications matching source code to every detail: "You draw diagrams to visualize a system from different perspectives, so a diagram is a projection into a system. For all but the most trivial systems, a diagram represents an elided view of the elements that make up a system." [#]_ : (could these stakeholders be identified and described to some extent somewhere? it might be interesting to think also towards users/customers, who in some methods (ways of using XP) control use cases themselves) .. [#] booch-jacobson-rumbaugh-uml-user-guide, p.24 Altough, even if the architectural documentation won't fully match the details of current implementation, we feel important to enable fast transition between them. In this article, we present the tools we developed for using UML diagrams as multiple-ended hyperlinks between the architectural documents and the documentation generated from source code by Javadoc. Background ========== Software development processes ------------------------------ In their article "A rational design process: how and why to fake it" [#]_ Parnas and Clements argue that rational software design process is not generally possible, but acceptable results could be achieved by faking "the ideal process". Their ideal software design process contains the following steps [#]_: A) establish and document requirements B) design and document the module structure C) design and document the module interfaces D) design and document the uses hierarchy E) design and document the module internal structures F) write programs G) maintain (redesign and redevelopment, keeping documentation up-to-date). .. [#] parnas-clements-rational .. [#] parnas-clements-rational, pp.252-255 It should be clear that documentation plays major role in the ideal software design process. Steps listed above should produce documentation, which records requirements or design decisions and could be used as reference manuals throughout the building of the software [#]_. In theory in an open project new programmers are always free to participate independently. In practice a new programmer could need a lot of mentoring by other group members, before one could be productive for the project. Sometimes sufficient mentoring could not even be possible, because members of such a projects could be geographically widely scattered. This results in the Mythical Man Month effect [#]_: adding a new member into software design process could rather delay than speed up the project: "Men and months are interchangeable commodities only when a task can be partitioned among many workers with no communication." [#]_ "The added burden of communication is made up of two parts, training and intercommunication. Each worker must be trained in the technology, the goal of the effort, the overall strategy, and the plan of work.-- --Intercommunication is worse." [#]_ .. [#] brooks-mythical-man-month .. [#] brooks-mythical-man-month p.16 .. [#] brooks-mythical-man-month p.18 .. [#] brooks-mythical-man-month p.18 Eventually, the greater is the amount of programmers or higher the turnover within them, the more important it is to have a good documentation. When new programmers join the project they shouldn't have to depend completely on the old staff for their information. An up-to-date and rational set of documents available for them could ameliorate the Mythical Man Month effect [#]_. .. [#] parnas-clements-rational p.255 Because our group is doing experimental research, we need a flexible model for software development process. From the classical software development process models, ours reminds most of Boehm's spiral model of software development and enhancement [#]_. The spiral model is cyclic and repetitive, and every cycle has the following phases [#]_: * determining objectives, alternatives, constraints * evaluating alternatives, identifying and resolving risks * developing, verifying * planning next phase .. [#] boehm-spiral-model .. [#] boehm-spiral-model p.64 ``XXXExtreme Programming`` [#]_ .. [#] beck99xp The nature of our software development process affects also strongly our documentation needs. As we proceed by prototyping and enhancing our software step by step, also our documentation should be updated in every cycle. Because the detailed implementation is in continual change, we prefer to emphasis the more constant and general architectural documentation. Of course, more detailed up-to-date documentation could be generated from the current source for those members who are working with particular details. Though, in general documentation rapidly changing details would be irrelevant and confusing. Since currently the main purpose for our documentation is to help intercommunication within our group, we believe manually made human abstracted documentation serves our purposes best. Javadoc ------- **Javadoc** [#]_ is a hypertext documentation tool developed by Sun Microsystems and delivered with Sun's Java Development Kit [#]_. Javadoc generates detailed and massively crosslinked hypertext documentation after analyzing class hierarchies and references, and includes documentation comments from java source code. Besides crosslinking, Javadoc creates alphabetical indexes and hierarchical trees to aid navigation into individual classes. ``XXX Little more about Javadoc's history...`` .. [#] friendly95javadoc .. [#] Java is a trademark of Sun Microsystems XXX Friendly, L. had a Java reference Sun's distribution of Javadoc tools is proprietary, but alternatives such as doc++ [#]_ exist as free software (licensed under GPL). Doc++ is designed to create hypertext documentation from other languages (e.g. C, C++, perl), in addition to Java. .. [#] http://docpp.sourceforge.net/ UML --- **Rational Rose** [#]_ is quite honored UML design tool. It provides a direct manipulation interface for drawing different kinds of UML diagrams into its database, which is also supported by other products of Rational Software Corporation's Rational product family. In addition to drawing diagrams from scratch, it allows reverse engineering diagrams from code [#]_. Rational Rose could be used with **Rational Soda** [#]_ to create more general architecture documentation. Soda is WYSIWYG (What You See Is What You Get) documentation writing tool, which allows including diagrams and documentation data from Rational Rose's database to the written documentation. Both Rational Rose and Rational Soda are proprietary software and quite expensive for use of small research groups. Problems of direct manipulation and WYSIWYG interfaces are discussed later. .. [#] http://www.rational.com/products/rose/ .. [#] pierce-tilley02connecting-documentation-rose .. [#] http://www.rational.com/products/soda/ **Dia** [#]_ is a free software (licensed under GPL) alternative for diagram drawing with direct manipulation user interface. Dia is still more just a drawing than design tool, but it can export diagrams in several formats and it has a plugin interface for new drawing object sets. .. [#] http://www.lysator.liu.se/~alla/dia/ **Doxygen** [#]_ is another Javadoc-like documentation tool, which generates hypertext documentation from source code of good variety of different languges. An intresting feature in Doxygen is that it can also generate and embed into class documentation UML diagrams representing software's class and inheritance hierarchies. Also the focused class is higlighted in diagrams on its documentation pages. Another enhancement to javadoc in Doxygen is that it creates a dynamic shrinkable treeview for class hierarchy. .. [#] http://www.doxygen.org/ **MetaPhor** [#]_ Project in University of Jyväskylä have looked for ways of using hypertext in CASE tools [#]_. Results of its research have been used in **MetaEdit+** [#]_ meta CASE tool developed by MetaCase Consulting. MetaEdit+ provides bi-directional linking between all its objects and an interface for reaching linking contexts from diagrams' objects. : XXX This should be checked, but metaphor.it.jyu.fi seems to be down :( ] .. [#] metaphor .. [#] lyytinen-kerola-metaphor .. [#] http://www.metacase.com/ Setting of problem ================== In our project, as probably in most projects that use Java programming language, the software development documentation is divided into two classes: the design documentation and Javadoc. The design documents cover the most important architectural features and are written either before coding (for design) or after (for exposition of the architecture). Javadoc, on the other hand, is detailed and fully generated documentation from javadoc comments of each class and method in the java source code. The two types of documentation are complementary, as demonstrated in the following table: +-----------------------------------+------------------------------------+ | Design docs | Javadoc | +-----------------------------------+------------------------------------+ | | | | - good overall picture. | - easy to find a given class, | | | easy to check all methods | | | | | - little detail, | - detailed | | | | | - may be slightly outdated at | - methods and classes *always* up | | any particular time. | to date (generated from | | | source), doc comments also | | | usually | | | | | - hard to find explanations | - no overall picture of classes' | | for a particular class. | roles | | | | | - UML diagrams | | | | | | - written and organized by humans | | | | | +-----------------------------------+------------------------------------+ Because Javadoc is fully generated, it is always up to date. Our design documentation, though, is updated manually. Of course, it should be updated regularly during every design cycle, but in practise that won't always happen. From our point of view it is still ridiculous even talk about generating the design documentation; the design should always be made by human, and if the design documentation is for human reading, it should be also human made. What we could do, is lower the treshold of getting into writing the design documentation. For detailed documentation we have decided to use Javadoc, since even though it's not free software it's still usable for free and quite easy to use. Also the likeness of documentation with most of the other Java based projects is a significant issue. A more difficult issue is to select tools for writing the design documentation and drawing diagrams into it. The solution should be cheap, and as a free software project we would prefer other free software. Also the solution should fit well to our current working customs. In addition to the tool selection problem we have to problem of two distinct pieces of documentation: the human written design documentation and the automatically generated detailed Javadoc documentation. During coding the Javadoc documentation is often necessary, but the architectural design documentation could be easily left in the dark reaches of the filesystem. It could be argued that the reason for architectural documents being left unused is that relevant parts linked to ongoing work are hard to find. The distinct pieces (javadoc and design documentation) cannot seen as a whole. Therefore we should make them work as a whole. "Obvious" question: can we increase the overall value by interconnecting the two distinct pieces of documentation? When looking at a design document, jumping to the javadocs to get the details would be useful, and when looking at a Javadoc, it would be useful to be able to see if any design documents refer to that class. We believe that the architecture document will be read more, after it's easily reachable from relevant parts of Javadoc. This is the starting point for the toolchain developed in this article. Design ====== The usability of hypertext-based documentation may suffer from user's disorientation: the tendency to lose one's sense of location and direction in a nonlinear document [#]_. That means, users don't know where they are in the documentation network or how to get to some other place that they know to exist in the network. In our documentation this gets even more complex, because we have two distinct pieces of documentation, which should have somehow unified navigation. .. [#] conklin-hypertext, pp.38-40 Edwards and Hardman [1989] argue that the most appropriate types of navigation devices would be based on spatiality. According to their research, individuals appear to be attempting to create cognitive representations of hypertext structures in the form of a survey-type map. They conlude that users should be allowed to develop a cognitive map of one view of the data structure before being given the opinion of navigating through the data some other way. [#]_ .. [#] edwards-hardman-lost-in-hyperspace, p.123 : See also 'Hypermedia and cognition: designing for comprehension' by Thuring et al. http://portal.acm.org/citation.cfm?id=208348&coll=portal&dl=ACM&CFID=7442809&CFTOKEN=15305251#FullText) We didn't need to look long for a common navigational metaphor to unify the two distinct pieces. Since the most of our UML diagrams included objects representing certain Java classes, it was clear to use those diagrams for cross-documential navigation. UML diagrams not only show the readers a map of documentation but also perform as spatial navigation menus. Looking closer, we can divide our UML diagrams into three classes: conceptual diagrams, specification diagrams and implementation diagrams. Different diagram types can be linked within the architectural documentation and to javadoc as follows: +-----------------------+------------------------+-----------------------+ | Conceptual | Specification |   Implementation | +-----------------------+------------------------+-----------------------+ | | | | | + probably no links | + can link Java | + can link all | | to javadoc | interface  | classes and | | | | packages | | + the design | + can link also Java | | | documentation | packages | | | packages could be | | | | linked withing the | | | | architectural | | | | documentation | | | | | | | | + concepts could be | |  | | linked with the | | | | classes that | | | | eventially | | | | implement it | | | | | | | +-----------------------+------------------------+-----------------------+ Documentation reader's point of view --------------------------------------- Before reaching the current state, our documentation evolved through several distinct steps, which will be viewed first from the reader's and then from the developer's point of view. **Step 0; In the beginning we had a distinct architecture documentation with UML diagrams and javadoc generated from the sourcecode.** This is probably the most common case. The both pieces of documentation could comprehensive and well navigable by their own, but their information is hard to combine, because there is no crosslinking between them. Advantage over paper: NONE **Step 1; Links from architecture documentation's UML diagrams are created into relevant javadoc class documentation.** After the first step javadoc was finally reachable from architecture documentation simply by clicking the relevant class or package within embedded UML diagrams. Though, after moving to the javadoc the context in architectural documentation would be immidiately lost. Also there was no links from javadoc to architecture documentation. Advantage over paper: MARGINAL: automatize cross-indexing **Step 2; UML diagrams are implicitly embedded into javadoc and embedded UML diagrams' objects also work as links.** After the second step we had diagrams working as menus between the objects that appeared in diagrams. Though, even though a single diagram could be used to traverse between all the documentation pages referred to in the diagram, the initiating design document was unreachable from the diagram itself. Advantage over paper: SOME: see the UML diagram contexts of a class, traverse them **Step 3; Backlinks to the originating architecture documentation are added.** Finally it's possible to "step back" from class documentation back to architecture documentation. After backlinks to the architecture documentation are added, a single diagram links to all architecture documents and javadoc pages where it is explicitly or implicitly included. Even though the architectural documents were not in roles in the original UML diagrams, they are included in the final image on top of the diagram. Putting all these into one graphical image makes the all links look a consistent whole - like a spatial menu. [XXXREFS??] In the image, the element that represents the current document page is also focused and the objective of focus+contex menu is achieved. Because the original location on the diagrams is easily reachable from its every implicit occurerrence in javadoc or in architecture documentation, the size of the implicitly embedded diagram could be reduced by 50%. The name of diagram object is also shown, hovering the pointing device over diagram - also on diminished diagrams. Advantage over paper: MAJOR: multi-end links easily traversable, structure can be understood Documentation developer's point of view ----------------------------------------- On the developer side, creating diagrams within the architecture documentation should be as easy and natural as possible; we want to minimun the barrier of drawing even only a small diagram to make documentation more clear. For Step 1, we keep it best to demand explicit links in the UML diagrams. Author has to decide for each object in UML diagram whether it should be linked -- for example, by giving a fully-qualified java class name or a relative path to architecture documentation page. After Step 1 from author's perspective, the computer has *all* the information needed to complete also steps 2 and 3, so *no further changes should be required*. For example javadoc comments should not need changes at all. During steps 2 and 3 computer simply embeds diagrams into previously linked documentation pages and creates backlinks to the initiating documentation pages. Analysis from a hypertext-theory point of view ============================================== ``UML diagram of how architectural documents, UML diagrams and javadoc pages work. Instead of browsing only in one direction, all associations shown in the diagram should be browsable in either direction! Show instances of arch. docs, javadoc pages &c. Show both the diagram and the final hypertext view.`` The pure Javadoc organizes class documentation hierarchically according to their containing packages, but using a single hierarchy is very limited way of organizing things. An illustrative example of this would be Borge's "taxonomy" for animals [#]_. XXX Do we need this for HT people? .. [#] Borges's "taxonomy" for animals; http://www.multicians.org/thvv/borges-animals.html In our software development documentation UML diagrams act as multi-ended nexus links, bringing together all documents they relate to. They function like navigation bars or context+focusing menus ``[REFSXXX]`` on the WWW, except that the same page may have several of them. For example, a particular class could be used in many different contexts and each context could be described by drawing a distinct diagram. Our documentation tool collects all those diagrams into the class documentation page to show all its documented usage contexts. Because every target document which a diagram links has also a version of the diagram, documents are linked bi-directionally. For example, if document A has a diagram B, which links to document C, document C has also a version of diagram B, which links back to document A. From our point of view, the links in the final version cease to be "links" and the UML diagrams become objects in their own right. Looking at the diagram itself, its links can be divided into two: 1) from the diagram into architecture documentation, and 2) from the diagram into javadoc source code documentation Note that also links from documentation pages to diagram images exist, but hypertext browsers usually interpret them implicitly by showing the embedded diagrams within the documentation pages. Diagrams embedded into documentation pages should still be also seen as part of their context. Therefore diagrams link documentation also in the following ways: 3) a javadoc page to an another javadoc page 4) an architectural documentation page to another architectural documentation page 5) a javadoc page to an architectural documentation page, and 6) an architectural documentation page to a javadoc page Finally, seeing links as bi-directional will combine links 5) and 6). The final list of links would be: 1) diagram <-> architecture doc 2) diagram <-> javadoc 3) architectural doc <-> architectural doc 4) javadoc <-> javadoc 5) architectural doc <-> javadoc Trigg Randall (1983) have collected a comprehensive taxonomy of different link types [#]_, which also covers link types found in our diagrams. Trigg notes that the physical direction of a link defines the manner in which readers are expected to follow the link. However, that won't rule off the possibility that link is typed depending only on its semantic direction. Trigg's taxonomy provides several pairs of such link types. In our documentation the physical direction of links in a diagram depends on the location of explicit inclusion of the diagram. Currently diagrams can be included explicitly only in architectural documents and all the links are physically directed from the including architectural document to the targets defined in the diagram. Most of the links from diagrams fall into Trigg's pairs of links mentioned above. (XXX Gets difficult to understand) An interesting situation comes when a diagram is implicitly embedded into two documents and provides an implicit link between them. Although on diagram level it includes directed links from a diagram object to its details, on document level the links are undirected. In our documentation this is the case always when a Javadoc page has an implicitly embedded diagram linking to another Javadoc page. Within architecture document this is not the case if the other document or both of the documents include the diagram explicitly by the author. +----------------------------------------------------------------------------------+ | Link in our documentation | Corresponding *normal* link type in | | | Trigg's taxonomy | +------------------------------------------+---------------------------------------+ | | | | 1) diagram <-> architecture doc | detail / summary | | 2) diagram <-> javadoc | detail / summary | | | | | 3) architectural doc <-> architectural | explicitly created depends on author | | doc | implicitly created is typed as | | | alternative view | | | | | 4) javadoc <-> javadoc | alternative view | | 5) architectural doc <-> javadoc | specification / generalization | | | | +------------------------------------------+---------------------------------------+ .. [#] trigg-link-taxonomy Implementation ============== (We found no Free Software alternative to our own development, which combines all the good parts we need from previous tools without reformin dramatically our workin culture.) + HTML/WWW browsers don't support bi-directional links Goals: - light-weight - using existing free software tools - plugins for documentation tool (javadoc could be replaced with free software, several documentation tools could be used simultanously...) - free software chain - outputs: easily adaptable to others (editing the source). - flow diagram of the implementation reST's filosophy - indentation for blocks - robust for erros - "natural" to write - "ASCII layout" (our UML-language is an exception, which is discussed later) Why UML-language - Doxygen can generate useless diagrams from the code class hierarchy should describe somewhere why these are useles... uses proprietary GraphViz tool to create diagrams - How should we discuss WYSIWYG, WYSIAYG/WYSIATI for reST &c? Someone will surely ask about that, PARTIAL RESOLUTION: - proprietarity (really?) - versioning much worse (changes in rtf could probably be compared?) - users = programmers --> used to editing text and compiling, it's the most practical to use the same editor as for coding - lack of good, **extensible** tools [gentner-nielsen-anti-mac, p.75] collects arguments/argues generally against WYSIWYG. - a document has a rich semantic structure that is poorly captured by its appearance on a sceen or printed page - other text representations, such as SGML, distinct the semantic structure and rules for converting the text and semantics into printablepage - WYSIWYG assumes there is only one useful representation of the information: that of the final reports - WYSIWYG assumes people want paper-style reports (and we already live within information flood) - blind users exists - Rational Soda is WYSIWYG, maybe this could be discussed under it - though, because we are coders, we would prefer UML script language over menus and dialogs when creating UML objects + we need a ref. of criticism of direct manipulation! - not necessarily improve performance, users must learn the meaning of the graphical components, graphic presentation could be misleading, graphical presentation could take excessive screen display space [Shneiderman-direct-manipulation, p. 64] probably only the first argument of performance is relevant anymore... though, creating the graphic presentation for problem could lead to using metaphors and those are often criticized (even by D. Norman :) - "The dark side of a direct manipulation interface is that you have to directly manipulate everything. Instead of an executive who gives high-level instructions, the user is reduced to an assembly line worker who must carry out the same task over and over." [gentner-nielsen-anti-mac, p.74] limits the precision of our actions to the precision achieved with eye-hand-mouse coordination (language and mathematics can be more precise) [gentner-nielsen-anti-mac, p.74] direct manipulation requires the user to be involved in every action, but sometimes the user may not know what to do [gentner-nielsen-anti-mac, p.74] "It's as if we have thrown away a million years of evolution, lost our facility with expressive language, and been reduced to pointing at objects in the immediate environment. -- We have lost all the power of language, and can no longer talk about objects that are not immidiately visible." [gentner-nielsen-anti-mac, p.74] - summaring: direct manipulation could be easy, if there is a natural graphic presentation for the problem. Even then it's not necesesserily the most efficient method. Also the practical point of using the same writing tool foor coding and documenting. + Rose's problem with direct manipulation: a lot of windows, menus, dialogies etc... to get direct manipulateable graphic representation for everything + though, directmanipulation is only relevant solution for replacing already created object - not strictly necessary; could have used another editor - minimum barrier to make a diagram (well, at first have to learn our UML-script) - direct manipulation when building / editing structure of diagrams clumsy - users are programmers --> a simple programming language is better than d.m. + elements are easy to create by script language + graphical placing could need an GUI - diagrams in the same reST source as the architecture document - separate UML structure from presentation, show structure in easily editable text - placement: currently metapost; looking at ways to make interactively editable. - Dia could be used instead of metapost, but direct manipulation problem again Practical use ============= The tool is currently in everyday use in the Gzz project ... - n classes, n diagrams, n pages architecture documentation, and works - we develop open software => good documentation is needed for lowering threshold for new people, in addition to the project members &c. Acknowledgments =============== antont Conclusion ========== - need easier placing (but still not automatic) - metapost's erros are fuzzy, when using scripting language for drawing diagrams, we need clearer errors - it's possible that we change metapost for something else + getting rid of problem with fuzzy error messages and slow compiling process + we can still continue using our UML language - However, less power: with mp, can draw arbitrary stuff on the diagrams as well; this is sometimes useful, but sometimes not desirable. - all in all, this is temporary system: using web pages for the presentation/UI layer of a hypertext system is useful and standard but very limiting. The system we are using these tools to develop is intended for better user interfaces to similar ... + antont's note about duplication of data problem - Currently, we only link UML classifiers to Java classes and other documents. Methods could also be linked, especially in interaction diagrams.