1. Introduction
JDosage2 is a webcomic downloader. It downloads comic images for offline reading, or to enable opening images in custom viewers for easier reading. It can also be used by webcomic authors to test their website for invalid links and missing images.
JDosage2 is inspired by Dosage, but tries to be easier to use, while remaining powerful and flexible enough to support most websites.
JDosage2 is written in Java and should work on all platforms where Java is supported, but it is developed and tested on GNU/Linux. Please, report any problems on other operating systems.
|
Most webcomics are protected by copyright and unauthorized copying is prohibited. This program is intended for personal use only. If you intend to distribute the downloaded webcomics, be sure to verify that the webcomic license allows it. |
2. Quick start
First, start JDosage2 from command line by invoking jdosage2.sh on GNU/Linux, or jdosage.cmd on Windows, without parameters. This will print the list of available command line parameters, and create the default configuration in the base directory.
JDosage2 base directory is .jdosage, located in the user home directory. This directory contains configuration, webcomic definitions and download logs.
User home directory is normally:
-
C:\Documents and Settings\<username> on Windows XP or older
-
C:\Users\<username> on Windows Vista or newer
-
/home/<username> on UNIX or GNU/Linux
Before downloading your first webcomic, you should edit the configuration according to your preferences. You will probably want to change the download directory, but the rest of the defaults should be fine.
To download a webcomic, you need to create a webcomic definition file. This file contains all the parameters needed for downloading comic images.
Finally, to start the download, run jdosage2.sh -d <webcomic>, where <webcomic> is the name of the definition file. The console will display download progress and possible errors.
3. Configuration
Configuration file config.properties is located in the base directory. The syntax is very simple: entries are name/value pairs and comments are prefixed by a hash mark. This is the default configuration:
TODO external file to be dropped
#JDosage2 configuration #Thu Oct 11 11:34:35 CEST 2012 download.dir=/home/fjagic/Webcomics connection.timeout=30 connection.retries=3 log.level.console=INFO log.level.file=DEBUG
3.1. Configuration parameters
This is a complete list of supported configuration parameters. Unknown entries are ignored.
Webcomic download directory. Default value is <user_home>/Webcomics. If you are a Windows user, you will probably want to change this to your Documents or Downloads directory.
HTTP connection timeout in seconds. Default value is 10. Increase if you have a slow or unstable internet connection.
Number of download retries in case of a timeout or a connection error. Default value is 3.
Logging level of console output. Default value is INFO. Set to DEBUG or TRACE when testing new webcomic definitions.
Logging level for the file log. Default level is DEBUG. Log files are located in the log subdirectory in the base directory.
3.2. Log levels
-
ALL - log everything (essentially the same as TRACE)
-
TRACE - log very detailed webcomic processing information, useful for debugging webcomic definitions
-
DEBUG - log less detailed webcomic processing information
-
INFO - log general information and comic downloads
-
WARN - log webcomic processing errors and external errors only
-
ERROR - log external errors only
-
OFF - log nothing
4. Webcomic definitions
Webcomic definition files are located in the webcomics subdirectory in the base directory. To add a webcomic definition, create a new text file with wcd extension. The syntax is the same as the configuration file: entries are name/value pairs and comments are prefixed by a hash mark.
4.1. Grouping
Webcomic definition files can be grouped in subdirectories inside the webcomics directory. When downloading webcomics, --download parameters accepts wildcards containg directory as well as definition filename. TODO see
4.2. Definition properties
This is a complete list of supported webcomic properties. Unknown entries are ignored.
Webcomic name is mandatory. It is usually used in file templates, so avoid non-ASCII characters.
Webcomic homepage is optional and not used when downloading comics. If it is set, it must be a valid URL.
Comic file template used to generate the path of the image file. Default value is ${name}/${filename}.${fileExt}. For more details on file template syntax see File templates
URL of the last comic webpage. If it is set, new downloads start from the last page, and download progress is cleared when download is finished. This is the preferred method because new downloads do not depend on previously saved progress to continue. If both first and last page are set, last page is used and first page is ignored.
URL of the first comic webpage. If it is set, new downloads start from the last downloaded comic. Download progress is saved and never cleared. Used when the latest comic cannot be reached directly through a static url.
Selector for comic images. Results must be images or links to image files. For more details on selector syntax see Selectors
Selector for the next webcomic page. Result must be a link. Multiple results are accepted only if they are links to the same URL. For more details on selector syntax see Selectors
4.3. File templates
When a comic image is downloaded, file template is used to generate the path and filename of the image file. File templates are always resolved relative to the download directory.
File templates are FreeMarker templates. For details on syntax, see FreeMarker manual.
The following variables are available in file templates:
-
name - name of the webcomic
-
imageUrl1 - url of the comic image
-
filename - comic image filename, without file extension
-
fileExt - comic image file extension
-
pageUrl1 - url of the current webpage
-
pageTitle2 - HTML title of the current webpage
-
imageText2 - comic image alt text, or link text if the image is linked
-
URL is decoded from the application/x-www-form-urlencoded format.
-
HTML entites are decoded, and text trimmed. Concurrent whitespace characters are replaced with a single space. Illegal filename characters are replaced with an underscore. Illegal filename characters are |, \, ?, *, <, ", :, >, +, [, ] and /.
TODO examples
4.4. Selectors
Selectors are HTML element matching expressions, used to find image and link tags when processing comic webpages.
JDosage2 uses JSoup selectors, similar to CSS selectors. For details, see JSoup documentation.
TODO examples
5. Download algorithm
JDosage2 scrapes webpage HTML and finds comic images and page links. Download algorithm works as follows:
-
get the current webpage URL from saved progress, lastPage or firstPage
-
download the webpage
-
find images and links matching the comic selector
-
if no comics are found, stop download with error
-
download comics
-
if the comic file already exists and webcomic is downloaded from the lastPage backwards, clear progress and stop download
-
-
-
find the link matching nextPage selector
-
if no next page link is found
-
if downloading from the lastPage backwards, clear progress
-
stop download
-
-
update download progress with the new URL
-
-
repeat until stop, error, or program termination
6. Running the program
JDosage2 is a command line program. On GNU/Linux, run jdosage2.sh from the Terminal. On Windows, run jdosage2.cmd from the Command Prompt. Run the program without parameters to see a list of parameters with their descriptions.
To stop the program, terminate it by pressing Ctrl + C.
6.1. Command line parameters
List all webcomics matching the wildcard expressions, or all if the expressions are ommitted.
Download new comics for all webcomics matching the wildcard expressions.
Do not stop the download if an existing comic image is found.
Do not stop the download if webpage contains no comic images.
6.2. Wildcard expression
Webcomics are selected by wildcard expressions matching webcomic definition filenames. ? matches one or zero characters, and * matches zero or more characters.
Expressions match the directory and filename inside the webcomic definitions directory.
* - matches all webcomics in the definitions directory (not subdirectories)
weekly/x* - matches all webcomics in the weekly subdirectory starting with g
|
If you are using BASH shell, escape * with backclash to prevent wildcard expansion. |
7. Licence
This program is licensed under GNU General Public License Version 3. For license details, see license.txt.
The licenses of dependencies are:
Apache Commons CLI - Apache License Version 2.0, https://commons.apache.org/cli/
Apache Commons IO - Apache License Version 2.0, https://commons.apache.org/io/
Apache Commons Lang - Apache License Version 2.0, https://commons.apache.org/lang/
FreeMarker - BSD-style license, http://freemarker.sourceforge.net/
JavaMail API - GNU General Public License Version 2, http://www.oracle.com/technetwork/java/javamail/index.html
JSoup - MIT License, http://jsoup.org/
Logback - GNU Lesser General Public License Version 2.1, http://logback.qos.ch/
Slf4j - MIT License, http://www.slf4j.org/
8. Future
Future version will include:
-
Installer
-
GUI
-
Parallel downloads