bugCompact Disc Input and Control Library - Bugs: bug #39373, wrong iso-read >4G .iso due to...

 
 

bug #39373: wrong iso-read >4G .iso due to 32bit offset when should 64bit

Submitter:  Leo Baschy <infrastrug>
Submitted:  Sun 30 Jun 2013 09:23:28 PM UTC
   
 
Category:  run-time error Severity:  5 - Average
Item Group:  iso-read Status:  Fixed
Privacy:  Public Assigned to:  rocky
Open/Closed:  Closed
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Sat 28 Sep 2013 10:37:16 AM UTC, comment #15: 

Hereby attaching a iso-read-test.sh that has been tested both in Enterprise Linux and Ubuntu.

Previous version in Ubuntu wasn't working right because of "which" command differences.

Also attached a new .tar.gz of an .iso (4KB for a 1MB .iso) ... if provided then is expanded and used automatically and the script for a 1M .iso doesn't depend on genisoimage or mkisofs.  Must match script (e.g. file names in it) or it is a mess.  Caching always comes with that risk.

(file #29250, file #29251)

Leo Baschy <infrastrug>
Sat 28 Sep 2013 09:00:12 AM UTC, comment #14: 

To be clear, the mystery problem I mentioned last couple posts on this topic has become a bug of its own 40130.

Leo Baschy <infrastrug>
Sat 28 Sep 2013 01:30:56 AM UTC, comment #13: 

Should have mentioned, the iso-read problems I saw today in Ubuntu 64 bit on that one not so large .iso file looked more like that other bug, 39354 than this one 39373.  Perplexing.

I only mentioned it to correctly report any issues I see, as much as I wish there weren't any and this could become 0.91.

Leo Baschy <infrastrug>
Sat 28 Sep 2013 01:01:42 AM UTC, comment #12: 

First I'd like to report a successful test on Ubuntu version 13.04 64 bit against a 5G .iso file, and against a 1M .iso file, i.e. passing iso-read-test.sh.

When running make check, however, I do notice both on SL 6.4 32 bit and Ubuntu 13.04 64 bit the following:


...
make[3]: `track' is up to date.
make[3]: `win32' is up to date.
make[3]: Leaving directory `/home/username/Projects/libcdio/test/driver'
make  check-TESTS
make[3]: Entering directory `/home/username/Projects/libcdio/test/driver'
PASS: abs_path
PASS: bincue
PASS: cdda
++ WARN: /home/username/Projects/libcdio/test/data/cdtext.toc line 47: SILENCE not fully implimented
++ WARN: /home/username/Projects/libcdio/test/data/t1.toc line 5: SILENCE not fully implimented
++ WARN: number of minutes (180) truncated to 99.
++ WARN: /home/username/Projects/libcdio/test/data/data1.toc line 7: SILENCE not fully implimented
++ WARN: /home/username/Projects/libcdio/test/data/data2.toc line 8: unimplimented keyword: ZERO
++ WARN: could not retrieve file info for `foo.bin': No such file or directory
PASS: cdrdao
SKIP: freebsd
PASS: gnu_linux
PASS: logger
PASS: mmc_read
PASS: mmc_write
PASS: nrg
SKIP: osx
-- Temp directory is /tmp/
PASS: realpath
SKIP: solaris
PASS: track
SKIP: win32
======================
All 11 tests passed
(4 tests were not run)
======================
make[3]: Leaving directory `/home/username/Projects/libcdio/test/driver'
make[2]: Leaving directory `/home/username/Projects/libcdio/test/driver'
make[2]: Entering directory `/home/username/Projects/libcdio/test'
ln: failed to create symbolic link ‘../test/data/isofs-m1.bin’: File exists
make[2]: *** [make-executable] Error 1
make[2]: Leaving directory `/home/username/Projects/libcdio/test'
make[1]: *** [check-recursive] Error 1
make[1]: Leaving directory `/home/username/Projects/libcdio/test'
make: *** [check-recursive] Error 1


While I had some major progress with iso-read from git HEAD now working better in Ubuntu ...

I am not ready to say all its problems are fixed.  Test files passed, one real use of a large .iso file passed, and then one real use of a smaller .iso file failed.  To early to say what it has been.  I should report when sure one way or the other.

Leo Baschy <infrastrug>
Fri 27 Sep 2013 01:21:30 AM UTC, comment #11: 

Ok. Commit 9c9f9d0 has this. Please double check.

Rocky Bernstein <rocky>
Group administrator
Thu 26 Sep 2013 07:10:17 AM UTC, comment #10: 

First line of iso-read-test.sh should be


#!/bin/bash


or else that test will not run in Ubuntu.

Leo Baschy <infrastrug>
Wed 18 Sep 2013 12:12:21 AM UTC, comment #9: 

Regarding tests for this bug 39373 and 39354.

Ran into problems (in another project) with genisoimage 1.1.11 crashing in Ubuntu 12.04 LTS and found out about issues around genisoimage and mkisofs.

Hence to avoid libcdio tests from failing on systems where genisoimage or mkisofs have problems (unrelated to libcdio) I have modified iso-read-test.sh so in case there isn't an .iso file but there is an .iso.tar.gz file it now extracts with tar instead of making anew from a directory.

Then if the needed .iso.tar.gz files (made on a first run) are in version control, then no dependency on genisoimage or mkisofs (working or not) exists.

I am attaching a new iso-read-test.sh and a 1 megabyte and a 5000 megabytes .iso.tar.gz, obviously compressed.

(file #29166, file #29167)

Leo Baschy <infrastrug>
Sun 15 Sep 2013 11:28:36 PM UTC, comment #8: 

Passes my visual verification of source code and testing.

That is in 32-bit Enterprise Linux 6.4.

Leo Baschy <infrastrug>
Sun 15 Sep 2013 10:39:27 PM UTC, comment #7: 

Attached patch applied. Please try with commit ed2c9d1 . Thanks.

Rocky Bernstein <rocky>
Group administrator
Sun 15 Sep 2013 04:26:08 PM UTC, comment #6: 

The email wrapped the diff, hence something got confused and the commit lost adding the p_iso->i_datastart.

The only change against the original code is supposed to be the type cast


(int64_t)(p_iso->i_framesize)


for a total assignment calculation of


i_byte_offset = (start * (int64_t)(p_iso->i_framesize)) + p_iso->i_fuzzy_offset + p_iso->i_datastart;


I should have attached a .diff file instead of just pasting it into the comment.  Luckily you requested a check after applying.

Leo Baschy <infrastrug>
Sun 15 Sep 2013 10:17:01 AM UTC, comment #5: 

Applied in commit c48003d . Please check to see that the patch was applied correctly.

To ensure a regression doesn't take place, a test (run in only in "maintainer mode") should be added.

Rocky Bernstein <rocky>
Group administrator
Sun 15 Sep 2013 12:52:10 AM UTC, comment #4: 

Here is a single line diff to make iso-read-test.sh pass for a 5 gigabytes large .iso file:


diff --git a/lib/iso9660/iso9660_fs.c b/lib/iso9660/iso9660_fs.c
index 3eb06b5..8974923 100644
--- a/lib/iso9660/iso9660_fs.c
+++ b/lib/iso9660/iso9660_fs.c
@@ -688,7 +688,7 @@ iso9660_seek_read_framesize (const iso9660_t *p_iso, void *ptr,
   int64_t i_byte_offset;

   if (!p_iso) return 0;
-  i_byte_offset = (start * p_iso->i_framesize) + p_iso->i_fuzzy_offset
+  i_byte_offset = (start * (int64_t)(p_iso->i_framesize)) + p_iso->i_fuzzy_offset
     + p_iso->i_datastart;

   ret = cdio_stream_seek (p_iso->stream, i_byte_offset, SEEK_SET);


I have tested on 32-bit Enterprise Linux 6.4.

For testing invoke attached iso-read-test.sh with 5000 as additional parameter on the command line.  Note that will use 5 gigabytes preparation directory and 5 gigabytes for .iso file for total disk use of 10 gigabytes.  Time consuming only on first run, reuses existing file on further runs.  Note this .iso file compresses very well, into a mere 5 megabytes (1/1000th) size .tar.gz file, look into directory after it ran.  It might be possible to construct a test using less disk space, not sure about that.

This was an "easy" fix now, after 39354 had been fixed, Rock Ridge working, since now it was possible to work on HEAD and apparently a lot of switching over to correct use of 64 bits for offset had been done already since 0.81.  Except this little oversight.

Leo Baschy <infrastrug>
Wed 14 Aug 2013 12:57:42 AM UTC, comment #3: 

Further improved test script.  Has been used to bisect 39354.

Given an integer argument of 5000, works for this defect 39373.

What is new:

  • exits with 0 on success, 1 on failure
  • takes two optional arguments in any order
  • one optional argument if an integer is used as size in megabytes of .iso file it makes, default is 1 for 1MB, and to get 5GB .iso file give it 5000
  • by default only uses 3 megabytes of disk space, no longer gigabytes
  • other optional argument is what iso-read to use, default is plain iso-read, but for example if using with git bisect may make sense to pass src/iso-read


As written here has been run in Enterprise Linux 6.4 and Mac OS X 10.8.


#!/bin/sh
#
# iso-read-test.sh
#
# takes zero, one, or two optional parameters in any order
#
# ONE optional parameter:
# an integer number for approximate size of .iso file in megabytes,
# default is 1, for 1 megabyte,
# to test approximately 5 gigabytes pass 5000
#
# OTHER optional parameter:
# the iso-read command you want to test,
# default is iso-read,
# has to be named iso-read to be recognized,
# could be for example src/iso-read

# in $1 or $2 accept integer number as total .iso size in megabytes
if [[ $1 =~ ^[0-9]+$ ]]
then
    size="$1"
elif [[ $2 =~ ^[0-9]+$ ]]
then
    size="$2"
else
    size="1" # default
fi

# in $1 or $2 accept a different location for iso-read,
# e.g. ~/libcdio/src/iso-read
if [[ $1 =~ iso-read ]]
then
    isoread="$1"
elif [[ $2 =~ iso-read ]]
then
    isoread="$2"
else
    isoread="iso-read" # default
fi
# make absolute before changing directory
isoread=`which "$isoread"`
isoreaddir=`dirname "$isoread"`
isoreaddir=`cd "$isoreaddir" && pwd`
isoread="$isoreaddir"/`basename "$isoread"`

echo "Going to use $isoread ."
"$isoread" --version

# this says to make test files in directory where this script is,
# and just so you know, those files will be megabytes large,
# but they will be reused on subsequent runs to be much faster.
# like a hundred times faster or more
datadir=`dirname "$0"`
datadir=`cd "$datadir" && pwd`
cd "$datadir"

# pick for your system,
# on a Mac mkisofs, otherwise people use newer genisoimage
isomakingcommand=""
which mkisofs && isomakingcommand="mkisofs"
which genisoimage && isomakingcommand="genisoimage"
if [ -z "$isomakingcommand" ]
then
    echo "NEITHER mkisofs NOR genisoimage FOUND !"
    exit 1
fi

echo "Working with a $size megabyte large .iso file in $datadir ."

isoname="exampleIso"$size"M.iso"

prepdir="prep"

if [ ! -f "$isoname" ]
then

    echo "Going to use " `which "$isomakingcommand"` " to prepare $isoname in $isoname.$prepdir ."

    mkdir "$isoname.$prepdir"
    mkdir "$isoname.$prepdir"/mixed

    # a large file with many of the same characters that will compress well
    dd if=/dev/zero bs=1024 count=$size"00" | tr '\0' '\060' > "$isoname.$prepdir/mixed/file0l.txt"
    # a small file we will want to iso-read
    echo "content of short text file #0 we are looking for" > "$isoname.$prepdir/mixed/file0s.txt"
    # a few times over now
    dd if=/dev/zero bs=1024 count=$size"00" | tr '\0' '\061' > "$isoname.$prepdir/mixed/file1l.txt"
    echo "content of short text file #1 we are looking for" > "$isoname.$prepdir/mixed/fileOneSmall.txt"
    dd if=/dev/zero bs=1024 count=$size"00" | tr '\0' '\062' > "$isoname.$prepdir/mixed/file2l.txt"
    echo "content of short text file #2 we are looking for" > "$isoname.$prepdir/mixed/file2small.txt"
    dd if=/dev/zero bs=1024 count=$size"00" | tr '\0' '\063' > "$isoname.$prepdir/mixed/file3l.txt"
    echo "content of short text file #3 we are looking for" > "$isoname.$prepdir/mixed/file3ThreeSmall.txt"
    dd if=/dev/zero bs=1024 count=$size"00" | tr '\0' '\064' > "$isoname.$prepdir/mixed/file4l.txt"
    echo "content of short text file #4 we are looking for" > "$isoname.$prepdir/mixed/fileFourS.txt"
    dd if=/dev/zero bs=1024 count=$size"00" | tr '\0' '\065' > "$isoname.$prepdir/mixed/file5l.txt"
    echo "content of short text file #5 we are looking for" > "$isoname.$prepdir/mixed/file5small.txt"
    dd if=/dev/zero bs=1024 count=$size"00" | tr '\0' '\066' > "$isoname.$prepdir/mixed/file6l.txt"
    echo "content of short text file #6 we are looking for" > "$isoname.$prepdir/mixed/file6s.txt"
    dd if=/dev/zero bs=1024 count=$size"00" | tr '\0' '\067' > "$isoname.$prepdir/mixed/file7l.txt"
    echo "content of short text file #7 we are looking for" > "$isoname.$prepdir/mixed/fileSevenSmall.txt"
    dd if=/dev/zero bs=1024 count=$size"00" | tr '\0' '\068' > "$isoname.$prepdir/mixed/file8l.txt"
    echo "content of short text file #8 we are looking for" > "$isoname.$prepdir/mixed/file8s.txt"
    dd if=/dev/zero bs=1024 count=$size"00" | tr '\0' '\069' > "$isoname.$prepdir/mixed/file9l.txt"
    echo "content of short text file #9 we are looking for" > "$isoname.$prepdir/mixed/file9s.txt"

    # a potentially huge .iso file, whatever you wrote above
    "$isomakingcommand" -r -J -T -f -V "EXAMPLEISO"$size"M" -o "$isoname" "$isoname.$prepdir"

    # if you want to archive a test file, this compressed one is much smaller,
    # seen 4KB instead of 1MB, seen 5MB instead of 5GB
    tar -cvzf "$isoname.tar.gz" "$isoname"

else
    echo "Using existing $isoname ."
fi

# remind of disk space use
ls -al
du -h .

echo "
####################################################
#
# Size $size megabyte .iso file read back:
#
"
for i in {0..9}
do
    echo "didn't read file #$i" > $size"Mfile"$i"s-read.txt"
done
#
"$isoread" --version
"$isoread" -i "$isoname" -e "mixed/file0s.txt" -o $size"Mfile0s-read.txt"
"$isoread" -i "$isoname" -e "mixed/fileOneSmall.txt" -o $size"Mfile1s-read.txt"
"$isoread" -i "$isoname" -e "mixed/file2small.txt" -o $size"Mfile2s-read.txt"
"$isoread" -i "$isoname" -e "mixed/file3ThreeSmall.txt" -o $size"Mfile3s-read.txt"
"$isoread" -i "$isoname" -e "mixed/fileFourS.txt" -o $size"Mfile4s-read.txt"
"$isoread" -i "$isoname" -e "mixed/file5small.txt" -o $size"Mfile5s-read.txt"
"$isoread" -i "$isoname" -e "mixed/file6s.txt" -o $size"Mfile6s-read.txt"
"$isoread" -i "$isoname" -e "mixed/fileSevenSmall.txt" -o $size"Mfile7s-read.txt"
"$isoread" -i "$isoname" -e "mixed/file8s.txt" -o $size"Mfile8s-read.txt"
"$isoread" -i "$isoname" -e "mixed/file9s.txt" -o $size"Mfile9s-read.txt"
#
anybad=0
for i in {0..9}
do
    readback=`cat $size"Mfile"$i"s-read.txt"`
    if [ "$readback" = "content of short text file #$i we are looking for" ]
    then
        echo "file #$i read back OK: $readback"
    else
        anybad=1
        echo "file #$i read DEFECT : $readback"
    fi
done
if [ $anybad -eq 0 ]
then
    echo "Overall success of $0 :)"
else
    echo "Overall FAILURE of $0 :("
fi
exit $anybad

# if it worked correctly you should see the lines we wrote into the short files:
#
# content of short text file #i we are looking for
#
# IF however defect https://savannah.gnu.org/bugs/?39373 is still
# in effect you might see lines such as:
# 111111111111111111111111111111111111
#
# furthermore, IF however defect https://savannah.gnu.org/bugs/?39354 is still
# in effect you might see lines such as:
#
# iso-read: Could not get ISO-9660 file information out of $isoname for file mixed/file2small.txt.
# iso-read: iso-info may be able to show the contents of $isoname.


Leo Baschy <infrastrug>
Mon 29 Jul 2013 07:46:56 PM UTC, comment #2: 

Here is a better test script for this bug and for bug 39354.

Runs on Mac (tested this version) and on Linux (should run).

Makes two .iso, one 5G large, one 3G large.

Reads back files to verify whether iso-read works correctly.

On repeated runs avoids making .iso again, uses .iso made on first run.


#!/bin/sh

prepdir="isoprep"

# pick for your system,
# on a Mac mkisofs, otherwise people use newer genisoimage
isomakingcommand=""
which mkisofs && isomakingcommand="mkisofs"
which genisoimage && isomakingcommand="genisoimage"
if [ -z $isomakingcommand]
then
    echo "NEITHER mkisofs NOR genisoimage FOUND"
    exit 1
fi

if [ ! -f exampleIsoGt4G.iso ]
then

    echo "going to prepare exampleIsoGt4G.iso in gt4G$prepdir"

    mkdir gt4G$prepdir
    mkdir gt4G$prepdir/mixed

    # a large file with many of the same characters that will compress well
    dd if=/dev/zero bs=1024 count=512000 | tr '\0' '\060' > gt4G$prepdir/mixed/file0l.txt
    # a small file we will want to iso-read
    echo "this is short text file #0 we are looking for" > gt4G$prepdir/mixed/file0s.txt
    # a few times over now
    dd if=/dev/zero bs=1024 count=512000 | tr '\0' '\061' > gt4G$prepdir/mixed/file1l.txt
    echo "this is short text file #1 we are looking for" > gt4G$prepdir/mixed/fileOneSmall.txt
    dd if=/dev/zero bs=1024 count=512000 | tr '\0' '\062' > gt4G$prepdir/mixed/file2l.txt
    echo "this is short text file #2 we are looking for" > gt4G$prepdir/mixed/file2small.txt
    dd if=/dev/zero bs=1024 count=512000 | tr '\0' '\063' > gt4G$prepdir/mixed/file3l.txt
    echo "this is short text file #3 we are looking for" > gt4G$prepdir/mixed/file3ThreeSmall.txt
    dd if=/dev/zero bs=1024 count=512000 | tr '\0' '\064' > gt4G$prepdir/mixed/file4l.txt
    echo "this is short text file #4 we are looking for" > gt4G$prepdir/mixed/fileFourS.txt
    dd if=/dev/zero bs=1024 count=512000 | tr '\0' '\065' > gt4G$prepdir/mixed/file5l.txt
    echo "this is short text file #5 we are looking for" > gt4G$prepdir/mixed/file5small.txt
    dd if=/dev/zero bs=1024 count=512000 | tr '\0' '\066' > gt4G$prepdir/mixed/file6l.txt
    echo "this is short text file #6 we are looking for" > gt4G$prepdir/mixed/file6s.txt
    dd if=/dev/zero bs=1024 count=512000 | tr '\0' '\067' > gt4G$prepdir/mixed/file7l.txt
    echo "this is short text file #7 we are looking for" > gt4G$prepdir/mixed/fileSevenSmall.txt
    dd if=/dev/zero bs=1024 count=512000 | tr '\0' '\068' > gt4G$prepdir/mixed/file8l.txt
    echo "this is short text file #8 we are looking for" > gt4G$prepdir/mixed/file8s.txt
    dd if=/dev/zero bs=1024 count=512000 | tr '\0' '\069' > gt4G$prepdir/mixed/file9l.txt
    echo "this is short text file #9 we are looking for" > gt4G$prepdir/mixed/file9s.txt

    # a huge .iso file about 5G or whatever you wrote above
    $isomakingcommand -r -J -T -f -V EXAMPLEISOGT4G -o exampleIsoGt4G.iso gt4G$prepdir

    # if you want to archive a test file, this compressed one is much smaller, 5MB
    tar -cvzf exampleIsoGt4G.iso.tar.gz exampleIsoGt4G.iso

fi

# AND now another .iso with less than 4G, about 3G
# for https://savannah.gnu.org/bugs/?39354

if [ ! -f exampleIsoLt4G.iso ]
then

    echo "going to prepare exampleIsoLt4G.iso in lt4G$prepdir"

    mkdir lt4G$prepdir
    mkdir lt4G$prepdir/mixed

    # a large file with many of the same characters that will compress well
    dd if=/dev/zero bs=1024 count=300000 | tr '\0' '\060' > lt4G$prepdir/mixed/file0l.txt
    # a small file we will want to iso-read
    echo "this is short text file #0 we are looking for" > lt4G$prepdir/mixed/file0s.txt
    # a few times over now
    dd if=/dev/zero bs=1024 count=300000 | tr '\0' '\061' > lt4G$prepdir/mixed/file1l.txt
    echo "this is short text file #1 we are looking for" > lt4G$prepdir/mixed/fileOneSmall.txt
    dd if=/dev/zero bs=1024 count=300000 | tr '\0' '\062' > lt4G$prepdir/mixed/file2l.txt
    echo "this is short text file #2 we are looking for" > lt4G$prepdir/mixed/file2small.txt
    dd if=/dev/zero bs=1024 count=300000 | tr '\0' '\063' > lt4G$prepdir/mixed/file3l.txt
    echo "this is short text file #3 we are looking for" > lt4G$prepdir/mixed/file3ThreeSmall.txt
    dd if=/dev/zero bs=1024 count=300000 | tr '\0' '\064' > lt4G$prepdir/mixed/file4l.txt
    echo "this is short text file #4 we are looking for" > lt4G$prepdir/mixed/fileFourS.txt
    dd if=/dev/zero bs=1024 count=300000 | tr '\0' '\065' > lt4G$prepdir/mixed/file5l.txt
    echo "this is short text file #5 we are looking for" > lt4G$prepdir/mixed/file5small.txt
    dd if=/dev/zero bs=1024 count=300000 | tr '\0' '\066' > lt4G$prepdir/mixed/file6l.txt
    echo "this is short text file #6 we are looking for" > lt4G$prepdir/mixed/file6s.txt
    dd if=/dev/zero bs=1024 count=300000 | tr '\0' '\067' > lt4G$prepdir/mixed/file7l.txt
    echo "this is short text file #7 we are looking for" > lt4G$prepdir/mixed/fileSevenSmall.txt
    dd if=/dev/zero bs=1024 count=300000 | tr '\0' '\068' > lt4G$prepdir/mixed/file8l.txt
    echo "this is short text file #8 we are looking for" > lt4G$prepdir/mixed/file8s.txt
    dd if=/dev/zero bs=1024 count=300000 | tr '\0' '\069' > lt4G$prepdir/mixed/file9l.txt
    echo "this is short text file #9 we are looking for" > lt4G$prepdir/mixed/file9s.txt

    # a huge .iso file about 5G or whatever you wrote above
    $isomakingcommand -r -J -T -f -V EXAMPLEISOLT4G -o exampleIsoLt4G.iso lt4G$prepdir

    # if you want to archive a test file, this compressed one is much smaller, 5MB
    tar -cvzf exampleIsoLt4G.iso.tar.gz exampleIsoLt4G.iso

fi

# remind of disk space use
ls -al
du -h .

echo "\n####################################################\n#\n# Greater than 4G .iso files read back:\n#\n"
iso-read --version
iso-read -i exampleIsoGt4G.iso -e mixed/file0s.txt -o gt4Gfile0s-read.txt
cat gt4Gfile0s-read.txt ; echo
iso-read -i exampleIsoGt4G.iso -e mixed/fileOneSmall.txt -o gt4Gfile1s-read.txt
cat gt4Gfile1s-read.txt ; echo
iso-read -i exampleIsoGt4G.iso -e mixed/file2small.txt -o gt4Gfile2s-read.txt
cat gt4Gfile2s-read.txt ; echo
iso-read -i exampleIsoGt4G.iso -e mixed/file3ThreeSmall.txt -o gt4Gfile3s-read.txt
cat gt4Gfile3s-read.txt ; echo
iso-read -i exampleIsoGt4G.iso -e mixed/fileFourS.txt -o gt4Gfile4s-read.txt
cat gt4Gfile4s-read.txt ; echo
iso-read -i exampleIsoGt4G.iso -e mixed/file5small.txt -o gt4Gfile5s-read.txt
cat gt4Gfile5s-read.txt ; echo
iso-read -i exampleIsoGt4G.iso -e mixed/file6s.txt -o gt4Gfile6s-read.txt
cat gt4Gfile6s-read.txt ; echo
iso-read -i exampleIsoGt4G.iso -e mixed/fileSevenSmall.txt -o gt4Gfile7s-read.txt
cat gt4Gfile7s-read.txt ; echo
iso-read -i exampleIsoGt4G.iso -e mixed/file8s.txt -o gt4Gfile8s-read.txt
cat gt4Gfile8s-read.txt ; echo
iso-read -i exampleIsoGt4G.iso -e mixed/file9s.txt -o gt4Gfile9s-read.txt
cat gt4Gfile9s-read.txt ; echo

# if it worked correctly you should see the lines we wrote into the short files:
#
# this is short text file #n we are looking for
#
# IF however defect https://savannah.gnu.org/bugs/?39373 is still
# in effect you might see lines such as:
# 111111111111111111111111111111111111
#
# furthermore, IF however defect https://savannah.gnu.org/bugs/?39354 is still
# in effect you might see lines such as:
#
# iso-read: Could not get ISO-9660 file information out of exampleIsoGt4G.iso for file mixed/file2small.txt.
# iso-read: iso-info may be able to show the contents of exampleIsoGt4G.iso.

echo "\n####################################################\n#\n# Less than 4G .iso files read back:\n#\n"
iso-read --version
iso-read -i exampleIsoLt4G.iso -e mixed/file0s.txt -o lt4Gfile0s-read.txt
cat lt4Gfile0s-read.txt ; echo
iso-read -i exampleIsoLt4G.iso -e mixed/fileOneSmall.txt -o lt4Gfile1s-read.txt
cat lt4Gfile1s-read.txt ; echo
iso-read -i exampleIsoLt4G.iso -e mixed/file2small.txt -o lt4Gfile2s-read.txt
cat lt4Gfile2s-read.txt ; echo
iso-read -i exampleIsoLt4G.iso -e mixed/file3ThreeSmall.txt -o lt4Gfile3s-read.txt
cat lt4Gfile3s-read.txt ; echo
iso-read -i exampleIsoLt4G.iso -e mixed/fileFourS.txt -o lt4Gfile4s-read.txt
cat lt4Gfile4s-read.txt ; echo
iso-read -i exampleIsoLt4G.iso -e mixed/file5small.txt -o lt4Gfile5s-read.txt
cat lt4Gfile5s-read.txt ; echo
iso-read -i exampleIsoLt4G.iso -e mixed/file6s.txt -o lt4Gfile6s-read.txt
cat lt4Gfile6s-read.txt ; echo
iso-read -i exampleIsoLt4G.iso -e mixed/fileSevenSmall.txt -o lt4Gfile7s-read.txt
cat lt4Gfile7s-read.txt ; echo
iso-read -i exampleIsoLt4G.iso -e mixed/file8s.txt -o lt4Gfile8s-read.txt
cat lt4Gfile8s-read.txt ; echo
iso-read -i exampleIsoLt4G.iso -e mixed/file9s.txt -o lt4Gfile9s-read.txt
cat lt4Gfile9s-read.txt ; echo

# if it worked correctly you should see the lines we wrote into the short files:
#
# this is short text file #n we are looking for
#
# IF however defect https://savannah.gnu.org/bugs/?39354 is still
# in effect you might see lines such as:
#
# iso-read: Could not get ISO-9660 file information out of exampleIsoGt4G.iso for file mixed/file2small.txt.
# iso-read: iso-info may be able to show the contents of exampleIsoGt4G.iso.


Leo Baschy <infrastrug>
Tue 02 Jul 2013 02:57:35 AM UTC, comment #1: 

This test script makes .iso 5G large, uses 10G or 15G while making it, then tries iso-read.  Watch for using either mkisofs or genisoimage on your system.


#!/bin/sh

prepdir="isoprep"

echo "going to prepare in $prepdir"

mkdir $prepdir
mkdir $prepdir/biggies
mkdir $prepdir/tail

# a few large files with many of the same characters in them that will compress well
dd if=/dev/zero bs=1024 count=512000 | tr '\0' '\060' > $prepdir/biggies/file0.txt
dd if=/dev/zero bs=1024 count=512000 | tr '\0' '\061' > $prepdir/biggies/file1.txt
dd if=/dev/zero bs=1024 count=512000 | tr '\0' '\062' > $prepdir/biggies/file2.txt
dd if=/dev/zero bs=1024 count=512000 | tr '\0' '\063' > $prepdir/biggies/file3.txt
dd if=/dev/zero bs=1024 count=512000 | tr '\0' '\064' > $prepdir/biggies/file4.txt
dd if=/dev/zero bs=1024 count=512000 | tr '\0' '\065' > $prepdir/biggies/file5.txt
dd if=/dev/zero bs=1024 count=512000 | tr '\0' '\066' > $prepdir/biggies/file6.txt
dd if=/dev/zero bs=1024 count=512000 | tr '\0' '\067' > $prepdir/biggies/file7.txt
dd if=/dev/zero bs=1024 count=512000 | tr '\0' '\068' > $prepdir/biggies/file8.txt
dd if=/dev/zero bs=1024 count=512000 | tr '\0' '\069' > $prepdir/biggies/file9.txt

# the file we will want to iso-read
echo "this is the file we are looking for" > $prepdir/tail/thefile.txt

# a huge .iso file about 5G or whatever you wrote above
# on a Mac mkisofs, otherwise people use newer genisoimage
mkisofs -r -J -T -f -V EXAMPLEISOGT4G -o exampleIsoGt4G.iso $prepdir

# if you want to archive a test file, this compressed one is much smaller, 5MB
tar -cvzf exampleIsoGt4G.iso.tar.gz exampleIsoGt4G.iso

# remind of disk space use
ls -al
du -h .

iso-read -i exampleIsoGt4G.iso -e tail/thefile.txt -o thefile-read.txt
# did it read correctly ?
cat thefile-read.txt

# if it worked correctly you should see the line from above:
# this is the file we are looking for
#
# IF however defect https://savannah.gnu.org/bugs/?39373 is still
# in effect you might see:
# 111111111111111111111111111111111111


Leo Baschy <infrastrug>
Sun 30 Jun 2013 09:23:28 PM UTC, original submission:  

Problem discussed 2010, 2011 at http://lists.gnu.org/archive/html/libcdio-help/2011-12/msg00001.html still a handicap for NrvrCommander.

Current work done with 0.81 because of indepedent problem in 0.90 in bug 39354.

The current test case is seeing nonsense read (4G off in .iso file) by


iso-read -i ~/Downloads/SL-64-x86_64-2013-03-18-Install-DVD.iso \
  -e repodata/repomd.xml \
  -o ~/repomd-gt4g.xml
cat ~/repomd-gt4g.xml


It does matter the downloaded .iso file is >4G such as http://ftp.scientificlinux.org/linux/scientific/6.4/x86_64/iso/SL-64-x86_64-2013-03-18-Install-DVD.iso

Working example for a slightly smaller .iso <4G, happens to other .iso of same distro, you get a nicely read file


iso-read -i ~/Downloads/SL-64-i386-2013-03-18-Install-DVD.iso \
  -e repodata/repomd.xml \
  -o ~/repomd-lt4g.xml
cat ~/repomd-lt4g.xml


I have dug into it and have "fixed" some issues but haven't succeeded at completing something that works.

First fix is in iso9660_fs.c in iso9660_seek_read_framesize to make i_byte_offset 64 bit, for which I happened to use


int64_t i_byte_offset;


because it seems is should be independent of platform it runs on, this is related to .iso files.  Either way, as long as it isn't 32 bit.

This library is supposed to run in 32 bit and 64 bit Linux.  We have seen problems on 64 bit machines but I have chosen to debug on a 32 bit VM.

Second fix is in same iso9660_fs.c in same iso9660_seek_read_framesize where it multiplies start * p_iso->framesize to first type cast to 64 bit before multiplying like so


i_byte_offset = (start * (int64_t)(p_iso->framesize)) + p_iso->i_fuzzy_offset + p_iso->i_datastart


Thirdly many functions like cdio_stream_seek must be fixed to allow offset of 64 bit.  That includes in _cdio_stream.h and _cdio_stream.c the function cdio_stream_seek parameter i_offset and function cdio_stream_getpos parameter *i_offset and the return value and typedef cdio_data_seek parameter offset.

This also includes in file _cdio_stream.c in struct _CdioDataSource the position must be 64 bit.

One more in _cdio_stdio.c for function parameter i_offset.

I cannot get this right so far, therefore I am filing this bug report so maybe someone else can look too.

Main idea is:  Offset into a file (an .iso file) to position can be 64 bit.  Remember to cast to 64 bit before multiplying 32 bit or smaller.  Everywhere there is a byte offset (also called a position) use 64 bit.

The example .iso are fine for making a fix.  If needed I (anyone) can make a special .iso that compresses to close to nothing but expands to >4G for testing purposes in a test suite.

Leo Baschy <infrastrug>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #30554:  iso-read-test.sh added by infrastrug (7KiB - application/x-shellscript - without carriage return characters, just line feed, so it runs in RHEL 7 beta)
file #29250:  iso-read-test.sh added by infrastrug (7KiB - application/octet-stream)
file #29251:  exampleIso1M.iso.tar.gz added by infrastrug (4KiB - application/x-gzip)
file #29166:  iso-read-test.sh added by infrastrug (7KiB - application/x-sh)
file #29167:  exampleIso1M.iso.tar.gz added by infrastrug (4KiB - application/x-gzip)
file #29145:  iso9660_fs_fix_fix.diff added by infrastrug (590B - application/octet-stream - diff for fix of fix, to avoid more errors in transfer)
file #28837:  iso-read-test.sh added by infrastrug (6KiB - application/octet-stream - generally useable iso-read-test.sh script)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by rocky (Posted a comment)
  • -email is unavailable- added by infrastrug (Submitted the item)
  • -email is unavailable- added by infrastrug
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 11 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2014-06-04 rocky Open/ClosedOpen Closed
    2014-02-15 infrastrug Attached File- Added iso-read-test.sh, #30554
    2013-10-10 rocky StatusNone Fixed
    2013-09-28 infrastrug Attached File- Added iso-read-test.sh, #29250
        Attached File- Added exampleIso1M.iso.tar.gz, #29251
    2013-09-27 rocky Assigned toNone rocky
    2013-09-18 infrastrug Attached File- Added iso-read-test.sh, #29166
        Attached File- Added exampleIso1M.iso.tar.gz, #29167
    2013-09-15 infrastrug Attached File- Added iso9660_fs_fix_fix.diff, #29145
    2013-08-14 infrastrug Attached File- Added iso-read-test.sh, #28837
    2013-06-30 infrastrug Carbon-Copy- Added infrastrug

    Back to the top

    Powered by Savane 3.13-cf05.
    Corresponding source code