Support for zipped binaries has been removed in 23ca8f5f942f831ec5be3667fd0a29873fae2912 and b2e27fcee4cec98ffc39273ecfaa73aace9da2c3.

Did Andrew's patch not get committed?  If so, can this bug be closed?  If not, can it be committed?  Thanks!

From http://bugs.debian.org/42079

From: -email is unavailable-
To: -email is unavailable-
Subject: hurd: exec doesn't like zip'ed binaries
Date: Tue, 27 Jul 1999 06:36:14 +0200

Package: hurd
Version: N/A
Severity: normal

Hello,

I tried the exec gzip'ed and bzip2'ed binaries on the fly feature, but it
didn't work quite right.

As far as I understood, I could take the hello world program "hello" and
expect the following to work:

$ gzip hello
$ ./gzip hello.gz

Right?

But it didn't. The command above did not return. Keyboard input was still
echoed, until I press ^C, when the system will "hang" and not even echo
keyboard input.

Same for bzip2'ed binaries.

On a related matter, zip'ed stores didn't work, too. Is there more to do
than creating a disk store with dd, running mke2fs on it, zip'ing it and
"settrans -c dummy /hurd/ext2fs `pwd`/dummy.fs.gz" ?
Accessing "dummy" resulted in some spiffy message about hypermetadata being
wrong (0 instead 0x53e7 or something.. This is from memory, and likely
wrong).

When this is fixed, I can take a look at boot disks...

Thanks,
Marcus


From: Gordon Matzigkeit <gord@fig.org>
To: -email is unavailable-
Cc: -email is unavailable-
Subject: Re: Bug #42079: hurd: exec doesn't like zip'ed binaries
Date: 28 Jul 1999 11:45:00 -0600

>>>>> brinkmd  writes:

 MB> As far as I understood, I could take the hello world program
 MB> "hello" and expect the following to work:

 MB> $ gzip hello
 MB> $ ./gzip hello.gz

I'm not at my Hurd machine right now, but do you mean

$ gzip hello
$ ./hello.gz

 MB> On a related matter, zip'ed stores didn't work, too. Is there
 MB> more to do than creating a disk store with dd, running mke2fs on
 MB> it, zip'ing it and "settrans -c dummy /hurd/ext2fs
 MB> `pwd`/dummy.fs.gz" ?

You may have to do:

$ settrans -c dummy /hurd/ext2fs -Tgunzip:file `pwd`/dummy.fs.gz

I'm not sure if libstore should automatically detect the type of
dummy.fs.gz or not.

From: Roland McGrath <roland@gnu.org>
To: Gordon Matzigkeit <gord@fig.org>, -email is unavailable-
Cc: -email is unavailable-
Subject: Re: Bug #42079: hurd: exec doesn't like zip'ed binaries
Date: Wed, 28 Jul 1999 14:11:05 -0400

> I'm not at my Hurd machine right now, but do you mean
>
> $ gzip hello
> $ ./hello.gz

Naturally he does.

>  MB> On a related matter, zip'ed stores didn't work, too. Is there
>  MB> more to do than creating a disk store with dd, running mke2fs on
>  MB> it, zip'ing it and "settrans -c dummy /hurd/ext2fs
>  MB> `pwd`/dummy.fs.gz" ?
>
> You may have to do:
>
> $ settrans -c dummy /hurd/ext2fs -Tgunzip:file `pwd`/dummy.fs.gz

That is correct.

> I'm not sure if libstore should automatically detect the type of
> dummy.fs.gz or not.

libstore does not do any "automatic detection" of this type.  If you don't
give a store-type (i.e. -T or --store-type option), then essentially there
is an implicit -Tfile (for the filesystems) or -Tdevice (for storeio).

If you wanted automatic detection based on file contents, the thing to do
would be a translator front-end program (or script) that examines the
contents and then selects the right translator and options to run.



From: Marcus Brinkmann <Marcus.Brinkmann@ruhr-uni-bochum.de>
To: Gordon Matzigkeit <gord@fig.org>, -email is unavailable-
Subject: Re: Bug #42079: hurd: exec doesn't like zip'ed binaries
Date: Wed, 28 Jul 1999 20:25:43 +0200

On Wed, Jul 28, 1999 at 11:45:00AM -0600, Gordon Matzigkeit wrote:

> I'm not at my Hurd machine right now, but do you mean
>
> $ gzip hello
> $ ./hello.gz

Yeah of course. I tried this and it failed :)

>  MB> On a related matter, zip'ed stores didn't work, too. Is there
>  MB> more to do than creating a disk store with dd, running mke2fs on
>  MB> it, zip'ing it and "settrans -c dummy /hurd/ext2fs
>  MB> `pwd`/dummy.fs.gz" ?
>
> You may have to do:
>
> $ settrans -c dummy /hurd/ext2fs -Tgunzip:file `pwd`/dummy.fs.gz
>
> I'm not sure if libstore should automatically detect the type of
> dummy.fs.gz or not.

I saw the -T option, but found no information about its syntax. I didn't try
hard, however. Any hints are welcome.

Thanks,
Marcus


From: Marcus Brinkmann <Marcus.Brinkmann@ruhr-uni-bochum.de>
To: Roland McGrath <roland@gnu.org>
Cc: -email is unavailable-
Subject: Re: Bug #42079: hurd: exec doesn't like zip'ed binaries
Date: Thu, 29 Jul 1999 17:56:27 +0200

On Wed, Jul 28, 1999 at 02:11:05PM -0400, Roland McGrath wrote:

> > You may have to do:
> >
> > $ settrans -c dummy /hurd/ext2fs -Tgunzip:file `pwd`/dummy.fs.gz
>
> That is correct.

I just read up the Store Classes section in the info manual. It seems there
is a lot of functionality in the Hurd that is well hidden in technical
documentation :) I will try to write something up for the web page as soon
as I have a better understanding of how stores work. My first question is
what the "XX:YY" notation means. I will also have to try out the various
store* utilities...

> > I'm not sure if libstore should automatically detect the type of
> > dummy.fs.gz or not.
>
> libstore does not do any "automatic detection" of this type.  If you don't
> give a store-type (i.e. -T or --store-type option), then essentially there
> is an implicit -Tfile (for the filesystems) or -Tdevice (for storeio).
>
> If you wanted automatic detection based on file contents, the thing to do
> would be a translator front-end program (or script) that examines the
> contents and then selects the right translator and options to run.

Ok. This is reasonable, and such front ends will emerge sooner or later.

For the bug report: Here is a more detailed list which things we can
compress currently and which not:

a) /boot/gnumach

I can compress this with gzip and GRUB boots it just fine. I haven't tried
bzip2, but I think GRUB doesn't support this anyway, right?

Any objections to provide the gzip'ed gnumach image in the Debian package
instead the uncompressed one? The linux kernel is compressed, too. It will
use up less disk space and uncompressing is done really fast (I didn't
notice a delay).

b) /boot/serverboot

This did work after updating my version of GRUB.

Are there disadvantages of serverboot being compressed by default? What
would be the impact on subsequent boot commands to boot subhurds? I don't
have strong feeling towards compressing it, but I ask just to get a complete
picture.

c) /hurd/ext2fs.static

Didn't work. I replaced /hurd/ext2fs.static with /hurd/ext2fs.static.gz in
servers.boot and got:

/dev/hd2s5/rverboot to load (the first word), and actions to be done: not found
bootstrap/default-pager panic: openi 5001

The strange error message in the first line indicates a major serverboot
confusion. It comes from the comment in servers.boot, which starts with:

# GNU Mach boot script for Debian GNU/Hurd.  Each line specifies a
# file for serverboot to load (the first word), and actions to be done
# with it.

Notice that it seems that serverboot has replaced the beginning of the
second line with /dev/hd2s5/ and tried to run the resulting bogus command
line. This bug can be switched on/off by only adding/removing the ".gz"
to/from /hurd/ext2fs.static invocation. Very strange indeed. (The complete
config file I used can be found in hurd/debian/servers.boot)

d) Loading a gzip'ed filesystem image with -Tgunzip:file

Worked par excellence.

Thanks,
Marcus





From: Szabó Zsolt <szzsolt@westel900.net>
To: "Roland McGrath" <roland@gnu.org>, <42079@bugs.debian.org>,
        "Gordon Matzigkeit" <gord@fig.org>
Cc: <brinkmd@debian.org>
Subject: Re: Bug #42079: hurd: exec doesn't like zip'ed binaries
Date: Thu, 29 Jul 1999 21:14:44 +0200

Dear All,

Could you please remove my address from the mailing list. Thank you in
advance.

Zsolt Szabó


----- Original Message -----
From: Roland McGrath <roland@gnu.org>
To: Gordon Matzigkeit <gord@fig.org>; <42079@bugs.debian.org>
Cc: <brinkmd@debian.org>
Sent: Wednesday, July 28, 1999 8:11 PM
Subject: Bug #42079: hurd: exec doesn't like zip'ed binaries

> > I'm not at my Hurd machine right now, but do you mean
> >
> > $ gzip hello
> > $ ./hello.gz
>
> Naturally he does.
>
> >  MB> On a related matter, zip'ed stores didn't work, too. Is there
> >  MB> more to do than creating a disk store with dd, running mke2fs on
> >  MB> it, zip'ing it and "settrans -c dummy /hurd/ext2fs
> >  MB> `pwd`/dummy.fs.gz" ?
> >
> > You may have to do:
> >
> > $ settrans -c dummy /hurd/ext2fs -Tgunzip:file `pwd`/dummy.fs.gz
>
> That is correct.
>
> > I'm not sure if libstore should automatically detect the type of
> > dummy.fs.gz or not.
>
> libstore does not do any "automatic detection" of this type.  If you don't
> give a store-type (i.e. -T or --store-type option), then essentially there
> is an implicit -Tfile (for the filesystems) or -Tdevice (for storeio).
>
> If you wanted automatic detection based on file contents, the thing to do
> would be a translator front-end program (or script) that examines the
> contents and then selects the right translator and options to run.
>
>

From: Roland McGrath <roland@gnu.org>
To: brinkmd@debian.org, -email is unavailable-
Cc: -email is unavailable-
Subject: Re: Bug #42079: hurd: exec doesn't like zip'ed binaries
Date: Thu, 29 Jul 1999 16:57:25 -0400

> I tried the exec gzip'ed and bzip2'ed binaries on the fly feature, but it
> didn't work quite right.

I will look into this when I get a chance.  But this feature is not a high
priority and we will probably drop it later on; I'm not even sure it's
worth fixing it now.

The reason this way of implementing compressed binaries is undesireable is
that it always uncompresses the entire binary into memory when it is
loaded.  This means there is no sharing of the uncompressed binary (not
even its read-only text segment) between multiple processes running the
same program.

If the motivation is boot disks, then I don't think that compressing
individual binaries is worthwhile.  You will get better compression by
compressing the entire disk image.  The whole disk image is uncompressed
into memory once at boot, and then the uncompressed contents in memory are
shared between multiple readers (just as pages mapped from disk are).
(Also, there is no further decompression overhead.)

> On a related matter, zip'ed stores didn't work, too. Is there more to do
> than creating a disk store with dd, running mke2fs on it, zip'ing it and
> "settrans -c dummy /hurd/ext2fs `pwd`/dummy.fs.gz" ?
> Accessing "dummy" resulted in some spiffy message about hypermetadata being
> wrong (0 instead 0x53e7 or something.. This is from memory, and likely
> wrong).

I think this has been addressed already, but no you cannot do it that
way--it's not automagic.  It is complaining because you asked it to read an
ext2fs out of that file and that file doesn't contain a proper ext2fs
format (it's gzip format).  To uncompress on the fly you need:

        /hurd/ext2fs -Tgunzip:file dummy.fs.gz

or

        /hurd/ext2fs -Tgunzip:device fd0

etc.  (Likewise -Tbunzip2.)


From: Roland McGrath <roland@gnu.org>
To: Marcus Brinkmann <Marcus.Brinkmann@ruhr-uni-bochum.de>
Cc: -email is unavailable-
Subject: Re: Bug #42079: hurd: exec doesn't like zip'ed binaries
Date: Thu, 29 Jul 1999 17:22:26 -0400

> I just read up the Store Classes section in the info manual. It seems there
> is a lot of functionality in the Hurd that is well hidden in technical
> documentation :) I will try to write something up for the web page as soon
> as I have a better understanding of how stores work. My first question is
> what the "XX:YY" notation means. I will also have to try out the various
> store* utilities...

Yes, storeinfo et al can be very informative.

The store-type notation "A:B" (as in `-Tgunzip:file') is for "filter" store
types, i.e. types whose underlying storage can be another store of any type.
It says the type of the store is A, and the type of the underlying store is B.
These can be further stacked, e.g.:

        -Tgunzip:remap:device 100+32,200+16:fd0

This says to open the device "fd0" (the floppy disk), take from it 32
blocks at block 100 prepended to 16 blocks at block 200, and gunzip that
48-block "virtual disk" into memory.

> I can compress this with gzip and GRUB boots it just fine. I haven't tried
> bzip2, but I think GRUB doesn't support this anyway, right?

It doesn't appear to, though I'm sure there were some patches floating
around to support it a while ago.

> Any objections to provide the gzip'ed gnumach image in the Debian package
> instead the uncompressed one? The linux kernel is compressed, too. It will
> use up less disk space and uncompressing is done really fast (I didn't
> notice a delay).

Sounds fine to me.  It is most likely in fact faster to read and uncompress
the kernel than just to read twice as many blocks from the disk.

> Are there disadvantages of serverboot being compressed by default?

I don't think so.  It gets loaded by GRUB into physical memory the same way
as the kernel.

> What would be the impact on subsequent boot commands to boot subhurds?

It is utterly unrelated.  The `boot' program does a similar job to
serverboot (and they share some source code), but they do not interact in
any way.

> c) /hurd/ext2fs.static
>
> Didn't work. I replaced /hurd/ext2fs.static with /hurd/ext2fs.static.gz in
> servers.boot and got:

Yup, looks like serverboot's gunzip code is buggy.


From: Marcus Brinkmann <Marcus.Brinkmann@ruhr-uni-bochum.de>
To: Roland McGrath <roland@gnu.org>
Cc: -email is unavailable-
Subject: Re: Bug #42079: hurd: exec doesn't like zip'ed binaries
Date: Fri, 30 Jul 1999 00:04:31 +0200

On Thu, Jul 29, 1999 at 04:57:25PM -0400, Roland McGrath wrote:

> > I tried the exec gzip'ed and bzip2'ed binaries on the fly feature, but it
> > didn't work quite right.
>
> I will look into this when I get a chance.  But this feature is not a high
> priority and we will probably drop it later on; I'm not even sure it's
> worth fixing it now.

Maybe just disable it for now...

> The reason this way of implementing compressed binaries is undesireable is
> that it always uncompresses the entire binary into memory when it is
> loaded.  This means there is no sharing of the uncompressed binary (not
> even its read-only text segment) between multiple processes running the
> same program.

Yes, you can't load pages on demand. It's better to have "real" compressed
disk stores and other stuff.

> If the motivation is boot disks, then I don't think that compressing
> individual binaries is worthwhile.  You will get better compression by
> compressing the entire disk image.

Right, I don't need this for bootdisks. I have some boot disks almost ready
(currently they are similar to Miles disks but use one floppy less).

I only need working gunzip in GRUB (which works) and in serverboot (which
doesn't). I also need a working bunzip in storeio (which should work, at
least gunzip did). The reason for bunzip is that it compresses slightly
better, ...

BTW: Any chance we can split a root fs on two disks? I can see probably
three ways to do this:

Using the concat store. Problem: We need to change the floppy disk
inbetween....

Using a two stage process: Load some data from the first disk, then boot,
and in the boot script load further data from the second disk. Problem:
shadowfs would be helpful here, or the seperation would be complicated.

Maybe there is a trick how we can extend the existing root fs mapped in
memory that I don't see. If you have any ideas, let me know. It is important
because the Hurd libraries and the C library are very big, and I am not
sure we can fit everything one disk, including the installation program and
what else we will need in the future on the installation disk.

Another solution would probably be to strip the unneeded functions from the
shared libraries. Is there a way to make lean libraries that only contain
the functions and symbols needed for a given set of binaries? Mmh.

[snip]

>       /hurd/ext2fs -Tgunzip:file dummy.fs.gz

Yup. Got it. Will also document this on the web page or so soon. This whole
store stuff is amazing and an interesting feature for users to play with.

Thanks,
Marcus

From: Erich Boleyn <erich@uruk.org>
To: Roland McGrath <roland@gnu.org>, 42079@bugs.debian.org,
    Marcus Brinkmann <Marcus.Brinkmann@ruhr-uni-bochum.de>
Cc: GNU Hurd Maintainers <bug-hurd@gnu.org>
Subject: Re: Bug #42079: hurd: exec doesn't like zip'ed binaries
Date: Thu, 29 Jul 1999 15:05:51 -0700

Roland McGrath <roland@gnu.org> wrote:

> > I can compress this with gzip and GRUB boots it just fine. I haven't tried
> > bzip2, but I think GRUB doesn't support this anyway, right?
>
> It doesn't appear to, though I'm sure there were some patches floating
> around to support it a while ago.

It doesn't.

> > Any objections to provide the gzip'ed gnumach image in the Debian package
> > instead the uncompressed one? The linux kernel is compressed, too. It will
> > use up less disk space and uncompressing is done really fast (I didn't
> > notice a delay).
>
> Sounds fine to me.  It is most likely in fact faster to read and uncompress
> the kernel than just to read twice as many blocks from the disk.

This was faster in the tests I did some time ago on a 486.  The floppy
overhead vastly dominated anything else.  The decompression code was such
a win performance-wise that it spoke for itself.

> > Are there disadvantages of serverboot being compressed by default?
>
> I don't think so.  It gets loaded by GRUB into physical memory the same way
> as the kernel.

Yes.  To the GRUB file system interface, it is exactly the same (in fact,
the consumer in general has no way to ask if a file was compressed, simply
to specify that auto-compression is on or off).

Message received at -email is unavailable- (full text, mbox):

From: Roland McGrath <roland@gnu.org>
To: Marcus Brinkmann <Marcus.Brinkmann@ruhr-uni-bochum.de>
Cc: -email is unavailable-
Subject: Re: Bug #42079: hurd: exec doesn't like zip'ed binaries
Date: Thu, 29 Jul 1999 20:29:41 -0400

> Maybe just disable it for now...

You can just delete the -DGZIP and -DBZIP2 from exec/Makefile.

> Right, I don't need this for bootdisks. I have some boot disks almost ready
> (currently they are similar to Miles disks but use one floppy less).

Great!

> BTW: Any chance we can split a root fs on two disks? I can see probably
> three ways to do this:

If we really need to do this, then the approach of using a store is what
appeals to me.  That gives the best compression benefit, since you have one
large compressed image spread across the two disks, read into memory.

> Using the concat store. Problem: We need to change the floppy disk
> inbetween....

Et voila, the "mvol" store!  (Miles thought of everything the first time. :-)

The "mvol" store type is like concat in that it produces a larger virtual
device; but rather than having multiple underlying stores that can all be
accessed at any time, it has a single underlying store and calls a callback
function to switch between volumes.

The gunzip and bunzip2 stores read their whole underlying store and
uncompress its contents into memory at the start.  So when using an mvol
store as the underlying store, it will always read each volume in sequence,
exactly once, giving the desired minimal and predictable number of requests
to the user to swap floppies.

A little more work is needed to make mvol easily usable, since there is no
default argp parser or default volume-switching function for it.  Probably
we want a special case type of store for install disks that uses mvol, but
has a foolproof volume switching function that is friendly to the user
and makes sure he inserted the right volume and so forth.

> Maybe there is a trick how we can extend the existing root fs mapped in
> memory that I don't see. If you have any ideas, let me know. It is important
> because the Hurd libraries and the C library are very big, and I am not
> sure we can fit everything one disk, including the installation program and
> what else we will need in the future on the installation disk.

Have you been using mksmallso?

> Another solution would probably be to strip the unneeded functions from the
> shared libraries. Is there a way to make lean libraries that only contain
> the functions and symbols needed for a given set of binaries? Mmh.

I'll take that as a "no, I haven't clued in to mksmallso", because that is
exactly what mksmallso is!  Please read ALL of the scripts and makefiles
in the hurd/release/ dir before you start repeating any more work.
(I told you Miles thought of everything the first time! ;)

> Yup. Got it. Will also document this on the web page or so soon. This whole
> store stuff is amazing and an interesting feature for users to play with.

Indeed.  I wrote the Linux-compatible `losetup' script with this in mind,
and I think we should also write mostly-compatible scripts for the Linux
`md' stuff and the BSD vnconfig (vnd devices) and ccdconfig (ccd devices).
These are a variety of features that people like a lot on Linux and BSD
(and other systems), that are all nicely unified in the Hurd's `store'
abstraction.  With these compatible scripts, people can use familiar ways
to set up things they understand from other systems (which in the Hurd all
down to setting a translator, usually storeio with some options).  Then
using Hurd tools like showtrans, fsysopts and storeinfo, they can see how
the various abstractions they know translate into the Hurd's store mechanism.

From: Marcus Brinkmann <Marcus.Brinkmann@ruhr-uni-bochum.de>
To: Roland McGrath <roland@gnu.org>
Cc: -email is unavailable-
Subject: Re: Bug #42079: hurd: exec doesn't like zip'ed binaries
Date: Fri, 30 Jul 1999 03:02:28 +0200

On Fri, Jul 30, 1999 at 12:04:31AM +0200, Marcus Brinkmann wrote:

>
> Another solution would probably be to strip the unneeded functions from the
> shared libraries. Is there a way to make lean libraries that only contain
> the functions and symbols needed for a given set of binaries? Mmh.

I just found hurd/release/mksmallso.sh, which should be perfect to do this.

Marcus


From: "Ashutosh S. Rajekar" <asr@giaspna.vsnl.net.in>
To: Szabó Zsolt <szzsolt@westel900.net>,
        -email is unavailable-
Subject: Re: Bug #42079: hurd: exec doesn't like zip'ed binaries
Date: Fri, 30 Jul 1999 12:20:51 -0500 (GMT)

Hello,

On Thu, 29 Jul 1999, [iso-8859-1] Szab? Zsolt wrote:

> Dear All,
>
> Could you please remove my address from the mailing list. Thank you in
> advance.
>

        Send an email to the address debian-hurd-request@lists.debian.org,
with the subject line "unsubscribe" without the quotes. You will get a
confirmation email, stating that you were unsubscribed.

        Good Luck ...

> Zsolt Szab?
>
>
> ----- Original Message -----
> From: Roland McGrath <roland@gnu.org>
> To: Gordon Matzigkeit <gord@fig.org>; <42079@bugs.debian.org>
> Cc: <brinkmd@debian.org>
> Sent: Wednesday, July 28, 1999 8:11 PM
> Subject: Bug #42079: hurd: exec doesn't like zip'ed binaries
>
>
> > > I'm not at my Hurd machine right now, but do you mean
> > >
> > > $ gzip hello
> > > $ ./hello.gz
> >
> > Naturally he does.
> >
> > >  MB> On a related matter, zip'ed stores didn't work, too. Is there
> > >  MB> more to do than creating a disk store with dd, running mke2fs on
> > >  MB> it, zip'ing it and "settrans -c dummy /hurd/ext2fs
> > >  MB> `pwd`/dummy.fs.gz" ?
> > >
> > > You may have to do:
> > >
> > > $ settrans -c dummy /hurd/ext2fs -Tgunzip:file `pwd`/dummy.fs.gz
> >
> > That is correct.
> >
> > > I'm not sure if libstore should automatically detect the type of
> > > dummy.fs.gz or not.
> >
> > libstore does not do any "automatic detection" of this type.  If you don't
> > give a store-type (i.e. -T or --store-type option), then essentially there
> > is an implicit -Tfile (for the filesystems) or -Tdevice (for storeio).
> >
> > If you wanted automatic detection based on file contents, the thing to do
> > would be a translator front-end program (or script) that examines the
> > contents and then selects the right translator and options to run.
> >
> >
>
>

--------------------
Ashutosh S. Rajekar




From: Marcus Brinkmann <Marcus.Brinkmann@ruhr-uni-bochum.de>
To: Roland McGrath <roland@gnu.org>, -email is unavailable-
Subject: Re: Bug #42079: hurd: exec doesn't like zip'ed binaries
Date: Mon, 2 Aug 1999 04:13:41 +0200

On Thu, Jul 29, 1999 at 05:22:26PM -0400, Roland McGrath wrote:

>
> > c) /hurd/ext2fs.static
> >
> > Didn't work. I replaced /hurd/ext2fs.static with /hurd/ext2fs.static.gz in
> > servers.boot and got:
>
> Yup, looks like serverboot's gunzip code is buggy.

No. It isn't even reached at this point. Actually, the gunzip code is fine.

The problem is reproducable with any filename, as long as /boot/servers.boot
is exactly 1024 byte (I didn't try multiple).

This is a very weird bug. Everthing's fine with a servers.boot of 1023 byte, even
1025 byte, but not with 1024 byte. It doesn't matter where you insert stub
bytes. The reason it failed with ".gz" is that my servers.boot file happened
to have 1021 byte before.

The error is that the "static structs cmd **cmds" in boot/boot_script.c
appears to be corrupt from anywhere but boot_script_parse_line().

I tried to debug this many hours now, and I know only this:

I inserted a 'printf("-> %s\n", cmds[0]->cmd);' before the "return 0" in
boot_script_parse_line, and for each command line in servers.boot it  gives
the correct output. But the SAME printf statement somewhere else (for
example in boot_script_exec in boot/boot_script.c) gives corrupted output.

It seems I am completely on the wrong track, and the memory corruption
happens elsewhere, for example somewhere in the filesystem code
(ext2_file_io). I don't know how to debug that, so I can't provide a
patch, sorry. At least ext2_read_file() seemed to do the correct thing
(reading 1024 bytes and not more).

This problem is not sever, as it can be worked around very easily (just
change the size of servers.boot). Still, it is very obscure and should be
fixed before people start to play around with servers.boot and don't know
what went wrong.

Thanks,
Marcus

From: Jeff Bailey <jbailey@nisa.net>
To: Erich Boleyn <erich@uruk.org>, Roland McGrath <roland@gnu.org>,
        42079@bugs.debian.org,
        Marcus Brinkmann <Marcus.Brinkmann@ruhr-uni-bochum.de>
Cc: GNU Hurd Maintainers <bug-hurd@gnu.org>
Subject: Re: Bug #42079: hurd: exec doesn't like zip'ed binaries
Date: Mon, 2 Aug 1999 18:04:11 -0700

On Thu, Jul 29, 1999 at 03:05:51PM -0700, Erich Boleyn wrote:

> Yes.  To the GRUB file system interface, it is exactly the same (in fact,
> the consumer in general has no way to ask if a file was compressed, simply
> to specify that auto-compression is on or off).

Not entirely true - GRUB could always look at the first few characters of
the file to identify it the same way file does.


From: Roland McGrath <roland@gnu.org>
To: Marcus Brinkmann <Marcus.Brinkmann@ruhr-uni-bochum.de>
Cc: -email is unavailable-
Subject: Re: Bug #42079: hurd: exec doesn't like zip'ed binaries
Date: Fri, 20 Aug 1999 01:34:29 -0400

> This is a very weird bug. Everthing's fine with a servers.boot of 1023 byte, even
> 1025 byte, but not with 1024 byte. It doesn't matter where you insert stub
> bytes. The reason it failed with ".gz" is that my servers.boot file happened
> to have 1021 byte before.

I've checked in a fix to serverboot that should fix the problem with boot
script files whose size is a multiple of 1024.



From: Marcus Brinkmann <Marcus.Brinkmann@ruhr-uni-bochum.de>
To: -email is unavailable-
Cc: -email is unavailable-
Subject: Re: Bug #42079: hurd: exec doesn't like zip'ed binaries
Date: Tue, 24 Aug 1999 16:58:07 +0200

severity 42079 wishlist
thanks

On Fri, Aug 20, 1999 at 01:34:29AM -0400, Roland McGrath wrote:

> > This is a very weird bug. Everthing's fine with a servers.boot of 1023 byte, even
> > 1025 byte, but not with 1024 byte. It doesn't matter where you insert stub
> > bytes. The reason it failed with ".gz" is that my servers.boot file happened
> > to have 1021 byte before.
>
> I've checked in a fix to serverboot that should fix the problem with boot
> script files whose size is a multiple of 1024.

Yes, this does work now.

For the record: All problems are fixed now beside the actual problem that
exec can't run compressed binaries. As this is low priority, I am
downgrading the severity to wishlist.

Thanks,
Marcus

From: Marcus Brinkmann <Marcus.Brinkmann@ruhr-uni-bochum.de>
To: Roland McGrath <roland@gnu.org>, -email is unavailable-
Cc: -email is unavailable-
Subject: Re: Bug #42079: hurd: exec doesn't like zip'ed binaries
Date: Tue, 18 Jul 2000 23:56:26 +0200

On Thu, Jul 29, 1999 at 04:57:25PM -0400, Roland McGrath wrote:

> > I tried the exec gzip'ed and bzip2'ed binaries on the fly feature, but it
> > didn't work quite right.
>
> I will look into this when I get a chance.  But this feature is not a high
> priority and we will probably drop it later on; I'm not even sure it's
> worth fixing it now.

No problem. It can be left for a newcomer as an exercise. I throwed gdb at
the failing exec process in a subhurd, and got an infinite loop somewhere,
which doesn't seem to make progress. Maybe it is even a libc problem, as it
hangs in a memcpy in fwrite().

For some very strange reason, the output in the transcript is NOT what I saw
on the screen (I can't explain this). On screen I saw some more levels in
the backtrace, below inflate_codes. They were (from my incomplete and
probably not 100% accurate logs):

memcpy()
realloc()
vscanf()
__flshp()
fflush_unlocked()
fwrite_unlocked()
zipwrite(buf=..., nwrite=32768) exec.c:1121
flush_window()                  util.c:210

After that the below trace followed.

When left running, the exec process was sucking up as much memory as it could,
and after reattaching a while later the backtrace looked exactly the same.

Something bogus going on, but maybe it is a problem that goes away with
libio?

Thanks,
Marcus

Script started on Tue Jul 18 23:20:06 2000
gdb hurd:/# gdb /d/debug/build/exec/exec 59
GNU gdb 19990928
Copyright 1998 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-unknown-gnu0.2"...

59: No such file or directory.
Attaching to program `/d/debug/build/exec/exec', pid 59

warning: Can't modify tracing state for pid 59: No signal thread
Reading symbols from /lib/libhurdbugaddr.so.0.2...done.
Reading symbols from /lib/libtrivfs.so.0.2...done.
Reading symbols from /lib/libfshelp.so.0.2...done.
Reading symbols from /lib/libiohelp.so.0.2...done.
Reading symbols from /lib/libports.so.0.2...done.
Reading symbols from /lib/libihash.so.0.2...done.
Reading symbols from /lib/libthreads.so.0.2...done.
Reading symbols from /lib/libshouldbeinlibc.so.0.2...done.
Reading symbols from /lib/libc.so.0.2...done.
Reading symbols from /lib/ld.so.1...done.
Reading symbols from /lib/libmachuser.so.1...done.
Reading symbols from /lib/libhurduser.so.0.0...done.
(gdb) bt
#0  0x8050612 in inflate_codes (tl=0x8084a68, td=0x80860b8, bl=9, bd=6)
    at ../../exec/inflate.c:543
#1  0x8051222 in inflate_dynamic () at ../../exec/inflate.c:855
#2  0x8051355 in inflate_block (e=0x1bb930) at ../../exec/inflate.c:900
#3  0x80513ba in inflate () at ../../exec/inflate.c:931
#4  0x804f3b4 in unzip (in=17, out=23) at ../../exec/unzip.c:120
#5  0x804be97 in check_gzip (earg=0x1bbbf8) at ../../exec/exec.c:1171
#6  0x804c200 in prepare_and_check (file=43, e=0x1bbbf8)
    at ../../exec/exec.c:1350
#7  0x804c446 in do_exec (file=43, oldtask=46, flags=0,
    argv=0x1bde58 "/bin/vmstat.gz", argvlen=15, argv_copy=1,
    envp=0x1bde74 "PWD=/", envplen=131, envp_copy=1, dtable=0x1bdf04,
    dtablesize=256, dtable_copy=1, portarray=0x1be310, nports=6,
    portarray_copy=1, intarray=0x1be334, nints=5, intarray_copy=1,
    deallocnames=0x1be354, ndeallocnames=263, destroynames=0x1be77c,
    ndestroynames=1) at ../../exec/exec.c:1397
#8  0x804d98a in S_exec_exec (protid=0x8084340, file=43, oldtask=46, flags=0,
    argv=0x1bde58 "/bin/vmstat.gz", argvlen=15, argv_copy=1,
    envp=0x1bde74 "PWD=/", envplen=131, envp_copy=1, dtable=0x1bdf04,
    dtablesize=256, dtable_copy=1, portarray=0x1be310, nports=6,
    portarray_copy=1, intarray=0x1be334, nints=5, intarray_copy=1,
    deallocnames=0x1be354, ndeallocnames=263, destroynames=0x1be77c,
    ndestroynames=1) at ../../exec/exec.c:2091
#9  0x804eb01 in _Xexec_exec (InHeadP=0x1bde1c, OutHeadP=0x1bbe1c)
---Type <return> to continue, or q <return> to quit---
    at execServer.c:255
#10 0x804ed4b in exec_server (InHeadP=0x1bde1c, OutHeadP=0x1bbe1c)
    at execServer.c:439
#11 0x804ace4 in exec_demuxer (inp=0x1bde1c, outp=0x1bbe1c)
    at ../../exec/main.c:61
#12 0x33e13 in ports_manage_port_operations_one_thread ()
   from /lib/libports.so.0.2
#13 0x68d3a in mach_msg_server_timeout () from /lib/libc.so.0.2
#14 0x33ef6 in ports_manage_port_operations_one_thread ()
   from /lib/libports.so.0.2
#15 0x33fe6 in ports_manage_port_operations_multithread ()
   from /lib/libports.so.0.2
#16 0x804a81f in main (argc=1, argv=0x1bffa8) at ../../exec/main.c:150
(gdb) quit
The program is running.  Quit anyway (and detach it)? (y or n) y
Detaching from program `/d/debug/build/exec/exec' pid 59
hurd:/# exit

Script done on Tue Jul 18 23:20:26 2000