reported by zhoulai.fu =at= gmail =dot= com

Running doc/examples/filt_edge.c with the Undefined Sanitizer enabled
reveals many unaligned memory access errors in deque.c. (See
https://www.kernel.org/doc/Documentation/unaligned-memory-access.txt
for unaligned memory access.)

deque.c:58:11: runtime error: member access within misaligned address
0x0000024010f4 for type 'struct deque', which requires 8 byte alignment
0x0000024010f4: note: pointer points here
  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00
 00 00 00 00 00 00 00 00
              ^
deque.c:59:11: runtime error: member access within misaligned address
0x0000024010f4 for type 'struct deque', which requires 8 byte alignment
0x0000024010f4: note: pointer points here
  00 00 00 00 ff ff ff ff  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00
 00 00 00 00 00 00 00 00
              ^
deque.c:60:11: runtime error: member access within misaligned address
0x0000024010f4 for type 'struct deque', which requires 8 byte alignment
0x0000024010f4: note: pointer points here
  00 00 00 00 ff ff ff ff  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00
 00 00 00 00 00 00 00 00
              ^
deque.c:61:12: runtime error: member access within misaligned address
0x0000024010f4 for type 'struct deque', which requires 8 byte alignment
0x0000024010f4: note: pointer points here
  00 00 00 00 ff ff ff ff  00 00 00 00 05 00 00 00  00 00 00 00 00 00 00 00
 00 00 00 00 00 00 00 00
              ^
...

To reproduce the error, compile GSL and filt_edge.c with
"-fsanitize=undefined" flag and run the binary.

See the attached error.txt for the full error messages.

Preliminary diagnosis:

Gdb shows the following backtrace at the place where the error occurs:

(gdb) bt
#0  deque_init (n=5, d=0x6d20f4) at deque.c:59
#1  0x000000000040cd71 in mmacc_init (n=4, vstate=0x6d2060) at mmacc.c:88
#2  0x0000000000401d59 in rmedian_init (n=4, vstate=0x6d2050) at
rmedian.c:175
#3  0x0000000000404aeb in gsl_movstat_apply_accum
(endtype=GSL_MOVSTAT_END_PADVALUE, x=0x7fffffffd960, accum=0x66d9c0
<rmedian_accum_type>,
    accum_params=0x7fffffffd950, y=0x7fffffffd990, z=0x0, w=0x6d2010) at
apply.c:74
#4  0x0000000000401bba in gsl_filter_rmedian
(endtype=GSL_FILTER_END_PADVALUE, x=0x6d4100, y=0x6d8040, w=0x6d1eb0) at
rmedian.c:147
#5  0x000000000040108d in main () at filt_edge.c:38


That is, the error occurs at in the function

 static int
 deque_init(const size_t n, deque * d)

of deque.c. In the function, the compiler expects the  pointer of type
deque, d, to be 8-aligned. Yet d got an address 0x6d20f4 that is not
8-aligned.

The cause of the error should closely relate to mmann.c:84

  state->maxque = (deque ) ((unsigned char ) state->minque + deque_size(n
+ 1));

In this operation of pointer casting, if we assume that the first term
above, state->minque, is a multiple of 8, then, the second term above,
deque_size(n + 1), should be a multiple of 8 as well, in order to meet the
8-alignment requirement. Now, take a look at the function deque_size in
deque.c:

 static size_t
 deque_size(const size_t n)
 {
   size_t size = 0;

   size += sizeof(deque);
   size += n sizeof(deque_type_t); / array */

   return size;
 }

The function returns 24 + n * 4, since sizeof(deque) is determined to be
24 at compile time, and sizeof(deque_type_t) is 4 as deque_type_t is a
typedef of int. This number, 24 + n * 4, is a multiple of 8 if and only if
n is even. In other words, the misalignment should occur whenever n
is odd.

Note that the kind of
misaligned memory access can cause undefined behavior, triggering
performance
penalty, crash or incorrect results. See
https://developer.ibm.com/technologies/systems/articles/pa-dalign/.