Currently, qammod always produces a column vector as output, regardless of the orientation of the input vector. As in the associated Matlab function it shall produce column or row vector output depending on the orientation of the input data vector.
A solution is to apply the reshape function to the y-vector at line 91. Please note that the size of the input vector needs to be taken before line 79 as the orientation of the x-vector might change in case of gray coding.