I fixed the issue with negative rho values on the stable branch in this cset (http://hg.savannah.gnu.org/hgweb/octave/rev/265d3a59da27).  It will be a part of the 4.4.1 bug fix release.  Alternatively, since this is just an m-file, you can fetch it from the Mercurial repository for Octave and substitute the m-file in your installation in order to see the change.

The following code does not plot the axis, it only plot the curve.
In matlab it works just fine.

clf;
x=linspace(-2*pi,2*pi,40);
polar (x,atan(-x.^2))

It produce the following error

error: set: "dataaspectratio" must be greater than 0

When changing the function slightly, it works without any issue.

clf;
x=linspace(-2*pi,2*pi,40);
polar (x,atan(+x.^2))