I agree that it would be nicer to have a warning message and stop appending, rather than corrupting the existing data file.

An easy way to check whether the file is an Octave binary data file is to look for the special pattern 'Octave' at the beginning of the file.  In an m-file, something like this would work

fid = fopen (filename, "rb");
sig = fread (fid, 6, "*uchar");
is_binary = strcmp (sig.', "Octave");

I think that may need to be converted to C++.

Consider the following code:

a = 1;
save -text binfile a
save -text textfile a
save -binary binfile a
a = 2;
save -text -append binfile a
save -binary -append textfile a
clear
load textfile a
a                 % -- a =  1
load binfile a    % -- error: load: trouble reading binary file 'binfile'

On linux I could check what kind of file it is before appending, e.g.:

system('file textfile')   % -- textfile: data
system('file binfile')    % -- binfile: Octave binary data (little endian)

but obviously, this is not a portable solution.

Proposal: If save is called with the append option, it should first check that the file type specified is the same as the target, otherwise exit with an error.