Ok, so the current file association seems like a quick and dirty hack, but there should be a more robust solution that works cross-platform. And presumably addresses this request, that the file will be opened in the editor in a running instance of Octave.

Association: This is done by the Windows installer. Option "Register .m file type with Octave". See screenshot attached.

Opening: Well, at least using Windows, clicking a .m file will open it in the Octave editor (not: run. This would take some more care because .m files are ambiguous between script / function). This is also what Matlab does. In addition, clicking a *.mat file will load data into the Matlab workspace.

FYI, here is an explanation of the file name extensions and what they are suppoosed to do:
http://www.mathworks.de/de/help/matlab/matlab_env/associate-files-with-matlab-on-windows-platforms.html

Reuse an existing instance or start a new one: Matlab would reuse the existing instance. It is commonly reported as an error if a new instance is opening, here an example for Windows 7:
http://www.mathworks.com/matlabcentral/answers/93468-why-is-a-new-instance-of-matlab-opened-when-i-double-click-on-a-matlab-file-in-windows-explorer-even

Two instances: I did not test it lately, but imho the most recently used instance will catch the event.

In the "File Browser" pane, when clicking on a *.m file, it will open in the editor. => OK

When double-clicking a *.m file in Windows Explorer (TM) / File Explorer (TM), a new instance of Octave GUI comprising another editor window will be created. The file will be opened in this second instance editor. => I would prefer to work with just one instance of Octave (unless lauchend explicitly).