As a small progress report, I am happy to say that the TAB completion has matured nicely and has already been fully implemented in 8 programs so far (12 more programs and scripts to go)!

The process of adding it to every program takes about an hour or two for me (which is hard to find, given my busy schedule!), so to avoid delays in testing, I have merged this branch into the 'master' branch (the latest program was Fits in Commit b947f79228).

Within the manual, a new "Shell TAB completion" title (Section 4.1.3) has been added to Chapter 4 on "Common program behavior". Also, the tutorial and general "for developers" description of the process has been improved for the current implementation in Section 12.8.

I am now using it in my daily usage of Gnuastro (for the programs that have it), and will try to find bugs and fix them. But it would be great if you (whom ever is reading this message!) can also update your installed Gnuastro and use it (I'm sure you'll love it!) and report any bugs or inconvenience you may find!

The programs that have TAB completion are the ones with an 'astPROGNAME-complete.bash' in their source directory.

Thanks for the encouragement Mohammad :)

Yes it looks like everything works just fine. Please let me know if you have any questions or concerns.

Hi again :)

I have submitted the new task #15924 under the name "Auto-completion: Output Formatting". Now, we can focus on core of the 'auto-completion feature' here, and work on formatting in its own dedicated task :)

Mohammad, what do you think about the current status of the auto- completion feature? Should we start implementing it in other sub-programs too?

Hello again Mohammad :) I have pushed 6 new atomic commits, so you can cherry pick if need be. I will try to work on your previous suggestions soon :)

Cheers

---

P.S.Commit Message

Auto-completion: Refactor FITS related functions

Until now, each function that had something to do with fits file had a
different approach of making sure the input file is actually a fits file.
However, Mohammad wrote a special function that checks if an input file
is an actual existing FITS file, regardless of its extension. With this
commit, I moved those functions below the '..._is_fits' function.

From now on, we should be careful to feed only valid FITS file to fits
accepting functions, or there will be confusing errors.

Also, I revised some comments on different functions and replaced some
variables with their human-readable substitutes.

---

Auto-completion: return value from last table

Until now, the _gnuastro_autocomplete_last_table function would not
return anything. It would just set a global variable. However, it might
be a better idea to use return values instead of breaking the loops.
Because later we can use the return value as a signal if need be.

---

Auto-completion: Rename boolean query functions

Until now, functions that validated if a file is FITS or a plaintext
table were named like: _gnuastro_autocomplete_x_is_extension
However, it might be a better naming style as if we were asking the
functions whether the input file is the extension we want or not as:
_gnuastro_autocomplete_is_x_extension (?) and we'd expect a 0 or 1 as
return. With this commit, I have renamed two functions accordingly.

---

Auto-completion: Human readable arguments

Until now, some functions just used the raw '$1', '$2', etc. to refer to
given arguments. With this commit, some of these arguments are given more
specific names to improve and maintain code readability.

Also, I moved some comment blocks out of functions to respect the coding
standards as Mohammad mentioned previously.

---

Auto-completion: Suggest plaintext tables too

Until now, autocomplete only suggested FITS files as input. With this
commit, plaintext files will be shown too.

Also, I have added the _gnuastro_autocomplete_list_all_valid_files
function to prevent redundant iteration over a list of files to check
for all acceptable file types.

---

Auto-completion: Call astfits/asttable consistently

With this commit, the auto-completion script calls for astfits and
asttable only using their stored name in the script beginning. They are
'$_gnuastro_astfits' and '$_gnuastro_asttable'.

Thanks a lot Pedram, I made some modifications in Commit 2f43d29151f (message in P.S., this was a good idea!)

In the case that '--information' is called before a file is given, we do indeed want to list all possible tables. But don't forget that tables can also be plain-text files (not just FITS). So we need a new function (maybe called '_gnuastro_autocomplete_file_is_table'). Can you implement it? All its components are already there ;-).

In short, we want to get a listing of all the files in this directory and pass them through this function so we only suggest files that are tables. The '_gnuastro_autocomplete_plaintext_is_table' function will already do the check for non-FITS files. For FITS files, you can use `astfits FILE.FITS -q` and look into the third column, if it has a 'table' in any of the rows/HDUs, you can show that file, otherwise, don't bother showing it!

This will be SO USEFUL! When we have a directory with image and table files, when we call 'asttable' and press TAB, we only see the tables :-D!

Another suggestion: after playing a little with it, I feel that mixing options and file names is very annoying! Now that we can print our own terminal prompt, can you implement a feature like this: when listing options, files and directories, we separate them like this:

astfits <TAB><TAB>

DIRECTORIES
a/    b/    c/

FILES
a.fits  b.fits c.fits

OPTIONS
--column --information --quiet ...

We can use the 'column' command to sort the outputs into columns internally. For the titles, we can also give different shell colors and make them bold. This will allow us to remove the empty lines. Ofcourse, when no directories, files or options are present, we can remove the title also.

It is becoming EXTREMELY USEFUL and I am really excited to be so close to implementing this in all the programs!


P.S.Commit message:

Auto-completion: Minor polishing of the warning message

Until now, the warning message for the '--information' option was a single
string that was repeated in two places (adding a check for it in general
was a good idea Pedram!). However, based on previous experience, this can
cause bugs if we later edit one of them and forget to the edit the other!

With this commit a new 'infowarning' variable is defined at the top of the
function and message is put there. That variable can now be used anywhere
in the code, and all the time it will have the same value.

Other changes with this commit:

 - When the 'last_table' variable is not empty, the existance of its file
   has already been checked. So we should just check if its an empty string
   or not. Don't forget that checking the existance of a file is
   computationally expensive (requires Bash to leave its internal
   evironment and go to the operating system!).

 - In my case the previous '$' wasn't enough to print the prompt. The new
   line simply started with a '$' sign, not my standard prompt. So I added
   '${PS1@P}'. As mentioned above it, there is a portability issue here
   with older Bash systems. We should find a more portable solution.

 - I removed the extra '-----' lines: they are distracting and waste a lot
   of valuable terminal lines ;-)!

[To help in future reference to commits related to auto-completion, let's
start the commit title with 'Auto-completion'.]

Thanks Pedram, I just pulled the changes and tried it. The problem is that in this mode, the user's prompt is no longer printed and the command stars at the beginning of the line! This is not what people expect.

Can you look into this and find a way to print the warning and then print the full prompt with the existing command?

Another point: Using the 'if' statement before the 'case' statement, only to check for something that is also checked later, makes the code very hard to read and un-elegant generally ;-).

We already have a case statement for '-i' or '--information'. So we should keep all checks related to this option under that case check (the same way it originally was).

Hi Mohammad, thank you for the swift response and code check!

I like the way you used the 'astfits' program instead of 'reinventing the wheel' ;-) Great!

There you go, problem solved in Commit 32084abdd7 ;-)

Just a tip, printing messages using 'printf' instead of 'COMPREPLY' might be a better approach. It can prevent confusions in the future. It might be better to distinguish communicating with the user (using 'echo', 'printf', 'read', etc.) and providing autocomplete suggestions (using 'COMPREPLY').

More explanation in the commit message. Please let me know if you have any questions :-)

I am happy you enjoyed the comments, a good work will have good feedback ;-).

I like your idea on adding some of these points within the "Developing" chapter. Since you have read it, can you find a good place to put them? I am too busy to focus on this right now :-(.

Since these tips are general, can you do this over a new branch on the most recent commit of the 'master' branch?

Since the GSoC application period is approaching, these tips can be very useful for new people who want to apply this year (and many people will be grateful to you ;-)).

I just went through the newly added section in the book and REALLY ENJOYED IT! Wonderful work Pedram. I really recommend the tutorial that Pedram has written to anyone using the shell and would like to customize their shell completions (even outside of Gnuastro!).

I just made Commit 69208a5aeb9ca64a over your branch in Codeberg with some minor edits in the book (see the commit message for full explanation on every change). In fact, I also added you as a member of this Gnuastro development fork on Codeberg, so its best that you commit to this repository/branch directly yourself from now on too ;-).

About this branch, you are the manager of this branch on Codeberg from now on. So feel free to commit to this branch on Codeberg, I'll send merge requests for you if parallel work comes up.

I will try to go over the actual implementation of the autocomplete features in the next few days and will post the results here.

Great work Pedram.

> It looks like arrays are not supported in POSIX.

No problem, the completion we are designing is only for Bash anyway. Things like Dash don't have completion (as far as I know), and really have no idea how other shells (like zsh) treat completion. So you just have to assume that the feature exists in a certain version of Bash.

Its fine to assume Bash 3.2! We can just add some checks at configure time and if certain features aren't available in the user's bash (or the user's shell isn't Bash at all, like on Debian systems that use Dash: a minimal shell), we won't activate autocomplete, and just print a warning. Don't worry about the test in './configure', I'll add it ;-)..

Of course, the same applies to 'compgen'. It is available in Bash, and that is the only important thing.

So you don't have to worry about POSIX here ;-).

The links you send were very good! I am sure I'm going to come back to this task a lot for those links ;-).

I'm a little busy in the next day or two, but I'll try my best to test the completed work! Very good job! In the meantime, you can define a new section in the Developing chapter of the book to explain the basic concepts. I'll start reading from that (as a new developer), then I'll look into the code and commits ;-).

I read a few articles about portability to Mac and previous versions. Keeping them here might be helpful.

This can be really confusing ... Should we keep things compatible with bash-3.2 [https://tiswww.case.edu/php/chet/bash/COMPAT (which is
still the `standard' version for Mac OS X)] or even older ones?

• A quick check on bash changes.
• A more extensive one.
• Bash compatibility with previous versions.
• Bash in GNU/Linux vs. MacOS.
• Major differences from the Bourne Shell.

As Mohammad pointed out, it is better to submit a new task for using the autocompletion in emacs. Now things are simpler, and we can focus on tasks one at a time.

I agree with you on portability Mohammad (comment #6)!

Thanks for keeping things simple, yet effective ^_^

I will keep you posted. Cheers.

This is very good progress, thanks Pedram ;-)!

Also, the link was interesting, thanks! I'd love to read it in detail but I am too busy these days :-(! I trust that you have read them and will follow the best practices ;-). But IT IS GREAT to have the links here for future reference, so please continue sharing relevant best practices here.

Another issue just occurred to me: the version of the host's Bash. For example I know some recent macOS systems that come with the Bash of 2007!!!! So to help in portability of the solution, please also check to see from when Bash completion was available. And within the functions/scripts, we should also avoid using too many recent shell constructs to help in the portability of this work.

Let me know when the full option set of the Table program is complete and I'll try it out and hopefully expand it to the other programs ;-).

I just came across this project: https://github.com/scop/bash-completion

It has autocomplete features for many common command-line programs. For example here is the Bash completion they have designed for Make: https://github.com/scop/bash-completion/blob/master/completions/make

Of course, they clearly aren't fans of writing comments (which is bad!). But there must be some good lessons to be learnt from their various solutions, or good conventions they may have adopted that may be useful in our scenario also.

As one step towards this, in Commit c826434952 the options that accept FITS (or FITS or TXT) are now marked as 'FITS' or 'FITS/TXT'.

This can help the auto-complete script to see what types of files it should show when a user presses TAB.

For the other string arguments (like HDU names in FITS files or column names in tables), the auto-complete script can call 'astfits' or 'asttable' and extract the list of HDUs or column names to show to the user.

Using autocomplete (using the <TAB> key) not only helps write commands faster, but also reduces the chance of making typing mistakes, and helps users explore all the options available.

I looked up the way one can achieve this feat, and found links and tutorials similar to Bash Programmable Completion Tutorial.

It might be a good idea to visit the GNU Programmable Completion Builtins too.