GNU Astronomy Utilities - Tasks: task #15799, Add the autocomplete feature to...
You are not allowed to post comments on this tracker with your current authentication level.
task #15799: Add the autocomplete feature to gnuastro
Submitter: | Pedram Ashofteh-Ardakani <pedram> | ||
Submitted: | Sun 25 Oct 2020 04:00:53 PM UTC | ||
Should Start On: | Sun 25 Oct 2020 12:00:00 AM UTC | Should be Finished on: | Sun 25 Oct 2020 12:00:00 AM UTC |
Category: | Development | Priority: | 5 - Normal |
Item Group: | New feature | Status: | None |
Privacy: | Public | Percent Complete: | 90% |
Assigned to: | pedram | Open/Closed: | Open |
Effort: | 0.00 |
( Jump to the original submission )
Sun 30 May 2021 04:34:45 PM UTC, comment #33: |
Mohammad Akhlaghi <makhlaghi>![]() |
Fri 14 May 2021 11:51:46 AM UTC, comment #32: As a small progress report, I am happy to say that the TAB completion has matured nicely and has already been fully implemented in 8 programs so far (12 more programs and scripts to go)!
The process of adding it to every program takes about an hour or two for me (which is hard to find, given my busy schedule!), so to avoid delays in testing, I have merged this branch into the 'master' branch (the latest program was Fits in Commit b947f79228).
Within the manual, a new "Shell TAB completion" title (Section 4.1.3) has been added to Chapter 4 on "Common program behavior". Also, the tutorial and general "for developers" description of the process has been improved for the current implementation in Section 12.8.
I am now using it in my daily usage of Gnuastro (for the programs that have it), and will try to find bugs and fix them. But it would be great if you (whom ever is reading this message!) can also update your installed Gnuastro and use it (I'm sure you'll love it!) and report any bugs or inconvenience you may find!
The programs that have TAB completion are the ones with an 'astPROGNAME-complete.bash' in their source directory. |
Mohammad Akhlaghi <makhlaghi>![]() |
Mon 03 May 2021 12:14:04 PM UTC, comment #31: Indeed, modularity is the key to doing any type of project (if you later want to allow the potential for growing and scaling up).
As one further step in modularity, I have now broken the file into two: bin/gnuastro-complete.bash and bin/table/asttable-complete.bash. The first one only has generic shell functions to be used by all programs, and the second one is only for Table. In the comments of the generic file I have explained how you can easily debug in this scenario ;-).
By the way, in the previous comment I forgot to mention why I defined a new branch: because Gnuastro had grew a lot in the last two months, I just "rebased" the old branch over the most recent 'master' branch and continued developing there. Hopefully we'll merge this in shortly and no further re-basing will be necessary.
Another minor note I forgot to mention is that four new options were added to the Fits program to greatly simplify and speedup the completion (in in Commit ffc760820041). With these, instead of having to parse plain-text outputs, 'astfits' will directly give us the answers we need :-). |
Mohammad Akhlaghi <makhlaghi>![]() |
Mon 03 May 2021 06:03:18 AM UTC, comment #30: Great! You made my day Mohammad :)
I had a glimpse of the new commit and it looks easier on the eye! The formatting is pretty neat. Defining helper functions is a smart move. I am learning a lot through this process :)
Thank you for the opportunity and the encouragements. |
Pedram Ashofteh-Ardakani <pedram>![]() ![]() |
Sun 02 May 2021 11:41:06 PM UTC, comment #29: I am happy to say that I was finally able to open up some time this weekend and do a complete review of your great work Pedram!
After studying it in detail, in order to make it readable and easily scalable to all programs, I done a full re-write in the 'auto-complete' branch of CodeBerg (see the new bin/table/completion.bash).
Fortunately the rest of the job (to install it and add completions for the other programs) is very easy and since it involves Automake, I guess its easier if I do it myself (but if you are interested and have time this week, let me know by tomorrow so I don't do it).
For everyone else reading this, I should say that this is a SUPER COOL and CONVENIENT new feature! It automatically suggests FITS files, tables, HDUs, column names based on the options!
For example if you are using 'asttable' and press [TAB][TAB], it only suggests files that have a table! Same for HDUs: if you are in 'asttable' and press [TAB][TAB] on '--hdu', you only get a list of HDUs that have a table!
Gone are the days that you had to run 'astfits' or 'asttable' to remember the example HDU or column name/number!
VERY NICE WORK PEDRAM :-). |
Mohammad Akhlaghi <makhlaghi>![]() |
Sun 28 Mar 2021 07:33:20 PM UTC, comment #28: Thanks for the encouragement Mohammad :)
Yes it looks like everything works just fine. Please let me know if you have any questions or concerns. |
Pedram Ashofteh-Ardakani <pedram>![]() ![]() |
Sat 27 Mar 2021 05:21:02 PM UTC, comment #27: Great! Its a really good point to discuss formattting issues independently of the actual job! Thanks for defining task #15924 for this.
About my review, I was waiting for your green light (if everything is complete from your side). From your last message, it appeared that you are still working on the task and its not ready for a review yet. If everything is ready on your side, I'll try to look into as soon as possible ;-). |
Mohammad Akhlaghi <makhlaghi>![]() |
Sat 27 Mar 2021 05:08:05 PM UTC, comment #26: Hi again :)
I have submitted the new task #15924 under the name "Auto-completion: Output Formatting". Now, we can focus on core of the 'auto-completion feature' here, and work on formatting in its own dedicated task :)
Mohammad, what do you think about the current status of the auto- completion feature? Should we start implementing it in other sub-programs too? |
Pedram Ashofteh-Ardakani <pedram>![]() ![]() |
Thu 04 Mar 2021 09:57:49 PM UTC, comment #25: Hi Mohammad!
I have pushed three new commits on the dev-auto-complete branch. Also, after lots of investigation to no avail, I started asking if it is possible to control and manipulate the bash autocomplete output formatting.
I have opened a new task to add a unit test script that checks on our completion script.
Please let me know if things are looking good :)
Cheers
P.S. Commit Messages:
Auto-completion: Refactor ..._is_table function
With this commit, I replaced 'return 0' (or 1) with 'break' as it used to
Also, I replaced an indented 'if' with 'elif' for better code readablity.
Auto-completion: Call ..._last_table where needed
Until now, the '..._last_table' function ran everytime. This function is
Also, I noticed and removed the extra space between the command prompt
Auto-completion: Debug -i, improve performance
Until now, autocompletion checked if there is a valid FITS file or a
Also, previously the 'awk' command in the '..._last_table' function would
|
Pedram Ashofteh-Ardakani <pedram>![]() ![]() |
Tue 02 Mar 2021 09:08:12 PM UTC, comment #24: Hello again Mohammad :) I have pushed 6 new atomic commits, so you can cherry pick if need be. I will try to work on your previous suggestions soon :)
Cheers
P.S.Commit Message
Auto-completion: Refactor FITS related functions
Until now, each function that had something to do with fits file had a
From now on, we should be careful to feed only valid FITS file to fits
Also, I revised some comments on different functions and replaced some
Auto-completion: return value from last table
Until now, the _gnuastro_autocomplete_last_table function would not
Auto-completion: Rename boolean query functions
Until now, functions that validated if a file is FITS or a plaintext
Auto-completion: Human readable arguments
Until now, some functions just used the raw '$1', '$2', etc. to refer to
Also, I moved some comment blocks out of functions to respect the coding
Auto-completion: Suggest plaintext tables too
Until now, autocomplete only suggested FITS files as input. With this
Also, I have added the _gnuastro_autocomplete_list_all_valid_files
Auto-completion: Call astfits/asttable consistently
With this commit, the auto-completion script calls for astfits and
|
Pedram Ashofteh-Ardakani <pedram>![]() ![]() |
Mon 01 Mar 2021 09:55:12 PM UTC, comment #23: My pleasure Mohammad! I sure learned some useful tricks coding with you :) I pushed two commits, latest commit b001e7c272.
I smartened up the way functions look inside directories for valid FITS files. Of course, I used 'astfits' since it does its job pretty well! -- as mentioned in 'Master Foo and the shell tools' ;)
I guess there is no need for that. I just use the 'astfits' return value as a reference. Is this OK?
Nice idea! I will see what I can do ASAP :) It is fun to play around with formatting options that make the outputs easier on the eye.
Same here! Thank you for the fast response and helpful tweaks!
P.S. Commit Message:
Auto-completion: Faster fits parse, use 'local'
With this commit, I added the '-q' option to 'astfits' program so it will
Since we can use the 'local' declaration, I removed the 'unset' at the
Also, there was a misleading typo in comments about return value of the
Auto-completion: Smart list fits files, fix var
Until now, the function responsible for listing the fits files only
With this commit, I used 'astfits' program return value to check their
Also, I replaced a hard-coded value with intended $inputfile variable. |
Pedram Ashofteh-Ardakani <pedram>![]() ![]() |
Mon 01 Mar 2021 01:18:46 AM UTC, comment #22: Thanks a lot Pedram, I made some modifications in Commit 2f43d29151f (message in P.S., this was a good idea!)
In the case that '--information' is called before a file is given, we do indeed want to list all possible tables. But don't forget that tables can also be plain-text files (not just FITS). So we need a new function (maybe called '_gnuastro_autocomplete_file_is_table'). Can you implement it? All its components are already there ;-).
In short, we want to get a listing of all the files in this directory and pass them through this function so we only suggest files that are tables. The '_gnuastro_autocomplete_plaintext_is_table' function will already do the check for non-FITS files. For FITS files, you can use `astfits FILE.FITS -q` and look into the third column, if it has a 'table' in any of the rows/HDUs, you can show that file, otherwise, don't bother showing it!
This will be SO USEFUL! When we have a directory with image and table files, when we call 'asttable' and press TAB, we only see the tables :-D!
Another suggestion: after playing a little with it, I feel that mixing options and file names is very annoying! Now that we can print our own terminal prompt, can you implement a feature like this: when listing options, files and directories, we separate them like this:
We can use the 'column' command to sort the outputs into columns internally. For the titles, we can also give different shell colors and make them bold. This will allow us to remove the empty lines. Ofcourse, when no directories, files or options are present, we can remove the title also.
It is becoming EXTREMELY USEFUL and I am really excited to be so close to implementing this in all the programs!
P.S.Commit message:
Auto-completion: Minor polishing of the warning message
Until now, the warning message for the '--information' option was a single
With this commit a new 'infowarning' variable is defined at the top of the
Other changes with this commit:
- When the 'last_table' variable is not empty, the existance of its file
- In my case the previous '$' wasn't enough to print the prompt. The new
- I removed the extra '-----' lines: they are distracting and waste a lot
[To help in future reference to commits related to auto-completion, let's
|
Mohammad Akhlaghi <makhlaghi>![]() |
Sun 28 Feb 2021 09:38:11 PM UTC, comment #21: Hi Mohammad! OK, I pushed the commit 968bb7c07b and changed the syntax a bit. Now it should be more clear :)
It should have printed the prompt in a new line :/ I used the command below to print the prompt:
A sample output:
Do you want me to print the part before '$' too? It is stored in '$PS1', I can look it up and see if we can prepend it to the $COMP_LINE.
Would you mind just checking it again and letting me know if it works?
Cheers
=== Commit Message: ===
/bin/table/completion.bash: Refactor code
Until now, I used a case inside an 'if' statement. The syntax looked
Also, I declared a new _gnuastro_autocomplete_print_message function.
Last but not least, I used 'grep' to scan the prompt. This is POSIX
|
Pedram Ashofteh-Ardakani <pedram>![]() ![]() |
Sun 28 Feb 2021 07:34:23 PM UTC, comment #20: Thanks Pedram, I just pulled the changes and tried it. The problem is that in this mode, the user's prompt is no longer printed and the command stars at the beginning of the line! This is not what people expect.
Can you look into this and find a way to print the warning and then print the full prompt with the existing command?
Another point: Using the 'if' statement before the 'case' statement, only to check for something that is also checked later, makes the code very hard to read and un-elegant generally ;-).
We already have a case statement for '-i' or '--information'. So we should keep all checks related to this option under that case check (the same way it originally was). |
Mohammad Akhlaghi <makhlaghi>![]() |
Sun 28 Feb 2021 05:39:49 AM UTC, comment #19: Oh I got caught up in replying comment #17 and forgot about comment #16. Sure! I will do it in a separate task soon :-) |
Pedram Ashofteh-Ardakani <pedram>![]() ![]() |
Sun 28 Feb 2021 05:35:59 AM UTC, comment #18: Hi Mohammad, thank you for the swift response and code check!
I like the way you used the 'astfits' program instead of 'reinventing the wheel' ;-) Great!
There you go, problem solved in Commit 32084abdd7 ;-)
Just a tip, printing messages using 'printf' instead of 'COMPREPLY' might be a better approach. It can prevent confusions in the future. It might be better to distinguish communicating with the user (using 'echo', 'printf', 'read', etc.) and providing autocomplete suggestions (using 'COMPREPLY').
More explanation in the commit message. Please let me know if you have any questions :-) |
Pedram Ashofteh-Ardakani <pedram>![]() ![]() |
Sun 28 Feb 2021 03:29:40 AM UTC, comment #17: Hi again Pedram, I made some good progress with the script, but I a problem that didn't let me progress fast and I didn't have time to figure out its solution :-(!
The problem (and all the changes I made) is described in Commit e18d46e476.
When you get a chance, can you look into it? I think you'll be able to find the solution to it pretty fast! This issue (of printing a warning when there is an inconsistency) would be a very good feature to have. |
Mohammad Akhlaghi <makhlaghi>![]() |
Sat 27 Feb 2021 04:08:59 PM UTC, comment #16: I am happy you enjoyed the comments, a good work will have good feedback ;-).
I like your idea on adding some of these points within the "Developing" chapter. Since you have read it, can you find a good place to put them? I am too busy to focus on this right now :-(.
Since these tips are general, can you do this over a new branch on the most recent commit of the 'master' branch?
Since the GSoC application period is approaching, these tips can be very useful for new people who want to apply this year (and many people will be grateful to you ;-)). |
Mohammad Akhlaghi <makhlaghi>![]() |
Sat 27 Feb 2021 06:59:45 AM UTC, comment #15: You made my day Mohammad! Thank you for the wise and informative feedback :) Mentioning the references points enthusiastic people - your truly ;-) - to the right direction. I am having a good time reading them. Thank you so much.
In fact, the Commit 69208a5aeb9ca64a taught me a good deal about both GNU conventions and Emacs shortcuts for better TexInfo content management. Maybe they can help other developers too! How about adding them to the GNU manual? Since we should respect the conventions and make sure every piece of code follows them anyway.
OK, I will continue the work on https://codeberg.org then :) Please let me know if I could help.
Cheers ^_^
|
Pedram Ashofteh-Ardakani <pedram>![]() ![]() |
Sat 27 Feb 2021 03:13:25 AM UTC, comment #14: I just went through the newly added section in the book and REALLY ENJOYED IT! Wonderful work Pedram. I really recommend the tutorial that Pedram has written to anyone using the shell and would like to customize their shell completions (even outside of Gnuastro!).
I just made Commit 69208a5aeb9ca64a over your branch in Codeberg with some minor edits in the book (see the commit message for full explanation on every change). In fact, I also added you as a member of this Gnuastro development fork on Codeberg, so its best that you commit to this repository/branch directly yourself from now on too ;-).
About this branch, you are the manager of this branch on Codeberg from now on. So feel free to commit to this branch on Codeberg, I'll send merge requests for you if parallel work comes up.
I will try to go over the actual implementation of the autocomplete features in the next few days and will post the results here. |
Mohammad Akhlaghi <makhlaghi>![]() |
Sat 20 Feb 2021 08:25:31 PM UTC, comment #13: Hi Mohammad!
Thanks for letting me know :) I got this far in the autocompletion code and the documentation.
I am looking forward to hearing your thoughts on this. Can't wait to have the autocompletion feature implemented in the entire Gnuastro!
It would be great if you could let me know the autocompletion is working as expected in a Mac device too. Because I am a bit skeptical about the 'compopt' command. I will just use another approach if 'compopt' does not work in Mac.
Please let me know if you have any questions or concerns. |
Pedram Ashofteh-Ardakani <pedram>![]() ![]() |
Mon 08 Feb 2021 12:41:50 AM UTC, comment #12: Great work Pedram.
No problem, the completion we are designing is only for Bash anyway. Things like Dash don't have completion (as far as I know), and really have no idea how other shells (like zsh) treat completion. So you just have to assume that the feature exists in a certain version of Bash.
Its fine to assume Bash 3.2! We can just add some checks at configure time and if certain features aren't available in the user's bash (or the user's shell isn't Bash at all, like on Debian systems that use Dash: a minimal shell), we won't activate autocomplete, and just print a warning. Don't worry about the test in './configure', I'll add it ;-)..
Of course, the same applies to 'compgen'. It is available in Bash, and that is the only important thing.
So you don't have to worry about POSIX here ;-).
The links you send were very good! I am sure I'm going to come back to this task a lot for those links ;-).
I'm a little busy in the next day or two, but I'll try my best to test the completed work! Very good job! In the meantime, you can define a new section in the Developing chapter of the book to explain the basic concepts. I'll start reading from that (as a new developer), then I'll look into the code and commits ;-). |
Mohammad Akhlaghi <makhlaghi>![]() |
Sun 07 Feb 2021 07:26:35 PM UTC, comment #11: Great news! check commit f8425b57..75b02a5d. I can not access gitlab right now, so please help yourself with the last commit on 'dev-auto-complete'. Try it out and let me know Mohammad :) Just download the 'completion.sh' file, and call 'source completion.sh'. Let me know if you have any tweaks in mind :) |
Pedram Ashofteh-Ardakani <pedram>![]() ![]() |
Sun 07 Feb 2021 09:09:06 AM UTC, comment #10: I read a few articles about portability to Mac and previous versions. Keeping them here might be helpful.
This can be really confusing ... Should we keep things compatible with bash-3.2 [https://tiswww.case.edu/php/chet/bash/COMPAT (which is
|
Pedram Ashofteh-Ardakani <pedram>![]() ![]() |
Sun 07 Feb 2021 08:32:28 AM UTC, comment #9: Hi Mohammad. Speaking of portability, I stumbled upon the program: shellcheck. This program shows warnings in case of any error or POSIX incompatibility. It looks like arrays are not supported in POSIX. Is it OK if I use them in autocompletion or should I find a replacement? E.g. using awk, known shell workarounds, etc.
Also, there is a warning on compgen, a bash built-in function for autocompletion. I guess I can replace this too. I am just wondering how much should I worry about POSIX compatibility and portability.
Your help always keeps me motivated and on track. Thanks in advance.
=== Helpful links ===
|
Pedram Ashofteh-Ardakani <pedram>![]() ![]() |
Fri 05 Feb 2021 02:39:12 PM UTC, comment #8: As Mohammad pointed out, it is better to submit a new task for using the autocompletion in emacs. Now things are simpler, and we can focus on tasks one at a time.
I agree with you on portability Mohammad (comment #6)!
Thanks for keeping things simple, yet effective ^_^
I will keep you posted. Cheers. |
Pedram Ashofteh-Ardakani <pedram>![]() ![]() |
Thu 04 Feb 2021 03:39:26 PM UTC, comment #7: As Samaeh pointed out, some Gnuastro users write their scripts in emacs. So it would be nice if the completions could work there too. Fortunately, that is possible already: https://github.com/szermatt/emacs-bash-completion |
Pedram Ashofteh-Ardakani <pedram>![]() ![]() |
Thu 04 Feb 2021 12:56:40 PM UTC, comment #6: This is very good progress, thanks Pedram ;-)!
Also, the link was interesting, thanks! I'd love to read it in detail but I am too busy these days :-(! I trust that you have read them and will follow the best practices ;-). But IT IS GREAT to have the links here for future reference, so please continue sharing relevant best practices here.
Another issue just occurred to me: the version of the host's Bash. For example I know some recent macOS systems that come with the Bash of 2007!!!! So to help in portability of the solution, please also check to see from when Bash completion was available. And within the functions/scripts, we should also avoid using too many recent shell constructs to help in the portability of this work.
Let me know when the full option set of the Table program is complete and I'll try it out and hopefully expand it to the other programs ;-). |
Mohammad Akhlaghi <makhlaghi>![]() |
Thu 04 Feb 2021 12:45:54 PM UTC, comment #5: I had some progress with the completion script in this commit. As we are thinking of more complicated output and helpers, this article might be useful: https://brbsix.github.io/2015/11/29/accessing-tab-completion-programmatically-in-bash/ |
Pedram Ashofteh-Ardakani <pedram>![]() ![]() |
Wed 03 Feb 2021 02:06:59 AM UTC, comment #4: I just came across this project: https://github.com/scop/bash-completion
It has autocomplete features for many common command-line programs. For example here is the Bash completion they have designed for Make: https://github.com/scop/bash-completion/blob/master/completions/make
Of course, they clearly aren't fans of writing comments (which is bad!). But there must be some good lessons to be learnt from their various solutions, or good conventions they may have adopted that may be useful in our scenario also. |
Mohammad Akhlaghi <makhlaghi>![]() |
Thu 28 Jan 2021 03:55:00 PM UTC, comment #3: Hi Mohammad, Thanks for the commit c826434952!
I have made some progress in Commit 2b52b3d3 setting up a bash completion script. For the time being, this works specifically for the CosmicCalculator program.
We can build up on these atomic commits step by step with the aim of creating an `auto-generated` bash completion script for the entire Gnuastro!
As you pointed out, `we can extract the list of HDUs or column names to show to the user` for the next step. |
Pedram Ashofteh-Ardakani <pedram>![]() ![]() |
Wed 27 Jan 2021 10:09:46 PM UTC, comment #2: As one step towards this, in Commit c826434952 the options that accept FITS (or FITS or TXT) are now marked as 'FITS' or 'FITS/TXT'.
This can help the auto-complete script to see what types of files it should show when a user presses TAB.
For the other string arguments (like HDU names in FITS files or column names in tables), the auto-complete script can call 'astfits' or 'asttable' and extract the list of HDUs or column names to show to the user. |
Mohammad Akhlaghi <makhlaghi>![]() |
Sun 25 Oct 2020 09:39:05 PM UTC, comment #1: This is indeed a very useful feature! Thanks a lot Pedram for suggesting it.
If you can get familiar with the completion concept and commands, you can write a "completion script" for one of the Gnuastro programs. The process for the other programs will be very similar.
We can then easily write a script to be included in Gnuastro's building steps (in particular in 'doc/Makefile.am') to automatically generate this script for all the programs when the developer is building Gnuastro. We can use the output of the '--help' option to do this, or at a lower-level, go and parse all the 'args.h' files of the programs.
In fact, 'doc/Makefile.am' already uses the output of '--help' to automatically construct the Man pages. Like the Man pages, the Bash completion scripts can also be installed in 'doc/Makefile.am'. |
Mohammad Akhlaghi <makhlaghi>![]() |
Sun 25 Oct 2020 04:00:53 PM UTC, original submission:
Using autocomplete (using the <TAB> key) not only helps write commands faster, but also reduces the chance of making typing mistakes, and helps users explore all the options available.
I looked up the way one can achieve this feat, and found links and tutorials similar to Bash Programmable Completion Tutorial.
It might be a good idea to visit the GNU Programmable Completion Builtins too.
|
Pedram Ashofteh-Ardakani <pedram>![]() ![]() |
No files currently attached
Depends on the following items: None found
Items that depend on this one: None found
There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.
Follow 5 latest changes.
Date | Changed by | Updated Field | Previous Value | => | Replaced by |
---|---|---|---|---|---|
2021-05-02 | makhlaghi | Percent Complete | 30% | ![]() |
90% |
2021-02-28 | makhlaghi | Assigned to | None | ![]() |
pedram |
2021-02-27 | makhlaghi | Percent Complete | 0% | ![]() |
30% |
2020-11-27 | pedram | Carbon-Copy | Removed makhlaghi | ![]() |
- |
2020-10-25 | pedram | Carbon-Copy | - | ![]() |
Added makhlaghi |
Due to bug #60618, in Commit 4c01082f8, I added notices above the relevant parts of the documentation. In the notices, we highlight that TAB completion is still under development and not yet complete. I also cite this task for interested people to see the progress on this.
Just for the record, as mentioned in the commit message, TAB completion is currently only available for these programs:
We'll continue with the rest once we find a solution to bug #60618.