Developers and LLMs need documentation
Giving documentation away so LLMs can use it for free sounds counter intuitive, but I hope my experience demonstrates that it's the most productive thing to propel any project forward. This article is how ChatGPT enabled me to improve my shell prompts and why documentation is key.
Setup, objective and prompt
My day to day box is FreeBSD with a C shell, it's tcsh version 6.22.04 (Astron) 2021-04-26 (x86_64-amd-FreeBSD) options wide,nls,dl,al,kan,sm,rh,color,filec.
The C shell has served me well for many years, yet since I wanted more from my prompt and consistency across shells I turned to ChatGPT for help, and help it did. As a starting point I asked it to convert a minimal Bash prompt to C shell and it offered a good answer to my question and then offered more, a mini tutorial with added value to make my prompt even more useful.
The first thing to say about shells is they are ancient, really ancient, and the syntax they use has grown from and still represents that era. They have many quirks and some aspects are painful.
To get started my question was:
convert bash prompt "PS1='(\[$(tput md)\]\t <\w>\[$(tput me)\]) $(echo $?) \$ '" to tcsh
ChatGPT output
It understood, explained what each tput and the other arguments were for, translated the bash prompt into tcsh and also explained in detail. This was useful, as were the follow on notes. It asked if I was interested in adding colour to which I said yes.
Although it worked after some tweaking to get quoting right, C shell precmd is unforgiving, though thankfully tcsh -X can be used to debug the script.
It's important to say that this tcsh prompt was successful.
ChatGPT was helpful and offered more:
Would you like me to add different colors for the directory or time as well (for example, cyan path, yellow time)? It’s easy to extend this setup.
me: Yes please
Limits to the tech
Alas I'm a heavy user of history and as there is no way to stop precmd barfing the history file, this was unacceptable. Nice try but the C shell is old and unlikely to be updated so update to another shell or accept the status quo. A useful exercise taking a few hours.
As a multiplatform developer it's necessary to develop for the minimum use case, and that's old Bourne shells. As a heavy user of history I find the C shell easier to work with than Bash and I mainly develop for a Bourne shell. All shells are frustrating, most are useful.
Why have that information in the prompt
As a developer I like to know where I am and two directories with %c2 are useful to differentiate between for example staging and production, I also like to know the return code. I could easily write a script to show this information but it's useful as part of a normal shell prompt.
Why documentation is essential
All the shells are well documented, and there are plenty of blogs and tutorials customising their many aspects, this is where ChatGPT and the LLMs get their information. It's useful.
As the conversation progressed, and I had to fix various aspects to get them to work, I found myself increasingly turning to manual pages: man tcsh and man sh. There are differences in having single line if/endif and the ability to test those in a script file with sh -x is enlightening.
If there's one takeaway from this it's that documentation is key.
Good documentation enabled ChatGPT to tailor what I wanted fairly easily, and to get it working close enough for what I wanted it to do, I needed to refer to the documentation and as it was clear and well structured I could easily identify if something was or wasn't available and how to use it.
By creating good docs the LLMs are able to get 95% of the way to a workable bespoke solution, great docs make closing those last few percentages easier.
Limitations
LLMs are getting better every day, the choice of ChatGPT was straightforward as a starting point and could be swapped for another if required.
It would have been helpful to have the various shells I use appear similar, but FreeBSD's Bourne shell, actually a lightweight clone called Almquist shell on BSDs since the 1990s, doesn't have as many options so it is what it is.
Asking ChatGPT to convert the prompt to a Bourne shell failed by hallucinating offering a bash one in the name of Bourne shell. This isn't surprising as those functions aren't available so there wouldn't be any documentation.
Other language models might be different, but the underlying tech is the ultimate limiting factor, no LLMs can get around that and Google searches confirmed the same.
In any case, the more documentation can be used to train LLMs and be available for when users need to reduce that final 5% the better.
It's no surprise that clear documentation with representative examples would be useful to both humans and LLMs.
Outcome
For the curious, here is a fragment of my current .tcshrc to show the prompt:
alan@sibelius:~/projects/dbmail 10:17 0 >
alias h history 1000
alias j jobs -l
alias la ls -a
alias lf ls -FA
alias ll ls -lA
# A righteous umask
umask 22
set path = (/sbin /bin /usr/sbin /usr/bin /usr/games /usr/local/sbin /usr/local/bin /usr/X11R6/bin $HOME/bin)
setenv EDITOR vi
setenv PAGER more
setenv BLOCKSIZE K
setenv LESS '--tabs=4'
setenv PRINTER EPSON_XP-960
setenv LANG en_GB.UTF-8
setenv MM_CHARSET UTF-8
setenv XDG_RUNTIME_DIR "/home/$USER/tmp"
if ($?prompt) then
# ==== COLOURFUL PROMPT ====
# Terminal colours via tput (portable)
set color_reset="`tput sgr0`"
set color_red="`tput setaf 1`"
set color_green="`tput setaf 2`"
set color_yellow="`tput setaf 3`"
set color_cyan="`tput setaf 6`"
set color_prompt="`tput md`"
set color_status = "$color_yellow"
# Prompt
set prompt="%{${color_prompt}%}%N@%M%{${color_reset}%}:%{${color_prompt}%}%c2 %{${color_green}%}%T %{${color_yellow}%}%?%{${color_reset}%} %# "
set filec
set histdup = erase
set history = 1000
set savehist = (1000 merge)
set mail = (/var/mail/$USER)
if ( $?tcsh ) then
bindkey "^W" backward-delete-word
bindkey -k up history-search-backward
bindkey -k down history-search-forward
endif
endif