Dotfile Management and Documentation with Org-Mode
Table of Contents
One day 'polm23' asked on Hacker News, what do the readers use to manage dotfiles. I was just experimenting with my method and contributed my two cents. Turns out, rare for me, I stumbled upon an original way to edit, document and deploy dotfiles. Although we Emacs users have been using Org-Mode to wrap our .emacs and document our settings for years, few people that I know have used it for managing more dotfiles.
There are at least 3 advantages of using Org-Mode to manage dotfiles, that I can think of right now.
- Every line of configuration can be documented. You can record why and when you add this line. You can also document other possible options for this particular setting right above or below the code block, so you don't always need to look up man pages or other documentation.
- You need fewer error-prone symlinks. We have all been bitten by broken links when we were young and type '
ln -s dotfiles/vimrc ~/.vimrc', right? Even if we have learnt the lesson, and always use the full path, what if you want to migrate the dotfiles from BSD to Linux, where~mean different paths? I know, you set branches in your deployment script. Not a requirement with Org-Mode. - You can manage remote dotfiles. Org-Mode can 'tangle' code blocks to a remote location, thus you can manage your server's
nginx.confas easily as your.bashrc.
1 Here's how
1.1 Requirements
Emacs 24.x up, Org-Mode 8.x or newer, and Your favourite keyboard.
Open a new file in Emacs, call it dotfiles.org or whatever you like.
1.2 "Normal" dotfiles
These files directly reside under your home directory. So managing them in Emacs is very straightforward. For ease of organization, you can put files of the same category, e.g. all things related to email, under a heading:
* Email ** Muttrc ** Aliases ** goobookrc
Then, under each subheading, put your files in an src block, like this:
* Email ** Muttrc #+BEGIN_SRC conf :tangle ~/.muttrc #source "/etc/Muttrc" # Not available on OS X source "gpg --batch --passphrase-file ~/.sec/.passphrase --textmode -d ~/.sec/mutt.gpg |" set realname="Haoyang Xu" set sig_dashes ... #+END_SRC
Then if you run M-x org-babel-tangle, or press C-c C-v t, the content of the above src block will be written to $HOME/.muttrc, overwriting the file's content if it exists already.
Similarly, write each file's content in an src block under the corresponding subheading:
* Email
** Muttrc ...
** Aliases
#+BEGIN_SRC conf :tangle ~/.aliases
alias mumon foobar@example.com
#+END_SRC
** goobookrc
#+BEGIN_SRC conf :tangle ~/.goobookrc
[DEFAULT]
# The following are optional, defaults are shown
# This file is written by the oauth library, and should be kept secure,
# it's like a password to your google contacts.
;oauth_db_filename: ~/.goobook_auth.json
# The client secret file is not really secret.
;client_secret_filename: ~/.goobook_client_secret.json
;cache_filename: ~/.goobook_cache
;cache_expiry_hours: 24
;filter_groupless_contacts: yes
#+END_SRC
Now when you invoke org-babel-tangle, the 3 dotfiles will be written. You can put the .org file under version control, edit the dotfiles within Emacs, and deploy them with one command.
1.3 Dotfiles in a dotdir
For example, you want to manage your SSH config file, which is under .ssh. Normally it will not be more difficult than the case above. But if you are setting up a new machine and don't have .ssh path yet, Org-Mode will complain when you tangle.
In that case, you can write a "magic" line at the beginning of the file:
# -- eval: (make-directory ".ssh" "~") --
Alternatively, you can write an src block before the file content block to run commands that create the directories needed.
#+BEGIN_SRC sh mkdir ~/.ssh #+END_SRC
The best approach may be setting a header argument mkdirp to yes, like this:
#+BEGIN_SRC sh :mkdirp yes :tangle ~/.ssh/config (your .ssh/config contents) #+END_SRC
If your .ssh directory does not exist yet when you tangle this block, Org-Mode will create it for you.
1.4 Emacs dotfiles
If your are using the good old .emacs or init.el to store your Emacs configuration, they are managed in the same manner as the above cases. However, if your configuration are already living in another .org file, you probably don't want to put it in an "org" src block in your dotfile.org. My solution is to put it in a directory under the directory where my home.org (my dotfile) resides. That makes it a little untidy but I don't have a better solution right now. Ah, the taste of irony.
1.5 Credentials and secrets
If you are putting your dotfiles online, you need to save dotfiles with passwords/secrets in an encrypted format. Luckily, Emacs has very good encryption/decryption support. You can put things you don't want others to read into a specific .org file, and use epa-encrypt-file to get encrypted file with .gpg suffix. After that you can delete the clear text .org file. Next time when you edit the encrypted .org.gpg file, Emacs will use gpg-agent to ask for the password, and decrypt it for you.
2 Documenting changes
The above method provides a simple and fast way to put all your dotfiles in a few .org files. But it does not fully justify a migration of all your dotfiles into Org-mode src blocks. The strength of Org-mode based dotfile management lies in seamless documentation and instant deploy of changes.
It is possible to breakdown a very long config file into multiple src blocks, and tangle them into one file for deployment. These src blocks can even be put into different subheadings, according to their categories and functions in the final tangled config.
In this case, the file to write to is not specified in each src block's attributes ('head arguments', as in the official document), but as a property of the subheading under which the contents of the file goes. It is better to illustrate with an example:
* Git
:PROPERTIES:
:tangle: ~/.gitconfig # <- all src blocks under this 'Git' subtree will be written to ~/.gitconfig
:END:
** personal information
#+BEGIN_SRC conf
[user]
name = John Doe
email = john.doe@example.net
#+END_SRC
** push settings
#+BEGIN_SRC conf
[push]
default = upstream
#+END_SRC
...
When you tangle this file, all src blocks under * Git subtree will be tangled into $HOME/.gitconfig.
Now, suppose we want to change Git's push settings, it is easy to locate that subtree, and do some editing.
** push settings
#+BEGIN_SRC conf
[push]
default = simple
#+END_SRC
It may become difficult to tell which part gets edited after a while. Why not write a little note about the change?
** push settings
#+BEGIN_SRC conf
[push]
default = simple
#+END_SRC
[2016-03-19 Sat 22:31] change push default from 'upstream' to 'simple'.
The timestamp can be inserted almost anywhere, by pressing C-u C-c !. If you only need to remember the date but not the time, you can use C-c !.
If you often need to change such settings, it is a good idea to document all possible options:
** push settings
With ~push.default~ set to ~simple~, ~git push~ will fail if the current local branch is not tracking a remote branch, even if remote has a branch with the same name. This seems to be the safest option. Other possible values are:
- ~upstream~: push the local branch to its upstream branch.
- ~current~: push the local branch to a branch of the same name.
#+BEGIN_SRC conf
[push]
default = simple
#+END_SRC
[2016-03-19 Sat 22:31] change push default from 'upstream' to 'simple'.
In other cases, you may want to experiment with various combinations of options. You can write them all out, and tell Org-Mode not to tangle some of them:
#+BEGIN_SRC conf :tangle no safe_threshold=1 encryption_mechanism=ECDHE_RSA #+END_SRC #+BEGIN_SRC conf safe_threshold=0 encryption_mechanism=HMAC-SHA1 #+END_SRC
Only the latter config will enter the config file.
3 Managing remote dotfiles and configs
In :tangle head argument or subtree property, you can specify a remote location, typically a remote server which you have SSH access. Suppose you are in charge of a web server, you can save yourself a lot of remote editing by using Org-Mode to manage its configuration:
* Nginx
:PROPERTIES:
:tangle: /webadmin@ssh.example.org:configs/nginx.conf
#+BEGIN_SRC conf
worker_processes 4;
events { worker_connections 1024; }
...
#+END_SRC
#+BEGIN_SRC sh :dir /ssh:webadmin@ssh.example.org|sudo:ssh.example.org :tangle no
cp /home/webadmin/configs/nginx.conf /etc/nginx/
chown nginx:nginx /etc/nginx/nginx.conf
#+END_SRC
The first code block get tangled into the remote file /home/webadmin/nginx.conf, the second code block has :tangle no and will not be tangled into any file, but you can run the code block from your local Emacs, it will ask you your sudo password, and copy the file to the right location and set owners.
4 Caveats
- Emacs is single-threaded. If you use Org-Mode to deal with files/shells on remote systems through a slow connection, you will have to wait during tangling remote files and executing remote commands.
- You don't have to deal with symlinks, and you don't get its benefits. For example, you change your Git settings through command
git config --global .... Such changes don't automatically get updated in your Org-Mode file. It is your responsibility to update yourdotfile.orgby hand.UPDATE: Ken Mankoff sends me a tip that partially solves the problem. You can add the following line to the top of your
dotfile.org:#+PROPERTY: header-args:conf :comments link :tangle-mode (identity #o444)
In Ken's own words: "This makes the files read-only, so I can't edit them by mistake. It also creates a commented link at the top of each, so I can jump from the dotfile to its Org origin if I open the dotfile by mistake." Kudos.
- Your
dotfile.orgmay become to big and unwieldy. For most people this is not a big deal. On my late-2012 MacBook Pro, opening Org files of a few hundred KB is as smooth as opening a new file. But if this system is used for a long time, the files may grow with all those logs and documents. In that case, you may want to split the files by, say, putting each top-level headline in a separate file. It is easy to create links to other files in Org-Mode, so you can still conveniently navigate through all the files.
5 Acknowledgements
This system is inspired by Sacha Chua's Emacs config. I did not realize Org-Mode was such a powerful tool for system administration until I see Howardism's talk on literate devops. Last but not least, thanks to all the people behind Org-Mode, Tramp and GNU Emacs.
