Combining ConfigParser and argparse with parse_known_args()

Edited to add: The best place to discuss this topic is on my original answer on StackOverflow.

I'm a fan of both the ConfigParser and argparse modules in my Python scripts and I've always thought it would be great to have a way to combine them. In other words allow the user of a script to provide a command line option that specified a configuration file that specified defaults for the command line options.

Recently I discovered the parse_known_args() method, which allows one to do just that.

Here's the script p.py that demonstrates this:

	#!/usr/bin/env python
	import argparse
	import ConfigParser
	parser = argparse.ArgumentParser()
	parser.add_argument("-c", "--conf_file",
	help="Specify config file", metavar="FILE")
	args, remaining_argv = parser.parse_known_args()
	defaults = {
	"option1" : "some default",
	"option2" : "some other default",
	}
	if args.conf_file:
	config = ConfigParser.SafeConfigParser()
	config.read([args.conf_file])
	defaults = dict(config.items("Defaults"))
	parser.set_defaults(**defaults)
	parser.add_argument("--option1", help="some option")
	parser.add_argument("--option2", help="some other option")
	args = parser.parse_args(remaining_argv)
	print args

view raw p.py hosted with ❤ by GitHub

Here's a configuration file and a demonstration of how it works:

	$ cat p.conf
	[Defaults]
	option1 = Hello World!
	option2 = Have a nice day.
	$ p.py --option1 "Hi there"
	Namespace(conf_file=None, option1='Hi there', option2='some other default')
	$ p.py --conf_file p.conf
	Namespace(conf_file=None, option1='Hello World!', option2='Have a nice day.')
	$ p.py --conf_file p.conf --option1="Hi there world."
	Namespace(conf_file=None, option1='Hi there world.', option2='Have a nice day.')

view raw gistfile1.txt hosted with ❤ by GitHub

So this is what I was looking for, the caller can specify a configuration file with defaults, but override those defaults with more command line options.

There's only one problem, the help option only shows the configuration file option:

	$ p.py -h
	usage: p.py [-h] [-c FILE]
	optional arguments:
	-h, --help show this help message and exit
	-c FILE, --conf_file FILE
	Specify config file

view raw gistfile1.txt hosted with ❤ by GitHub

That's because the '-h' is processed by the parse_known_args() instead of the final parse_args(). The way to fix this is to create two ArgumentParsers and use the add_help parameter when creating first to suppress it from parsing -h:

	#!/usr/bin/env python
	import argparse
	import ConfigParser
	conf_parser = argparse.ArgumentParser(
	# Turn off help, so we print all options in response to -h
	add_help=False
	)
	conf_parser.add_argument("-c", "--conf_file",
	help="Specify config file", metavar="FILE")
	args, remaining_argv = conf_parser.parse_known_args()
	defaults = {
	"option1" : "some default",
	"option2" : "some other default",
	}
	if args.conf_file:
	config = ConfigParser.SafeConfigParser()
	config.read([args.conf_file])
	defaults = dict(config.items("Defaults"))
	# Don't surpress add_help here so it will handle -h
	parser = argparse.ArgumentParser(
	# Inherit options from config_parser
	parents=[conf_parser],
	# print script description with -h/--help
	description=__doc__,
	# Don't mess with format of description
	formatter_class=argparse.RawDescriptionHelpFormatter,
	)
	parser.set_defaults(**defaults)
	parser.add_argument("--option1", help="some option")
	parser.add_argument("--option2", help="some other option")
	args = parser.parse_args(remaining_argv)
	print args

view raw p-with-fixed-help.py hosted with ❤ by GitHub

Now help works like you would expect:

	$ p2.py -h
	usage: p2.py [-h] [-c FILE] [--option1 OPTION1] [--option2 OPTION2]
	optional arguments:
	-h, --help show this help message and exit
	-c FILE, --conf_file FILE
	Specify config file
	--option1 OPTION1 some option
	--option2 OPTION2 some other option

view raw gistfile1.txt hosted with ❤ by GitHub