From [OperationsSoftware]

Runtime Configuration Utility

This page documents a general runtime configuration utility provided in the opscore.utility.config python module. The basic design is to extend the standard python command-line options parser with a transparent back end that reads default option values from a configuration file. The result is that run-time options are specified in files and can be overridden on the command line. One extra feature is support for secret options such as passwords.

Configuration Files

Configuration files follow the loose INI standard, as implemented in the standard python ConfigParser module. Files are organized into named sections of parameter-value assignments. For example, this file contains three sections, named DEFAULT, server and client:

# This is a comment

[DEFAULT]

tcp-port: 1966

[server]

log-dir: /tmp/archiver-PID
interactive: no
db-password: fd6092344341fa78ed6af926cc8ab4d6

[client]

timeout: 1000 # in milliseconds

An INI file is queried by specifying a section name and an option name. In case the option is not in the named section, a value in an optional DEFAULT section will be used.

The runtime configuration utility will search for INI files in two locations (which might be the same): first, in the current working directory and, second, in the installed product's etc/ subdirectory. In case both files are present, the first one containing a value for a given option name will be used. To establish how option values will be retrieved from INI files, your program creates a ConfigOptionParser at run time:

if __name__ == '__main__':
    from opscore.utility import config
    cli = config.ConfigOptionParser(product_name='archiver',config_file='archiver.ini',config_section='server')
    ...

The config_file parameter specifies the INI file name to search for in the two directories. The config_section parameter specifies which section of the INI file(s) to read option values from. The defaults for these parameters are config.ini and DEFAULT, respectively. The optional product_name is used to locate the etc/ subdirectory of an installed product via an environment variable $PRODUCTNAME_DIR that will normally have been established using the setup tool. If no product_name is provided, only the current working directory will be searched for a config file. ConfigOptionParser is a subclass of the standard python OptionParser and the two options above can be combined with the standard options.

Command Line Parser

Configure your program to process command line options using a ConfigOptionParser just as if it were a standard OptionParser object. The only special handling occurs for the add_option and parse_args methods, as described below. Here is a simple example, following on from the code above:

    cli.add_option('-l','--log-dir',dest='logDir',help='Directory that log files will be stored in (PID replaced by actual value)')
    cli.add_option('-p','--tcp-port',dest='tcpPort',type='int',default=0,help='TCP port that server will listen to')
    cli.add_option('-i','--interactive',action='store_true',help='Is program running interactively?')
    cli.add_option('--db-password',dest='dbPassword',type='secret',help='Database password to use')
    ...
    (options,args) = cli.parse_args()
    ...
    # can now use options.logDir, options.tcpPort, etc

The add_option method accepts an optional default keyword. The ConfigOptionParser extends the OptionParser by always providing this keyword when a suitable value is available from an INI file, using the long-form of the option name (without the double-dash prefix) as the parameter name to lookup. In case a default keyword is already provided, as for log-dir in the example above, an INI value will replace it. There is no requirement that every command line option have a corresponding INI value.

Secret Options

This module adds a new secret option type to the built-in OptionParser, to contain encrypted strings such as passwords. The values for such options are stored in INI files (or provided on the command line) as a hex string representing the encrypted value, for example:

db-password: fd6092344341fa78ed6af926cc8ab4d6

Decrypting requires a pass phrase that the user is prompted for when the parse_args method is called. Use an optional keyword to control the prompt:

    (options,args) = cli.parse_args(prompt='How big is the Universe?')
    ...
    # options.dbPassword now contains the decrypted value if the correct response is received

Use the secret.py program to generate encrypted values suitable for storing in INI files (or entering on the command line). Users are only prompted for a single pass phrase, so all secret data used by a program should be encrypted using the same phrase.

Since python does not provide a built-in cryptography library, using any secret options requires that you have the pycrypto module installed. However, this is a runtime dependency and does not apply to programs with no secret options.

Documentation

There are potentially two places that a run-time configuration option can be documented: in the help metadata provided to the ConfigOptionParser add_option method, and in comments embedded in the INI file. The recommended convention is that the help metadata should be used as the primary description of an option since this will appear directly in the top-level program and can be conveniently formatted using the built-in --help command-line option. There is also no guarantee that a given option is contained in an INI file, or it could appear multiple times. With this convention, comments in INI files should be reserved for explaining the specific values being assigned, as appropriate.

The ConfigOptionModule supplements the built-in print_help() function to append information about run-time configuration, for example:

% ./server.py --help
Usage: server.py [options]

Options:
  -h, --help            show this help message and exit
  --log-dir=LOGDIR      temporary path for server log and buffer files
  --tcp-port=TCPPORT    TCP port number to listen to or zero for none
  --interactive         running interactively?
  --db-password=DBPASSWORD  database engine to use
  ...

Runtime configuration defaults provided by the following files:

  /.../.../archiver.ini

Default values are:

            log-dir: /tmp/archiver-PID
           tcp-port: 1966
        interactive: False
        db-password: fd6092344341fa78ed6af926cc8ab4d6