From Core
The formal grammars that define valid command and reply strings enable a parser to perform a first level of validation, but additional validation is generally required to ensure that keyword values can be interpreted according to their expected types (the grammar considers all keyword values to be strings) and, for commands, to test for valid keyword sequences.
This page describes a python framework for declaring valid keywords and commands, and associating them with user-defined callbacks. The framework is implemented in the python modules opscore.protocols.validation, opscore.protocols.keysformat and opscore.protocols.keys.
A keyword validator associates a keyword name with a list of expected value types, for example:
Key('Position',
Float(units='deg',strFmt='%.3f')*2,
Enum('AltAz','RADec',help='Coordinate system'),
help='Object position.')
The resulting Key object serves as a template for valid Keyword objects. The general format is
Key(<name>,<type-1>,<type-2>,...<type-n>,help=...)
where the help value is optional and type declarations are documented elsewhere.
Note that the Key object stores the name with whatever capitalization is provided, for improved readability of the code and any automatically generated documentation. However, for the purposes of validation, keyword name matches are case insensitive and names must be unique within the namespace of a Keys dictionary, which usually groups together all reply or command keywords of a single actor.
A Key object serves as a template for creating new Keyword instances from a list of values. Values can be either strings to be interpreted or else already typed, but must be compatible with the types declared for the Key object. Values can either be provided in a list:
key.create(['-1.2',0xdead,'beef']) key.create([-1.2,'dead',0xbeef])
or else as individual arguments:
key.create('-1.2',0xdead,'beef')
key.create(-1.2,'dead',0xbeef)
In case the wrong number of values is provided or the provided values have the wrong types, the create method raises a KeysError exception. The generated Keyword will used the stored capitalization of the keyword name, rather than a canonical lower/upper case form, for improved readability.
Although you will not normally use a Key to directly validate a Keyword, this is how it works:
from opscore.protocols import types,keys,messages
key = keys.Key('keyname',types.Float(),types.Hex()*2)
keyword = messages.Keyword('keyname',['-1.23','dead','beef'])
if key.consume(keyword):
x,j,k = keyword.values
The consume method returns True if its argument is a matching keyword and also replaces the string keyword values with their typed equivalents in the keyword object. No keyword values will be replaced unless they all can be successfully converted to the appropriate types, so the typing operation is guaranteed to be atomic. This ensures that the keyword values always represent the best available knowledge without requiring users to keep track of the processing history of a keyword. In case you care about whether values have been successfully converted to types, you can test the keyword's matched attribute.
If you want to understand how the consume action works in detail, you can enable its debug output using:
keys.Consumer.debug = True
Any subsequent consume commands will now generate output like this:
Key(keyname) << KEY(keyname)=['-1.23', 'dead', 'beef']
Types[Float*1, Hex*2] << ['-1.23', 'dead', 'beef']
PASS >> [Float(-1.23), Hex(0xdead), Hex(0xbeef)]
PASS >> {KEY(keyname)=[Float(-1.23), Hex(0xdead), Hex(0xbeef)]}
Each pair of lines at the same indentation level shows the input stack fed to a consumer (<<) and the resulting output (>>). Nested indentation shows the consumer calling chain.
A keyword validator can describe the keywords it matches in either plain text:
key = Key('AltLim',
types.Float(units='deg',strFormat='%.2f',help='min/max position limits')*2,
types.Float(units='deg/s',strFormat='%.1f',help='max velocity'),
types.Float(units='deg/s^2',strFormat='%.1f',help='max acceleration'),
types.Float(units='deg/s^3',strFormat='%.1f',help='max jerk'),
help='TCC altitude motion limits (not the limits used by the axis controller itself).'
)
print key.describe()
Keyword: AltLim
Description: TCC altitude motion limits (not the limits used by the axis controller itself).
Values: 5
Repeated: 2 times
Description: min/max position limits
Type: Float (float)
Units: deg
Repeated: once
Description: max velocity
Type: Float (float)
Units: deg/s
Repeated: once
Description: max acceleration
Type: Float (float)
Units: deg/s^2
Repeated: once
Description: max jerk
Type: Float (float)
Units: deg/s^3
or else in HTML format:
print >> htmlfile, key.describeAsHTML()
<div class="key">
<div class="descriptor"><span class="label">Keyword</span><span class="value">AltLim</span></div>
<div class="descriptor"><span class="label">Description</span><span class="value">TCC altitude motion limits (not the limits used by the axis controller itself).</span></div>
<div class="vtypes">
<div class="descriptor"><span class="label">Values</span><span class="value">5</span></div>
<div class="vtype">
<div class="descriptor"><span class="label">Repeated</span><span class="value">2 times</span></div>
<div class="descriptor"><span class="label">Description</span><span class="value">min/max position limits</span></div>
<div class="descriptor"><span class="label">Type</span><span class="value">Float (float)</span></div>
<div class="descriptor"><span class="label">Units</span><span class="value">deg</span></div>
</div>
...
<div class="vtype">
<div class="descriptor"><span class="label">Repeated</span><span class="value">once</span></div>
<div class="descriptor"><span class="label">Description</span><span class="value">max jerk</span></div>
<div class="descriptor"><span class="label">Type</span><span class="value">Float (float)</span></div>
<div class="descriptor"><span class="label">Units</span><span class="value">deg/s^3</span></div>
</div>
</div>
</div>
In either case, the optional help metadata is included as the keyword description. The HTML output is tagged with CSS class names to allow user-defined styling.
Multi-line keyword descriptions will be wrapped for a total output width of 80 columns using the python textwrap module. Leading white space in triple-quoted strings will also be removed.
A command has three components to be validated:
The general form of a command validator is:
Cmd(<verb>,<type-1>,<type-2>,...,<type-n>,<keys-format>,help="...")
where only the initial verb is required. Command values are specified in exactly the same way as keyword values and the help argument should be used to provide a general description of a command. The new ingredient required to specify a valid command is a keywords format string, described in detail in the following section. Some examples of command validators are:
Cmd('expose','@(object|flat|dark|sky|calib) <time>',
help='Starts an exposure and does not save the resuting image.')
Cmd('expose','@(object|flat|dark|sky|calib) <time> <filename> [<window>] [<bin>]',
help='Starts an exposure and saves the result image to a file.')
Cmd('expose','abort',
help='Aborts any current exposure.')
There are three ways to match a single keyword in a format string:
The second form looks up Key definitions by name in a keys dictionary that has been assigned using:
CmdKey.setKeys(keysdict)
Multiple keys dictionaries can be searched using:
CmdKey.addKeys(keysdict1) CmdKey.addKeys(keysdict2) ...
Here are some examples of valid single-key format strings:
key <key> key1|key2 <key1>|<key2> key1|<key2>
Single keys are combined into a key group by listing them with intervening space, for example:
key1 key2 key3 <key1> key2 <key3a|key3b>
By default, all keywords listed in a group must be present in the command but not necessarily in the order they are listed in the format string. Single keys in a group can be decorated in two ways to change their default matching behavior:
Here are all the valid two-key groups with positioned keys:
@key1 key2 @key2 key1 @key1 @key2 @key2 @key1
and here are the valid distinct two-key groups with optional keys (with no positioned keys, order does not matter and permutations are not distinct):
[key1] key2 key1 [key2] [key1] [key2]
Finally, a format string supports a limited notion of hierarchical keywords via subgroups that must match as a unit:
key1 (@key2 [key2a] [key2b]) key3
The subgroup is treated as a single-key unit of its parent group and so can be positioned @(...) or made optional [(...)] in its parent context. [...] can be written in place of [(...)]. Positioning within a subgroup is relative to the first keyword where the subgroup match is attempted, so that the following command keywords would be valid matches for the example above:
key1 key2 key3 key2 key2a key3 key1 key3 key1 key2 key2b key2a
while the following would not match:
key1 key2a key2 key3 key2 key1 key2b key3 key3 key2 key1 key2b key2a
The parsing code for keys format strings is in the ops.core.protocols.keysformat module.
A Cmd object serves as a template for creating new Command instances from either a list of keywords or a sequence of keyword parameters:
command = Cmd.create([<key-1>,<key-2>,...]) command = Cmd.create(<key-1>,<key-2>,...)
Keywords that have no associated values can be specified simply as strings:
command = Cmd.create("key1","key2",...)
All other keywords should already have a corresponding Key object that has been registered with CmdKey.setKeys(...) or CmdKey.addKeys(...), and are specified with a tuple:
command = Cmd.create(("key1",<value-1>,<value-2>,...),...)
or, equivalently,
command = Cmd.create(("key1",[<value-1>,<value-2>,...]),...)
In case the wrong number of values is provided or the provided values have the wrong types, the create method raises a KeysError exception. Other errors, such as an incorrect sequence of valid keywords, will raise a ValidationError exception.
If a command takes values that are not associated with any keyword, these should be appended as a named list:
command = Cmd.create(...,values=[1.23,'0xbeef'])
As always, each value can either be provided as a string or already typed. Invalid command values will raise a ValidationError.
Although you will not normally use a Cmd to directly validate a Command, this is how it works (help metadata has been omitted for clarity). First, register the keys dictionary that defines any keywords referenced in the command's format string:
CmdKey.setKeys(keysdict)
Next, define the command validator to use:
cmd = Cmd('expose','@(object|flat|dark|sky|calib) <time> <filename> [<window>] [<offset>] [<size>] [<bin>]')
A validation target can either be built using the validator as a template, for example:
command = cmd.create("calib",("filename","sky.fits"),("bin",16,16),("time",12.345),("offset",0,0))
or else by parsing a valid command string:
from ops.core.protocols.parser import CommandParser
parser = CommandParser()
command = parser.parse("expose calib filename='sky.fits' bin=16,16 time=12.345 offset=0,0")
Finally, test the target command for a valid match using:
if cmd.consume(command):
exposureType = command.keywords[0]
filename = command.keywords['filename'].values[0]
if 'window' in command.keywords:
llx,urx,lly,ury = command.keywords['window'].values
Note that after a successful validation, all keyword values will have been overwritten with their typed equivalents (but no values will be overwritten for a partial match).
All keywords present in a command must be matched against the keys format string and no unmatched keywords are allowed for a valid match. Keywords with the same name may be repeated in a command, with either the same or different value types (although actor designers are discouraged from using this feature without good reason).
A command validator can describe the commands it matches in either plain text:
print cmd.describe()
or else in HTML format:
print >> htmlfile, cmd.describeAsHTML()
Refer to the description of keyword documentation for details. Here is an example of a plain text command description:
Command: expose
Description: Take an exposure and save the results in a FITS file
Values: none
Keywords: @(object|flat|dark|sky|calib) <time> <filename> [<window>] [<offset>] [<size>] [<bin>]
Note that the keywords are not described here since they would normally be described once for a group of command sharing a keys dictionary.
A CommandHandler consumes commands as strings, attempts to parse them and, if the parse is successful, tests them against a list of command validators. Each validator can be associated with a callback chain using '>>' operator notation (borrowed from C++ iostreams):
handler = CommandHandler(
Cmd('expose','@(object|flat|dark|sky|calib) <time>') >> exposeNoFile,
Cmd('expose','@(object|flat|dark|sky|calib) <time> <filename> [<window>] [<offset>] [<size>] [<bin>]') >> exposeWithFile,
Cmd('expose','@abort') >> exposeAbort
)
Feed a command handler using:
handler.consume("""
expose object time=12.345
expose flat filename='flat.fits' time=12.345
expose dark filename='dark.fits' time=12.345 window=0,0,100,100
expose sky time=12.345 filename='sky.fits' bin=16,16
expose calib filename='sky.fits' bin=16,16 time=12.345 offset=0,0
expose abort
""")
Callbacks are invoked with a single argument: a validated and parsed Command object with typed command and keyword values. Implementations of the callbacks used above might look like this:
def exposeNoFile(cmd):
exposureType = cmd.keywords[0]
def exposeWithFile(cmd):
exposureType = cmd.keywords[0]
filename = cmd.keywords['filename'].values[0]
if 'window' in cmd.keywords:
llx,urx,lly,ury = cmd.keywords['window'].values
def exposeAbort(cmd):
print 'aborting...'
Callbacks can be any callable python object (in particular you can pass in a functor; this allows you to pass your own data to the callback). The validation module defines a callable Trace class that prints out any command it receives, prefixed by an optional tag provided to the constructor:
Cmd(...) >> Trace()
Cmd(...) >> Trace('Got expose command')
Any exception raised by a callback will be passed on by the handler's consume method. Any callback return value is ignored.
Mutiple callbacks can be chained together on a single validator:
Cmd(...) >> callback1 >> callback2 >> callback3
They will all be called (unless one raises an exception) in the order listed. Any changes to the attributes of the command object passed to one callback will be visible to subsequent callbacks in the same chain.
An alternative way to use callbacks is via the match method, which takes a single command and returns the parsed command and a list of callbacks:
cmd, funcList = handler.match("expose flat filename='flat.fits' time=12.345")
for func in funcList:
func(cmd, 666)
Here's an example of your various command parsing options.
The following describes a simple reply callback mechanism that is suitable for testing and simple applications. TUI uses a more sophisticated dispatch mechanism.
Reply messages are handled using a combination of the ingredients describe above. Valid reply keywords are collected in a keys dictionary and registered using:
ReplyKey.setKeys(keysdict)
or, with key validators spread across multiple dictionaries,
ReplyKey.addKeys(keysdict1) ReplyKey.addKeys(keysdict2) ...
A ReplyHandler consumes replies as strings, attempts to parse them and, if the parse is successful, tests them against a list of reply keyword validators. Each validator can be associated with a callback chain using '>>' operator notation:
handler = ReplyHandler(
ReplyKey('dspload') >> Trace('REPLY') >> dspload,
ReplyKey('arrayPower') >> Trace('REPLY'),
ReplyKey('exposureState') >> Trace('REPLY') >> exposureState,
ReplyKey('exposureMode') >> Trace('REPLY')
)
Feed a reply handler using:
handler.consume("""
dspload='myfile.dat'
arrayPower = off ; exposureState = processing , 12.34 , 0.123
exposureMode='fowler',12;arrayPower='?'
""")
Reply keyword callbacks are invoked with a validated and parsed Keyword object with typed values. Implementations of the callbacks used above might look like this:
def dspload(key):
filename = key.values[0]
def exposureState(key):
state,completionTime,remainingTime = key.values
Reply keyword callbacks can be chained and modify their argument, as described above for command callbacks.