KPF Recipe AST Parser

class kpfpipe.pipelines.kpf_parse_ast.KpfPipelineNodeVisitor(pipeline=None, context=None)

KpfPipelineNodeVisitor is a node visitor class derived from ast.NodeVisitor to convert KPF pipeline recipes expressed in python syntax into operations on the KPF Framework. This class implements methods with names of the form visit_<syntax_element>, each of which handles the actual processing of <syntax_element> instances found in a recipe. ast.NodeVisitor implements visit(), which is the code that calls the appropriate visit_<syntax_element> method, or generic_visit() if no such method is found. For recipes containing only supported syntax elements, generic_visit() is never called.

There are three class member flags that keep the state of node walking across calls to data processing primitives and loop iterations. They are:

self.awaiting_call_return:

This flag is set just before the visit_Call() method returns, after queueing a data processing primitive to run in the Framework. Any node visiting method that can have a Call within it must “return” immediately when this flag is set. It is cleared by resume_recipe() after the Framework finishes executing the called primitive.

self.returning_from_call:

This flag is set by resume_recipe() just before it starts walking the recipe syntax tree, which starts from the top each time. (Each visit_<syntax_element> method is responsible for behaving correctly in the face of these multiple “duplicate” calls, either by redoing work quickly, or by saving state and results of work already done. See the notes about the kpf_completed flag below.) visit_Call() uses this flag to trigger the processing of the return value(s) from the data processing primitive previously enqueued to the framework that resume_recipe() is following This flag distinguishes returning from a finished call from preparing for a new call.

self._reset_visited_states:

This flag is set by reset_visited_states(), which is called from visit_For() and potentially other loop control methods to be implemented in the future. It causes the visit_<syntax_element> methods to reset their saved state to prepare for another iteration of a loop.

In addition, all visit_<syntax_element>() methods doing work that takes a significant amount of time use an internal attribute, kpf_completed, to indicate that the work represented by that node and any of its children has been completed, and as necessary the result has been stored in an attribute of the AST node itself. Some visit_<syntax_element>() methods store additional state and parameters. Those behaviors are covered below in the method documentation.

generic_visit(node)

generic_visit() is called if no explicit visitor function exists for a node. It logs a message in the pipeline logger noting that the recipe contained an unsupported syntax element, and then raises RecipeError with the same message.

register_builtin(key, func, nargs)

register_builtin() registers a function so that the function so registered can be called from within a recipe directly, without using the Framework’s event queue mechanism. This is useful for recipe support functions like converting floats to ints and splitting strings.

Parameters:

• key – the name of the built-in function as a string

• func – the python function itself

• nargs – the number of input args the function expects

Note

Functions that can take a variable number of arguments are not supported by the recipe parser.

reset_visited_states(node)

reset_visited_states() walks the tree below the node where the call is made after setting the class member self._reset_visited_states. Each visit_<syntax_element>, when called with that flag set, should reset its state so that an enclosing loop can be properly executed again with new parameters.

reset_visited_states() is called in visit_<> methods that control looping, such as visit_For()

visit_Add(node)

visit_Add() implements the binary addition operator by invoking the internal method _binary_op_impl() with an appropriate lambda function. See also visit_BinOp()

visit_And(node)

visit_And() implements the boolean “and” function by invoking the internal method _binary_op_impl() with an appropriate lambda function. See also visit_BinOp()

visit_Assign(node)

visit_Assign() processes the assignment of one or more constant or calculated values to named variables. The variable names to assign into come from the _store stack, while the values come from the _load stack. Calls to visit_call() may occur in the tree that is walked to generate the value to be assigned, and therefore may set self._awaiting_call_return, in which case visit_Assign() we need to immediately return, and pick up where we left off later, completing the assignment. See also visit_call() and resume_recipe().

visit_Attribute(node)

visit_Attribute() implements the syntax e.g. of an object attribute access, e.g. “a.key”. It does so by getting the name of the attribute, and then testing to see of the object at the top of the _load stack has an attribute of that name. If it does, the value of the attribute replaces the original object at the top of the _load stack. If no such attribute exists on the object, an message is logged and RecipeError is raised.

visit_BinOp(node)

visit_BinOp() implements binary operations, i.e. “x + y”, “x - y”, “x * y”, “x / y”. The actual work is done in the operator visitor method, e.g. visit_Add or visit_Mult.

Implementor Note:

This implementation doesn’t support calls within binOp expressions, so we don’t bother guarding for self.awaiting_call_return here, nor in the binary operator methods.

visit_BoolOp(node)

visit_BoolOp() implements boolean binary operations, i.e. “x and y” and “x or y”. The actual work is done in the operator visitor method, e.g. visit_And.

Implementor Note:

This implementation doesn’t support calls within boolOp expressions, so we don’t bother guarding for self.awaiting_call_return here, nor in the binary operator methods.

visit_Call(node)

visit_Call() implements function call syntax. The function can be one of several built-in functions, or can enqueue a data processing primitive to be run in the Framework. In the latter case, it cooperates with return_recipe() to make available to the recipe values returned from data processing primitives.

Whatever the function type, arguments are popped from the _load stack, and results of the call are pushed back onto the _load stack.

If the function’s name is the special case “invoke_subrecipe”, the one expected argument is a path to a recipe file. If it has not already been, it is parsed into an abstract syntax tree (AST) and hung on the “call” tree node for future use. The subtree is then “visited” just like any other subtree. The head of the subrecipe tree is a “module” node, as for all recipes. See visit_module() for more details on how subrecipes are handled.

If the name of the function to be called is not “invoke_subrecipe”, the registry of built-in functions is checked. (See the documentation for “register_recipe_builtins()” for a list of built-ins and their behavior.)

If the function name is not found among the registered built-ins, it is presumed to be a data processing primitive to be run on the Keck framework. The list of arguments popped from the _load stack is wrapped in an Arguments class object, an event is constructed containing the primitive name and the argument object. That event is appended to the Framework’s priority event queue. The awaiting_call_return flag is set, and visit_Call() immediately returns. That flag causes an immediate return from each of the visit_<syntax_element>() functions all the way up the call stack, and the instance of start_recipe() or resume_recipe() at the top of the call stack also returns. The Framework is then free to run the next queued event, which is the primitive named in node being processed by this visit_Call().

When the primitive has been run by the framework, the next primitive will be “resume_recipe”, which will set the returning_from_call flag and start traversing the previously saved AST tree from the top again. Because of the various kpf_completed attributes set on nodes of the tree, processing will quickly get back to here, with the state of the various tree nodes the same as before. resume_recipe() will have placed the output of the primitive, which is an Arguments class object, on the class instance’s call_output property and set the returning_from_call flag. visit_Call() will extract each positional argument value from the Arguments object and push it onto the _load stack, becoming the results of the call. (Any keyword arguments in the Arguments object returned from the primitive are ignored.)

visit_Compare(node)

visit_Compare() implements comparison expressions, e.g. “x <= y”. It walks the AST subtrees for the left and right side of the comparison, pushing the corresponding values on the _load stack. It “visits” the comparison operator itself, which results in popping off the values for the two sides and pushing either “True” or “False” on the _load stack as a Bool. See also the comparison operators, e.g. visit_Eq(), visit_NotEq(), visit_Lt(), visit_LtE(), visit_Is(), visit_IsNot(), visit_In().

visit_Div(node)

visit_Div() implements the binary division operator by invoking the internal method _binary_op_impl() with an appropriate lambda function. See also visit_BinOp()

visit_Eq(node)

visit_Eq() implements the equality comparison operator by invoking the internal method _compare_op_impl() with an appropriate lambda function. See also visit_Compare().

visit_Expr(node)

visit_Expr() implements an expression by pushing the resulting value on the _load stack. Expression syntax elements are produced by the AST parser only rarely in recipe situations, for example when an expression stands alone, not as part of an assignment statement.

visit_For(node)

visit_For() processes the “for” node of an AST.

Handling looping correctly in a recipe is made more complex because of the need to support calls to data processing primitives within the loop, which cause start_recipe() or resume_recipe() to stop walking the AST tree nodes and return so that the Framework can run the primitive. When resume_recipe() picks up walking the parse tree after the primitive returns, it must be able to resume with the same state as when it was interrupted. Furthermore, state flags and results of operations stored on the various nodes within the loop must be appropriately reset for each iteration, and nesting of loops must work correctly.

In addition to using node attributes as described in the documentation for the class as a whole, visit_for() also uses the following attributes stored on the AST node to keep track of the state of loop processing and loop parameters.

kpf_started:

Processing has started for this “for” node and it’s children, but has not completed unless kpf_completed is also set.

kpf_params:

A local dictionary used to store the values of loop variables and assignment targets. The principal stored parameters are “target”, “args_iter” and “current_arg”

visit_Gt(node)

visit_Gt() implements the greater than comparison operator by invoking the internal method _compare_op_impl() with an appropriate lambda function.

visit_GtE(node)

visit_GtE() implements the greater than or equal comparison operator by invoking the internal method _compare_op_impl() with an appropriate lambda function. See also visit_Compare().

visit_If(node)

visit_If() implements conditional execution triggered by an “if” or “if … else” statement. Evaluate the test and visit one of the two branches, body or orelse.

Note

The python “if” expression, e.g. x = a if <condition> else b, is not supported in recipes.

visit_ImportFrom(node)

visit_ImportFrom() processes the “from … import” statement in a recipe. It causes the imported function names to be added to the pipeline’s event_table, and the paths in the “from” section to be added to the Framework’s module search path. In combination, these two actions allow data reduction pipeline primitives to be successfully run when they are invoked within a recipe. The actual queueing of a pipeline primitive onto the Framework’s event queue occurs in visit_Call().

Note

An “import” clause without a corresponding “from” clause is not supported in recipes, since it would not provide a mechanism to represent a module search path. A bare “import” clause would want to call “visit_Import()”, which this class does not implement, so generic_visit() would be called instead. generic_visit() logs and raises an error.

Note

Because “from … import” statements in subrecipes could be imported more than once if the subrecipe is within a loop, inclusing of “from … import” statements should only appear in the primary recipe, not subrecipes.

visit_In(node)

visit_In() implements the “in” range comparison operator by invoking See also visit_Compare(). the internal method _compare_op_impl() with an appropriate lambda function.

visit_Index(node)

visit_Index() doesn’t do anything special. It simply visits the subtree corresponding to the index value. That (or those) nodes will have the result of pushing some appropriate value onto the _load stack. visit_Index() doesn’t need to alter that value in any way. Typically visit_Subscript() will use that value to perform the actual indexing operation.

visit_Is(node)

visit_Is() implements the “is” comparison operator by invoking the internal method _compare_op_impl() with an appropriate lambda function. See also visit_Compare().

visit_IsNot(node)

visit_Eq() implements the “is not” comparison operator by invoking the internal method _compare_op_impl() with an appropriate lambda function. See also visit_Compare().

visit_JoinedStr(node)

Evaluate a Python f-string (ast.JoinedStr) and push the resulting string onto the _load stack so that built-ins like print() can use it.

Only simple {expr} interpolation is supported; conversion flags (!s, !r, !a) and format specs (‘:…’) are ignored. That is enough for recipe logging.

visit_List(node)

visit_List() implements the “[<list item>, <list item>, …]” list syntax from Python. It does this by visiting the tree node representing each list element in turn. Visiting each node results in a new item pushed onto the _load stack. After visiting each subtree, the new items on the _load stack are popped and appended onto a list object. Finally, the list object is pushed onto the _load stack.

visit_Lt(node)

visit_Lt() implements the less than comparison operator by invoking the internal method _compare_op_impl() with an appropriate lambda function. See also visit_Compare().

visit_LtE(node)

visit_LtE() implements the less than or equal comparison operator by invoking the internal method _compare_op_impl() with an appropriate lambda function. See also visit_Compare().

visit_Module(node)

visit_Module() processes “module” node of a parsed recipe. A Module node is always at the top of an AST tree returned by ast.parse(), and there is only one per recipe file. The parameters dictionary self._params used to store the values of recipe variables is initialized or reset things when processing the module node, and cleaned up (releasing allocated memory) at the end. Cleaning up has the effect of freeing storage used by variables from a previous pipeline recipe.

The recipe parser supports invocation of sub-recipes through the special-case function name “invoke_subrecipe()” (see visit_Call()). Care is taken here to avoid reinitializing the parsing state when processing the “module” node of a sub-recipe.

visit_Mult(node)

visit_Mult() implements the binary multiplication operator by invoking the internal method _binary_op_impl() with an appropriate lambda function. See also visit_BinOp()

visit_Name(node)

visit_Name() processes variable names encountered in a recipe. A Name can occur on either the left or right side of an assignment. If it’s on the left side the context is Store, and the Name is the variable name into which to store a value. visit_Name() pushes that variable name on the _store stack. If the Name is on the right side, e.g. as part of an expression, visit_Name() looks up the name in the params dict, and pushes the corresponding value on the _load stack. There are three special cases to be aware of. If the name “None” is encountered, the value None is pushed on the _load stack. If the name “config” is encountered, the config object is pushed on the _load stack. This object supports access to attributes, so a subsequent call to visit_attribute(), which is provided from the “.” in an expression like “config.ARGUMENT”, will work correctly to extract the ARGUMENT dictionary from the config. The name is looked up in an environment dictionary that has been preloaded from the shell environment that invoked the framework and pipeline. If none of the previous cases produced a successful match, the name is looked up in the internal _params dictionary, which stores by name the values of variables that have been previously encountered in the recipe on the left side of assignment statements. If the name is not found in any of the above places, an error is logged and raised. Note that “config” and keywords from the environment will hide variables defined in recipes of the same name.

Implementer note: The same instance of a Name can appear as different nodes in an AST, so nothing should be stored in the node as a node-specific attribute.

visit_NameConstant(node)

visit_NameConstant() implements python NameConstant syntax element by pushing the value on the _load stack.

visit_Not(node)

visit_Not() implements the operator for not, as in “ not x” by invoking the internal method _unary_op_impl() with an appropriate lambda function. See also visit_UnaryOp().

visit_NotEq(node)

visit_NotEq() implements the inequality comparison operator by invoking the internal method _compare_op_impl() with an appropriate lambda function. See also visit_Compare().

visit_Num(node)

visit_Num() implements a numeric constant by pushing it on the _load stack.

visit_Or(node)

visit_Or() implements the boolean “or” function by invoking the internal method _binary_op_impl() with an appropriate lambda function. See also visit_BinOp()

visit_Str(node)

visit_Str() implements a string constant by pushing its value on the _load stack. Multiline strings delimited by three double-quotes will result in string values with embedded line breaks. Multiple quoted strings with no separator character e.g. comma, are not automatically concatenated, as would be in Python.

visit_Sub(node)

visit_Sub() implements the binary subtraction operator by invoking the internal method _binary_op_impl() with an appropriate lambda function. See also visit_BinOp()

visit_Subscript(node)

visit_Subscript() implements subscript syntax of the form a[i], where the subscript value is a single index. The behavior is to replace the object on the _load stack with the value of the indexed item. See also visit_Index().

Note

Slice subscripts of the form a[i:j] are not supported.

visit_Tuple(node)

visit_Tuple() implements the “(<tuple item>, <tuple item>, …)” tuple syntax from Python. It does this by visiting the tree node representing each tuple element in turn. Visiting each node results in a new item pushed onto the _load stack. After visiting each subtree, the new items on the _load stack are popped and appended onto a list object. Finally, the list object is pushed onto the _load stack.

Internally, the tuple is generated by calling visit_List() to build a list on the _load stack, and then converting it into a tuple.

visit_UAdd(node)

visit_UAdd() implements the operator for unary +, as in “+x” by invoking the internal method _unary_op_impl() with an appropriate lambda function. See also visit_UnaryOp().

visit_USub(node)

visit_USub() implements the operator for unary -, as in “-x” by invoking the internal method _unary_op_impl() with an appropriate lambda function. See also visit_UnaryOp().

visit_UnaryOp(node)

visit_UnaryOp() implements the UnaryOps “-x”, “+x” and “not x”. The actual work is done in the operator visitor method, e.g. visit_UAdd or visit_USub.

Implementor Note:

This implementation doesn’t support calls within unaryOp expressions, so we don’t bother guarding for self.awaiting_call_return here, nor in the Unary Operator methods.

visit_alias(node)

visit_alias() processes an “as” alias node, which can only appear as part of an “from … import … as …” construct. The “as” clause is parsed, but ignored, and so should not be used in recipes. visit_alias() puts the name and asname on the _load stack as a tuple. visit_importFrom() handles the heavy lifting. Note: asname is currently not supported and is ignored.

visit_keyword(node)

visit_keyword() implements the syntax of keyword arguments (as opposed to positional arguments) within function calls. It does so by generating tuples of (keyword, value), generating the value by walking the corresponding AST subtree. The resulting tuple is stored on the _load stack, replacing the keyword name item.

exception kpfpipe.pipelines.kpf_parse_ast.RecipeError

RecipeError is raised whenever an error in the recipe or recipe processing is encountered. The most common reason for raising RecipeError is an error in the recipe itself, such as a typo or undefined variable or function name, or the use of syntactic elements from Python that are not supported in recipes.