1. API documentation¶

1.1. Library¶

1.1.1. pbs_api_to_cli¶

1.1.2. pbs_ifl_mock¶

1.1.3. pbs_testlib¶

class ptl.lib.pbs_testlib.PtlConfig(conf=None)¶

Holds configuration options The options can be stored in a file as well as in the OS environment variables.When set, the environment variables will override definitions in the file.By default, on Unix like systems, the file read is /etc/ptl.conf, the environment variable PTL_CONF_FILE can be used to set the path to the file to read.

The format of the file is a series of <key> = <value> properties. A line that starts with a ‘#’ is ignored and can be used for comments

Parameters:	conf (str or None) – Path to PTL configuration file

exception ptl.lib.pbs_testlib.PtlException(rv=None, rc=None, msg=None, post=None, *args, **kwargs)¶

Generic errors raised by PTL operations. Sets a return value, a return code, and a message A post function and associated positional and named arguments are available to perform any necessary cleanup.

Parameters:	rv (int or None.) – Return value set for the error occured during PTL operation rc (int or None.) – Return code set for the error occured during PTL operation msg (str or None.) – Message set for the error occured during PTL operation post (callable or None.) – Execute given post callable function if not None.
Raises:	PTL exceptions

exception ptl.lib.pbs_testlib.PtlFailureException(rv=None, rc=None, msg=None, post=None, *args, **kwargs)¶

Generic failure exception raised by PTL operations. Sets a return value, a return code, and a message A post function and associated positional and named arguments are available to perform any necessary cleanup.

Parameters:	rv (int or None.) – Return value set for the failure occured during PTL operation rc (int or None.) – Return code set for the failure occured during PTL operation msg (str or None.) – Message set for the failure occured during PTL operation post (callable or None.) – Execute given post callable function if not None.
Raises:	PTL exceptions

exception ptl.lib.pbs_testlib.PbsServiceError(rv=None, rc=None, msg=None, post=None, *args, **kwargs)¶

exception ptl.lib.pbs_testlib.PbsConnectError(rv=None, rc=None, msg=None, post=None, *args, **kwargs)¶

exception ptl.lib.pbs_testlib.PbsStatusError(rv=None, rc=None, msg=None, post=None, *args, **kwargs)¶

exception ptl.lib.pbs_testlib.PbsSubmitError(rv=None, rc=None, msg=None, post=None, *args, **kwargs)¶

exception ptl.lib.pbs_testlib.PbsManagerError(rv=None, rc=None, msg=None, post=None, *args, **kwargs)¶

exception ptl.lib.pbs_testlib.PbsDeljobError(rv=None, rc=None, msg=None, post=None, *args, **kwargs)¶

exception ptl.lib.pbs_testlib.PbsDelresvError(rv=None, rc=None, msg=None, post=None, *args, **kwargs)¶

exception ptl.lib.pbs_testlib.PbsDeleteError(rv=None, rc=None, msg=None, post=None, *args, **kwargs)¶

exception ptl.lib.pbs_testlib.PbsRunError(rv=None, rc=None, msg=None, post=None, *args, **kwargs)¶

exception ptl.lib.pbs_testlib.PbsSignalError(rv=None, rc=None, msg=None, post=None, *args, **kwargs)¶

exception ptl.lib.pbs_testlib.PbsMessageError(rv=None, rc=None, msg=None, post=None, *args, **kwargs)¶

exception ptl.lib.pbs_testlib.PbsHoldError(rv=None, rc=None, msg=None, post=None, *args, **kwargs)¶

exception ptl.lib.pbs_testlib.PbsReleaseError(rv=None, rc=None, msg=None, post=None, *args, **kwargs)¶

exception ptl.lib.pbs_testlib.PbsOrderError(rv=None, rc=None, msg=None, post=None, *args, **kwargs)¶

exception ptl.lib.pbs_testlib.PbsRerunError(rv=None, rc=None, msg=None, post=None, *args, **kwargs)¶

exception ptl.lib.pbs_testlib.PbsMoveError(rv=None, rc=None, msg=None, post=None, *args, **kwargs)¶

exception ptl.lib.pbs_testlib.PbsAlterError(rv=None, rc=None, msg=None, post=None, *args, **kwargs)¶

exception ptl.lib.pbs_testlib.PbsResourceError(rv=None, rc=None, msg=None, post=None, *args, **kwargs)¶

exception ptl.lib.pbs_testlib.PbsSelectError(rv=None, rc=None, msg=None, post=None, *args, **kwargs)¶

exception ptl.lib.pbs_testlib.PbsSchedConfigError(rv=None, rc=None, msg=None, post=None, *args, **kwargs)¶

exception ptl.lib.pbs_testlib.PbsMomConfigError(rv=None, rc=None, msg=None, post=None, *args, **kwargs)¶

exception ptl.lib.pbs_testlib.PbsFairshareError(rv=None, rc=None, msg=None, post=None, *args, **kwargs)¶

exception ptl.lib.pbs_testlib.PbsQdisableError(rv=None, rc=None, msg=None, post=None, *args, **kwargs)¶

exception ptl.lib.pbs_testlib.PbsQenableError(rv=None, rc=None, msg=None, post=None, *args, **kwargs)¶

exception ptl.lib.pbs_testlib.PbsQstartError(rv=None, rc=None, msg=None, post=None, *args, **kwargs)¶

exception ptl.lib.pbs_testlib.PbsQstopError(rv=None, rc=None, msg=None, post=None, *args, **kwargs)¶

exception ptl.lib.pbs_testlib.PtlExpectError(rv=None, rc=None, msg=None, post=None, *args, **kwargs)¶

exception ptl.lib.pbs_testlib.PbsInitServicesError(rv=None, rc=None, msg=None, post=None, *args, **kwargs)¶

exception ptl.lib.pbs_testlib.PbsQtermError(rv=None, rc=None, msg=None, post=None, *args, **kwargs)¶

exception ptl.lib.pbs_testlib.PtlLogMatchError(rv=None, rc=None, msg=None, post=None, *args, **kwargs)¶

exception ptl.lib.pbs_testlib.PbsResvAlterError(rv=None, rc=None, msg=None, post=None, *args, **kwargs)¶

class ptl.lib.pbs_testlib.PbsTypeSize(value=None)¶

Descriptor class for memory as a numeric entity. Units can be one of b, kb, mb, gb, tb, pt

Parameters:	unit (str) – The unit type associated to the memory value value (int or None) – The numeric value of the memory
Raises:	ValueError and TypeError

encode(value=None, valtype='kb', precision=1)¶

Encode numeric memory input in kilobytes to a string, including unit

Parameters:	value (int or None.) – The numeric value of memory to encode valtype (str) – The unit of the input value, defaults to kb precision (int) – Precision of the encoded value, defaults to 1
Returns:	Encoded memory in kb to string

class ptl.lib.pbs_testlib.PbsTypeDuration(val)¶

Descriptor class for a duration represented as hours, minutes, and seconds,in the form of [HH:][MM:]SS

Parameters:	as_seconds (int) – HH:MM:SS represented in seconds as_str (str) – duration represented in HH:MM:SS

class ptl.lib.pbs_testlib.PbsTypeArray(value=None, sep=', ')¶

Descriptor class for a PBS array list type, e.g. String array

Parameters:	value – Array value to be passed sep (str) – Separator for two array elements
Returns:	List

class ptl.lib.pbs_testlib.PbsTypeList(value=None, sep=', ', kvsep='=')¶

Descriptor class for a generic PBS list that are key/value pairs delimited

Parameters:	value – List value to be passed sep (str) – Separator for two key/value pair kvsep (str) – Separator for key and value
Returns:	Dictionary

class ptl.lib.pbs_testlib.PbsTypeLicenseCount(value=None)¶

Descriptor class for a PBS license_count attribute.

It is a specialized list where key/values are ‘:’ delimited, separated by a ‘ ‘ (space)

Parameters:	value – PBS license_count attribute value
Returns:	Specialized list

class ptl.lib.pbs_testlib.PbsTypeVariableList(value=None)¶

Descriptor class for a PBS Variable_List attribute

It is a specialized list where key/values are ‘=’ delimited, separated by a ‘,’ (space)

Parameters:	value – PBS Variable_List attribute value
Returns:	Specialized list

class ptl.lib.pbs_testlib.PbsTypeSelect(s=None)¶

Descriptor class for PBS select/schedselect specification. Select is of the form:

<select> ::= <m>":"<chunk> | <select>"+"<select>

<m> ::= <digit> | <digit><m>

<chunk> ::= <resc_name>":"<resc_value> | <chunk>":"<chunk>

<m> is a multiplying factor for each chunk requested

<chunk> are resource key/value pairs

The type populates a list of single chunk of resource key/value pairs, the list can be walked by iterating over the type itself.

Parameters:	num_chunks – The total number of chunks in the select resources (Dictionary) – A dictionary of all resource counts in the select

class ptl.lib.pbs_testlib.PbsTypeChunk(vnode=None, resources=None, chunkstr=None)¶

Descriptor class for a PBS chunk associated to a PbsTypeExecVnode.This type of chunk corresponds to a node solution to a resource request,not to the select specification.

chunk ::= <subchk> | <chunk>"+"<chunk>

subchk ::= <node>":"<resource>

resource ::= <key>":"<val> | <resource>":"<resource>

A chunk expresses a solution to a specific select-chunk request. If multiple chunks are needed to solve a single select-chunk, e.g., on a shared memory system, the chunk will be extended into virtual chunk,vchunk.

Parameters:	vnode (str or None) – the vnode name corresponding to the chunk resources (Dictionary or None) – the key value pair of resources in dictionary form vchunk (list) – a list of virtual chunks needed to solve the select-chunk, vchunk is only set if more than one vchunk are required to solve the select-chunk

add(vnode, resources)¶

Add a chunk specificiation. If a chunk is already defined, add the chunk as a vchunk.

Parameters:	vnode (str) – The vnode to add resources (str) – The resources associated to the vnode
Returns:	Added chunk specification

class ptl.lib.pbs_testlib.PbsTypeExecVnode(s=None)¶

Execvnode representation, expressed as a list of PbsTypeChunk

Parameters:	vchunk (List) – List of virtual chunks, only set when more than one vnode is allocated to a host satisfy a chunk requested num_chunks (int) – The number of chunks satisfied by this execvnode vnodes (List) – List of vnode names allocated to the execvnode resource – method to return the amount of a named resource satisfied by this execvnode

resource(name=None)¶

Parameters:	name (str or None) – Name of the resource

class ptl.lib.pbs_testlib.PbsTypeExecHost(s=None)¶

Descriptor class for exec_host attribute

Parameters:	hosts (List) – List of hosts in the exec_host. Each entry is a host info dictionary that maps the number of cpus and its task number

class ptl.lib.pbs_testlib.PbsTypeJobId(value=None)¶

Descriptor class for a Job identifier

Parameters:	id (int) – The numeric portion of a job identifier server_name (str) – The pbs server name server_shortname (str) – The first portion of a FQDN server name

class ptl.lib.pbs_testlib.PbsUser(name, uid=None, groups=None)¶

The PbsUser type augments a PBS username to associate it to groups to which the user belongs

Parameters:	name (str) – The user name referenced uid (int or None) – uid of user groups (List or None) – The list of PbsGroup objects the user belongs to

class ptl.lib.pbs_testlib.PbsGroup(name, gid=None, users=None)¶

The PbsGroup type augments a PBS groupname to associate it to users to which the group belongs

Parameters:	name (str) – The group name referenced gid (int or None) – gid of group users (List or None) – The list of PbsUser objects the group belongs to

class ptl.lib.pbs_testlib.BatchUtils¶

Utility class to create/convert/display various PBS data structures

list_to_attrl(l)¶

Convert a list to a PBS attribute list

Parameters:	l (List) – List to be converted
Returns:	PBS attribute list

list_to_attropl(l, op=0)¶

Convert a list to a PBS attribute operation list

Parameters:	l (List) – List to be converted
Returns:	PBS attribute operation list

str_to_attrl(s)¶

Convert a string to a PBS attribute list

Parameters:	s (str) – String to be converted
Returns:	PBS attribute list

str_to_attropl(s, op=0)¶

Convert a string to a PBS attribute operation list

Parameters:	s (str) – String to be converted
Returns:	PBS attribute operation list

dict_to_attrl(d={})¶

Convert a dictionary to a PBS attribute list

Parameters:	d (Dictionary) – Dictionary to be converted
Returns:	PBS attribute list

dict_to_attropl(d={}, op=0)¶

Convert a dictionary to a PBS attribute operation list

Parameters:	d (Dictionary) – Dictionary to be converted
Returns:	PBS attribute operation list

convert_to_attrl(attrib)¶

Generic call to convert Python type to PBS attribute list

Parameters:	attrib (List or tuple or dictionary or str) – Attributes to be converted
Returns:	PBS attribute list

convert_to_attropl(attrib, cmd=13, op=None)¶

Generic call to convert Python type to PBS attribute operation list

Parameters:	attrib (List or tuple or dictionary or str) – Attributes to be converted
Returns:	PBS attribute operation list

command_to_op(cmd=None)¶

Map command to a SET or UNSET Operation. An unrecognized command will return SET. No command will return None.

Parameters:	cmd (str) – Command to be mapped
Returns:	`SET` or `UNSET` operation for the command

display_attrl(a=None, writer=<open file '<stdout>', mode 'w'>)¶

Display an attribute list using writer, defaults to sys.stdout

Parameters:	a (List) – Attributes
Returns:	Displays attribute list

display_attropl(attropl=None, writer=<open file '<stdout>', mode 'w'>)¶

Display an attribute operation list with writer, defaults to sys.stdout

Parameters:	attropl (List) – Attribute operation list
Returns:	Displays an attribute operation list

display_dict(d, writer=<open file '<stdout>', mode 'w'>)¶

Display a dictionary using writer, defaults to sys.stdout

Parameters:	d (Dictionary) – Dictionary
Returns:	Displays a dictionary

batch_status_to_dictlist(bs=None, attr_names=None, id=None)¶

Convert a batch status to a list of dictionaries. version 0.1a6 added this conversion as a typemap(out) as part of the swig wrapping itself so there are fewer uses for this function.Returns a list of dictionary representation of batch status

Parameters:	bs – Batch status attr_names – Attribute names
Returns:	List of dictionaries

display_batch_status(bs=None, attr_names=None, writer=<open file '<stdout>', mode 'w'>)¶: Display a batch status using writer, defaults to sys.stdout :param bs: Batch status :param attr_name: Attribute name :type attr_name: str :returns: Displays batch status

display_dictlist(l=[], writer=<open file '<stdout>', mode 'w'>, fmt=None)¶

Display a list of dictionaries using writer, defaults to sys.stdout

Parameters:	l (List) – The list to display writer – The stream on which to write fmt (str or None) – An optional formatting string
Returns:	Displays list of dictionaries

dictlist_to_file(l=[], filename=None, mode='w')¶

write a dictlist to file

Parameters:	l (List) – Dictlist filename (str) – File to which dictlist need to be written mode (str) – Mode of file
Raises:	Exception writing to file

batch_status_as_dictlist_to_file(l=[], writer=<open file '<stdout>', mode 'w'>)¶

Write a dictlist to file

Parameters:	l (List) – Dictlist
Raises:	Exception writing to file

file_to_dictlist(file=None, attribs=None, id=None)¶

Convert a file to a batch dictlist format

Parameters:	file (str) – File to be converted attribs – Attributes
Returns:	File converted to a batch dictlist format

file_to_vnodedef(file=None)¶

Convert a file output of pbsnodes -av to a vnode definition format

Parameters:	file – File to be converted
Returns:	Vnode definition format

show(l=[], name=None, fmt=None)¶

Alias to display_dictlist with sys.stdout as writer

Parameters:	name (str) – if specified only show the object of that name fmt (str) – Optional formatting string, uses %n for object name, %a for attributes, for example a format of ‘%nE{}nE{}t%aE{}n’ will display objects with their name starting on the first column, a new line, and attributes indented by a tab followed by a new line at the end.

get_objtype(d={})¶

Get the type of a given object

Parameters:	d (Dictionary) – Dictionary
Returns:	Type of the object

display_batch_status_as_dictlist(l=[], writer=<open file '<stdout>', mode 'w'>, fmt=None)¶

Display a batch status as a list of dictionaries using writer, defaults to sys.stdout

Parameters:	l (List) – List fmt (str or None) – Optional format string
Returns:	Displays batch status as a list of dictionaries

batch_status_as_dict_to_str(d={}, fmt=None)¶

Return a string representation of a batch status dictionary

Parameters:	d (Dictionary) – Dictionary fmt (str or None) – Optional format string
Returns:	String representation of a batch status dictionary

display_batch_status_as_dict(d={}, writer=<open file '<stdout>', mode 'w'>, fmt=None)¶

Display a dictionary representation of a batch status using writer, defaults to sys.stdout

Parameters:	d (Dictionary) – Dictionary fmt – Optional format string fmt – str
Returns:	Displays dictionary representation of a batch status

decode_dictlist(l=None, json=True)¶

decode a list of dictionaries

Parameters:	l (List) – List of dictionaries json – The target of the decode is meant for `JSON` formatting
Returns:	Decoded list of dictionaries

convert_to_dictlist(l, attribs=None, mergelines=True, id=None)¶

Convert a list of records into a dictlist format.

Parameters:	l (List) – array of records to convert mergelines – merge qstat broken lines into one
Returns:	Record list converted into dictlist format

convert_to_batch(l, mergelines=True)¶

Convert a list of records into a batch format.

Parameters:	l (List) – array of records to convert mergelines (bool) – qstat breaks long lines over multiple lines, merge them to one by default.
Returns:	A linked list of batch status

file_to_batch(file=None)¶

Convert a file to batch format

Parameters:	file (str or None) – File to be converted
Returns:	File converted into batch format

batch_to_file(bs=None, file=None)¶

Write a batch object to file

Parameters:	bs – Batch status file (str) – File to which batch object is to be written

batch_to_vnodedef(bs)¶

Parameters:	bs – Batch status
Returns:	The vnode definition string representation of nodes batch_status

dictlist_to_vnodedef(dl=None)¶

Parameters:	dl (List) – Dictionary list
Returns:	The vnode definition string representation of a dictlist

objlist_to_dictlist(objlist=None)¶

Convert a list of PBS/PTL objects (e.g. Server/Job...) into a dictionary list representation of the batch status

Parameters:	objlist (List) – List of `PBS/PTL` objects
Returns:	Dictionary list representation of the batch status

obj_to_dict(obj)¶

Convert a PBS/PTL object (e.g. Server/Job…) into a dictionary format

Parameters:	obj – `PBS/PTL` object
Returns:	Dictionary of `PBS/PTL` objects

parse_execvnode(s=None)¶

Parse an execvnode string into chunk objects

Parameters:	s (str or None) – Execvnode string
Returns:	Chunk objects for parsed execvnode string

anupbs_exechost_numhosts(s=None)¶

Parameters:	s (str or None) – Exechost string

parse_exechost(s=None)¶

Parse an exechost string into a dictionary representation

Parameters:	s (str or None) – String to be parsed
Returns:	Dictionary format of the exechost string

parse_select(s=None)¶

Parse a select/schedselect string into a list of dictionaries.

Parameters:	s (str or None) – select/schedselect string
Returns:	List of dictonaries

classmethod isfloat(value)¶

returns true if value is a float or a string representation of a float returns false otherwise

Parameters:	value (str or int or float) – value to be checked
Returns:	True or False

classmethod decode_value(value)¶

Decode an attribute/resource value, if a value is made up of digits only then return the numeric value of it, if it is made of alphanumeric values only, return it as a string, if it is of type size, i.e., with a memory unit such as b,kb,mb,gb then return the converted size to kb without the unit

Parameters:	value (str or int) – attribute/resource value
Returns:	int or float or string

convert_time(val, fmt='%a %b %d %H:%M:%S %Y')¶

Convert a date time format into number of seconds since epoch

Parameters:	val – date time value fmt (str) – date time format
Returns:	seconds

convert_duration(val)¶

Convert HH:MM:SS into number of seconds If a number is fed in, that number is returned If neither formatted data is fed in, returns 0

Parameters:	val (str) – duration value
Raises:	Incorrect format error
Returns:	seconds

convert_seconds_to_datetime(tm, fmt=None, seconds=True)¶

Convert time format to number of seconds since epoch

Parameters:	tm (str) – the time to convert fmt (str or None) – optional format string. If used, the seconds parameter is ignored.Defaults to `%Y%m%d%H%M` seconds (bool) – if True, convert time with seconds granularity. Defaults to True.
Returns:	Number of seconds

convert_stime_to_seconds(st)¶

Convert a time to seconds, if we fail we return the original time

Parameters:	st (str) – Time to be converted
Returns:	Number of seconds

convert_dedtime(dtime)¶

Convert dedicated time string of form %m/%d/%Y %H:%M.

Parameters:	dtime (str) – A datetime string, as an entry in the dedicated_time file
Returns:	A tuple of (from,to) of time since epoch

convert_datetime_to_epoch(mdyhms, fmt='%m/%d/%Y %H:%M:%S')¶

Convert the date time to epoch

Parameters:	mdyhms (str) – date time fmt (str) – Format for date time
Returns:	Epoch time

compare_versions(v1, v2, op=None)¶

Compare v1 to v2 with respect to operation op

Parameters:	v1 – If not a looseversion, it gets converted to it v2 – If not a looseversion, it gets converted to it op (str) – An operation, one of `LT`, `LE`, `EQ`, `GE`, `GT`
Returns:	True or False

convert_arglist(attr)¶

strip the XML attributes from the argument list attribute

Parameters:	attr (List) – Argument list attributes
Returns:	Stripped XML attributes

convert_to_cli(attrs, op=None, hostname=None, dflt_conf=True, exclude_attrs=None)¶

Convert attributes into their CLI format counterpart. This method is far from complete, it grows as needs come by and could use a rewrite, especially going along with a rewrite of pbs_api_to_cli

Parameters:

attrs (List or str or dictionary) – Attributes to convert
op (str or None) – The qualifier of the operation being performed, such as IFL_SUBMIT, IFL_DELETE, IFL_TERMINUTE…
hostname (str or None) – The name of the host on which to operate
dflt_conf (bool) – Whether we are using the default PBS configuration
exclude_attrs (List) – Optional list of attributes to not convert

Returns:

CLI format of attributes

filter_batch_status(bs, attrib)¶

Filter out elements that don’t have the attributes requested This is needed to adapt to the fact that requesting a resource attribute returns all '<resource-name>.*' attributes so we need to ensure that the specific resource requested is present in the stat’ed object.

This is needed especially when calling expect with an op=NE because we need to filter on objects that have exactly the attributes requested

Parameters:	bs – Batch status attrib (str or dictionary) – Requested attributes
Returns:	Filtered batch status

convert_attributes_by_op(attributes, setattrs=False)¶

Convert attributes by operator, i.e. convert an attribute of the form

<attr_name><op><value> (e.g. resources_available.ncpus>4)

to

<attr_name>: (<op>, <value>) (e.g. resources_available.ncpus: (GT, 4))

Parameters:	attributes (List) – the attributes to convert setattrs (bool) – if True, set the attributes with no operator as (SET, ‘’)
Returns:	Converted attributes by operator

operator_in_attribute(attrib)¶

Returns True if an operator string is present in an attribute name

Parameters:	attrib (str) – Attribute name
Returns:	True or False

list_resources(objtype=None, objs=[])¶

Lists the resources

Parameters:	objtype (str) – Type of the object objs (List) – Object list
Returns:	List of resources

compare(obj1, obj2, showdiff=False)¶

Compare two objects.

Parameters:	showdiff (bool) – whether to print the specific differences, defaults to False
Returns:	0 if objects are identical and non zero otherwise

classmethod random_str(length=1, prefix='')¶

Generates the random string

Parameters:	length (int) – Length of the string prefix (str) – Prefix of the string
Returns:	Random string

update_attributes_list(obj)¶

Updates the attribute list

Parameters:	obj – Objects
Returns:	Updated attribute list

parse_fgc_limit(limstr=None)¶

Parse an FGC limit entry, of the form:

<limtype>[.<resource>]=\[<entity_type>:<entity_name> =<entity_value>\]

Parameters:	limstr (str or None) – FGC limit string
Returns:	Parsed FGC string in given format

is_job_array(jobid)¶

If a job array return True, otherwise return False

Parameters:	jobid – PBS jobid
Returns:	True or False

is_subjob(jobid)¶

If a subjob of a job array, return the subjob id otherwise return False

Parameters:	jobid (str) – PBS job id
Returns:	True or False

class ptl.lib.pbs_testlib.PbsTypeFGCLimit(attr, val)¶

FGC limit entry, of the form:

<limtype>[.<resource>]=\[<entity_type>:<entity_name>= <entity_value>\]

Parameters:	attr (str) – FGC limit attribute value (int) – Value of attribute
Returns:	FGC limit entry of given format

class ptl.lib.pbs_testlib.PbsBatchStatus(bs)¶

Wrapper class for Batch Status object Converts a batch status (as dictlist) into a list of PbsBatchObjects

Parameters:	bs (List or dictionary) – Batch status
Returns:	List of PBS batch objects

class ptl.lib.pbs_testlib.PbsBatchObject(bs)¶

set_batch_status(bs)¶

Sets the batch status

Parameters:	bs – Batch status

class ptl.lib.pbs_testlib.PbsAttribute(name=None, value=None)¶

Descriptor class for PBS attribute

Parameters:	name (str) – PBS attribute name value (str or int or float) – Value for the attribute

set_name(name)¶

Set PBS attribute name

Parameters:	name (str) – PBS attribute

set_value(value)¶

Set PBS attribute value

Parameters:	value (str or int or float) – Value of PBS attribute

obfuscate_name(a=None)¶: Obfuscate PBS attribute name

obfuscate_value(v=None)¶: Obfuscate PBS attribute value

class ptl.lib.pbs_testlib.Entity(etype=None, name=None)¶

Abstract representation of a PBS consumer that has an external relationship to the PBS system. For example, a user associated to an OS identifier (uid) maps to a PBS user entity.

Entities may be subject to policies, such as limits, consume a certain amount of resource and/or fairshare usage.

Parameters:	etype (str or None) – Entity type name (str or None) – Entity name

set_limit(limit=None)¶

Parameters:	limit (str or None) – Limit to be set

set_resource_usage(container=None, resource=None, usage=None)¶

Set the resource type

Parameters:	resource (str or None) – PBS resource usage (str or None) – Resource usage value

set_fairshare_usage(usage=0)¶

Set fairshare usage

Parameters:	usage (int) – Fairshare usage value

class ptl.lib.pbs_testlib.Policy¶: Abstract PBS policy. Can be one of limits, access control, scheduling policy, etc…this class does not currently support any operations

class ptl.lib.pbs_testlib.Limit(limit_type=None, resource=None, entity_obj=None, value=None, container=None, container_id=None)¶

Representation of a PBS limit Limits apply to containers, are of a certain type (e.g., max_run_res.ncpus) associated to a given resource (e.g., resource), on a given entity (e.g.,user Bob) and have a certain value.

Parameters:	limit_type (str or None) – Type of the limit resource (str or None) – PBS resource entity_obj – Entity object value (int) – Limit value

set_container(container, container_id)¶

Set the container

Parameters:	container (str) – Container which is to be set container_id – Container id

set_limit_type(t)¶

Set the limit type

Parameters:	t (str) – Limit type

set_resource(resource)¶

Set the resource

Parameters:	resource (str) – resource value to set

set_value(value)¶

Set the resource value

Parameters:	value (str) – Resource value

class ptl.lib.pbs_testlib.ExpectActions(action=None, level=20)¶

List of action handlers to run when Server’s expect function does not get the expected result

Parameters:	action (str) – Action to run level – Logging level

add_action(action=None, hostname=None, level=20)¶

Add an action

Parameters:	action – Action to add hostname (str) – Machine hostname level – Logging level

has_action(name)¶

check whether action exists or not

Parameters:	name (str) – Name of action

get_action(name)¶

Get an action if exists

Parameters:	name (str) – Name of action

list_actions(level=20)¶

List an actions

Parameters:	level – Logging level

get_all_actions()¶: Get all the action

get_actions_by_type(atype=None)¶

Get an action by type

Parameters:	atype (str) – Action type

disable_action(action=None, name=None)¶: Disable an action

enable_action(action=None, name=None)¶: Enable an action

disable_all_actions()¶: Disable all actions

enable_all_actions()¶: Enable all actions

class ptl.lib.pbs_testlib.ExpectAction(name=None, enabled=True, atype=None, action=None, level=20)¶

Action function to run when Server’s expect function does not get the expected result

Parameters:	atype (str) – Action type

set_name(name, level=20)¶

Set the actione name

Parameters:	name (str) – Action name

class ptl.lib.pbs_testlib.PbsTypeAttribute¶: Experimental. This is a placeholder object that will be used in the future to map attribute information and circumvent the error-pron dynamic type detection that is currently done using decode_value()

class ptl.lib.pbs_testlib.PBSObject(name, attrs={}, defaults={})¶

Generic PBS Object encapsulating attributes and defaults

Parameters:	name (str) – The name associated to the object attrs (Dictionary) – Dictionary of attributes to set on object defaults (Dictionary) – Dictionary of default attributes. Setting this will override any other object’s default

set_attributes(a={})¶

set attributes and custom attributes on this object. custom attributes are used when converting attributes to CLI

Parameters:	a (Dictionary) – Attribute dictionary

unset_attributes(attrl=[])¶

Unset attributes from object’s attributes and custom attributes

Parameters:	attrl (List) – Attribute list

class ptl.lib.pbs_testlib.PBSService(name=None, attrs={}, defaults={}, pbsconf_file=None, diagmap={}, diag=None)¶

Generic PBS service object to hold properties of PBS daemons

Parameters:

name (str or None) – The name associated to the object
attrs (Dictionary) – Dictionary of attributes to set on object
defaults (Dictionary) – Dictionary of default attributes. Setting this will override any other object’s default
pbsconf_file (str or None) – Optional path to the pbs configuration file
diagmap (Dictionary) – A dictionary of PBS objects (node,server,etc) to mapped files from PBS diag directory
diag (str or None) – path to PBS diag directory (This will overrides diagmap)

init_logfile_path(conf=None)¶

Initialize path to log files for this service

Parameters:	conf (Dictionary) – PBS conf file parameters

initialise_service()¶: Purpose of this method is to override and initialise the service

log_lines(logtype, id=None, n=50, tail=True, starttime=None, endtime=None)¶

Return the last <n> lines of a PBS log file, which can be one of server, scheduler, MoM, or tracejob

Parameters:

logtype (str or object) – The entity requested, an instance of a Scheduler, Server or MoM object, or the string ‘tracejob’ for tracejob
id – The id of the object to trace. Only used for tracejob
n (str or int) – One of ‘ALL’ of the number of lines to process/display, defaults to 50.
tail (bool) – if True, parse log from the end to the start, otherwise parse from the start to the end. Defaults to True.
day (int) – Optional day in YYYMMDD format. Defaults to current day
starttime – date timestamp to start matching
endtime – date timestamp to end matching

Returns:

Last <n> lines of logfile for Server, Scheduler, MoM or tracejob

accounting_match(msg, id=None, n=50, tail=True, allmatch=False, regexp=False, max_attempts=None, interval=None, starttime=None, endtime=None, level=20, existence=True)¶

Match given msg in given n lines of accounting log

Parameters:	msg (str) – log message to match, can be regex also when `regexp` is True id (str) – The id of the object to trace. Only used for tracejob n (str or int) – ‘ALL’ or the number of lines to search through, defaults to 50 tail (bool) – If true (default), starts from the end of the file allmatch (bool) – If True all matching lines out of then parsed are returned as a list. Defaults to False regexp (bool) – If true msg is a Python regular expression. Defaults to False max_attempts (int) – the number of attempts to make to find a matching entry interval (int) – the interval between attempts starttime (int) – If set ignore matches that occur before specified time endtime (int) – If set ignore matches that occur after specified time level (int) – The logging level, defaults to INFO existence (bool) – If True (default), check for existence of given msg, else check for non-existence of given msg.
Returns:	(x,y) where x is the matching line number and y the line itself. If allmatch is True, a list of tuples is returned.
Return type:	tuple
Raises:	PtlLogMatchError – When `existence` is True and given `msg` is not found in `n` line Or When `existence` is False and given `msg` found in `n` line.

Note

The matching line number is relative to the record number, not the absolute line number in the file.

tracejob_match(msg, id=None, n=50, tail=True, allmatch=False, regexp=False, max_attempts=None, interval=None, starttime=None, endtime=None, level=20, existence=True)¶

Match given msg in given n lines of tracejob log

Parameters:	msg (str) – log message to match, can be regex also when `regexp` is True id (str) – The id of the object to trace. n (str or int) – ‘ALL’ or the number of lines to search through, defaults to 50 tail (bool) – If true (default), starts from the end of the file allmatch (bool) – If True all matching lines out of then parsed are returned as a list. Defaults to False regexp (bool) – If true msg is a Python regular expression. Defaults to False max_attempts (int) – the number of attempts to make to find a matching entry interval (int) – the interval between attempts starttime (int) – If set ignore matches that occur before specified time endtime (int) – If set ignore matches that occur after specified time level (int) – The logging level, defaults to INFO existence (bool) – If True (default), check for existence of given msg, else check for non-existence of given msg.
Returns:	(x,y) where x is the matching line number and y the line itself. If allmatch is True, a list of tuples is returned.
Return type:	tuple
Raises:	PtlLogMatchError – When `existence` is True and given `msg` is not found in `n` line Or When `existence` is False and given `msg` found in `n` line.

Note

The matching line number is relative to the record number, not the absolute line number in the file.

get_tempdir()¶: platform independent call to get a temporary directory

cleanup_files()¶: This function removes any dynamic resource files created by server/mom objects

class ptl.lib.pbs_testlib.Comm(name=None, attrs={}, pbsconf_file=None, diagmap={}, diag=None, server=None, db_access=None)¶

PBS Comm configuration and control

isUp()¶: Check for comm up

signal(sig)¶: Send signal to comm

get_pid()¶: Get the comm pid

all_instance_pids()¶: Get all pids of given instance

start(args=None, launcher=None)¶

Start the comm

Parameters:	args (str) – Argument required to start the comm launcher (str or list) – Optional utility to invoke the launch of the service

stop(sig=None)¶

Stop the comm.

Parameters:	sig (str) – Signal to stop the comm

restart()¶: Restart the comm.

log_match(msg=None, id=None, n=50, tail=True, allmatch=False, regexp=False, max_attempts=None, interval=None, starttime=None, endtime=None, level=20, existence=True)¶

Match given msg in given n lines of Comm log

Parameters:	msg (str) – log message to match, can be regex also when `regexp` is True id (str) – The id of the object to trace. Only used for tracejob n (str or int) – ‘ALL’ or the number of lines to search through, defaults to 50 tail (bool) – If true (default), starts from the end of the file allmatch (bool) – If True all matching lines out of then parsed are returned as a list. Defaults to False regexp (bool) – If true msg is a Python regular expression. Defaults to False max_attempts (int) – the number of attempts to make to find a matching entry interval (int) – the interval between attempts starttime (int) – If set ignore matches that occur before specified time endtime (int) – If set ignore matches that occur after specified time level (int) – The logging level, defaults to INFO existence (bool) – If True (default), check for existence of given msg, else check for non-existence of given msg.
Returns:	(x,y) where x is the matching line number and y the line itself. If allmatch is True, a list of tuples is returned.
Return type:	tuple
Raises:	PtlLogMatchError – When `existence` is True and given `msg` is not found in `n` line Or When `existence` is False and given `msg` found in `n` line.

Note

The matching line number is relative to the record number, not the absolute line number in the file.

class ptl.lib.pbs_testlib.Server(name=None, attrs={}, defaults={}, pbsconf_file=None, diagmap={}, diag=None, client=None, client_pbsconf_file=None, db_access=None, stat=True)¶

PBS server configuration and control

The Server class is a container to PBS server attributes and implements wrappers to the IFL API to perform operations on the server. For example to submit, status, delete, manage, etc… jobs, reservations and configurations.

This class also offers higher-level routines to ease testing, see functions, for example: revert_to_defaults, init_logging, expect, counter.

The ptl_conf dictionary holds general configuration for the framework’s operations, specifically, one can control:

mode: set to PTL_CLI to operate in CLI mode or PTL_API to operate in API mode

expect_max_attempts: the default maximum number of attempts to be usedby expect. Defaults to 60

expect_interval: the default time interval (in seconds) between expectrequests. Defaults to 0.5

update_attributes: the default on whether Object attributes should beupdated using a list of dictionaries. Defaults to True

Parameters:

name (str) – The hostname of the server. Defaults to calling pbs_default()
attrs (Dictionary) – Dictionary of attributes to set, these will override defaults.
defaults (Dictionary) – Dictionary of default attributes. Default: dflt_attributes
pbsconf_file (str) – path to config file to parse for PBS_HOME, PBS_EXEC, etc
diagmap (Dictionary) – A dictionary of PBS objects (node,server,etc) to mapped files from PBS diag directory
diag (str) – path to PBS diag directory (This will overrides diagmap)
client (str) – The host to use as client for CLI queries. Defaults to the local hostname.
client_pbsconf_file (str) – The path to a custom PBS_CONF_FILE on the client host. Defaults to the same path as pbsconf_file.
db_acccess – set to either file containing credentials to DB access or dictionary containing {‘dbname’:…,’user’:…,’port’:…}
stat (bool) – if True, stat the server attributes

update_version_info()¶: Update the version information.

classmethod set_update_attributes(val)¶: Set update attributes

classmethod set_expect_max_attempts(val)¶: Set expect max attempts

classmethod set_expect_interval(val)¶: Set expect interval

set_client(name=None)¶

Set server client

Parameters:	name (str) – Client name

set_connect_timeout(timeout=0)¶

Set server connection timeout

Parameters:	timeout (int) – Timeout value

get_op_mode()¶: Returns operating mode for calls to the PBS server. Currently, two modes are supported, either the API or the CLI. Default is API

set_op_mode(mode)¶

set operating mode to one of either PTL_CLI or PTL_API.Returns the mode that was set which can be different from the value requested, for example, if requesting to set PTL_API, in the absence of the appropriate SWIG wrappers, the library will fall back to CLI, or if requesting PTL_CLI and there is no PBS_EXEC on the system, None is returned.

Parameters:	mode (str) – Operating mode

add_expect_action(name=None, action=None)¶

Add an action handler to expect. Expect Actions are custom handlers that are triggered when an unexpected value is encountered

Parameters:	name (str or None) – Action name action – Action to add

set_attributes(a={})¶

set server attributes

Parameters:	a (Dictionary) – Attribute dictionary

isUp()¶: returns True if server is up and False otherwise

signal(sig)¶

Send signal to server

Parameters:	sig (str) – Signal to send

get_pid()¶: Get the server pid

all_instance_pids()¶: Get all pids for a given instance

start(args=None, launcher=None)¶

Start the PBS server

Parameters:	args (str) – Argument required to start the server launcher (str or list) – Optional utility to invoke the launch of the service

stop(sig=None)¶

Stop the PBS server

Parameters:	sig (str) – Signal to stop PBS server

restart()¶: Terminate and start a PBS server.

log_match(msg=None, id=None, n=50, tail=True, allmatch=False, regexp=False, max_attempts=None, interval=None, starttime=None, endtime=None, level=20, existence=True)¶

Match given msg in given n lines of Server log

Parameters:	msg (str) – log message to match, can be regex also when `regexp` is True id (str) – The id of the object to trace. Only used for tracejob n (str or int) – ‘ALL’ or the number of lines to search through, defaults to 50 tail (bool) – If true (default), starts from the end of the file allmatch (bool) – If True all matching lines out of then parsed are returned as a list. Defaults to False regexp (bool) – If true msg is a Python regular expression. Defaults to False max_attempts (int) – the number of attempts to make to find a matching entry interval (int) – the interval between attempts starttime (int) – If set ignore matches that occur before specified time endtime (int) – If set ignore matches that occur after specified time level (int) – The logging level, defaults to INFO existence (bool) – If True (default), check for existence of given msg, else check for non-existence of given msg.
Returns:	(x,y) where x is the matching line number and y the line itself. If allmatch is True, a list of tuples is returned.
Return type:	tuple
Raises:	PtlLogMatchError – When `existence` is True and given `msg` is not found in `n` line Or When `existence` is False and given `msg` found in `n` line.

Note

The matching line number is relative to the record number, not the absolute line number in the file.

revert_to_defaults(reverthooks=True, revertqueues=True, revertresources=True, delhooks=True, delqueues=True, delscheds=True, delnodes=True, server_stat=None)¶

reset server attributes back to out of box defaults.

Parameters:	reverthooks (bool) – If True disable all hooks. Defaults to True revertqueues (bool) – If True disable all non-default queues. Defaults to True revertresources (bool) – If True, resourcedef file is removed. Defaults to True. Reverting resources causes a server restart to occur. delhooks (bool) – If True, hooks are deleted, if deletion fails, fall back to reverting hooks. Defaults to True. delqueues (bool) – If True, all non-default queues are deleted, will attempt to delete all jobs first, if it fails, revertqueues will be honored, otherwise,revertqueues is ignored. Defaults to True delscheds (bool) – If True all non-default schedulers are deleted The sched_priv and sched_logs directories will be deleted. delnodes (bool) – If True all vnodes are deleted
Returns:	True upon success and False if an error is encountered.
Raises:	PbsStatusError or PbsManagerError

save_configuration(outfile, mode='a')¶

Save a server configuration, this includes:

server_priv/resourcedef

qmgr -c "print server"

qmgr -c "print sched"

qmgr -c "print hook"

Parameters:	outfile (str) – the output file to which onfiguration is saved mode (str) – The mode in which to open outfile to save configuration. The first object being saved should open this file with ‘w’ and subsequent calls from other objects should save with mode ‘a’ or ‘a+’. Defaults to a+
Returns:	True on success, False on error

load_configuration(infile)¶: load configuration from saved file infile

get_hostname()¶: return the default server hostname

status_db(obj_type=None, attrib=None, id=None, db_access=None, logerr=True)¶

Status PBS objects from the SQL database

Parameters:

obj_type – The type of object to query, one of the * objects,Default: SERVER
attrib (str or list or dictionary) – Attributes to query, can a string, a list, a dictionaryDefault: None. All attributes will be queried
id (str) – An optional identifier, the name of the object to status
db_access (str or dictionary) – information needed to access the database, can be either a file containing user, port, dbname, password info or a dictionary of key/value entries

status(obj_type=0, attrib=None, id=None, extend=None, level=20, db_access=None, runas=None, resolve_indirectness=False, logerr=True)¶

Stat any PBS object [queue, server, node, hook, job, resv, sched].If the Server is setup from diag input, see diag or diagmap member, the status calls are routed directly to the data on files from diag.

The server can be queried either through the ‘qstat’ command line tool or through the wrapped PBS IFL api, see set_op_mode.

Return a dictionary representation of a batch status object raises PbsStatsuError on error.

Parameters:

obj_type – The type of object to query, one of the * objects.Default: SERVER
attrib (str or list or dictionary) – Attributes to query, can be a string, a list, a dictionary.Default is to query all attributes.
id (str) – An optional id, the name of the object to status
extend – Optional extension to the IFL call
level (str) – The logging level, defaults to INFO
db_acccess – set to either file containing credentials to DB access or dictionary containing {'dbname':...,'user':...,'port':...}
runas (str) – run stat as user
resolve_indirectness (bool) – If True resolves indirect node resources values
logerr (bool) – If True (default) logs run_cmd errors

In addition to standard IFL stat call, this wrapper handles a few cases that aren’t implicitly offered by pbs_stat*, those are for Hooks,Resources, and a formula evaluation.

submit_interactive_job(job, cmd)¶

submit an interactive job. Returns a job identifier or raises PbsSubmitError on error

Parameters:

cmd (str) – The command to run to submit the interactive job
job – the job object. The job must have the attribute ‘interactive_job’ populated. That attribute is a list of tuples of the form: (<command>, <expected output>, <…>) for example to send the command hostname and expect ‘myhost.mydomain’ one would set:job.interactive_job = [(‘hostname’, ‘myhost.mydomain’)] If more than one lines are expected they are appended to the tuple.

Raises:

PbsSubmitError

submit(obj, script=None, extend=None, submit_dir=None)¶

Submit a job or reservation. Returns a job identifier or raises PbsSubmitError on error

Parameters:	obj – The Job or Reservation instance to submit script (str or None) – Path to a script to submit. Default: None as an executable/bin/sleep 100 is submitted extend (str or None) – Optional extension to the IFL call. see pbs_ifl.h submit_dir (str or None) – directory from which job is submitted. Defaults to temporary directory
Raises:	PbsSubmitError

deljob(id=None, extend=None, runas=None, wait=False, logerr=True, attr_W=None)¶

delete a single job or list of jobs specified by id raises PbsDeljobError on error

Parameters:

id (str or list) – The identifier(s) of the jobs to delete
extend (str or None) – Optional parameters to pass along to PBS
runas (str or None) – run as user
wait (bool) – Set to True to wait for job(s) to no longer be reported by PBS. False by default
logerr (bool) – Whether to log errors. Defaults to True.
attr_w (str) – -W args to qdel (Only for cli mode)

Raises:

PbsDeljobError

delresv(id=None, extend=None, runas=None, wait=False, logerr=True)¶

delete a single job or list of jobs specified by id raises PbsDeljobError on error

Parameters:	id (str or list) – The identifier(s) of the jobs to delete extend (str or None) – Optional parameters to pass along to PBS runas (str or None) – run as user wait (bool) – Set to True to wait for job(s) to no longer be reported by PBS. False by default logerr (bool) – Whether to log errors. Defaults to True.
Raises:	PbsDeljobError

delete(id=None, extend=None, runas=None, wait=False, logerr=True)¶

delete a single job or list of jobs specified by id raises PbsDeleteError on error

Parameters:	id (str or list) – The identifier(s) of the jobs/resvs to delete extend (str or none) – Optional parameters to pass along to PBS runas (str) – run as user wait (bool) – Set to True to wait for job(s)/resv(s) to no longer be reported by PBS. False by default logerr (bool) – Whether to log errors. Defaults to True.
Raises:	PbsDeleteError

select(attrib=None, extend=None, runas=None, logerr=True)¶

Select jobs that match attributes list or all jobs if no attributes raises PbsSelectError on error

Parameters:	attrib (str or list or dictionary) – A string, list, or dictionary of attributes extend (str or None) – the extended attributes to pass to select runas (str or None) – run as user logerr (bool) – If True (default) logs run_cmd errors
Returns:	A list of job identifiers that match the attributes specified
Raises:	PbsSelectError

selstat(select_list, rattrib, runas=None, extend=None)¶

stat and filter jobs attributes.

Parameters:	select_list – The filter criteria rattrib (List) – The attributes to query runas (str or None) – run as user

Note

No CLI counterpart for this call

manager(cmd, obj_type, attrib=None, id=None, extend=None, level=20, sudo=None, runas=None, logerr=True)¶

issue a management command to the server, e.g to set an attribute

Returns 0 for Success and non 0 number for Failure

Parameters:

cmd (str) – The command to issue, MGR_CMD_[SET,UNSET, LIST,...] see pbs_ifl.h
obj_type – The type of object to query, one of the * objects
attrib (str or list or dictionary) – Attributes to operate on, can be a string, a list,a dictionary
id (str or list) – The name or list of names of the object(s) to act upon.
extend (str or None) – Optional extension to the IFL call. see pbs_ifl.h
level – logging level
sudo (bool) – If True, run the manager command as super user. Defaults to None. Some attribute settings should be run with sudo set to True, those are acl_roots, job_sort_formula, hook operations, no_sched_hook_event, in those cases, setting sudo to False is only needed for testing purposes
runas (str) – run as user
logerr (bool) – If False, CLI commands do not log error, i.e. silent mode

Raises:

PbsManagerError

sigjob(jobid=None, signal=None, extend=None, runas=None, logerr=True)¶

Send a signal to a job. Raises PbsSignalError on error.

Parameters:	jobid (str or list) – identifier of the job or list of jobs to send the signal to signal (str or None) – The signal to send to the job, see pbs_ifl.h extend – extend options runas (str or None) – run as user logerr (bool) – If True (default) logs run_cmd errors
Raises:	PbsSignalError

msgjob(jobid=None, to_file=None, msg=None, extend=None, runas=None, logerr=True)¶

Send a message to a job. Raises PbsMessageError on error.

Parameters:	jobid (str or List) – identifier of the job or list of jobs to send the message to msg (str or None) – The message to send to the job to_file (str or None) – one of `MSG_ERR` or `MSG_OUT` or `MSG_ERR\|MSG_OUT` extend – extend options runas (str or None) – run as user logerr (bool) – If True (default) logs run_cmd errors
Raises:	PbsMessageError

alterjob(jobid=None, attrib=None, extend=None, runas=None, logerr=True)¶

Alter attributes associated to a job. Raises PbsAlterError on error.

Parameters:	jobid (str or list) – identifier of the job or list of jobs to operate on attrib (dictionary) – A dictionary of attributes to set extend – extend options runas (str or None) – run as user logerr (bool) – If False, CLI commands do not log error, i.e. silent mode
Raises:	PbsAlterError

holdjob(jobid=None, holdtype=None, extend=None, runas=None, logerr=True)¶

Hold a job. Raises PbsHoldError on error.

Parameters:	jobid (str or list) – identifier of the job or list of jobs to hold holdtype (str or None) – The type of hold to put on the job extend – extend options runas (str or None) – run as user logerr (bool) – If True (default) logs run_cmd errors
Raises:	PbsHoldError

rlsjob(jobid, holdtype, extend=None, runas=None, logerr=True)¶

Release a job. Raises PbsReleaseError on error.

Parameters:	jobid (str or list) – job or list of jobs to release holdtype (str) – The type of hold to release on the job extend – extend options runas (str or None) – run as user logerr (bool) – If True (default) logs run_cmd errors
Raises:	PbsReleaseError

rerunjob(jobid=None, extend=None, runas=None, logerr=True)¶: Rerun a job. Raises PbsRerunError on error. :param jobid: job or list of jobs to release :type jobid: str or list :param extend: extend options :param runas: run as user :type runas: str or None :param logerr: If True (default) logs run_cmd errors :type logerr: bool :raises: PbsRerunError

orderjob(jobid1=None, jobid2=None, extend=None, runas=None, logerr=True)¶

reorder position of jobid1 and jobid2. Raises PbsOrderJob on error.

Parameters:	jobid1 (str or None) – first jobid jobid2 (str or None) – second jobid extend – extend options runas (str or None) – run as user logerr (bool) – If True (default) logs run_cmd errors
Raises:	PbsOrderJob

runjob(jobid=None, location=None, async=False, extend=None, runas=None, logerr=False)¶

Run a job on given nodes. Raises PbsRunError on error.

Parameters:	jobid (str or list) – job or list of jobs to run location (str or None) – An execvnode on which to run the job async (bool) – If true the call will return immediately assuming success. extend – extend options runas (str or None) – run as user logerr (bool) – If True (default) logs run_cmd errors
Raises:	PbsRunError

movejob(jobid=None, destination=None, extend=None, runas=None, logerr=True)¶

Move a job or list of job ids to a given destination queue. Raises PbsMoveError on error.

Parameters:	jobid (str or list) – A job or list of job ids to move destination (str or None) – The destination queue@server extend – extend options runas (str or None) – run as user logerr (bool) – If True (default) logs run_cmd errors
Raises:	PbsMoveError

qterm(manner=None, extend=None, server_name=None, runas=None, logerr=True)¶

Terminate the pbs_server daemon

Parameters:	manner – one of `(SHUT_IMMEDIATE \| SHUT_DELAY \| SHUT_QUICK)` and can be combined with SHUT_WHO_SCHED, SHUT_WHO_MOM, SHUT_WHO_SECDRY, SHUT_WHO_IDLESECDRY, SHUT_WHO_SECDONLY. :param extend: extend options server_name (str or None) – name of the pbs server runas (str or None) – run as user logerr (bool) – If True (default) logs run_cmd errors
Raises:	PbsQtermError

teminate(manner=None, extend=None, server_name=None, runas=None, logerr=True)¶

Terminate the pbs_server daemon

Parameters:	manner – one of `(SHUT_IMMEDIATE \| SHUT_DELAY \| SHUT_QUICK)` and can be combined with SHUT_WHO_SCHED, SHUT_WHO_MOM, SHUT_WHO_SECDRY, SHUT_WHO_IDLESECDRY, SHUT_WHO_SECDONLY. :param extend: extend options server_name (str or None) – name of the pbs server runas (str or None) – run as user logerr (bool) – If True (default) logs run_cmd errors
Raises:	PbsQtermError

geterrmsg()¶: Get the error message

qdisable(queue=None, runas=None, logerr=True)¶

Disable queue. CLI mode only

Parameters:	queue (str or list) – The name of the queue or list of queue to disable runas (str or None) – Optional name of user to run command as logerr (bool) – Set to False ot disable logging command errors.Defaults to True.
Raises:	PbsQdisableError

qenable(queue=None, runas=None, logerr=True)¶

Enable queue. CLI mode only

Parameters:	queue (str or list) – The name of the queue or list of queue to enable runas (str or None) – Optional name of user to run command as logerr (bool) – Set to False ot disable logging command errors.Defaults to True.
Raises:	PbsQenableError

qstart(queue=None, runas=None, logerr=True)¶

Start queue. CLI mode only

Parameters:	queue (str or list) – The name of the queue or list of queue to start runas (str or None) – Optional name of user to run command as logerr (bool) – Set to False ot disable logging command errors.Defaults to True.
Raises:	PbsQstartError

qstop(queue=None, runas=None, logerr=True)¶

Stop queue. CLI mode only

Parameters:	queue (str or list) – The name of the queue or list of queue to stop runas (str or None) – Optional name of user to run command as logerr (bool) – Set to False ot disable logging command errors. Defaults to True.
Raises:	PbsQstopError

parse_resources()¶

Parse server resources as defined in the resourcedef file Populates instance variable self.resources

Returns:	The resources as a dictionary

remove_resource(name)¶

Remove an entry from resourcedef

Parameters:

name (str) – The name of the resource to remove
restart – Whether to restart the server or not. Applicable to update_mode ‘file’ operations only.
update_mode – one of ‘file’ or ‘auto’ (the default). If ‘file’, updates the resourcedef file only and will not use the qmgr operations on resources introduced in 12.3. If ‘auto’, will automatically handle the update on resourcedef or using qmgr based on the version of the Server.

add_resource(name, type=None, flag=None)¶

Define a server resource

Parameters:

name (str) – The name of the resource to add to the resourcedef file
type – The type of the resource, one of string, long, boolean, float
flag (str or None) – The target of the resource, one of n, h, q, or none
restart – Whether to restart the server after adding a resource.Applicable to update_mode ‘file’ operations only.
update_mode – one of ‘file’ or ‘auto’ (the default). If ‘file’, updates the resourcedef file only and will not use the qmgr operations on resources introduced in 12.3. If ‘auto’, will automatically handle the update on resourcedef or using qmgr based on the version of the Server.

Returns:

True on success False on error

write_resourcedef(resources=None, filename=None, restart=True)¶

Write into resource def file

Parameters:	resources (dictionary) – PBS resources filename (str or None) – resourcedef file name

parse_resourcedef(file=None)¶

Parse an arbitrary resource definition file passed as input and return a dictionary of resources

Parameters:	file (str or None) – resource definition file
Returns:	Dictionary of resource
Raises:	PbsResourceError

pbs_api_as(cmd=None, obj=None, user=None, **kwargs)¶

Generic handler to run an API call impersonating a given user.This method is only used for impersonation over the API because CLI impersonation takes place through the generic DshUtils run_cmd mechanism.

Parameters:	cmd (str or None) – PBS command user (str or None) – PBS user or current user
Raises:	eval

alterresv(resvid, attrib, extend=None, runas=None, logerr=True)¶

Alter attributes associated to a reservation. Raises PbsResvAlterError on error.

Parameters:	resvid (str.) – identifier of the reservation. attrib (dictionary.) – A dictionary of attributes to set. extend – extend options. runas (str or None.) – run as user. logerr (bool.) – If False, CLI commands do not log error, i.e. silent mode.
Raises:	PbsResvAlterError.

expect(obj_type, attrib=None, id=None, op=4, attrop=1, attempt=0, max_attempts=None, interval=None, count=None, extend=None, offset=0, runas=None, level=20, msg=None)¶

expect an attribute to match a given value as per an operation.

Parameters:	obj_type (str) – The type of object to query, JOB, SERVER, SCHEDULER, QUEUE NODE attrib (str or list or dictionary) – Attributes to query, can be a string, a list, or a dict id – The id of the object to act upon op – An operation to perform on the queried data, e.g., EQ, SET, LT,.. attrop – Operation on multiple attributes, either PTL_AND, PTL_OR when an PTL_AND is used, only batch objects having all matches are returned, otherwise an OR is applied attempt (int) – The number of times this function has been called max_attempts (int or None) – The maximum number of attempts to perform.C{param_max_attempts}: 5 interval – The interval time btween attempts. C{param_interval}: 1s count (bool) – If True, attrib will be accumulated using function counter extend – passed to the stat call offset (int) – the time to wait before the initial check. Defaults to 0. runas (str or None) – query as a given user. Defaults to current user msg (str or None) – Message from last call of this function, this message will be used while raising PtlExpectError.
Returns:	True if attributes are as expected
Raises:	PtlExpectError if attributes are not as expected

is_history_enabled()¶: Short-hand method to return the value of job_history_enable

cleanup_jobs(extend=None, runas=None)¶

Helper function to delete all jobs. By default this method will determine whether job_history_enable is on and will cleanup all history jobs. Specifying an extend parameter could override this behavior.

Parameters:	runas (str or None) – Clean the job as

cleanup_reservations(extend=None, runas=None)¶: Helper function to delete all reservations

cleanup_jobs_and_reservations(extend='forcedeletehist')¶

Helper function to delete all jobs and reservations

Parameters:	extend – Optional extend parameter that is passed to delete. It defaults to ‘deletehist’ which is used in qdel and pbs_deljob() to force delete all jobs, including history jobs extend – str

update_attributes(obj_type, bs)¶: Populate objects from batch status data

counter(obj_type=None, attrib=None, id=None, extend=None, op=None, attrop=None, bslist=None, level=20, idonly=True, grandtotal=False, db_access=None, runas=None, resolve_indirectness=False)¶

Accumulate properties set on an object. For example, to count number of free nodes: server.counter(VNODE,{'state':'free'})

Parameters:

obj_type – The type of object to query, one of the * objects
attrib (str or list or dictionary) – Attributes to query, can be a string, a list, a dictionary
id – The id of the object to act upon
extend – The extended parameter to pass to the stat call
op (str or None) – The operation used to match attrib to what is queried. SET or None
attrop – Operation on multiple attributes, either PTL_AND, PTL_OR
bslist – Optional, use a batch status dict list instead of an obj_type
idonly (bool) – if true, return the name/id of the matching objects
db_access (str or dictionary) – credentials to access db, either a path to file or dictionary
runas (str or None) – run as user

filter(obj_type=None, attrib=None, id=None, extend=None, op=None, attrop=None, bslist=None, idonly=True, grandtotal=False, db_access=None, runas=None, resolve_indirectness=False)¶

Filter objects by properties. For example, to filter all free nodes:server.filter(VNODE,{'state':'free'})

For each attribute queried, if idonly is True, a list of matching object names is returned; if idonly is False, then the value of each attribute queried is returned.

This is unlike Python’s built-in ‘filter’ that returns a subset of objects matching from a pool of objects. The Python filtering mechanism remains very useful in some situations and should be used programmatically to achieve desired filtering goals that can not be met easily with PTL’s filter method.

Parameters:

obj_type – The type of object to query, one of the * objects
attrib (str or list or dictionary) – Attributes to query, can be a string, a list, a dictionary
id – The id of the object to act upon
extend – The extended parameter to pass to the stat call
op (str or None) – The operation used to match attrib to what is queried. SET or None
bslist (List or None) – Optional, use a batch status dict list instead of an obj_type
idonly (bool) – if true, return the name/id of the matching objects
db_access (str or dictionary) – credentials to access db, either path to file or dictionary
runas (str or None) – run as user

logit(msg, obj_type, attrib, id, level=20)¶

Generic logging routine for IFL commands

Parameters:	msg (str) – The message to log obj_type – object type, i.e * attrib – attributes to log id (str or list) – name of object to log level – log level, defaults to `INFO`

equivalence_classes(obj_type=None, attrib={}, bslist=None, op=0, show_zero_resources=True, db_access=None, resolve_indirectness=False)¶

Parameters:

obj_type – PBS Object to query, one of *
attrib (dictionary) – attributes to build equivalence classes out of.
bslist (List) – Optional, list of dictionary representation of a batch status
op – set to RESOURCES_AVAILABLE uses the dynamic amount of resources available, i.e., available - assigned, otherwise uses static amount of resources available
db_acccess – set to either file containing credentials to DB access or dictionary containing {'dbname':...,'user':...,'port':...}

show_equivalence_classes(eq=None, obj_type=None, attrib={}, bslist=None, op=0, show_zero_resources=True, db_access=None, resolve_indirectness=False)¶

helper function to show the equivalence classes

Parameters:	eq – equivalence classes as compute by equivalence_classes see equivalence_classes for remaining parameters description db_acccess – set to either file containing credentials to DB access or dictionary containing `{'dbname':...,'user':...,'port':...}`

whats_available(attrib=None, jobs=None, resvs=None, nodes=None)¶

Returns what’s available as a list of node equivalence classes listed by availability over time.

Parameters:	attrib (List) – attributes to consider jobs – jobs to consider, if None, jobs are queried locally resvs – reservations to consider, if None, they are queried locally nodes – nodes to consider, if None, they are queried locally

show_whats_available(wa=None, attrib=None, jobs=None, resvs=None, nodes=None)¶

helper function to show availability as computed by whats_available

Parameters:	wa (Dictionary) – a dictionary of available attributes. see whats_available for a description of the remaining parameters

utilization(resources=None, nodes=None, jobs=None, entity={})¶

Return utilization of consumable resources on a set of nodes

Parameters:

nodes (List) – A list of dictionary of nodes on which to compute utilization.Defaults to nodes resulting from a stat call to the current server.
resources (List) – comma-separated list of resources to compute utilization on. The name of the resource is for example, ncpus or mem
entity (Dictionary) – An optional dictionary of entities to compute utilization of, e.g. {'user':u1, 'group':g1, 'project'=p1}

The utilization is returned as a dictionary of percentage utilization for each resource.

Non-consumable resources are silently ignored.

create_vnodes(name=None, attrib=None, num=1, mom=None, additive=False, sharednode=True, restart=True, delall=True, natvnode=None, usenatvnode=False, attrfunc=None, fname=None, vnodes_per_host=1, createnode=True, expect=True)¶

helper function to create vnodes.

Parameters:

name (str or None) – prefix name of the vnode(s) to create
attrib – attributes to assign to each node
num (int) – the number of vnodes to create. Defaults to 1
mom – the MoM object on which the vnode definition is to be inserted
additive (bool) – If True, vnodes are added to the existing vnode defs.Defaults to False.
sharednode (bool) – If True, all vnodes will share the same host.Defaults to True.
restart (bool) – If True the MoM will be restarted.
delall (bool) – If True delete all server nodes prior to inserting vnodes
natvnode (str or None) – name of the natural vnode.i.e. The node name in qmgr -c “create node <name>”
usenatvnode (bool) – count the natural vnode as an allocatable node.
attrfunc – an attribute=value function generator, see create_vnode_def
fname (str or None) – optional name of the vnode def file
vnodes_per_host (int) – number of vnodes per host
createnode (bool) – whether to create the node via manage or not. Defaults to True
expect (bool) – whether to expect attributes to be set or not. Defaults to True

Returns:

True on success and False otherwise

create_moms(name=None, attrib=None, num=1, delall=True, createnode=True, conf_prefix='pbs.conf_m', home_prefix='pbs_m', momhosts=None, init_port=15011, step_port=2)¶

Create MoM configurations and optionall add them to the server. Unique pbs.conf files are defined and created on each hosts on which MoMs are to be created.

Parameters:

name (str or None) – Optional prefix name of the nodes to create. Defaults to the name of the MoM host.
attrib – Optional node attributes to assign to the MoM.
num (int) – Number of MoMs to create
delall (bool) – Whether to delete all nodes on the server. Defaults to True.
createnode (bool) – Whether to create the nodes and add them to the server.Defaults to True.
conf_prefix (str) – The prefix of the PBS conf file.Defaults to pbs.conf_m
home_prefix (str) – The prefix of the PBS_HOME directory. Defaults to pbs_m
momhosts (List) – A list of hosts on which to deploy num MoMs.
init_port (int) – The initial port number to start assigning PBS_MOM_SERIVCE_PORT to. Default 15011.
step_port (int) – The increments at which ports are allocated. Defaults to 2.

Note

Since PBS requires that PBS_MANAGER_SERVICE_PORT = PBS_MOM_SERVICE_PORT+1 The step number must be greater or equal to 2.

create_hook(name, attrs)¶

Helper function to create a hook by name.

Parameters:	name (str) – The name of the hook to create attrs (str) – The attributes to create the hook with.
Returns:	False if hook already exists
Raises:	PbsManagerError, otherwise return True.

import_hook(name, body, level=20)¶

Helper function to import hook body into hook by name. The hook must have been created prior to calling this function.

Parameters:	name (str) – The name of the hook to import body to body (str) – The body of the hook as a string.
Returns:	True on success.
Raises:	PbsManagerError

create_import_hook(name, attrs=None, body=None, overwrite=True, level=20)¶

Helper function to create a hook, import content into it, set the event and enable it.

Parameters:	name (str) – The name of the hook to create attrs (str) – The attributes to create the hook with. Event and Enabled are mandatory. No defaults. body (str) – The hook body as a string overwrite – If True, if a hook of the same name already exists, bypass its creation. Defaults to True
Returns:	True on success and False otherwise

evaluate_formula(jobid=None, formula=None, full=True, include_running_jobs=False, exclude_subjobs=True)¶

Evaluate the job sort formula

Parameters:

jobid (str or None) – If set, evaluate the formula for the given jobid, if not set,formula is evaluated for all jobs in state Q
formula – If set use the given formula. If not set, the server’s formula, if any, is used
full (bool) – If True, returns a dictionary of job identifiers as keys and the evaluated formula as values. Returns None if no formula is used. Each job id formula is returned as a tuple (s,e) where s is the formula expression associated to the job and e is the evaluated numeric value of that expression, for example, if job_sort_formula is ncpus + mem a job requesting 2 cpus and 100kb of memory would return (‘2 + 100’, 102). If False, if a jobid is specified, return the integer value of the evaluated formula.
include_running_jobs (bool) – If True, reports formula value of running jobs. Defaults to False.
exclude_subjobs (bool) – If True, only report formula of parent job array

parse_server_limits(server=None, db_access=None)¶

Parse all server limits

Parameters:	server (List) – list of dictionary of server data db_acccess – set to either file containing credentials to DB access or dictionary containing `{'dbname':...,'user':...,'port':...}`

parse_queue_limits(queues=None, id=None, db_access=None)¶

Parse queue limits

Parameters:	queues (List) – list of dictionary of queue data id – The id of the queue to parse limit for. If None, all queue limits are parsed db_acccess – set to either file containing credentials to DB access or dictionary containing `{'dbname':...,'user':...,'port':...}`

parse_all_limits(server=None, queues=None, db_access=None)¶

Parse all server and queue limits

Parameters:	server (List) – list of dictionary of server data queues (List) – list of dictionary of queue data db_acccess – set to either file containing credentials to DB access or dictionary containing `{'dbname':...,'user':...,'port':...}`

limits_info(etype=None, ename=None, server=None, queues=None, jobs=None, db_access=None, over=False)¶

Collect limit information for each entity on which a server/queue limit is applied.

Parameters:

etype (str or None) – entity type, one of u, g, p, o
ename (str or None) – entity name
server (List) – optional list of dictionary representation of server object
queues (List) – optional list of dictionary representation of queues object
jobs (List) – optional list of dictionary representation of jobs object
db_acccess – set to either file containing credentials to DB access or dictionary containing {'dbname':...,'user':...,'port':...}
over (bool) – If True, show only entities that are over their limit.Default is False.

Returns:

A list of dictionary similar to that returned by a converted batch_status object, i.e., can be displayed using the Utils.show method

clusterize(conf_file=None, hosts=None, import_jobs=False, db_creds_file=None)¶

Mimic a pbs_diag snapshot onto a set of hosts running a PBS server,``scheduler``, and MoM.

This method clones the following information from the diag:

Server attributes Server resourcedef Hooks Scheduler configuration Scheduler resource_group Scheduler holiday file Per Queue attributes

Nodes are copied as a vnode definition file inserted into each host’s MoM instance.

Currently no support for cloning the server ‘sched’ object, nor to copy nodes to multi-mom instances.

Jobs are copied over only if import_jobs is True, see below for details

Parameters:

asdiag (str) – Path to the pbs_diag snapshot to use
conf_file – Configuration file for the MoM instance
hosts (List) – List of hosts on which to clone the diag snapshot
include_jobs (bool) – [Experimental] if True jobs from the pbs_diag are imported into the host’s database. There are several caveats to this option: The scripts are not imported The users and groups are not created on the local system.There are no actual processes created on the MoM for each job so operations on the job such as signals or delete will fail (delete -W force will still work)
db_creds_file (str or None) – Path to file containing credentials to access the DB

class ptl.lib.pbs_testlib.EquivClass(name, attributes={}, entities=[])¶

Equivalence class holds information on a collection of entities grouped according to a set of attributes :param attributes: Dictionary of attributes :type attributes: Dictionary :param entities: List of entities :type entities: List

add_entity(entity)¶

Add entities

Parameters:	entity (str) – Entity to add

show(showobj=False)¶

Show the entities

Parameters:	showobj (bool) – If true then show the entities

class ptl.lib.pbs_testlib.Resource(name=None, type=None, flag=None)¶

PBS resource referenced by name, type and flag

Parameters:	name (str or None) – Resource name type – Type of resource

set_name(name)¶: Set the resource name

set_type(type)¶: Set the resource type

set_flag(flag)¶: Set the flag

class ptl.lib.pbs_testlib.Holidays¶: Descriptive calss for Holiday file.

class ptl.lib.pbs_testlib.Scheduler(hostname=None, server=None, pbsconf_file=None, diagmap={}, diag=None, db_access=None, id='default', sched_priv=None)¶

Container of Scheduler related properties

Parameters:

hostname (str or None) – The hostname on which the scheduler instance is operating
server – A PBS server instance to which this scheduler is associated
pbsconf_file (str or None) – path to a PBS configuration file
diagmap (Dictionary) – A dictionary of PBS objects (node,server,etc) to mapped files from PBS diag directory
diag (str or None) – path to PBS diag directory (This will overrides diagmap)
db_acccess – set to either file containing credentials to DB access or dictionary containing {'dbname':...,'user':...,'port':...}

setup_sched_priv(sched_priv=None)¶: Initialize Scheduler() member variables on initialization or if sched_priv changes

initialise_service()¶: initialise the scheduler object

isUp()¶: Check for PBS scheduler up

signal(sig)¶: Send a signal to PBS scheduler

get_pid()¶: Get the PBS scheduler pid

all_instance_pids()¶: Get the all pids for the instance

start(sched_home=None, args=None, launcher=None)¶: Start the scheduler :param sched_home: Path to scheduler log and home directory :type sched_home: str :param args: Arguments required to start the scheduler :type args: str :param launcher: Optional utility to invoke the launch of the service :type launcher: str or list

stop(sig=None)¶

Stop the PBS scheduler

Parameters:	sig (str) – Signal to stop the PBS scheduler

restart()¶: Restart the PBS scheduler

log_match(msg=None, id=None, n=50, tail=True, allmatch=False, regexp=False, max_attempts=None, interval=None, starttime=None, endtime=None, level=20, existence=True)¶

Match given msg in given n lines of Scheduler log

Parameters:	msg (str) – log message to match, can be regex also when `regexp` is True id (str) – The id of the object to trace. Only used for tracejob n (str or int) – ‘ALL’ or the number of lines to search through, defaults to 50 tail (bool) – If true (default), starts from the end of the file allmatch (bool) – If True all matching lines out of then parsed are returned as a list. Defaults to False regexp (bool) – If true msg is a Python regular expression. Defaults to False max_attempts (int) – the number of attempts to make to find a matching entry interval (int) – the interval between attempts starttime (int) – If set ignore matches that occur before specified time endtime (int) – If set ignore matches that occur after specified time level (int) – The logging level, defaults to INFO existence (bool) – If True (default), check for existence of given msg, else check for non-existence of given msg.
Returns:	(x,y) where x is the matching line number and y the line itself. If allmatch is True, a list of tuples is returned.
Return type:	tuple
Raises:	PtlLogMatchError – When `existence` is True and given `msg` is not found in `n` line Or When `existence` is False and given `msg` found in `n` line.

Note

The matching line number is relative to the record number, not the absolute line number in the file.

pbs_version()¶: Get the version of the scheduler instance

parse_sched_config(schd_cnfg=None)¶

Parse a sceduling configuration file into a dictionary. Special handling of identical keys (e.g., node_sort_key) is done by appending a delimiter, ‘%’, between each value of the key. When printed back to file, each delimited entry gets written on a line of its own. For example, the python dictionary entry:

{'node_sort_key': ["ncpus HIGH unusued" prime", "node_priority HIH" non-prime"]}

will get written as:

node_sort_key: "ncpus HIGH unusued" prime node_sort_key: "node_priority HIGH" non-prime

Returns sched_config dictionary that gets reinitialized every time this method is called.

check_defaults(config)¶: Check the values in argument config against default values

apply_config(config=None, validate=True, path=None)¶

Apply the configuration specified by config

Parameters:

config – Configurations to set. Default: self. sched_config
validate (bool) – If True (the default) validate that settings did not yield an error. Validation is done by parsing the scheduler log which, in some cases may be slow and therefore undesirable.
path (str) – Optional path to file to which configuration is written. If None, the configuration is written to PBS_HOME/sched_priv/sched_config

Returns:

True on success and False otherwise. Success means that upon applying the new configuration the scheduler did not emit an “Error reading line” in its log file.

set_sched_config(confs={}, apply=True, validate=True)¶

set a sched_config property

Parameters:	confs (Dictionary) – dictionary of key value sched_config entries apply (bool) – if True (the default), apply configuration. validate (bool) – if True (the default), validate the configuration settings.

add_server_dyn_res(custom_resource, script_body=None, res_file=None, apply=True, validate=True, dirname=None, host=None, perm=448, prefix='PtlPbsSvrDynRes', suffix='.scr')¶

Add a root owned server dynamic resource script or file to the scheduler configuration.

Parameters:

custom_resource (str) – The name of the custom resource to define
script_body – The body of the server dynamic resource
res_file (str or None) – Alternatively to passing the script body, use the file instead
apply (bool) – if True (the default), apply configuration.
validate (bool) – if True (the default), validate the configuration settings.
dirname (str or None) – the file will be created in this directory
host (str or None) – the hostname on which dyn res script is created
perm – perm to use while creating scripts (must be octal like 0777)
prefix (str) – the file name will begin with this prefix
suffix (str) – the file name will end with this suffix

:return Absolute path of the dynamic resource script

unset_sched_config(name, apply=True)¶

Delete a sched_config entry

Parameters:	name (str) – the entry to delete from sched_config apply (bool) – if True, apply configuration. Defaults to True

set_dedicated_time_file(file)¶: Set the path to a dedicated time

revert_to_defaults()¶

Revert scheduler configuration to defaults.

Returns:	True on success, False otherwise

create_scheduler(sched_home=None)¶: Start scheduler with creating required directories for scheduler :param sched_home: path of scheduler home and log directory :type sched_home: str

save_configuration(outfile, mode='a')¶

Save scheduler configuration

Parameters:	outfile (str) – Path to a file to which configuration is saved mode (str) – mode to use to access outfile. Defaults to append, ‘a’.
Returns:	True on success and False otherwise

load_configuration(infile)¶: load configuration from saved file infile

get_resources(exclude=[])¶

returns a list of allocatable resources.

Parameters:	exclude (List) – if set, excludes the named resources, if they exist, from the resulting list

add_resource(name, apply=True)¶

Add a resource to sched_config.

Parameters:	name (str) – the resource name to add apply (bool) – if True, apply configuration. Defaults to True
Returns:	True on success and False otherwise. Return True if the resource is already defined.

remove_resource(name, apply=True)¶

Remove a resource to sched_config.

Parameters:	name (str) – the resource name to remove apply (bool) – if True, apply configuration. Defaults to True
Returns:	True on success and False otherwise

holidays_revert_to_default(level=20)¶: Revert holidays file to default

holidays_parse_file(path=None, obj=None, level=20)¶

Parse the existing holidays file

Parameters:	path (str or None) – optional path to the holidays file to parse obj – optional holidays object to be used instead of internal
Returns:	The content of holidays file as a list of lines

holidays_set_day(day_id, prime='', nonprime='', apply=True, obj=None, level=20)¶

Set prime time values for a day

Parameters:	day_id (str) – the day to be set (string) prime – the prime time value nonprime – the non-prime time value apply (bool) – to reflect the changes to file obj – optional holidays object to be used instead of internal
Returns:	The position `(0-7)` of the set day

holidays_get_day(day_id, obj=None, level=20)¶

Parameters:	obj – optional holidays object to be used instead of internal day_id (str) – either a day’s name or “all”
Returns:	A copy of info about a day/all set days

holidays_reposition_day(day_id, new_pos, apply=True, obj=None, level=20)¶

Change position of a day (0-7) as it appears in the holidays file

Parameters:	day_id (str) – name of the day new_pos – new position apply (bool) – to reflect the changes to file obj – optional holidays object to be used instead of internal
Returns:	The new position of the day

holidays_unset_day(day_id, apply=True, obj=None, level=20)¶

Unset prime time values for a day

Parameters:	day_id (str) – day to unset (string) apply – to reflect the changes to file obj – optional holidays object to be used instead of internal

Note

we do not unset the ‘valid’ field here so the entry will still be displayed but without any values

holidays_invalidate_day(day_id, apply=True, obj=None, level=20)¶

Remove a day’s entry from the holidays file

Parameters:	day_id (str) – the day to remove (string) apply (bool) – to reflect the changes to file obj – optional holidays object to be used instead of internal

holidays_validate_day(day_id, apply=True, obj=None, level=20)¶

Make valid a previously set day’s entry

Parameters:	day_id (str) – the day to validate (string) apply (bool) – to reflect the changes to file obj – optional holidays object to be used instead of internal

Note

The day will retain its previous position in the file

holidays_delete_entry(entry_type, idx=None, apply=True, obj=None, level=20)¶

Delete one/all entries from holidays file

Parameters:	entry_type (str) – ‘y’:year, ‘d’:day, ‘h’:holiday or ‘a’: all idx (str or None) – either a day of week (monday, tuesday etc.) or Julian date of a holiday apply (bool) – to reflect the changes to file obj – optional holidays object to be used instead of internal
Returns:	False if entry_type is invalid, otherwise True

Note

The day cannot be validated and will lose it’s position in the file

holidays_set_year(new_year='', apply=True, obj=None, level=20)¶

Set the year value

Parameters:	newyear (str) – year value to set apply (bool) – to reflect the changes to file obj – optional holidays object to be used instead of internal

holidays_unset_year(apply=True, obj=None, level=20)¶

Unset the year value

Parameters:	apply (bool) – to reflect the changes to file obj – optional holidays object to be used instead of internal

holidays_get_year(obj=None, level=20)¶

Parameters:	obj – optional holidays object to be used instead of internal
Returns:	The year entry of holidays file

holidays_add_holiday(date=None, apply=True, obj=None, level=20)¶

Add a calendar holiday to the holidays file

Parameters:	date – Date value for the holiday apply (bool) – to reflect the changes to file obj – optional holidays object to be used instead of internal

holidays_get_holidays(obj=None, level=20)¶

Parameters:	obj – optional holidays object to be used instead of internal
Returns:	The list of holidays in holidays file

holidays_write_file(content=None, out_path=None, hup=True, obj=None, level=20)¶

Write to the holidays file with content given/generated

Parameters:	hup (bool) – SIGHUP the scheduler after writing the holidays file obj – optional holidays object to be used instead of internal

parse_dedicated_time(file=None)¶

Parse the dedicated_time file and populate dedicated times as both a string dedicated_time array of dictionaries defined as [{'from': datetime, 'to': datetime}, ...] as well as a dedicated_time_as_str array with a string representation of each entry

Parameters:	file (str or None) – optional file to parse. Defaults to the one under `PBS_HOME/sched_priv`
Returns:	The dedicated_time list of dictionaries or None on error.Return an empty array if dedicated time file is empty.

clear_dedicated_time(hup=True)¶: Clear the dedicated time file

add_dedicated_time(as_str=None, start=None, end=None, hup=True)¶

Append a dedicated time entry. The function can be called in one of two ways, either by passing in start and end as time values, or by passing as_str, a string that gets appended to the dedicated time entries and formatted as follows, note that no check on validity of the format will be made the function uses strftime to parse the datetime and will fail if the strftime can not convert the string. MM/DD/YYYY HH:MM MM/DD/YYYY HH:MM

Returns:	True on success and False otherwise

valgrind()¶: run scheduler instance through valgrind

alloc_to_execvnode(chunks)¶: convert a resource allocation to an execvnode string representation

cycles(start=None, end=None, firstN=None, lastN=None)¶

Analyze scheduler log and return cycle information

Parameters:	start – Optional setting of the start time to consider end – Optional setting of the end time to consider firstN – Optional setting to consider the given first N cycles lastN – Optional setting to consider only the given last N cycles

query_fairshare(name=None, id=None)¶: Parse fairshare data using pbsfs and populates fairshare_tree.If name or id are specified, return the data associated to that id.Otherwise return the entire fairshare tree

set_fairshare_usage(name=None, usage=None)¶

Set the fairshare usage associated to a given entity.

Parameters:	name (str or `PbsUser` or None) – The entity to set the fairshare usage of usage – The usage value to set

decay_fairshare_tree()¶: Decay the fairshare tree through pbsfs

cmp_fairshare_entities(name1=None, name2=None)¶

Compare two fairshare entities. Wrapper of pbsfs -c e1 e2

Parameters:	name1 (str or `PbsUser` or None) – name of first entity to compare name2 (str or `PbsUser` or None) – name of second entity to compare
Returns:	the name of the entity of higher priority or None on error

parse_resource_group(hostname=None, resource_group=None)¶

Parse the Scheduler’s resource_group file

Parameters:	hostname (str or None) – The name of the host from which to parse resource_group resource_group (str or None) – The path to a resource_group file
Returns:	A fairshare tree

add_to_resource_group(name, fairshare_id, parent, nshares, validate=True)¶

Add an entry to the resource group file

Parameters:	name (str or `PbsUser`) – The name of the entity to add fairshare_id (int) – The numeric identifier of the entity to add parent (str) – The name of the parent group nshares (int) – The number of shares associated to the entity validate (bool) – if True (the default), validate the configuration settings.

job_formula(jobid=None, starttime=None, max_attempts=5)¶

Extract formula value out of scheduler log

Parameters:	jobid (str or int) – Optional, the job identifier for which to get the formula. starttime – The time at which to start parsing the scheduler log max_attempts (int) – The number of attempts to search for formula in the logs
Returns:	If jobid is specified, return the formula value associated to that job if no jobid is specified, returns a dictionary mapping job ids to formula

class ptl.lib.pbs_testlib.FairshareTree(hostname=None, resource_group=None)¶

Object representation of the Scheduler’s resource_group file and pbsfs data

Parameters:	hostname (str) – Hostname of the machine

add_node(node, apply=True)¶: add node to the fairshare tree

create_node(name, id, parent_name, nshares)¶

Add an entry to the resource_group file

Parameters:	name (str) – The name of the entity to add id (int) – The uniqe numeric identifier of the entity parent (str) – The name of the parent/group of the entity nshares (int) – The number of shares assigned to this entity
Returns:	True on success, False otherwise

get_node(name=None, id=None)¶

Return a node of the fairshare tree identified by either name or id.

Parameters:	name (str or None) – The name of the entity to query id – The id of the entity to query
Returns:	The fairshare information of the entity when found, if not, returns None

Note

The name takes precedence over the id.

class ptl.lib.pbs_testlib.FairshareNode(name=None, id=None, parent_name=None, parent_id=None, nshares=None, usage=None, perc=None)¶

Object representation of the fairshare data as queryable through the command pbsfs.

Parameters:	name (str or None) – Name of fairshare node nshares (int or None) – Number of shares usage – Fairshare usage perc – Percentage the entity has of the tree

class ptl.lib.pbs_testlib.MoM(name=None, attrs={}, pbsconf_file=None, diagmap={}, diag=None, server=None, db_access=None)¶

Container for MoM properties. Provides various MoM operations, such as creation, insertion, deletion of vnodes.

Parameters:

name (str or None) – The hostname of the server. Defaults to calling pbs_default()
attrs (Dictionary) – Dictionary of attributes to set, these will override defaults.
pbsconf_file (str or None) – path to config file to parse for PBS_HOME, PBS_EXEC, etc
diagmap (Dictionary) – A dictionary of PBS objects (node,server,etc) to mapped files from PBS diag directory
diag (str or None) – path to PBS diag directory (This will overrides diagmap)
server – A PBS server instance to which this mom is associated
db_acccess – set to either file containing credentials to DB access or dictionary containing {‘dbname’:…,’user’:…,’port’:…}

isUp(max_attempts=20)¶: Check for PBS mom up

signal(sig)¶: Send signal to PBS mom

get_pid()¶: Get the PBS mom pid

all_instance_pids()¶: Get all pids of a instance

start(args=None, launcher=None)¶

Start the PBS mom

Parameters:	args (str or None) – Arguments to start the mom launcher (str or list or None) – Optional utility to invoke the launch of the service

stop(sig=None)¶

Stop the PBS mom

Parameters:	sig (str) – Signal to stop the PBS mom

restart()¶: Restart the PBS mom

log_match(msg=None, id=None, n=50, tail=True, allmatch=False, regexp=False, max_attempts=None, interval=None, starttime=None, endtime=None, level=20, existence=True)¶

Match given msg in given n lines of MoM log

Parameters:	msg (str) – log message to match, can be regex also when `regexp` is True id (str) – The id of the object to trace. Only used for tracejob n (str or int) – ‘ALL’ or the number of lines to search through, defaults to 50 tail (bool) – If true (default), starts from the end of the file allmatch (bool) – If True all matching lines out of then parsed are returned as a list. Defaults to False regexp (bool) – If true msg is a Python regular expression. Defaults to False max_attempts (int) – the number of attempts to make to find a matching entry interval (int) – the interval between attempts starttime (int) – If set ignore matches that occur before specified time endtime (int) – If set ignore matches that occur after specified time level (int) – The logging level, defaults to INFO existence (bool) – If True (default), check for existence of given msg, else check for non-existence of given msg.
Returns:	(x,y) where x is the matching line number and y the line itself. If allmatch is True, a list of tuples is returned.
Return type:	tuple
Raises:	PtlLogMatchError – When `existence` is True and given `msg` is not found in `n` line Or When `existence` is False and given `msg` found in `n` line.

Note

The matching line number is relative to the record number, not the absolute line number in the file.

pbs_version()¶: Get the PBS version

revert_to_defaults(delvnodedefs=True)¶

Revert MoM configuration to defaults.
Remove epilogue and prologue

3. Delete all vnode definitions HUP MoM

Parameters:	delvnodedefs (bool) – if True (the default) delete all vnode definitions and restart the MoM
Returns:	True on success and False otherwise

save_configuration(outfile, mode='a')¶

Save a MoM mom_priv/config

Parameters:	outfile (str) – the output file to which onfiguration is saved mode (str) – the mode in which to open outfile to save configuration.
Returns:	True on success, False on error

Note

first object being saved should open this file with ‘w’ and subsequent calls from other objects should save with mode ‘a’ or ‘a+’. Defaults to a+

load_configuration(infile)¶: load configuration from saved file infile

is_cray()¶: Returns True if the version of PBS used was built for Cray platforms

is_cpuset_mom()¶: Check for cpuset mom

create_vnode_def(name, attrs={}, numnodes=1, sharednode=True, pre='[', post=']', usenatvnode=False, attrfunc=None, vnodes_per_host=1)¶

Create a vnode definition string representation

Parameters:

name (str) – The prefix for name of vnode to create, name of vnode will be prefix + pre + <num> + post
attrs (Dictionary) – Dictionary of attributes to set on each vnode
numnodes (int) – The number of vnodes to create
sharednode (bool) – If true vnodes are shared on a host
pre (str) – The symbol preceding the numeric value of that vnode.
post (str) – The symbol following the numeric value of that vnode.
usenatvnode (bool) – use the natural vnode as the first vnode to allocate this only makes sense starting with PBS 11.3 when natural vnodes are reported as a allocatable
attrfunc – function to customize the attributes, signature is (name, numnodes, curnodenum, attrs), must return a dict that contains new or modified attrs that will be added to the vnode def. The function is called once per vnode being created, it does not modify attrs itself across calls.
vnodes_per_host (int) – number of vnodes per host

Returns:

A string representation of the vnode definition file

parse_config()¶

Parse mom config file into a dictionary of configuration options.

Returns:	A dictionary of configuration options on success, and None otherwise

add_config(conf={}, hup=True)¶

Add config options to mom_priv_config.

Parameters:	conf (Dictionary) – The configurations to add to `mom_priv/config` hup (bool) – If True (default) `HUP` the MoM
Returns:	True on success and False otherwise

unset_mom_config(name, hup=True)¶

Delete a mom_config entry

Parameters:	name (String) – The entry to remove from `mom_priv/config` hup (bool) – if True (default) `HUP` the MoM
Returns:	True on success and False otherwise

apply_config(conf={}, hup=True, restart=False)¶

Apply configuration options to MoM.

Parameters:	conf (Dictionary) – A dictionary of configuration options to apply to MoM hup (bool) – If True (default) , HUP the MoM to apply the configuration
Returns:	True on success and False otherwise.

get_vnode_def(vnodefile=None)¶

Returns:	A vnode def file as a single string

insert_vnode_def(vdef, fname=None, additive=False, restart=True)¶

Insert and enable a vnode definition. Root privilege is required

Parameters:

vdef (str) – The vnode definition string as created by create_vnode_def
fname (str or None) – The filename to write the vnode def string to
additive (bool) – If True, keep all other vnode def files under config.d Default is False
delete (bool) – If True, delete all nodes known to the server. Default is True
restart (bool) – If True, restart the MoM. Default is True

has_vnode_defs()¶: Check for vnode definition(s)

delete_vnode_defs(vdefname=None)¶

delete vnode definition(s) on this MoM

Parameters:	vdefname (str) – name of a vnode definition file to delete, if None all vnode definitions are deleted
Returns:	True if delete succeed otherwise False

has_pelog(filename=None)¶: Check for prologue and epilogue

has_prologue()¶: Check for prologue

has_epilogue()¶: Check for epilogue

delete_pelog()¶: Delete any prologue and epilogue files that may have been defined on this MoM

create_pelog(body=None, src=None, filename=None)¶

create prologue and epilogue files, functionality accepts either a body of the script or a source file.

Returns:	True on success and False on error

prologue(body=None, src=None)¶: create prologue

epilogue(body=None, src=None)¶: Create epilogue

action(act, script)¶: Define action script. Not currently implemented

add_mom_dyn_res(custom_resource, script_body=None, res_file=None, dirname=None, host=None, perm=448, prefix='PtlPbsMomDynRes', suffix='.scr')¶

Add a root owned mom dynamic resource script or file to the mom configuration

Parameters:

custom_resource (str) – The name of the custom resource to define
script_body – The body of the mom dynamic resource
res_file (str or None) – Alternatively to passing the script body, use the file instead
dirname (str or None) – the file will be created in this directory
host (str or None) – the hostname on which dyn res script is created
perm – perm to use while creating scripts (must be octal like 0777)
prefix (str) – the file name will begin with this prefix
suffix (str) – the file name will end with this suffix

:return Absolute path of the dynamic resource script

class ptl.lib.pbs_testlib.Hook(name=None, attrs={}, server=None)¶

PBS hook objects. Holds attributes information and pointer to server

Parameters:	name (str or None) – Hook name attrs (Dictionary) – Hook attributes server – Pointer to server

class ptl.lib.pbs_testlib.ResourceResv(name, attrs={}, defaults={})¶

Generic PBS resource reservation, i.e., job or advance/standing reservation

execvnode(attr='exec_vnode')¶: PBS type execution vnode

exechost()¶: PBS type execution host

resvnodes()¶: nodes assigned to a reservation

classmethod get_hosts(exechost=None)¶

Returns:	The hosts portion of the exec_host

get_vnodes(execvnode=None)¶

Returns:	The unique vnode names of an execvnode as a list

class ptl.lib.pbs_testlib.Job(username=None, attrs={}, jobname=None)¶

PBS Job. Attributes and Resources

Parameters:	username (str or None) – Job username attrs (Dictionary) – Job attributes jobname (str or None) – Name of the PBS job

add_cray_vntype(select=None)¶

Cray specific function to add vntype as cray_compute to each select chunk

Parameters:	select (str or None) – PBS select statement

set_attributes(a={})¶

set attributes and custom attributes on this job. custom attributes are used when converting attributes to CLI. In case of Cray platform if ‘Resource_List.vntype’ is set already then remove it and add vntype value to each chunk of a select statement.

Parameters:	a (Dictionary) – Attribute dictionary

set_variable_list(user=None, workdir=None)¶: Customize the Variable_List job attribute to <user>

set_sleep_time(duration)¶

Set the sleep duration for this job.

Parameters:	duration (int) – The duration, in seconds, to sleep

set_execargs(executable, arguments=None)¶

Set the executable and arguments to use for this job

Parameters:	executable (str) – path to an executable. No checks are made. arguments (str or list or int) – arguments to executable.

create_script(body=None, asuser=None, hostname=None)¶

Create a job script from a given body of text into a temporary location

Parameters:	body (str or None) – the body of the script asuser (str or None) – Optionally the user to own this script, defaults ot current user hostname (str or None) – The host on which the job script is to be created

create_subjob_id(job_array_id, subjob_index)¶

insert subjob index into the square brackets of job array id

Parameters:	job_array_id (str) – PBS parent array job id subjob_index (int) – index of subjob
Returns:	subjob id string

create_eatcpu_job(duration=None)¶: Create a job that eats cpu indefinitely or for the given duration of time

class ptl.lib.pbs_testlib.Reservation(username=None, attrs={})¶

PBS Reservation. Attributes and Resources

Parameters:	attrs (Dictionary) – Reservation attributes

class ptl.lib.pbs_testlib.InteractiveJob(job, cmd, host)¶

An Interactive Job thread

Interactive Jobs are submitted as a thread that sets the jobid as soon as it is returned by qsub -I, such that the caller can get back to monitoring the state of PBS while the interactive session goes on in the thread.

The commands to be run within an interactive session are specified in the job’s interactive_script attribute as a list of tuples, where the first item in each tuple is the command to run, and the subsequent items are the expected returned data.

Implementation details:

Support for interactive jobs is currently done through the pexpect module which must be installed separately from PTL. Interactive jobs are submitted through CLI only, there is no API support for this operation yet.

The submission of an interactive job requires passing in job attributes,the command to execute (i.e. path to qsub -I) and the hostname

when not impersonating:

pexpect spawns the qsub -I command and expects a prompt back, for each tuple in the interactive_script, it sends the command and expects to match the return value.

when impersonating:

pexpect spawns sudo -u <user> qsub -I. The rest is as described in non- impersonating mode.

run()¶: Run the interactive job

class ptl.lib.pbs_testlib.Queue(name=None, attrs={}, server=None)¶

PBS Queue container, holds attributes of the queue and pointer to server

Parameters:	name (str or None) – Queue name attrs (Dictionary) – Queue attributes

revert_to_defaults()¶: reset queue attributes to defaults

class ptl.lib.pbs_testlib.PBSInitServices(hostname=None, conf=None)¶

PBS initialization services

Parameters:	hostname (str or None) – Machine hostname conf (str or None) – PBS configuaration file

initd(hostname=None, op='status', conf_file=None, init_script=None, daemon='all')¶

Run the init script for a given operation

Parameters:	hostname (str or None) – hostname on which to execute the init script op (str) – one of status, start, stop, restart conf_file (str or None) – optional path to a configuration file init_script (str or None) – optional path to a PBS init script daemon (str) – name of daemon to operate on. one of server, mom, sched, comm or all

restart(hostname=None, init_script=None)¶

Run the init script for a restart operation

Parameters:	hostname (str or None) – hostname on which to execute the init script init_script (str or None) – optional path to a PBS init script

restart_server(hostname=None, init_script=None)¶

Run the init script for a restart server

Parameters:	hostname (str or None) – hostname on which to restart server init_script (str or None) – optional path to a PBS init script

restart_mom(hostname=None, init_script=None)¶

Run the init script for a restart mom

Parameters:	hostname (str or None) – hostname on which to restart mom init_script (str or None) – optional path to a PBS init script

restart_sched(hostname=None, init_script=None)¶

Run the init script for a restart sched

Parameters:	hostname (str or None) – hostname on which to restart sched init_script (str or None) – optional path to a PBS init script

restart_comm(hostname=None, init_script=None)¶

Run the init script for a restart comm

Parameters:	hostname (str or None) – hostname on which to restart comm init_script (str or None) – optional path to a PBS init script

start(hostname=None, init_script=None)¶

Run the init script for a start operation

Parameters:	hostname (str or None) – hostname on which to execute the init script init_script (str or None) – optional path to a PBS init script

start_server(hostname=None, init_script=None)¶

Run the init script for a start server

Parameters:	hostname (str or None) – hostname on which to start server init_script (str or None) – optional path to a PBS init script

start_mom(hostname=None, init_script=None)¶

Run the init script for a start mom

Parameters:	hostname (str or None) – hostname on which to start mom init_script (str or None) – optional path to a PBS init script

start_sched(hostname=None, init_script=None)¶

Run the init script for a start sched

Parameters:	hostname (str or None) – hostname on which to start sched init_script (str or None) – optional path to a PBS init script

start_comm(hostname=None, init_script=None)¶

Run the init script for a start comm

Parameters:	hostname (str or None) – hostname on which to start comm init_script (str or None) – optional path to a PBS init script

stop(hostname=None, init_script=None)¶

Run the init script for a stop operation

Parameters:	hostname (str or None) – hostname on which to execute the init script init_script (str or None) – optional path to a PBS init script

stop_server(hostname=None, init_script=None)¶

Run the init script for a stop server

Parameters:	hostname (str or None) – hostname on which to stop server init_script (str or None) – optional path to a PBS init script

stop_mom(hostname=None, init_script=None)¶

Run the init script for a stop mom

Parameters:	hostname (str or None) – hostname on which to stop mom init_script (str or None) – optional path to a PBS init script

stop_sched(hostname=None, init_script=None)¶

Run the init script for a stop sched

Parameters:	hostname (str or None) – hostname on which to stop sched init_script (str or None) – optional path to a PBS init script

stop_comm(hostname=None, init_script=None)¶

Run the init script for a stop comm

Parameters:	hostname (str or None) – hostname on which to stop comm init_script (str or None) – optional path to a PBS init script

status(hostname=None, init_script=None)¶

Run the init script for a status operation

Parameters:	hostname (str or None) – hostname on which to execute the init script init_script (str or None) – optional path to a PBS init script

status_server(hostname=None, init_script=None)¶

Run the init script for a status server

Parameters:	hostname (str or None) – hostname on which to status server init_script (str or None) – optional path to a PBS init script

status_mom(hostname=None, init_script=None)¶

Run the init script for a status mom

Parameters:	hostname (str or None) – hostname on which to status mom init_script (str or None) – optional path to a PBS init script

status_sched(hostname=None, init_script=None)¶

Run the init script for a status sched

Parameters:	hostname (str or None) – hostname on which to status sched init_script (str or None) – optional path to a PBS init script

status_comm(hostname=None, init_script=None)¶

Run the init script for a status comm

Parameters:	hostname (str or None) – hostname on which to status comm init_script (str or None) – optional path to a PBS init script

switch_version(hostname=None, version=None)¶

Switch to another version of PBS installed on the system

Parameters:	hostname (str or None) – The hostname to operate on version – version to switch

1.2. Utilities¶

1.2.1. pbs_cliutils¶

class ptl.utils.pbs_cliutils.CliUtils¶

Command line interface utility

classmethod get_logging_level(level)¶

Get the logging levels

Parameters:	level (str) – Name of the logging level

static check_bin(bin_name)¶

Check for command exist

Parameters:	bin_name (str) – Command to be checked
Returns:	True if command exist else return False

static expand_abs_path(path)¶: Expand the path to absolute path

static priv_ports_info(hostname=None)¶

Return a list of privileged ports in use on a given host

Parameters:	hostname (str or None) – The host on which to query privilege ports usage. Defaults to the local host

1.2.2. pbs_covutils¶

class ptl.utils.pbs_covutils.LcovUtils(cov_bin=None, html_bin=None, cov_out=None, data_dir=None, html_nosrc=False, html_baseurl=None)¶

Coverage Utils

Parameters:	cov_bin – Coverage binary html_bin – Coverage html binary cov_out (str or None) – Coverage output directory data_dir (str or None) – Coverage data directory html_nosrc – HTML reports without PBS source html_baseurl (str or None) – HTML base url

set_html_baseurl(baseurl)¶: Set HTML base url

set_html_nosource(nosource=False)¶: Set HTML nosource parameter to False.

set_coverage_bin(cov_bin=None)¶

Set the coverage binary

Parameters:	cov_binary – Coverage bin to set

set_genhtml_bin(html_bin=None)¶

Set HTML generation utility.

Parameters:	html_bin – HTML bin to set

set_coverage_out(cov_out=None)¶

Set the coverage output directory

Parameters:	cov_out – Coverage output directory path.

set_coverage_data_dir(data=None)¶

Set the coverage data directory

Parameters:	data – Data directory path
Returns:	True if file name ends with .gcno else return False

add_trace(trace)¶

Add coverage trace

Parameters:	trace – Coverage trace

create_coverage_data_files(path)¶: Create .gcda counterpart files for every .gcno file and give it read/write permissions

initialize_coverage(out=None, name=None)¶

Initialize coverage

Parameters:	out (str or None) – Output path name (str or None) – name of the command

capture_coverage(out=None, name=None)¶: Capture the coverage parameters

zero_coverage()¶: Zero the data counters. Note that a process would need to be restarted in order to collect data again, running --initialize will not get populate the data counters

merge_coverage_traces(out=None, name=None, exclude=None)¶: Merge the coverage traces

generate_html(out=None, html_out=None, html_nosrc=False)¶: Generate the HTML report

change_baseurl(html_out=None, html_baseurl=None)¶: Change the HTML base url

summarize_coverage(out=None)¶: Summarize the coverage output

1.2.3. pbs_dshutils¶

exception ptl.utils.pbs_dshutils.PbsConfigError(message=None, rv=None, rc=None, msg=None)¶: Initialize PBS configuration error

exception ptl.utils.pbs_dshutils.PtlUtilError(message=None, rv=None, rc=None, msg=None)¶: Initialize PTL Util error

class ptl.utils.pbs_dshutils.DshUtils¶

PBS shell utilities

A set of tools to run commands, copy files, get process information and parse a PBS configuration on an arbitrary host

get_platform(hostname=None, pyexec=None)¶

Get a local or remote platform info, essentially the value of Python’s sys.platform, in case of Cray it will return a string as “cray” for actual Cray cluster and “craysim” for Cray ALPS simulator

Parameters:	hostname (str or None) – The hostname to query for platform info pyexec (str or None) – A path to a Python interpreter to use to query a remote host for platform info

For efficiency the value is cached and retrieved from the cache upon subsequent request

get_uname(hostname=None, pyexec=None)¶

Get a local or remote platform info in uname format, essentially the value of Python’s platform.uname :param hostname: The hostname to query for platform info :type hostname: str or None :param pyexec: A path to a Python interpreter to use to query

a remote host for platform info

For efficiency the value is cached and retrieved from the cache upon subsequent request

get_os_info(hostname=None, pyexec=None)¶

Get a local or remote OS info

Parameters:	hostname (str or None) – The hostname to query for platform info pyexec (str or None) – A path to a Python interpreter to use to query a remote host for platform info
Returns:	a ‘str’ object containing os info

get_pbs_conf_file(hostname=None)¶

Get the path of the pbs conf file. Defaults back to /etc/pbs.conf if unsuccessful

Parameters:	hostname (str or None) – Hostname of the machine
Returns:	Path to pbs conf file

parse_pbs_config(hostname=None, file=None)¶

initialize pbs_conf dictionary by parsing pbs config file

Parameters:	file (str or None) – PBS conf file

set_pbs_config(hostname=None, fin=None, fout=None, append=True, confs=None)¶

Set environment/configuration variables in a pbs.conf file

Parameters:	hostname (str or None) – the name of the host on which to operate fin (str or None) – the input pbs.conf file fout (str or None) – the name of the output pbs.conf file, defaults to `/etc/pbs.conf` append (boolean) – whether to append to fout or not, defaults to True confs (Dictionary or None) – The `key/value` pairs to create

unset_pbs_config(hostname=None, fin=None, fout=None, confs=None)¶

Unset environment/configuration variables in a pbs.conf file

Parameters:	hostname (str or None) – the name of the host on which to operate fin (str or None) – the input pbs.conf file fout (str or None) – the name of the output pbs.conf file, defaults to `/etc/pbs.conf` confs (List or str or dict or None) – The configuration keys to unset

get_pbs_server_name(pbs_conf=None)¶: Return the name of the server which may be different than PBS_SERVER,in order, this method looks at PBS_PRIMARY, PBS_SERVER_HOST_NAME, and PBS_LEAF_NAME, and PBS_SERVER

parse_pbs_environment(hostname=None, file='/var/spool/pbs/pbs_environment')¶: Initialize pbs_conf dictionary by parsing pbs config file

set_pbs_environment(hostname=None, fin='/var/spool/pbs/pbs_environment', fout=None, append=True, environ=None)¶

Set the PBS environment

Parameters:	environ (dict or None) – variables to set hostname (str or None) – Hostname of the machine fin (str) – pbs_environment input file fout (str or None) – pbs_environment output file append (bool) – whether to append to fout or not, defaults defaults to true

unset_pbs_environment(hostname=None, fin='/var/spool/pbs/pbs_environment', fout=None, environ=None)¶

Unset environment variables in a pbs_environment file

Parameters:	hostname (str or None) – the name of the host on which to operate fin (str) – the input pbs_environment file fout (str or None) – the name of the output pbs.conf file, defaults to `/var/spool/pbs/pbs_environment` environ (List or str or dict or None) – The environment keys to unset

parse_rhosts(hostname=None, user=None)¶

Parse remote host

Parameters:	hostname (str or None) – Hostname of the machine user (str or None) – User name

set_rhosts(hostname=None, user=None, entry={}, append=True)¶

Set the remote host attributes

Parameters:	entry (Dictionary) – remote hostname user dictionary append (boolean) – If true append key value else not

map_pbs_conf_to_cmd(cmd_map={}, pconf={})¶

Map PBS configuration parameter to command

Parameters:	cmd_map (Dictionary) – command mapping pconf (Dictionary) – PBS conf parameter dictionary

get_current_user()¶: helper function to return the name of the current user

check_user_exists(username=None, hostname=None)¶

Check if user exist or not

Parameters:	username (str or None) – Username to check hostname (str or None) – Machine hostname
Returns:	True if exist else return False

check_group_membership(username=None, uid=None, grpname=None, gid=None)¶

Checks whether a user, passed in as username or uid, is a member of a group, passed in as group name or group id.

Parameters:	username (str or None) – The username to inquire about uid – The uid of the user to inquire about (alternative to username) grpname (str or None) – The groupname to check for user membership gid – The group id to check for user membership (alternative to grpname)

group_memberships(group_list=[])¶: Returns all group memberships as a dictionary of group names and associated memberships

get_id_info(user)¶

Return user info in dic format obtained by "id -a <user>" command for given user

Parameters: user (str) – The username to inquire about

Returns:

dic format:

{

“uid”: <uid of given user>,
”gid”: <gid of given user’s primary group>,

”name”: <name of given user>,

”pgroup”: <name of primary group of given user>,

”groups”: <list of names of groups of given user>

}

get_tempdir(hostname=None)¶

Returns:	The temporary directory on the given host Default host is localhost.

run_cmd(hosts=None, cmd=None, sudo=False, stdin=None, stdout=-1, stderr=-1, input=None, cwd=None, env=None, runas=None, logerr=True, as_script=False, wait_on_script=True, level=18)¶

Run a command on a host or list of hosts.

Parameters:

hosts (str or None) – the name of hosts on which to run the command, can be a comma-separated string or a list. Defaults to localhost
cmd (str or None) – the command to run
sudo (boolean) – whether to run the command as root or not. Defaults to False.
stdin – custom stdin. Defaults to PIPE
stdout – custom stdout. Defaults to PIPE
stderr – custom stderr. Defaults to PIPE
input – input to pass to the pipe on target host, e.g. PBS answer file
cwd – working directory on local host from which command is run
env – environment variables to set on local host
runas – run command as given user. Defaults to calling user
logerr (boolean) – whether to log error messages or not. Defaults to True
as_script (boolean) – if True, run the command in a script created as a temporary file that gets deleted after being run. This is used mainly to circumvent some implementations of sudo that prevent passing environment variables through sudo.
wait_on_script (boolean) – If True (default) waits on process launched as script to return.

Returns:

error, output, return code as a dictionary: {'out':...,'err':...,'rc':...}

run_copy(hosts=None, src=None, dest=None, sudo=False, uid=None, gid=None, mode=None, env=None, logerr=True, recursive=False, runas=None, preserve_permission=True, level=18)¶

copy a file or directory to specified target hosts.

Parameters:

hosts (str or None) – the host(s) to which to copy the data. Can be a comma-separated string or a list
src (str or None) – the path to the file or directory to copy. If src is remote,it must be prefixed by the hostname. e.g. remote1:/path,remote2:/path
dest (str or None) – the destination path.
sudo (boolean) – whether to copy as root or not. Defaults to False
uid – optionally change ownership of dest to the specified user id,referenced by uid number or username
gid – optionally change ownership of dest to the specified group name/id
mode – optinoally set mode bits of dest
env – environment variables to set on the calling host
logerr – whether to log error messages or not. Defaults to True.
recursive (boolean) – whether to copy a directory (when true) or a file.Defaults to False.
runas (str or None) – run command as user
preserve_permission – Preserve file permission while copying file (cp cmd with -p flag) Defaults to True

:type preserve_permission:boolean :param level: logging level, defaults to DEBUG :type level: int :returns: {‘out’:<outdata>, ‘err’: <errdata>, ‘rc’:<retcode>}

upon and None if no source file specified

run_ptl_cmd(hostname, cmd, sudo=False, stdin=None, stdout=-1, stderr=-1, input=None, cwd=None, env=None, runas=None, logerr=True, as_script=False, wait_on_script=True, level=18)¶: Wrapper method of run_cmd to run PTL command

classmethod set_sudo_cmd(cmd)¶: set the sudo command

classmethod set_copy_cmd(cmd)¶: set the copy command

classmethod set_rsh_cmd(cmd)¶: set the remote shell command

is_localhost(host=None)¶

Parameters:	host (str or None) – Hostname of machine
Returns:	true if specified host (by name) is the localhost all aliases matching the hostname are searched

isdir(hostname=None, path=None, sudo=False, runas=None, level=18)¶

Parameters:	hostname (str or None) – The name of the host on which to check for directory path (str or None) – The path to the directory to check sudo (boolean) – Whether to run the command as a privileged user runas (str or None) – run command as user level – Logging level
Returns:	True if directory pointed to by path exists and False otherwise

isfile(hostname=None, path=None, sudo=False, runas=None, level=18)¶

Parameters:	hostname (str or None) – The name of the host on which to check for file path (str or None) – The path to the file to check sudo (boolean) – Whether to run the command as a privileged user runas (str or None) – run command as user level – Logging level
Returns:	True if file pointed to by path exists, and False otherwise

getmtime(hostname=None, path=None, sudo=False, runas=None, level=18)¶

Parameters:	hostname (str or None) – The name of the host on which file exists path (str or None) – The path to the file to get mtime sudo (boolean) – Whether to run the command as a privileged user runas (str or None) – run command as user level – Logging level
Returns:	Modified time of given file

listdir(hostname=None, path=None, sudo=False, runas=None, fullpath=True, level=18)¶

Parameters:	hostname (str or None) – The name of the host on which to list for directory path (str or None) – The path to directory to list sudo (bool) – Whether to chmod as root or not. Defaults to False runas (str or None) – run command as user fullpath (bool) – Return full paths? level (int) – Logging level.
Returns:	A list containing the names of the entries in the directory

chmod(hostname=None, path=None, mode=None, sudo=False, runas=None, recursive=False, logerr=True, level=18)¶

Generic function of chmod with remote host support

Parameters:

hostname (str or None) – hostname (default current host)
path (str or None) – the path to the file or directory to chmod
mode – mode to apply as octal number like 0777, 0666 etc.
sudo (boolean) – whether to chmod as root or not. Defaults to False
runas (str or None) – run command as user
recursive (boolean) – whether to chmod a directory (when true) or a file.Defaults to False.
logerr (boolean) – whether to log error messages or not. Defaults to True.
level – logging level, defaults to INFOCLI2

Returns:

True on success otherwise False

chown(hostname=None, path=None, uid=None, gid=None, sudo=False, recursive=False, runas=None, logerr=True, level=18)¶

Generic function of chown with remote host support

Parameters:

hostname (str or None) – hostname (default current host)
path (str or None) – the path to the file or directory to chown
uid – uid to apply (must be either user name or uid or -1)
gid – gid to apply (must be either group name or gid or -1)
sudo (boolean) – whether to chown as root or not. Defaults to False
recursive (boolean) – whether to chmod a directory (when true) or a file.Defaults to False.
runas (str or None) – run command as user
logerr (boolean) – whether to log error messages or not. Defaults to True.
level – logging level, defaults to INFOCLI2

Returns:

True on success otherwise False

chgrp(hostname=None, path=None, gid=None, sudo=False, recursive=False, runas=None, logerr=True, level=18)¶

Generic function of chgrp with remote host support

Parameters:

hostname (str or None) – hostname (default current host)
path (str or None) – the path to the file or directory to chown
gid – gid to apply (must be either group name or gid or -1)
sudo (boolean) – whether to chgrp as root or not. Defaults to False
recursive (boolean) – whether to chmod a directory (when true) or a file.Defaults to False.
runas (str or None) – run command as user
logerr (boolean) – whether to log error messages or not. Defaults to True.
level – logging level, defaults to INFOCLI2

Returns:

True on success otherwise False

which(hostname=None, exe=None, level=18)¶

Generic function of which with remote host support

Parameters:	hostname (str or None) – hostname (default current host) exe (str or None) – executable to locate (can be full path also) (if exe is full path then only basename will be used to locate) level – logging level, defaults to INFOCLI2

rm(hostname=None, path=None, sudo=False, runas=None, recursive=False, force=False, cwd=None, logerr=True, as_script=False, level=18)¶

Generic function of rm with remote host support

Parameters:

hostname (str or None) – hostname (default current host)
path (str or None) – the path to the files or directories to remove for more than one files or directories pass as list
sudo (boolean) – whether to remove files or directories as root or not.Defaults to False
runas – remove files or directories as given user Defaults to calling user
recursive (boolean) – remove files or directories and their contents recursively
force (boolean) – force remove files or directories
cwd – working directory on local host from which command is run
logerr (boolean) – whether to log error messages or not. Defaults to True.
as_script (boolean) – if True, run the rm in a script created as a temporary file that gets deleted after being run. This is used mainly to handle wildcard in path list. Defaults to False.
level – logging level, defaults to INFOCLI2

Returns:

True on success otherwise False

mkdir(hostname=None, path=None, mode=None, sudo=False, runas=None, parents=True, cwd=None, logerr=True, as_script=False, level=18)¶

Generic function of mkdir with remote host support

Parameters:

hostname (str or None) – hostname (default current host)
path (str or None) – the path to the directories to create for more than one directories pass as list
mode – mode to use while creating directories (must be octal like 0777)
sudo (boolean) – whether to create directories as root or not. Defaults to False
runas – create directories as given user. Defaults to calling user
parents (boolean) – create parent directories as needed. Defaults to True
cwd (str or None) – working directory on local host from which command is run
logerr (boolean) – whether to log error messages or not. Defaults to True.
as_script (boolean) – if True, run the command in a script created as a temporary file that gets deleted after being run. This is used mainly to handle wildcard in path list. Defaults to False.
level – logging level, defaults to INFOCLI2

Returns:

True on success otherwise False

cat(hostname=None, filename=None, sudo=False, runas=None, logerr=True, level=18)¶

Generic function of cat with remote host support

Parameters:

hostname (str or None) – hostname (default current host)
filename (str or None) – the path to the filename to cat
sudo (boolean) – whether to create directories as root or not. Defaults to False
runas (str or None) – create directories as given user. Defaults to calling user
logerr (boolean) – whether to log error messages or not. Defaults to True.

Returns:

output of run_cmd

cmp(hostname=None, fileA=None, fileB=None, sudo=False, runas=None, logerr=True)¶

Compare two files and return 0 if they are identical or non-zero if not

Parameters:

hostname (str or None) – the name of the host to operate on
fileA (str or None) – the first file to compare
fileB (str or None) – the file to compare fileA to
sudo (boolean) – run the command as a privileged user
runas (str or None) – run the cmp command as given user
logerr (boolean) – whether to log error messages or not. Defaults to True.

useradd(name, uid=None, gid=None, shell='/bin/bash', create_home_dir=True, home_dir=None, groups=None, logerr=True, level=18)¶

Add the user

Parameters:	name (str) – User name shell – shell to use create_home_dir (boolean) – If true then create home directory home_dir (str or None) – path to home directory groups – User groups

userdel(name, del_home=True, force=True, logerr=True, level=18)¶

Delete the user

Parameters:	del_home (boolean) – If true then delete user home force (boolean) – If true then delete forcefully

groupadd(name, gid=None, logerr=True, level=18)¶: Add a group

create_temp_file(hostname=None, suffix='', prefix='PtlPbs', dirname=None, text=False, asuser=None, body=None, level=18)¶

Create a temp file by calling tempfile.mkstemp

Parameters:

hostname (str or None) – the hostname on which to query tempdir from
suffix (str) – the file name will end with this suffix
prefix (str) – the file name will begin with this prefix
dirname (str or None) – the file will be created in this directory
text (boolean) – the file is opened in text mode is this is true else in binary mode
asuser (str or None) – Optional username or uid of temp file owner
body (str or None) – Optional content to write to the temporary file
level (int) – logging level, defaults to INFOCLI2

mkdtemp(hostname=None, suffix='', prefix='PtlPbs', dir=None, uid=None, gid=None, mode=None, level=18)¶

Create a temp dir by calling tempfile.mkdtemp

Parameters:

hostname (str or None) – the hostname on which to query tempdir from
suffix (str) – the directory name will end with this suffix
prefix (str) – the directory name will begin with this prefix
dir (str or None) – the directory will be created in this directory
uid – Optional username or uid of temp directory owner
gid – Optional group name or gid of temp directory group owner
mode – Optional mode bits to assign to the temporary directory
level – logging level, defaults to INFOCLI2

parse_strace(lines)¶: strace parsing. Just the regular expressions for now

1.2.4. pbs_fileutils¶

class ptl.utils.pbs_fileutils.FileUtils(f, mode='head')¶

Utility to walk a file from 'head' or 'tail' on the local filesystem

Parameters:	f (str) – File to process mode – One of FILE_HEAD or FILE_TAIL, which respectively set the file for processing from head or tail. Defaults to head.

get_file_descriptor()¶: Return the file descriptor associated to the file being processed

set_buf_size(bs=1024)¶: Set the buffer size to read blocks of file into

set_mode(m=None)¶

Parameters:	m – `FILE_TAIL` if file to be tailed, and `FILE_HEAD` to read from head

tell()¶

Returns:	The current file `'cursor'`

get_size()¶

Returns:	The size of the file

get_num_lines()¶

Returns:	No of lines for the file

next(n=1)¶

Get the next n lines of the file

Parameters:	n (int) – the numer of lines to retrieve

get_line(n=1)¶

Returns:	The nth line from file

get_block(from_n=1, to_n=1)¶

Returns:	A block of lines between `from_n` and `to_n`

next_head(n=1)¶: Next line(s) from head

next_tail(n=1)¶: Next line(s) from tail

head(n=1)¶

Returns:	n lines of head

tail(n=1)¶

Returns:	n lines of tail

1.2.5. pbs_logutils¶

class ptl.utils.pbs_logutils.PBSLogUtils¶

Miscellaneous utilities to process log files

classmethod convert_date_time(dt=None, fmt=None)¶

convert a date time string of the form given by fmt into number of seconds since epoch (with possible microseconds). it considers the current system’s timezone to convert the datetime to epoch time

Parameters:	dt (str or None) – the datetime string to convert fmt (str) – Format to which datetime is to be converted
Returns:	None if conversion fails

get_num_lines(log, hostname=None, sudo=False)¶

Get the number of lines of particular log

Parameters:	log (str) – the log file name

open_log(log, hostname=None, sudo=False)¶

Parameters:	log (str) – the log file name to read from hostname (str or None) – the hostname from which to read the file sudo (boolean) – Whether to access log file as a privileged user.
Returns:	A file instance

get_timestamps(logfile=None, hostname=None, num=None, sudo=False)¶

Helper function to parse logfile

Returns:	Each timestamp in a list as number of seconds since epoch

match_msg(lines, msg, allmatch=False, regexp=False, starttime=None, endtime=None)¶

Returns (x,y) where x is the matching line y, or None if nothing is found.

Parameters:	allmatch (boolean) – If True (False by default), return a list of matching tuples. regexp (bool) – If True, msg is a Python regular expression. Defaults to False. starttime – If set ignore matches that occur before specified time endtime – If set ignore matches that occur after specified time

static convert_resv_date_time(date_time)¶: Convert reservation datetime to seconds

static convert_hhmmss_time(tm)¶: Convert datetime in hhmmss format to seconds

get_rate(l=[])¶

Returns:	The frequency of occurrences of array l The array is expected to be sorted

in_range(tm, start=None, end=None)¶

Parameters:	tm – time to check within a provided range start – Lower limit for the time range end – Higer limit for the time range
Returns:	True if time is in the range else return False

static get_day(tm=None)¶

Parameters:	tm – Time for which to get a day

static percentile(N, percent)¶

Find the percentile of a list of values.

Parameters:	N (List) – A list of values. Note N MUST BE already sorted. percent (Float) – A float value from 0.0 to 1.0.
Returns:	The percentile of the values

static process_intervals(intervals, groups, frequency=60)¶: Process the intervals

get_log_files(hostname, path, start, end, sudo=False)¶

Parameters:	hostname (str) – Hostname of the machine path (str) – Path for the log file start – Start time for the log file end – End time for the log file
Returns:	list of log file(s) found or an empty list

class ptl.utils.pbs_logutils.PBSLogAnalyzer(schedlog=None, serverlog=None, momlog=None, acctlog=None, genericlog=None, hostname=None, show_progress=False)¶

Utility to analyze the PBS logs

set_custom_match(pattern, frequency=None)¶

Set the custome matching

Parameters:	pattern (str) – Matching pattern frequency (int) – Frequency of match

set_conditional_match(conditions)¶

Set the conditional match

Parameters:	conditions – Conditions for macthing

analyze_scheduler_log(filename=None, start=None, end=None, hostname=None, summarize=True)¶

Analyze the scheduler log

Parameters:	filename (str or None) – Scheduler log file name start – Time from which log to be analyzed end – Time till which log to be analyzed hostname (str or None) – Hostname of the machine summarize (bool) – Summarize data parsed if True else not

analyze_server_log(filename=None, start=None, end=None, hostname=None, summarize=True)¶: Analyze the server log

analyze_accounting_log(filename=None, start=None, end=None, hostname=None, summarize=True)¶: Analyze the accounting log

analyze_mom_log(filename=None, start=None, end=None, hostname=None, summarize=True)¶: Analyze the mom log

parse_conditional(rec, start, end)¶

Match a sequence of regular expressions against multiple consecutive lines in a generic log. Calculate the number of conditional matching lines.

Example usage: to find the number of times the scheduler stat’ing the server causes the scheduler to miss jobs ending, which could possibly indicate a race condition between the view of resources assigned to nodes and the actual jobs running, one would call this function by setting re_conditional to ['Type 20 request received from Scheduler', 'Exit_status'] Which can be read as counting the number of times that the Type 20 message is preceded by an Exit_status message

analyze(path=None, start=None, end=None, hostname=None, summarize=True, sudo=False)¶

Parse any log file. This method is not context-specific to each log file type.

Parameters:

path (str or None) – name of file/dir to parse
start – optional record time at which to start analyzing
end – optional record time after which to stop analyzing
hostname (str or None) – name of host on which to operate. Defaults to localhost
summarize (bool) – if True, summarize data parsed. Defaults to True.
sudo (bool) – If True, access log file(s) as privileged user.

analyze_logs(schedlog=None, serverlog=None, momlog=None, acctlog=None, genericlog=None, start=None, end=None, hostname=None, showjob=False)¶: Analyze logs

class ptl.utils.pbs_logutils.PBSServerLog(filename=None, hostname=None, show_progress=False)¶

Parameters:	filename (str or None) – Server log filename hostname (str or None) – Hostname of the machine

parse_runjob(line)¶: Parse server log for run job records. For each record keep track of the job id, and time in a dedicated array

parse_endjob(line)¶: Parse server log for run job records. For each record keep track of the job id, and time in a dedicated array

parse_nodeup(line)¶: Parse server log for nodes that are up

parse_enquejob(line)¶: Parse server log for enqued jobs

class ptl.utils.pbs_logutils.JobEstimatedStartTimeInfo(jobid)¶

Information regarding Job estimated start time

add_estimate(tm)¶

Add a job’s new estimated start time If the new estimate is now later than any preivous one, we add that difference to the drift time. If the new drift time is pulled earlier it is not added to the drift time.

drift time is a measure of "negative perception" that comes along a job being estimated to run at a later date than earlier "advertised".

class ptl.utils.pbs_logutils.PBSSchedulerLog(filename=None, hostname=None, show_progress=False)¶

get_cycles(start=None, end=None)¶

Get the scheduler cycles

Parameters:	start – Start time end – End time
Returns:	Scheduling cycles

estimated_info_parsing(line)¶: Parse Estimated start time information for a job

summarize_estimated_analysis(estimated_jobs=None)¶: Summarize estimated job analysis

summary(cycles=None, showjobs=False)¶: Scheduler log summary

class ptl.utils.pbs_logutils.PBSMoMLog(filename=None, hostname=None, show_progress=False)¶

Container and Parser of a PBS MoM log

summary()¶: Mom log summary

class ptl.utils.pbs_logutils.PBSAccountingLog(filename=None, hostname=None, show_progress=False)¶

Container and Parser of a PBS accounting log

enable_running_jobs_parsing()¶: Enable parsing for running jobs

enable_utilization_parsing(hostname=None, nodesfile=None, jobsfile=None)¶

Enable utilization parsing

Parameters:	hostname (str or None) – Hostname of the machine nodesfile (str or None) – optional file containing output of pbsnodes -av jobsfile (str or None) – optional file containing output of qstat -f

enable_job_info_parsing()¶: Enable job information parsing

enable_accounting_workload_parsing()¶: Enable accounting workload parsing

process_nodes_data(hostname=None, nodesfile=None, jobsfile=None)¶

Get job and node information by stat’ing and parsing node data from the server. Compute the number of nodes and populate a list of running job ids on those nodes.

Parameters:	hostname (str or None) – The host to query nodesfile (str or None) – optional file containing output of pbsnodes -av jobsfile (str or None) – optional file containing output of qstat -f

The node data is needed to compute counts of nodes and cpus The job data is needed to compute the amount of resources requested

accounting_parsing(rec, start, end)¶: Parsing accounting log

job_info(rec)¶: PBS Job information

finished_jobs_nodes(last=None, tm_range=None)¶

Parameters:	tm_range – a tuple of time where the first item is the start time last – If tm_range is None and last is specified, a time range from now till ‘last’ seconds from now is used as a range of time to consider and the second item is the end time to consider
Returns:	A dictionary of jobs that ended in the time range as keys,and nodes (hostnames) on which those jobs were running as values.

summary()¶: Accounting log summary

1.2.6. pbs_procutils¶

class ptl.utils.pbs_procutils.ProcUtils¶

Utilities to query process information

get_ps_cmd(hostname=None)¶

Get the ps command

Parameters:	hostname (str or None) – hostname of the machine

get_proc_info(hostname=None, name=None, pid=None, regexp=False)¶

Return process information from a process name, or pid, on a given host

Parameters:	hostname (str or none) – The hostname on which to query the process info. On Windows,only localhost is queried. name (str or None) – The name of the process to query. pid (int or None) – The pid of the process to query regexp (bool) – Match processes by regular expression. Defaults to True. Does not apply to matching by PID.
Returns:	A list of ProcInfo objects, one for each matching process.

Note

If both, name and pid, are specified, name is used.

get_proc_state(hostname=None, pid=None)¶

Returns:	PID’s process state on host hostname

On error the empty string is returned.

get_proc_children(hostname=None, ppid=None)¶

Returns:	A list of children PIDs associated to `PPID` on host hostname.

On error, an empty list is returned.

class ptl.utils.pbs_procutils.ProcInfo(name=None, pid=None)¶: Process information reports PID, RSS, VSZ, Command and Time at which process information is collected

class ptl.utils.pbs_procutils.ProcMonitor(name=None, regexp=False, frequency=60)¶

A background process monitoring tool

set_frequency(value=60)¶

Set the frequency

Parameters:	value (int) – Frequency value

run()¶: Run the process monitoring

stop()¶: Stop the process monitoring

1.2.7. pbs_testsuite¶

ptl.utils.pbs_testsuite.skip(reason='Skipped test execution')¶

Unconditionally skip a test.

Parameters:	reason (str or None) – Reason for the skip

ptl.utils.pbs_testsuite.timeout(val)¶: Decorator to set timeout value of test case

ptl.utils.pbs_testsuite.checkModule(modname)¶: Decorator to check if named module is available on the system and if not skip the test

ptl.utils.pbs_testsuite.skipOnCray(function)¶: Decorator to skip a test on a Cray system

ptl.utils.pbs_testsuite.skipOnCpuSet(function)¶: Decorator to skip a test on a CpuSet system

ptl.utils.pbs_testsuite.requirements(*args, **kwargs)¶: Decorator to provide the cluster information required for a particular testcase.

class ptl.utils.pbs_testsuite.PBSServiceInstanceWrapper(*args, **kwargs)¶

A wrapper class to handle multiple service (i.e., mom, server, scheduler)``instances as passed along through the test harness ``(pbs_benchpress).Returns an ordered dictionary of PBS service instances (i.e., mom/server/ scheduler)

Users may invoke PTL using pointers to multiple services per host, for example:

pbs_benchpress -p moms=hostA@/etc/pbs.conf,hostB,hostA@/etc/pbs.conf3

In such cases, the moms instance variable must be able to distinguish the self.moms['hostA'] instances, each instance will be mapped to a unique configuration file

keys() → list of D's keys¶

itervalues() → an iterator over the values of D¶

values() → list of D's values¶

exception ptl.utils.pbs_testsuite.setUpClassError¶

exception ptl.utils.pbs_testsuite.tearDownClassError¶

class ptl.utils.pbs_testsuite.PBSTestSuite(methodName='runTest')¶

Generic setup, teardown, and logging functions to be used as parent class for most tests. Class instantiates:

server object connected to localhost

scheduler objected connected to localhost

mom object connected to localhost

Custom parameters:

Parameters:

server – The hostname on which the PBS server/scheduler are running
mom – The hostname on which the PBS MoM is running
servers – Colon-separated list of hostnames hosting a PBS server. Servers are then accessible as a dictionary in the instance variable servers.
client – For CLI mode only, name of the host on which the PBS client commands are to be run from. Format is <host>@<path-to-config-file>
moms – Colon-separated list of hostnames hosting a PBS MoM. MoMs are made accessible as a dictionary in the instance variable moms.
comms – Colon-separated list of hostnames hosting a PBS Comm. Comms are made accessible as a dictionary in the instance variable comms.
nomom=<host1>:<host2>... – expect no MoM on given set of hosts
mode – Sets mode of operation to PBS server. Can be either 'cli' or 'api'.Defaults to API behavior.
conn_timeout – set a timeout in seconds after which a pbs_connect IFL call is refreshed (i.e., disconnected)
skip-setup – Bypasses setUp of PBSTestSuite (not custom ones)
skip-teardown – Bypasses tearDown of PBSTestSuite (not custom ones)
procinfo – Enables process monitoring thread, logged into ptl_proc_info test metrics. The value can be set to _all_ to monitor all PBS processes,including pbs_server, pbs_sched, pbs_mom, or a process defined by name.
revert-to-defaults=<True|False> – if False, will not revert to defaults.True by default.
revert-hooks=<True|False> – if False, do not revert hooks to defaults.Defaults to True. revert-to-defaults set to False overrides this setting.
del-hooks=<True|False> – If False, do not delete hooks. Defaults to False.``revert-to-defaults`` set to False overrides this setting.
revert-queues=<True|False> – If False, do not revert queues to defaults.Defaults to True. revert-to-defaults set to False overrides this setting.
revert-resources=<True|False> – If False, do not revert resources to defaults. Defaults to True. revert-to-defaults set to False overrides this setting.
del-queues=<True|False> – If False, do not delete queues. Defaults to False.``revert-to-defaults`` set to Falseoverrides this setting.
del-vnodes=<True|False> – If False, do not delete vnodes on MoM instances.Defaults to True.
server-revert-to-defaults=<True|False> – if False, don’t revert Server to defaults
comm-revert-to-defaults=<True|False> – if False, don’t revert Comm to defaults
mom-revert-to-defaults=<True|False> – if False, don’t revert MoM to defaults
sched-revert-to-defaults=<True|False> – if False, don’t revert Scheduler to defaults
procmon – Enables process monitoring. Multiple values must be colon separated. For example to monitor server, sched, and mom use procmon=pbs_server:pbs_sched:pbs_mom
procmon-freq – Sets a polling frequency for the process monitoring tool.Defaults to 10 seconds.
test-users – colon-separated list of users to use as test users. The users specified override the default users in the order in which they appear in the PBS_USERS list.
default-testcase-timeout – Default test case timeout value.
data-users – colon-separated list of data users.
oper-users – colon-separated list of operator users.
mgr-users – colon-separated list of manager users.
root-users – colon-separated list of root users.
build-users – colon-separated list of build users.
clienthost – the hostnames to set in the MoM config file

classmethod setUpClass()¶: Hook method for setting up class fixture before running tests in the class.

setUp()¶: Hook method for setting up the test fixture before exercising it.

classmethod check_users_exist()¶: Check whether the user is exist or not

classmethod kicksched_action(server, obj_type, *args, **kwargs)¶: custom scheduler action to kick a scheduling cycle when expectig a job state change

classmethod parse_param()¶

get test configuration parameters as a comma-separated list of attributes.

Attributes may be '=' separated key value pairs or standalone entries.

Multi-property attributes are colon-delimited.

classmethod is_server_licensed(server)¶: Check if server is licensed or not

classmethod init_from_conf(conf, single=None, multiple=None, skip=None, func=None)¶

Helper method to parse test parameters for`` mom/server/scheduler`` instances.

The supported format of each service request is:

hostname@configuration/path

For example:

pbs_benchpress -p server=remote@/etc/pbs.conf.12.0

initializes a remote server instance that is configured according to the remote file /etc/pbs.conf.12.0

classmethod init_servers(init_server_func=None, skip=None)¶: Initialize servers

classmethod init_comms(init_comm_func=None, skip=None)¶: Initialize comms

classmethod init_schedulers(init_sched_func=None, skip=None)¶: Initialize schedulers

classmethod init_moms(init_mom_func=None, skip='nomom')¶: Initialize moms

classmethod init_server(hostname, pbsconf_file=None)¶

Initialize a server instance

Define custom expect action to trigger a scheduling cycle when job is not in running state

Returns:	The server instance on success and None on failure

classmethod init_comm(hostname, pbsconf_file=None, server=None)¶

Initialize a Comm instance associated to the given hostname.

This method must be called after init_server

Parameters:	hostname (str) – The host on which the Comm is running pbsconf_file (str or None) – Optional path to an alternate pbs config file server (str) – The server name associated to the Comm
Returns:	The instantiated Comm upon success and None on failure.

Return the instantiated Comm upon success and None on failure.

classmethod init_scheduler(hostname, pbsconf_file=None, server=None)¶

Initialize a Scheduler instance associated to the given server. This method must be called after init_server

Parameters:	server (str) – The server name associated to the scheduler pbsconf_file (str or None) – Optional path to an alternate config file hostname (str) – The host on which Sched is running
Returns:	The instantiated scheduler upon success and None on failure

classmethod init_mom(hostname, pbsconf_file=None, server=None)¶

Initialize a MoM instance associated to the given hostname.

This method must be called after init_server

Parameters:	hostname (str) – The host on which the MoM is running pbsconf_file (str or None) – Optional path to an alternate pbs config file
Returns:	The instantiated MoM upon success and None on failure.

init_proc_mon()¶: Initialize process monitoring when requested

revert_pbsconf()¶: Revert contents of the pbs.conf file Also start/stop the appropriate daemons

revert_servers(force=False)¶: Revert the values set for servers

revert_comms(force=False)¶: Revert the values set for comms

revert_schedulers(force=False)¶: Revert the values set for schedulers

revert_moms(force=False)¶: Revert the values set for moms

revert_server(server, force=False)¶: Revert the values set for server

revert_comm(comm, force=False)¶: Revert the values set for comm

revert_scheduler(scheduler, force=False)¶: Revert the values set for scheduler

revert_mom(mom, force=False)¶

Revert the values set for mom

Parameters:	mom (MoM object) – the MoM object whose values are to be reverted force (bool) – Option to reverse forcibly

analyze_logs()¶: analyze accounting and scheduler logs from time test was started until it finished

set_test_measurements(mdic=None)¶

set dictionary of analytical results of the test in order to include it in test report

Parameters:	mdic (dict) – dictionary with analytical data
Returns:	True on successful append or False on failure

add_additional_data_to_report(datadic=None)¶

set dictionary that will be merged with the test report for the overall test run

Parameters:	datadic (dict) – dictionary with analytical data
Returns:	True on succssful update or False on failure

start_proc_monitor(name=None, regexp=False, frequency=60)¶

Start the process monitoring

Parameters:	name (str or None) – Process name regexp (bool) – Regular expression to match frequency (int) – Frequency of monitoring

stop_proc_monitor()¶: Stop the process monitoring

skipTest(reason=None)¶

Skip Test

Parameters:	reason (str or None) – message to indicate why test is skipped

skip_test(reason=None)¶

Skip Test

Parameters:	reason (str or None) – message to indicate why test is skipped

tearDown()¶: verify that server and scheduler are up clean up jobs and reservations

classmethod tearDownClass()¶: Hook method for deconstructing the class fixture after running all tests in the class.

1.2.8. Plugins¶

1.2.8.1. ptl_test_data¶

class ptl.utils.plugins.ptl_test_data.PTLTestData¶

Save post analysis data on test cases failure or error

options(parser, env)¶: Register command line options

configure(options, config)¶: Configure the plugin and system, based on selected options

1.2.8.2. ptl_test_db¶

exception ptl.utils.plugins.ptl_test_db.PTLDbError(rv=None, rc=None, msg=None, post=None, *args, **kwargs)¶

PTL database error class

Parameters:	rv (str or None) – Return value for the database error rc (str or None) – Return code for the database error msg (str or None) – Error message

class ptl.utils.plugins.ptl_test_db.DBType(dbtype, dbpath, dbaccess)¶

Base class for each database type Any type of database must inherit from me

Parameters:	dbtype (str) – Database type dbpath (str) – Path to database dbaccess (str) – Path to a file that defines db options