toyz.utils package¶

Subpackages¶

toyz.utils.db_interfaces package

Submodules¶

toyz.utils.core module¶

class toyz.utils.core.Toy(toy, module=None, path=None, config=None, key=None)¶

A toy built on the toyz framework.

This class has not yet been setup

reload(module)¶

class toyz.utils.core.ToyzClass(dict_in)¶: I often prefer to work with classes rather than dictionaries. To allow these objects to be pickled they must be defined as a class, so this is simply a class that converts a dictionary into a class.

class toyz.utils.core.ToyzJobQueue(queue_name, process_count)¶

An environment for a single user to run tasks.

No longer implemented, but may be in the future

add_job(job)¶

close_env(interrupt_jobs=False)¶

initialize()¶

job_worker(queue)¶

class toyz.utils.core.ToyzSettings(config_root_path=None)¶

Settings for the current toyz application.

first_time_setup(config_root_path)¶

Initial setup of the Toyz application and creation of files the first time it’s run.

Parameters

config_root_path (string ): Default root path of the new Toyz instance

load_settings(config_path, security_key=None)¶

Load settings. If toyz_settings.security.encrypt_config is True, the settings file will be decrypted (this requires a securiity key).

Security not yet implemented

save_settings(security_key=None)¶

Save the toyz settings to disk. If toyz_settings.security.encrypt_config is True, the settings file will be encrypted (this requires a security key).

Security not yet implemented

toyz.utils.core.check4keys(myDict, keys)¶

Checks a dictionary for a set of required keys.

Parameters

myDict (dict ): Dictionary to be searched
keys (list ): List of keys to search for in the dictionary

Raises

Raises a toyz.utils.errors.ToyzJobError if any keys are missing from myDict and lists all of the missing keys

toyz.utils.core.check_instance(obj, instances)¶

Check if an object is an instance of another object

Parameters

obj (object ): Object to check
instances (list ): List of objects to crosscheck with obj

Returns

is_instance (bool ): True if the obj is an instances of one of the objects in the list, False otherwise

toyz.utils.core.check_pwd(toyz_settings, user_id, pwd)¶

Check to see if a users password matches the one stored in the database.

Parameters

toyz_settings ( toyz.utils.core.ToyzSettings ): Settings for the current application
user_id (string ): Id of the user logging in
pwd: (string ): password the user has entered

Returns

valid_login (bool ): True if the user name and password match

toyz.utils.core.check_user_modules(toyz_settings, user_id, module)¶

Check to see if a module is available for the user

Parameters

toyz_settings ( toyz.utils.core.ToyzSettings ):

Settings for the application runnning the job (may be needed to load user info or check permissions)

user_id ( string ): id of the current user module ( string ): name of the module to search for

Returns

module_present ( bool ):

true if the user has access to the module, false otherwise

toyz.utils.core.check_user_shortcuts(toyz_settings, user_id, shortcuts=None)¶

Check that a user has all of the default shortcuts.

Parameters

toyz_settings ( toyz.utils.core.ToyzSettings ): Toyz Settings
user_id (string ): User id to check for shortcuts
shortcuts (dict, optional): Dictionary of shortcuts for a user. Shortcuts are always of the form shortcut_name:path .

Returns

shortcuts: dictionary of shortcuts for the user

toyz.utils.core.check_version(db_settings)¶: Since version changes may update the API or the database, check that the current version is compatible with the users configuration and database

toyz.utils.core.create_paths(paths)¶

Search for paths on the server. If a path does not exist, create the necessary directories. For example, if paths=['~/Documents/images/2014-6-5_data/'] and only the path ‘~/Documents’ exists, both ‘~/Documents/images/’ and ‘~/Documents/images/2014-6-5_data/’ are created.

Parameters: paths (string or list of strings): If paths is a string, this is the path to search for and create. If paths is a list, each one is a path to search for and create

toyz.utils.core.encrypt_pwd(toyz_settings, pwd)¶

Use the passlib module to create a hash for the given password.

Parameters

toyz_settings ( toyz.utils.core.ToyzSettings ): Settings for the current application
pwd (string ): password the user has entered

Returns

pwd_hash (string ): Password hash to be stored for the given password. If passwords are not encrypted, this will just return the pwd passed to the function.

toyz.utils.core.get_all_user_modules(toyz_settings, user_id)¶

Get all modules available for the user.

Parameters

toyz_settings ( toyz.utils.core.ToyzSettings ):

Settings for the application runnning the job (may be needed to load user info or check permissions)

user_id ( string ): id of the current user

Returns

user_modules ( list of strings ):

list of all toyz modules that the user has access to, including permissions granted by member groups

toyz.utils.core.get_all_user_toyz(toyz_settings, user_id)¶

Get all available Toyz paths for the current user

Parameters

toyz_settings ( toyz.utils.core.ToyzSettings ):

Settings for the application runnning the job (may be needed to load user info or check permissions)

user_id ( string ): id of the current user

Returns

user_toyz ( dict ):

All of the local python directories that the user can use to run jobs.

toyz.utils.core.get_bool(prompt)¶: Prompt a user for a boolean expression and repeat until a valid boolean has been entered. prompt is the text to prompt the user with.

toyz.utils.core.get_module_info(toyz_settings, tid, params)¶: Get information about modules accessible by the current user

toyz.utils.core.get_toyz_module(toyz_settings, user_id, module)¶

Get a toyz module from either installed modules or one in a users toyz paths. An uninstalled toy will take precedence if it is in the toy paths as opposed to installed modules.

Parameters

toyz_settings ( toyz.utils.core.ToyzSettings ):

Settings for the application runnning the job (may be needed to load user info or check permissions)

user_id ( string ): id of the current user module ( string ): name of the module to search for

Returns

toyz_module ( module ): python module if it exists, otherwise None.

Raises

Raises a toyz.utils.errors.ToyzJobError if the module is not found in the users approved modules

toyz.utils.core.get_user_toyz(toyz_settings, user_id, toy)¶

If a toy is contained in a users toyz paths, get the module and return it

Parameters

toyz_settings ( toyz.utils.core.ToyzSettings ):

Settings for the application runnning the job (may be needed to load user info or check permissions)

user_id ( string ): id of the current user toy ( string ): name of the toy to search for

Returns

toy ( module ): python module if it exists, otherwise None.

toyz.utils.core.get_user_type(params)¶

Since user properties and group properties are kept in the same table and use the same functions, many functions use the keyword user_id or group_id to figure out if they are operating on groups or users.

Parameters

params (dict ): Dictionary of parameters sent to a function

Returns

user (dict ): Dictionary with user_type. This is always either {user_id: params['user_id']} or {group_id: params['group_id']}

toyz.utils.core.get_workspace_permissions(toyz_settings, tid, params)¶

toyz.utils.core.is_int(str_in)¶: Check whether or not a string is an integer.

toyz.utils.core.is_number(str_in)¶: Check whether or not a string is a number

toyz.utils.core.merge_dict(d1, d2, copy=False, keys=[])¶: Python version of jQuery.extend that recursively merges two python dictionaries. If copy is True then the new object is a copy, otherwise d2 is merged into d1 (the same as $.extend). keys should never be passed on the first call as it is used to keep track of which elements have been used (to avoid infinite loops).

toyz.utils.core.normalize_path(path)¶: Format a path with bash symbols like ‘~‘ , ‘.‘ , ‘..‘ into a full absolute path. This simply returns os.path.abspath(os.path.expanduser(path)) .

toyz.utils.core.progress_log(msg)¶

Send a notification to the client to update on the progress of a job

Parameters

msg ( string ): message to send to client

toyz.utils.core.run_job(toyz_settings, pipe, job)¶

Loads modules and runs a job (function) sent from a client. Any errors will be trapped and flagged as a toyz.utils.errors.ToyzError and sent back to the client who initiated the job.

All job functions will take exactly 3 parameters: toyz_settings, tid, params. The tid is the task id (user, session, and request_id ) information (see below), and params is a dictionary of parameters sent by the client.

Each job is run as a new process, so any modules imported should be removed from memory when the job has completed.

Parameters

toyz_settings ( toyz.utils.core.ToyzSettings ):

Settings for the application runnning the job (may be needed to load user info or check permissions)

pipe: multiprocessing.Pipe

Communication with the parent process.
It may be useful to pass progress notifications to a client as a job is run. To send a notifaction the pipe requires a dictionary that is of the same form as the result (see Returns below), but usually with id='notification'.

job: dict

The job received from the user. Each job will contain the following keys:
id (dict): Unique values for the given job. These are the user_id, the id of the user loaded from a secure cookie; the session_id, a unique identifier for the websocket; and the request_id, a unique identifier for the current request, sent from the client
module (str): Name of the Python module that contains the function called by the client. In order for a module to work for a user, he/she must either have permissions set to view the module or belong to a group with permissions set to view the module (including all_users).
task (str): Name of the function called by the client
parameters (dict): Required and optional parameters passed to the function.

Returns

result: dict

The result is returned to the application running the job and is composed of two keys: an id: the job id for the completed job, and a response that is sent to the client.
The response is either an empty dictionary or one that contains (at a minimum) the key id, which is used by the client to identify the type of response it is receiving. Including the key request_completed with a True value tells the client that the current request has finished and may be removed from the queue.

Example

A client might send the following job to the server:
job = {
    id : {
        user_id : 'Iggy',
        session_id : '12',
        request_id : 305
    },
    module : 'toyz.web.tasks',
    task : 'load_directory',
    parameters : {
        path: '~/images'
    }
}
In this case, after receiving the job, this function will import the toyz.web.tasks module (if it has not been imported already) and run the function load_directory(toyz_settings, job['id'],job['parameters']). If there are any errors in loading the directory, a response of the form
response = {
    'id' : 'ERROR',
    'error' : 'Error message here for unable to lead directory',
    'traceback' : traceback.format_exec()
}
is sent. If the directory is loaded correctly a response of the form
response={
    'id': 'directory',
    'path': '~/images/proj1',
    'shortcuts': ['user', 'temp', 'home'],
    'folders': ['proj1', 'proj2'],
    'files': [],
    'parent': '~/images'
}
is sent to the client.

toyz.utils.core.str_2_bool(bool_str)¶

Case independent function to convert a string representation of a boolean ('true'/'false', 'yes'/'no') into a bool. This is case insensitive, and will also accept part of a boolean string ('t'/'f', 'y'/'n').

Raises a toyz.utils.errors.ToyzError if an invalid expression is entered.

toyz.utils.db module¶

To keep Toyz database agnostic this module works as a stardard API to access any database interfaces written into the framework. Currently on the sqlite interface is supported.

toyz.utils.db.check_chars(err_flag, *lists)¶

Check if every value in every list is alpha-numeric.

Parameters

err_flag (bool ):
- If err_flag is True, an exception will be raised if the field is not alpha_numeric.
- If err_flag is False, the function will return False
lists (lists):
- Each list in lists must be a list of strings, each one a keyword to verify if it is alpha-numeric

toyz.utils.db.create_toyz_database(db_settings)¶: Create a new Toyz database. This should only be done when creating a new instance of a Toyz application.

toyz.utils.db.db_func(db_settings, func, **params)¶: Function from a database outside of the database API.

toyz.utils.db.delete_param(db_settings, param_type, **params)¶: Delete a parameter from the database.

toyz.utils.db.delete_workspace(db_settings, user_id, work_id)¶: Delete a workspace

toyz.utils.db.get_all_ids(db_settings, user_type)¶: Get all user_id’s or group_id’s in the database.

toyz.utils.db.get_db_info(db_settings)¶: Get the info for the database, including the version it was created with, updates made, and the latest version of toyz it was configured for

toyz.utils.db.get_param(db_settings, param_type, **params)¶: Get a parameter from the database. This may be either a single value, dictionary, or list of values, depending on the parameter.

toyz.utils.db.get_path_info(db_settings, path)¶: Get the permissions for all users and groups for a given path.

toyz.utils.db.get_table_names(db_settings)¶: Get the names of tables in the database (this can be useful when the user has changed versions)

toyz.utils.db.get_workspace_sharing(db_settings, **params)¶: Get sharing permissions for a users workspaces

toyz.utils.db.group_fields = ['pwd', 'users', 'paths', 'modules', 'toyz', 'workspaces']¶

toyz.utils.db.param_formats is a dictionary that sets a standard for getting and updating rows in the toyz database. Each param format has the following keys:

tbl ( string ):

Name of the table

required ( string ):

Required parameters to get a row (or rows) from the table

get ( string or list ):

When a user calls a get_param command, the get columns (along with the required columns) are loaded from the table

format ( string ):

The type of object to be added to the table. This can be single, dict, or list.

update ( string ):

Name of key to use when updating the table (see below for different formats)

single:

Only a single value can be store/retrieved

update is just the same as the get column name

list:

A list of values can be stored or retrieved, but the table only has a single non-required column

update is the variable name of the list that needs to be added to the table

dict:

A dictionary is stored in the table

get is a list where the first variable is the key of the dictionary and the second is the value

update is the variable name of the dictionary that is being added to the table

rows:

Some tables have multiple columns that may be stored/retrieved in different ways

Tables with format='rows' cannot make user of the get/update/delete_param methods

The keywords of the dictionary are the columns of the table

columns is a list of column names in the table

json ( list, optional):

A list of any columns that need to be converted to/from a json string for storage/retrieval

toyz.utils.db.init(**params)¶: For some databases it might be necessary to initialize them on startup, so this function call init in the current database interface

toyz.utils.db.update_all_params(db_settings, param_type, **params)¶: Update a parameter with a single value, a list of values, or a dictionary. In the case of a list or a dictionary, this function also removes any entries in the database not contained in params .

toyz.utils.db.update_param(db_settings, param_type, **params)¶: Update a parameter with a single value, list of values, or dictionary.

toyz.utils.db.update_version(db_settings, params)¶: It may be necessary to update a database when the Toyz version changes. This function will be unique to each DB and describe how to implement a given version change

toyz.utils.db.update_workspace(db_settings, user_id, work_id, **kwargs)¶: Update sharing for a workspace

toyz.utils.errors module¶

Error classes used in Toyz.

exception toyz.utils.errors.Error¶

Bases: exceptions.Exception

Base class for custom errors related to the running of Toyz

exception toyz.utils.errors.ToyzDataError(msg)¶

Bases: toyz.utils.errors.ToyzError

Class for Toyz DataSource Errors

exception toyz.utils.errors.ToyzDbError(msg)¶

Bases: toyz.utils.errors.ToyzError

Class for errors initiating in the Toyz Database Interface

exception toyz.utils.errors.ToyzError(msg)¶

Bases: toyz.utils.errors.Error

Class for custom errors related to the running of Toyz.

exception toyz.utils.errors.ToyzIoError(msg)¶

Bases: toyz.utils.errors.ToyzError

Class for I/O Errors

exception toyz.utils.errors.ToyzJobError(msg)¶

Bases: toyz.utils.errors.ToyzError

Class for errors initiating in the Toyz Job Queue

class toyz.utils.errors.ToyzWarning(msg)¶: Class for warnings

exception toyz.utils.errors.ToyzWebError(msg)¶

Bases: toyz.utils.errors.ToyzError

Class for errors initiating in the Toyz Web Application

toyz.utils.file_access module¶

Utilities to access path and file permissions for the server.

toyz.utils.file_access.get_all_parents(path)¶

Create a list with all of the parent directories of the file or path.

For example /Users/penny/Documents would be returned as ['/Users/penny/Documents', '/Users/penny', '/Users/']

toyz.utils.file_access.get_file_permissions(db_settings, path, **user)¶

Get all of the permissions for a given path. Returns None type if no permissions have been set.

Parameters

db_settings (object ): Database settings
path (string ): Path to check for permissions
user (dict ): Key is either user_id or group_id, value is the user_id or group_id

Return

permissions (string ): Permissions for the given user for the given path. Returns None if no permissions have been set.

toyz.utils.file_access.get_parent_permissions(db_settings, path, **user)¶

Find the permissions of the given path. If it doesn’t have any permissions explicitely set, descend a tree and find the first parent directory with permissions set. If no permissions can be found, None is returned.

Parameters

db_settings (object ): Database settings
path (string ): Path to check for permissions
user (dict ): Key is either user_id or group_id, value is the user_id or group_id

Return

permissions (string ): Permissions for the given user for the given path. Returns None if no permissions have been set for any parent paths.

toyz.utils.file_access.get_path_tree(path)¶: Get all of the sub directories of path.

toyz.utils.file_access.split_path(path_in)¶

Splits a path into a list of its folders.

For example /Users/penny/Documents would be returned as ['Users', 'penny', 'Documents']

toyz.utils.security module¶

toyz.utils.security.decrypt_pickle(app_settings, key)¶: Use the provided key to decrypt the config file. Not yet supported

toyz.utils.security.encrypt_pickle(app_settings)¶: Encrypt the app settings using a config file. Not yet supported

toyz.utils.third_party_settings module¶

Module contents¶

Common non-package specific utility functions