API Reference

If you are looking for information on a specific function, class or method, this part of the documentation is for you.

Python API for Environmental Simulator Quest (QUEST).

This module defines the Python API for the Environmental Simulator Data Services Library.

quest.api.add_datasets(collection, catalog_entries)

Adds new datasets (created from catalog_entries) to collection

Parameters

• collection (string, Required) – name of collection

• catalog_entries (string, comma separated strings, list of strings, or pandas.DataFrame, Required) – list of catalog entry uris from which to create new datasets to add to the collection.

Returns

uris of newly created datasets

Return type

uris (list)

quest.api.add_project(name, path, activate=True)

Add a existing QUEST project to the list of available projects

Parameters

• name (string, Required) – name of project; existing name can be used or project can be renamed

• path (string, Required) – path to existing project

• activate (bool, Optional, Default=True) – if True, the added project is set as the currently active project

quest.api.add_user_provider(uri)

Add a custom web service created from a file or http folder.

Converts a local/network or http folder that contains a quest.yml and associated data into a service that can be accessed through quest

Parameters

uri (string, Required) – uri of new ‘user’ service

Returns

status of adding service (i.e. failed/success)

Return type

message (string)

quest.api.authenticate_provider(uri, **kwargs)

Authenticate the user.

Parameters

uri – uri of ‘user service’

quest.api.cancel_tasks(task_ids)

Cancel tasks.

Parameters

task_ids (string or list of strings, Required) – id of tasks to be cancelled

quest.api.copy(uris, destination_collection, as_dataframe=None, expand=None)

quest.api.delete(uris)

Delete metadata for resource(s)

Warning

deleting a collection deletes all associated datasets

Parameters

uris (string, comma separated string or list of strings, Required) – uri(s) of collection, and/or dataset to delete

Returns

True on success

Return type

status (bool)

quest.api.delete_project(name)

Delete a project.

Deletes a project and all data in the project folder.

Parameters

name (string, Required) – name of a project

Returns

all remaining projects and their respective folders

Return type

projects (dict)

quest.api.delete_user_provider(uri)

Remove ‘user’ service.

Parameters

uri – uri of ‘user service’

quest.api.download(catalog_entry, file_path, dataset=None, **kwargs)

Download dataset and save it locally.

Parameters

• catalog_entry (string, Required) – uri of catalog_entry within a service or collection

• file_path (string, Required) – path location to save downloaded data

• dataset (string, Optional, Default=None) – maybe only be used by some providers

• async – (bool, Optional, Default=False) if True, download in background

• kwargs – optional download kwargs

Returns

details of downloaded data

Return type

data (dict)

quest.api.download_datasets(datasets, options=None, raise_on_error=False)

Download datasets and save them in the Quest project.

Parameters

• datasets (string or list, Required) – datasets to download

• options (dict, Optional, Default=None) – Dictionary of download options to stage the datasets with before downloading (see quest.api.stage_for_download())

• raise_on_error (bool, Optional, Default=False) – if True, if an error occurs raise an exception

• async (bool, Optional, Default=False) – if True, download in background

Note

If options are not provided then the datasets should already download options set by calling quest.api.stage_for_download().

Returns

download status of datasets

Return type

status (dict)

quest.api.get_download_options(uris, fmt='json')

List optional kwargs that can be specified when downloading a dataset.

Parameters

• uris (string or list, Required) – uris of catalog_entries or datasets

• fmt – format in which to return download_options. One of [‘json’, ‘param’]

quest.api.get_active_project()

Get active project name.

Returns

name of currently active project

Return type

name (string)

quest.api.get_api_version()

Get QUEST API version.

Returns

version of QUEST API being used

Return type

QUEST version (string)

quest.api.get_auth_status(uri)

Check to see if a provider has been authenticated

Parameters

uri – uri of ‘user service’

quest.api.get_collections(expand=False, as_dataframe=False)

Get available collections.

Collections are folders on the local disk that contain downloaded or created data along with associated metadata.

Parameters

• expand (bool, Optional, Default=False) – include collection details and format as dict

• as_dataframe (bool, Optional, Default=False) – include collection details and format as pandas dataframe

Returns

all available collections

Return type

collections (list, dict, or pandas dataframe, Default=list)

quest.api.get_data(service_uri, search_filters=None, search_queries=None, download_options=None, collection_name='default', expand=False, use_cache=True, max_catalog_entries=10, as_open_datasets=True, raise_on_error=False)

Downloads data from source uri and adds to a quest collection.

Parameters

• service_uri (string, required) – uri for service to get data from

• search_filters (dict, optional, default=None) – dictionary of search filters to filter the catalog search. At least one of search_filters or search_queries should be specified. (see docs for quest.api.search_catalog())

• search_queries (list, optional, default=None) – list of string arguments to pass to pandas.DataFrame.query to filter the catalog search. At least one of search_filters or search_queries should be specified. (see docs for quest.api.search_catalog())

• download_options (dict or Parameterized, optional, default=None) – dictionary or Parameterized object with download options for service (see docs for quest.api.download_datasets)

• collection_name (string, optional, default='default') – name of collection to add downloaded data to. If collection doesn’t exist it will be created.

• expand (bool, optional, default=False) – include dataset details and format as dict

• use_cache (bool, optional, default=True) – if True then previously downloaded datasets with the same download options will be returned rather than downloading new datasets

• max_catalog_entries (int, optional, default=10) – the maximum number of datasets to allow in the search. If exceeded a Runtime error is raised.

• as_open_datasets (bool, optional, default=False) – if True return datasets as Python data structures rather than as dataset ids (see docs for quest.api.open_dataset)

• raise_on_error (bool, optional, default=False) – if True then raise an exception if no datasets are returned in the search, or if there is an error while downloading any of the datasets.

Returns

the quest dataset name, or an python data structure if open_dataset=True.

quest.api.get_datasets(expand=None, filters=None, queries=None, as_dataframe=None)

Return all available datasets in active project.

Parameters

• expand (bool, Optional, Default=None) – include dataset details and format as dict

• filters (dict, Optional, Default=None) – filter dataset by any metadata field

• queries (list, Optional, Default=None) – list of string arguments to pass to pandas.DataFrame.query to filter the datasets

• as_dataframe (bool or None, Optional, Default=None) – include dataset details and format as pandas dataframe

Returns

staged dataset uids

Return type

uris (list, dict, pandas Dataframe, Default=list)

quest.api.get_mapped_parameters()

Get list of common parameters.

Returns

list of common parameters

Return type

parameters (list)

quest.api.get_metadata(uris, as_dataframe=False)

Get metadata for uris.

Parameters

• uris (string, comma separated string, or list of strings, Required) – list of uris to retrieve metadata for

• as_dataframe (bool, Optional, Default=False) – include details of newly created dataset as a pandas Dataframe

Returns

metadata at each uri keyed on uris

Return type

metadata (dict or pd.DataFrame, Default=dict)

quest.api.get_parameters(service_uri, update_cache=False)

Get available parameters, even unmapped ones, for specified service.

Parameters

• service_uri (string, Required) – uri of service to get parameters for

• update_cache (bool, Optional, Default=True) – if True, update metadata cache

Returns

all available parameters for specified service

Return type

parameters (list)

quest.api.get_pending_tasks(**kwargs)

Return list of pending tasks

calls get_tasks with filter -> status=pending, passes through other kwargs (filters={}, expand=None, as_dataframe=None, with_future=None)

quest.api.get_projects(expand=False, as_dataframe=False)

Get list of available projects.

Parameters

expand – include collection details and format as dict

Returns

all available projects

Return type

projects (list, dict, or pandas Dataframe,Default=list)

quest.api.get_providers(expand=None, update_cache=False)

Return list of Providers.

Parameters

• expand (bool, Optional, Default=None) – include providers’ details and format as dict

• update_cache (bool, Optional, Default=False) – reload the list of providers

Returns

list of all available providers

Return type

providers (list or dict, Default=list)

quest.api.get_publishers(expand=None, publisher_type=None)

This method returns a list of avaliable publishers.

The method first gets a dictionary filled with the available providers in Quest. Then we loop through grabbing the keys and the objects within the dictionary. Then we loop again, accessing each service getting another dictionary with the provider as the key and the metadata as the values. Then we create a publish uri, and get the publisher class name for the service. We return a list of publishers.

Parameters

• expand (bool, Optional, Default=False) – include providers’ details and format as dict

• publisher_type (string, Optional, Default=None') – filter to only include specific type

Returns

list of all available providers

Return type

providers (list or dict,Default=list)

quest.api.get_quest_version()

Get QUEST version.

Returns

version of QUEST being used

Return type

QUEST version (string)

quest.api.get_seamless_data(service_uri, bbox, search_filters=None, search_queries=None, download_options=None, collection_name='default', expand=False, use_cache=True, max_catalog_entries=10, as_open_dataset=True, raise_on_error=False)

Downloads raster data from source uri and adds to a quest collection.

If multiple raster tiles are retrieved for the given bounds it calls a quest tool to merge the tiles into a single raster.

Parameters

• service_uri (string, required) – uri for service to get data from

• bbox (list, required) – list of lat/lon coordinates representing the bounds of the data in for form [lon_min, lat_min, lon_max, lat_max].

• search_filters (dict, optional, default=None) – dictionary of search filters to filter the catalog search (see docs for quest.api.search_catalog())

• search_queries (list, optional, default=None) – list of string arguments to pass to pandas.DataFrame.query to filter the catalog search (see docs for quest.api.search_catalog())

• download_options (dict or Parameterized, optional, default=None) – dictionary or Parameterized object with download options for service (see docs for quest.api.download_datasets)

• collection_name (string, optional, default='default') – name of collection to add downloaded data to. If collection doesn’t exist it will be created.

• expand (bool, optional, default=False) – include dataset details and format as dict

• use_cache (bool, optional, default=True) – if True then previously downloaded datasets with the same download options will be returned rather than downloading new datasets

• max_catalog_entries (int, optional, default=10) – the maximum number of datasets to allow in the search. If exceeded a Runtime error is raised.

• as_open_dataset (bool, optional, default=False) – if True return dataset as Python data structure rather than as a dataset id (see docs for quest.api.open_dataset)

• raise_on_error (bool, optional, default=False) – if True then raise an exception if no datasets are returned in the search, or if there is an error while downloading.

Returns

the quest dataset name.

quest.api.get_services(expand=None, parameter=None, service_type=None)

Return list of Services.

Parameters

• expand (bool, Optional, Default=False) – include providers’ details and format as dict

• parameter (string, Optional, Default=None) –

• service_type (string, Optional, Default=None') – filter to only include specific type

Returns

all available providers

Return type

providers (list or dict, Default=dict)

quest.api.get_settings()

Get the settings currently being used by QUEST.

Returns

A dictionary of the current settings.

Example

{‘BASE_DIR’: ‘/Users/dharhas/’, ‘CACHE_DIR’: ‘cache’, ‘PROJECTS_DIR’: ‘projects’, ‘USER_SERVICES’: [], }

quest.api.get_tags(service_uris, update_cache=False, filter=None, as_count=False)

Get searchable tags for a given service.

Parameters

• service_uris (string or list, Required) – uris of providers

• update_cache (bool, Optional) – if True, update metadata cache

• filter (list, Optional) – list of tags to include in return value

• as_count (bool, Optional) – if True, return dictionary with the number of values rather than a list of possible values

Returns

dict keyed by tag name and list of possible values

Note: nested dicts are parsed out as a multi-index tag where keys for nested dicts are joined with ‘:’.

Return type

tags (dict)

quest.api.get_task(task_id, with_future=None)

Get details for a task.

Parameters

• task_id (string,Required) – id of a task

• with_future (bool, Optional, Default=None) – If true include the task future objects in the returned dataframe/dictionary

quest.api.get_tasks(filters=None, expand=None, as_dataframe=None, with_future=None)

Get all available tasks.

Parameters

• filters (dict, Optional, Default=None) –

  filter tasks by one or more of the available filters

  available filters:

  • task_ids (str or list): task id or list of task ids

  • status (str or list): single status or list of statuses. Must be subset of

    [‘pending’, ‘cancelled’, ‘finished’, ‘lost’, ‘error’]

  • fn (str): name of the function a task was assigned

  • args (list): list of arguments that were passed to the task function

  • kwargs (dict): dictionary of keyword arguments that were passed to the task function

  • result (object): result of the task function

• expand (bool, Optional, Default=None) – include details of tasks and format as a dict

• as_dataframe (bool, Optional, Default=None) – include details of tasks and format as a pandas dataframe

• with_future (bool, Optional, Default=None) – If true include the task future objects in the returned dataframe/dictionary

Returns

all available tasks

Return type

tasks (list, dict, or pandas dataframe, Default=list)

quest.api.get_tools(filters=None, expand=False, **kwargs)

List available tool plugins

Parameters

• filters (dict, Optional, Default=None) –

  filter the list of tools by one or more of the available filters

  Available Filters:

  • dataset

  • group

  • geotype

  • datatype

  • parameter

  Note: If a dataset filter is used, all other filters are overridden and set from the dataset’s metadata.

• expand (bool, Optional, Default=None) – if True, return details of the filters as a dict

• kwargs – optional filter kwargs

Returns

all available tools

Return type

tools (list or dict, Default=list)

quest.api.get_tool_options(name, fmt='json', **kwargs)

Retrieve kwarg options for run_tool.

Parameters

• name (string, Required) – name of filter

• fmt (string, Required, Default='json') – format in which to return options. One of [‘json’, ‘param’]

• kwargs – keyword arguments of options to set and exclude from return value.

Returns

tool options that can be applied when calling quest.api.run_filter

Return type

tool options (json scheme)

quest.api.move(uris, destination_collection, as_dataframe=None, expand=None)

quest.api.new_catalog_entry(geometry=None, geom_type=None, geom_coords=None, metadata=None)

Add a new entry to a catalog either a quest local catalog (table) or file.

Parameters

• geometry (string or Shapely.geometry.shape, optional, Default=None) – well-known-text or Shapely shape representing the geometry of the catalog_entry. Alternatively geom_type and geom_coords can be passed.

• geom_type (string, Optional, Default=None) – geometry type of catalog_entry (i.e. point/line/polygon)

• geom_coords (string or list, Optional, Default=None) – geometric coordinates specified as valid geojson coordinates (i.e. a list of lists i.e. ‘[[-94.0, 23.2], [-94.2, 23.4] …]’ ——— OR ——— [[-94.0, 23.2], [-94.2, 23.4] …] etc)

• metadata (dict, Optional, Default=None) – optional metadata at the new catalog_entry

Returns

uri of newly created entry

Return type

uri (string)

quest.api.new_collection(name, display_name=None, description=None, metadata=None, exists_ok=False)

Create a new collection.

Create a new collection by creating a new folder in project directory and adding collection metadata in project database.

Parameters

• name (string, Required) – Name of the collection used in all quest function calls,must be unique. Will also be the folder name of the collection

• display_name (string, Optional, Default=None) – display name for collection

• description (string, Optional, Default=None) – description of collection

• metadata (dict, Optional, Default=None) – user defined metadata

• exists_ok (bool, Optional, Default=False) – If True then ValueError is not raised if the collection already exits. Rather the metadata of the existing colleciton is returned.

Returns

details of the newly created collection

Return type

dict

Raises

ValueError – If collection with name already exists.

quest.api.new_dataset(catalog_entry, collection, source=None, display_name=None, description=None, file_path=None, metadata=None, name=None)

Create a new dataset in a collection.

Parameters

• catalog_entry (string, Required) – catalog_entry uri

• collection (string, Required) – name of collection to create dataset in

• source (string, Optional, Default=None) – type of the dataset such as timeseries or raster

• display_name (string, Optional, Default=None) – display name for dataset

• description (string, Optional, Default=None) – description of dataset

• file_path (string, Optional, Default=None) – path location to save new dataset’s data

• metadata (dict, Optional, Default=None) – user defined metadata

• name (dict, Optional, Default=None) – optionally pass in a UUID starting with d as name, otherwise it will be generated

Returns

uid of dataset

Return type

uri (string)

quest.api.new_parameter(uri, parameter_name)

Add new parameter to collection.

quest.api.new_project(name, display_name=None, description=None, metadata=None, folder=None, activate=True)

Create a new QUEST project and add it to list of available projects.

Parameters

• name (string, Required) – name of newly created project

• display_name (string, Optional, Default=None) – display name for project

• description (string, Optional, Default=None) – description of project

• metadata (dict, Optional, Default=None) – user defined metadata

• folder (string, Optional, Default=None) – folder where all project data will be saved

• activate (bool, Optional, Default=True) – if True, set newly created project as currently active project

quest.api.open_dataset(dataset, fmt=None, **kwargs)

Open the dataset and return in format specified by fmt

Parameters

• dataset (string, Required) – uid of dataset to be opened

• fmt (string, Optional, Default=None) – format in which dataset should be returned will raise NotImplementedError if format requested is not possible

Returns

contents of dataset

Return type

data (pandas dataframe, json, or dict, Default=dataframe)

quest.api.publish(publisher_uri, options=None, **kwargs)

quest.api.get_publish_options(publish_uri, fmt='json')

quest.api.remove_project(name)

Remove a project from the list of available projects.

This does not delete the project folder or data, just removes it from the index of available projects.

Parameters

name – name of project

quest.api.remove_tasks(task_ids=None, status=None)

Remove tasks.

Parameters

• task_ids (string or list, Optional, Default=None) – tasks with specified id(s) will be removed

• status (string or list, Optional, Default=None) –

  tasks with specified status(es) will be removed. Valid statuses are:

  • cancelled

  • finished

  • lost

  • error

  NOTE: pending is not a valid option and will be ignored,

  since pending tasks must be canceled before they can be removed.

• no status is specified, remove tasks with (If) –

• = ['cancelled', 'finished', 'lost', 'error'] from task list (status) –

quest.api.run_tool(name, options=None, as_dataframe=None, expand=None, as_open_datasets=None, **kwargs)

Apply Tool to dataset.

Parameters

• name (string,Required) – name of filter

• options (dict, Required) – a dictionary of arguments to pass to the filter formatted as specified by get_tool_options

• expand (bool, Optional, Default=False) – include details of newly created dataset and format as a dict

• as_dataframe (bool, Optional, Default=False) – include details of newly created dataset and format as a pandas dataframe

• as_open_datasets (bool, Optional, Default=False) – returns datasets as Python data structures rather than Quest IDs

• async (bool,Optional) – if True, run filter in the background

• kwargs – keyword arguments that will be added to options

Returns

resulting datasets and/or catalog_entries

Return type

dataset/catalog_entry uris (dict or pandas dataframe, Default=dict)

quest.api.save_settings(filename=None)

Save settings currently being used by QUEST to a yaml file.

Parameters

filename (string) – Path to the yaml file to save the settings.

Returns

A true boolean if settings were saved successfully.

quest.api.set_active_project(name)

Set active QUEST project.

Parameters

name (string, Required) – name of a project

Returns

name of project currently set as active

Return type

project (string)

quest.api.search_catalog(uris=None, expand=False, as_dataframe=False, as_geojson=False, update_cache=False, filters=None, queries=None)

Retrieve list of catalog entries from resources.

Parameters

• uris (string or list, Required) – uris of service_uris

• expand (bool, Optional, Default=False) – if true then return metadata along with catalog entries

• as_dataframe (bool, Optional, Default=False) – include catalog_entry details and format as a pandas DataFrame indexed by catalog_entry uris

• as_geojson (bool, Optional, Default=False) – include catalog_entry details and format as a geojson scheme indexed by catalog_entry uris

• update_cache (bool, Optional,Default=False) – if True, update metadata cache

• filters (dict, Optional, Default=None) –

  filter catalog_entries by one or more of the available filters

  Note

  available filters:

  • bbox (string, optional): filter catalog_entries by bounding box

  • geom_type (string, optional): filter catalog_entries by geom_type, i.e. point/line/polygon

  • parameter (string, optional): filter catalog_entries by parameter

  • display_name (string, optional): filter catalog_entries by display_name

  • description (string, optional): filter catalog_entries by description

  • search_terms (list, optional): filter catalog_entries by search_terms

  catalog_entries can also be filtered by any other metadata fields

• queries (list, Optional, Default=None) – list of string arguments to pass to pandas.DataFrame.query to filter the catalog_entries

Returns

datasets of specified service(s), collection(s) or catalog_entry(s)

Return type

datasets (list, geo-json dict or pandas.DataFrame, Default=list)

quest.api.stage_for_download(uris, options=None)

Apply download options before downloading

Parameters

• uris (string or list, Required) – uris of datasets to stage for download

• options (dict or list of dicts, Optional, Default=None) –

  options to be passed to quest.api.download function specified for each dataset

  If options is a dict, then apply same options to all datasets, else each dict in list is used for each respective dataset

Returns

staged dataset uids

Return type

uris (list)

quest.api.unauthenticate_provider(uri)

Un-Authenticate the user.

Parameters

uri – uri of ‘user service’

quest.api.update_metadata(uris, display_name=None, description=None, metadata=None, quest_metadata=None)

Update metadata for resource(s)

Parameters

• uris (string, comma separated string, or list of strings, Required) – list of uris to update metadata for.

• display_name (string or list, Optional,Default=None) – display name for each uri

• description (string or list, Optional,Default=None) – description for each uri

• metadata (dict or list of dicts, Optional, Default=None) – user defiend metadata

• quest_metadata (dict or list of dicts, Optional, Default=None) – metadata used by QUEST

Returns

metadata of each uri keyed on uris

Return type

metadata (dict)

quest.api.update_project_metadata(name, display_name=None, description=None, metadata=None)

Updates a project’s metadata

Parameters

• name (string, required) – name of project to update

• display_name (string, optional, default=None) – new display name for the project. If None then the display name will not be modified.

• (string, optional, default=None (description) – new description for the project. If None then the description will not be modified.

• metadata (dict, optional, default=None) – new metadata dict with which to update the project’s current metadata. If None then the metadata will not be modified.

Returns

dictionary of updated metadata for project

Return type

dict

quest.api.update_settings(config={})

Update the settings file that is being stored in the Quest

settings directory.

Notes

Only key/value pairs that are provided are updated, any other existing pairs are left unchanged or defaults are used.

Parameters

config (dict) – Key/value pairs of settings that are to be updated.

Returns

Updated Settings

Example

{‘BASE_DIR’: ‘/Users/dharhas/’, ‘CACHE_DIR’: ‘cache’, ‘PROJECTS_DIR’: ‘projects’, ‘USER_SERVICES’: [], }

quest.api.update_settings_from_file(filename)

Update the settings from a new yaml file.

Notes

Only key/value pairs that are provided are updated, any other existing pairs are left unchanged or defaults are used.

Parameters

filename (string) – Path to the yaml file containing the new settings.

Returns

Updated settings

Example

{‘BASE_DIR’: ‘/Users/dharhas/’, ‘CACHE_DIR’: ‘cache’, ‘PROJECTS_DIR’: ‘projects’, ‘USER_SERVICES’: [], }

quest.api.visualize_dataset(dataset, update_cache=False, **kwargs)

Visualize the dataset as a matplotlib/bokeh plot.

Check for existence of dataset on disk and call appropriate file format driver.

Parameters

• dataset (string, Required) – uri of dataset to be visualized

• update_cache (bool, Optional, Default=False) – currently unused

• kwargs – optional download kwargs

Returns

path to the newly visualized dataset

Return type

path (string)

quest.api.get_visualization_options(dataset, fmt='json')

Return visualization available options for dataset.

Parameters

• dataset (string, Required) – uid of dataset

• fmt (string, Required, Default='json') – format in which to return options

Returns

options that can be specified when calling quest.api.visualize_dataset

Return type

get_visualization_options (dict)