Source¶

class deepint.core.Source(organization_id: str, workspace_id: str, credentials: Credentials, info: SourceInfo, features: List[SourceFeature])¶

A Deep Intelligence source.

Note: This class should not be instanced directly, and it’s recommended to use the deepint.core.source.Source.build or deepint.core.source.Source.from_url methods.

organization_id¶: organization where source is located.

workspace_id¶: workspace where source is located.

info¶: deepint.core.source.SourceInfo to operate with source’s information.

instances¶: deepint.core.source.SourceInstances to operate with source’s instances.

features¶: deepint.core.source.SourceFeatures to operate with source’s features.

credentials¶: credentials to authenticate with Deep Intelligence API and be allowed to perform operations over the source. If not provided, the credentials are generated with the deepint.auth.credentials.Credentials.build.

classmethod build(organization_id: str, workspace_id: str, source_id: str, credentials: Optional[Credentials] = None) → Source¶

Builds a source.

Note: when source is created, the source’s information and features are retrieved from API.

Parameters:

organization_id – organization where source is located.
workspace_id – workspace where source is located.
source_id – source’s id.
credentials – credentials to authenticate with Deep Intelligence API and be allowed to perform operations over the source. If not provided, the credentials are generated with the deepint.auth.credentials.Credentials.build.

Returns:

the source build with the given parameters and credentials.

clone(name: Optional[str] = None) → Source¶

Clones a source.

Parameters:: name – name for the new source. If not providen the name will be Copy of <current visualization’s name>
Returns:: the cloned source instance.

delete()¶: Deletes a source.

classmethod from_url(url: str, organization_id: Optional[str] = None, credentials: Optional[Credentials] = None) → Source¶

Builds a source from it’s API or web associated URL.

The url must contain the workspace’s id and the source’s id as in the following examples:

Example

Note: when source is created, the source’s information and features are retrieved from API.: Also it is remmarkable that if the API URL is providen, the organization_id must be provided in the optional parameter, otherwise this ID won’t be found on the URL and the Organization will not be created, raising a value error.

Parameters:

url – the source’s API or web associated URL.
organization_id – the id of the organziation. Must be providen if the API URL is used.
credentials – credentials to authenticate with Deep Intelligence API and be allowed to perform operations over the source. If not provided, the credentials are generated with the deepint.auth.credentials.Credentials.build.

Returns:

the source build with the URL and credentials.

load()¶

Loads the source’s information.

If the source’s information is already loaded, is replace by the new one after retrieval.

to_dict() → Dict[str, Any]¶

Builds a dictionary containing the information stored in current object.

Returns:: dictionary contining the information stored in the current object.

update(name: Optional[str] = None, description: Optional[str] = None)¶

Updates a source’s name and description.

Parameters:

name – source’s name. If not provided the source’s name stored in the deepint.core.source.Source.source_info attribute is taken.
descrpition – source’s description. If not provided the source’s description stored in the deepint.core.source.Source.source_info attribute is taken.

class deepint.core.RealTimeSource(organization_id: str, workspace_id: str, credentials: Credentials, info: SourceInfo, features: List[SourceFeature])¶

Operates over a Deep Intelligence Real Time Source.

Note: This class should not be instanced directly, and it’s recommended to use the deepint.core.source.Source.build or deepint.core.source.Source.from_url methods.

organization_id¶: organization where source is located.

workspace_id¶: workspace where source is located.

info¶: deepint.core.source.SourceInfo to operate with source’s information.

instances¶: deepint.core.source.SourceInstances to operate with source’s instances.

features¶: deepint.core.source.SourceFeatures to operate with source’s features.

credentials¶: credentials to authenticate with Deep Intelligence API and be allowed to perform operations over the source. If not provided, the credentials are generated with the deepint.auth.credentials.Credentials.build.

classmethod build(source: Source) → RealTimeSource¶

Builds a Real-Time source from an deepint.core.source.Source

This allows to use the Real Time data sources extra funcionality.

Parameters:: source – original source.
Returns:: the source build from the given source and credentials.

fetch_connection() → Dict[str, Any]¶

Retrieves Real Time source connection details. Currently on MQTT.

Returns:: a dictionary containing max_age number (Max age of registers in milliseconds. Set to 0 or negative for unlimited), mqtt_url (Connection URl to the MQTT service), mqtt_user (Username to authenticate to the source), mqtt_password (Password to authenticate to the source), mqtt_topic (Topic to publish registers to the source)

update_connection(max_age: int, regenerate_password: bool = False) → None¶

Updates the Realtime source connection details. Currently on MQTT.

Parameters:

max_age – maximum age of registers in milliseconds. Set to 0 or negative for unlimited
regenerate_password – set to true to regenerate the MQTT password, if it was compromised. By default is false.

class deepint.core.ExternalSource(organization_id: str, workspace_id: str, credentials: Credentials, info: SourceInfo, features: List[SourceFeature])¶

Operates over a Deep Intelligence External Source.

To learn more about external sources, please check the (External Sources documentation)[https://deepintdev.github.io/deepint-documentation/EXTERNAL-SOURCES.html].

Note: This class should not be instanced directly, and it’s recommended to use the deepint.core.source.Source.build or deepint.core.source.Source.from_url methods.

organization_id¶: organization where source is located.

workspace_id¶: workspace where source is located.

info¶: deepint.core.source.SourceInfo to operate with source’s information.

instances¶: deepint.core.source.SourceInstances to operate with source’s instances.

features¶: deepint.core.source.SourceFeatures to operate with source’s features.

credentials¶: credentials to authenticate with Deep Intelligence API and be allowed to perform operations over the source. If not provided, the credentials are generated with the deepint.auth.credentials.Credentials.build.

classmethod build(source: Source) → ExternalSource¶

Builds an External source from an deepint.core.source.Source

This allows to use the External sources extra funcionality.

Parameters:: source – original source.
Returns:: the source build from the given source and credentials.

fetch_connection() → str¶

Gets external source connection URL.

Returns:: the URL to connect to external source

update_connection(url: str) → None¶

Gets external source connection URL.

Parameters:: url – the URL to connect to external source

class deepint.core.AutoUpdatedSource(organization_id: str, workspace_id: str, credentials: Credentials, info: SourceInfo, features: List[SourceFeature])¶

Operates over a Deep Intelligence autoupdated source.

Note: This class should not be instanced directly, and it’s recommended to use the deepint.core.source.Source.build or deepint.core.source.Source.from_url methods.

organization_id¶: organization where source is located.

workspace_id¶: workspace where source is located.

info¶: deepint.core.source.SourceInfo to operate with source’s information.

instances¶: deepint.core.source.SourceInstances to operate with source’s instances.

features¶: deepint.core.source.SourceFeatures to operate with source’s features.

credentials¶: credentials to authenticate with Deep Intelligence API and be allowed to perform operations over the source. If not provided, the credentials are generated with the deepint.auth.credentials.Credentials.build.

classmethod build(source: Source) → AutoUpdatedSource¶

Builds an External source from an deepint.core.source.Source

This allows to use the External sources extra funcionality.

Parameters:: source – original source.
Returns:: the source build from the given source and credentials.

fetch_actualization_config() → Dict[str, Any]¶

Retrieves autoupdate configuration.

Returns:: a dictionary containing the autoupdate configuration

update_actualization_config(is_json_content: Optional[bool] = None, is_csv_content: Optional[bool] = None, auto_update: Optional[bool] = None, auto_update_period: Optional[int] = None, replace_on_update: Optional[bool] = None, pk_for_update: Optional[str] = None, update_duplicates: Optional[bool] = None, separator: Optional[str] = None, quotes: Optional[str] = None, has_csv_header: Optional[bool] = None, json_fields: Optional[List[str]] = None, json_prefix: Optional[str] = None, is_single_json_obj: Optional[bool] = None, date_format: Optional[str] = None, url: Optional[str] = None, http_headers: Optional[Dict[str, str]] = None, ignore_security_certificates: Optional[bool] = None, enable_store_data_parameters: Optional[bool] = None, stored_data_parameters_name: Optional[str] = None, stored_data_parameters_sorting_desc: Optional[bool] = None, database_name: Optional[str] = None, database_user: Optional[str] = None, database_password: Optional[str] = None, database_table: Optional[str] = None, database_query: Optional[str] = None, mongodb_sort: Optional[Dict[str, Any]] = None, mongodb_project: Optional[str] = None, database_query_limit: Optional[int] = None, database_host: Optional[str] = None, database_port: Optional[str] = None, mqtt_topics: Optional[List[str]] = None, mqtt_fields: Optional[List[Dict[str, str]]] = None, database_type: Optional[SourceType] = None) → None¶

Updates the auto udpate source configuration.

Note: the not providen configuration, is taken from the current source configuration, fetched from Deep Intelligence.

Parameters:

auto_update – set to true to enable auto update
auto_update_period – auto update delay in milliseconds. Minimum is 5 minutes.
replace_on_update – set to true to replace the entire data set with each update. False to append the data.
pk_for_update – Name of the primary key field. In order to check for duplicates when appending.
update_duplicates – Set to true to update existing rows (by primary key). Set to false to skip duplicate rows. If you set dyn_replace to true. This option does not have any effect.
separator – separator character for csv files
quotes – quotes character for csv files.
has_csv_header – Set to false if the csv files does not have a header.
json_fields – List of fileds to get, in order, for json files or mongo databases.
json_prefix – Prefix to tell the engine where the data is in the JSON file. Use dots to split levels.
is_single_json_obj – Set to true in case there is a single instance in the JSON.
date_format – Date format in the CSV of JSON file. By default is the ISO format. This uses the Moment.js formats.
url – URL for url/any and ckan source types. In case of S3. This is the URI of the object inside the bucket. For mongo and influx, this is the connection URL.
is_csv_content – Set to True to indicate that is a CSV content. Otherwise the content will be considered as JSON.
http_headers – Custom headers to send by Deep Intelligence for requesting the data. example: “example: Header1: Value1 Header2: Value2”
ignore_security_certificates – Set to true to ignore invalid certificates for HTTPs
enable_store_data_parameters – Set to true to enable stored data parameter in the Query. Any instances of ${SDP} will be replaced.
stored_data_parameters_name – Name of the field to use for SDP.
stored_data_parameters_sorting – Sorting direction to calc the SDP. Must be asc or desc.
database_name – Name of the database or the S3 bucket.
database_user – User / Access key ID
database_password – Password / Secret key
database_table – Name of the table / collection
database_query – Database Query. For mongo, this is a JSON.
database_type – the type of database for relational database.
mongodb_sort – For MongoDB. Sorting
mongodb_project – MongoDB project.
database_query_limit – Limit of results per Deep Intelligent data retrieval query against source.
database_host – Database host
database_port – Port number
mqtt_topics – For MQTT, list of topics split by commas.
mqtt_fields – List of expected fields for MQTT. Read Deep Intelligence advanced documentation for more information.

class deepint.core.SourceType(value)¶

Available source types in the system.

classmethod all() → List[str]¶

Returns all available model types serialized to str.

Returns:: all available source types.

classmethod from_string(_str: str) → SourceType¶

Builds the deepint.core.source.SourceType from a str.

Parameters:: _str – name of the source type.
Returns:: the model type converted to deepint.core.source.SourceType.

class deepint.core.DerivedSourceType(value)¶

Available derived source types in the system.

classmethod all() → List[str]¶

Returns all available model types serialized to str.

Returns:: all available source types.

classmethod from_string(_str: str) → SourceType¶

Builds the deepint.core.source.SourceType from a str.

Parameters:: _str – name of the source type.
Returns:: the model type converted to deepint.core.source.SourceType.

class deepint.core.SourceFeature(index: int, name: str, feature_type: FeatureType, indexed: bool, date_format: str, computed: bool, null_count: int, min_value: float, max_value: float, mean_value: float, deviation: float, mapped_to: int)¶

Stores the index, name, type and stats of a feature associated with a deepint.net source.

index¶: Feature index, starting with 0.

name¶: Feature name (max 120 characters).

feature_type¶: The type of the feature. Must be one of the values given in deepint.core.source.FeatureType.

date_format¶: format used to parse the date if this feature is the type deepint.core.source.FeatureType.date.

computed¶: True if the feature is computed (value based on operations over other features).

null_count¶: Number of instances with value null.

min_value¶: Min value.

max_value¶: Max value..

mean_value¶: Average value.

deviation¶: Standard deviation.

mapped_to¶: Index of the feature to map the existing data. You can specify -1 for no mapping.

classmethod from_dataframe(df: DataFrame, date_formats: Optional[Dict[str, str]] = None, min_text_length: int = 1000) → List[SourceFeature]¶

Given an pandas.DataFrame buils the list of deepint.core.source.SourceFeature associated with each of its columns.

The given resulting ditionary only contains fields for the name, type, indexed, date_format and mapped_to attributes.

Note: The index values are assigned with the order of the columns in the given pandas.DataFrame

Parameters:

df – pandas.DataFrame from which the data types for each of its columns will be constructed.
date_formats – dicionary contianing the association between column name and date format like the ones specified in [#/date_formats]. Is optional to provide value for any column, but if not provided will be considered as null and the date format (in case of being a date type) will be the default one assigned by Deep Intelligence.
min_text_length – the minimun length of an element to consider the type as text instead of nominal.

Returns:

collection of features in the format of Deep Intellgience corresponding to the given DataFrame.

static from_dict(obj: Any) → SourceFeature¶

Builds a SourceFeature with a dictionary.

Parameters:: obj – object or dict containing the a serialized SourceFeature.
Returns:: SourceFeature containing the information stored in the given dictionary.

to_dict() → Dict[str, Any]¶

Builds a dictionary containing the information stored in current object.

Returns:: dictionary containing the information stored in the current object.

to_dict_minimized() → Dict[str, Any]¶

Builds a dictionary containing the minimal information stored in current object.

The given resulting dictionary only contains fields for the name, type, indexed, date_format and mapped_to attributes.

Returns:: dictionary containing the information stored in the current object.

class deepint.core.FeatureType(value)¶

Available feature types in the system.

classmethod from_pandas_type(column: Series, min_text_size: int = 256) → List[FeatureType]¶

Builds a deepint.core.source.FeatureType from a pandas.Series.

Checks the type of the elements stored in the column attribute, to detect the python native type or the pandas type, and build the corresponding deepint.core.source.FeatureType.

Parameters:

column – column of a pd.DataFrame to obtain the associated deepint.core.source.FeatureType
min_text_size – the minimum length of an element to consider the type as text instead of nominal.

Returns:

The feature type associated to the given column

classmethod from_string(_str: str) → FeatureType¶

Builds the deepint.core.source.FeatureType from a str.

Parameters:: _str – name of the feature type.
Returns:: the feature type converted to deepint.core.source.FeatureType.

class deepint.core.source.SourceInfo(source_id: str, created: datetime, last_modified: datetime, last_access: datetime, name: str, description: str, source_type: SourceType, instances: int, size_bytes: int)¶

Stores the information of a Deep Intelligence source.

source_id¶: source’s id in format uuid4.

created¶: Creation date.

last_modified¶: Last modified date.

last_access¶: Last access date.

name¶: source’s name.

description¶: source’s description.

source_type¶: type of source (mongodb, SQL, CSV, etc).

instances¶: Number of instances.

size_bytes¶: Source size in bytes.

static from_dict(obj: Any) → SourceInfo¶

Builds a SourceInfo with a dictionary.

Parameters:: obj – object or dict containing the a serialized SourceInfo.
Returns:: SourceInfo containing the information stored in the given dictionary.

to_dict() → Dict[str, Any]¶

Builds a dictionary containing the information stored in current object.

Returns:: dictionary containing the information stored in the current object.

class deepint.core.source.SourceFeatures(source: Source, features: List[SourceFeature])¶

Operates over the features of a concrete source.

Note: This class should not be instanced, and only be used within an deepint.core.source.Source.

source¶: the source with which to operate with its features.

fetch(index: Optional[int] = None, name: Optional[str] = None, force_reload: bool = False) → Optional[SourceFeature]¶

Search for a feature in the source.

Note: if no name or index is provided, the returned value is None.

Parameters:

index – feature’s index to search by.
name – feature’s name to search by.
force_reload – if set to True, features are reloaded before the search with the deepint.core.source.SourceFeature.load method.

Returns:

retrieved feature if found, and in other case None.

fetch_all(force_reload: bool = False) → List[SourceFeature]¶

Retrieves all source’s features.

Parameters:: force_reload – if set to True, features are reloaded before the search with the deepint.core.source.SourceFeature.load method.
Returns:: the source’s features.

load()¶

Loads a source’s features.

If the features were already loaded, this ones are replace by the new ones after retrieval.

update(features: Optional[List[SourceFeature]] = None) → Task¶

Updates a source’s features.

If the features were already loaded, this ones are replace by the new ones after retrieval.

Parameters:: features – the new eatures to update the source. If not provided the source’s internal ones are used.
Returns:: reference to task created to perform the source features update operation.

class deepint.core.source.SourceInstances(source: Source)¶

Operates over the instances of a concrete source.

Note: This class should not be instanced, and only be used within an deepint.core.source.Source

source¶: the source with which to operate with its instances

clean(where: Optional[str] = None) → Task¶

Removes a source’s instances.

Parameters:: where – query in Deepint Query Language, to select which instances delete.
Returns:: reference to task created to perform the source instances deletion operation.

fetch(select: Optional[str] = None, where: Optional[str] = None, order_by: Optional[str] = None, order_type: Optional[str] = None, offset: Optional[int] = None, limit: Optional[int] = None) → DataFrame¶

Retrieves a source’s instances.

Parameters:

select – features to retrieve. Note: all features must belon to the source.
where – query in Deepint Query Language.
order_by – feature by which to sort instances during retrieval.
order_type – must be asc or desc.
offset – number of instances to ignore during the retrieval.
limit – maximum number of instances to retrieve.

Returns:

containing the retrieved data.

Return type:

pd.DataFrame

update(data: DataFrame, replace: bool = False, pk: Optional[str] = None, date_format_feature: Optional[int] = None, send_with_index: bool = False, **kwargs) → Task¶

Updates a source’s instances.

Parameters:

data – data to update the instances. The column names must correspond to source’s feature names.
replace – if set to True the source’s content is replaced by the new insertions.
pk – feature used primary key during the instances insertion, to update the existing values and insert the not existing ones. If is provided with the replace set to True, all the instances will be replaced.
date_format_feature – the input pk for the request body.
send_with_index – if set to False the data is send without the index as first field. Else index is send.

Returns:

reference to task created to perform the source instances update operation.

class deepint.core.source.RealTimeSourceInstances(source: Source)¶

Operates over a Deep Intelligence Real Time Source’s instances.

Note: This class should not be instanced, and only be used within an deepint.core.source.Source

source¶: the source with which to operate with its instances

classmethod build(source: Source) → RealTimeSourceInstances¶

Builds a Real-Time source instances from an deepint.core.source.SourceInstances

This allows to use the Real Time data source instances extra funcionality.

Parameters:: source – original source.
Returns:: the source instances build from the given source and credentials.

clear_queued_instances(from_time: datetime, to_time: datetime) → None¶

Clears instances of a real time source between the limit of a time span.

Parameters:

from_time – start of clear range
to_time – start of clear range

update(data: DataFrame, **kwargs) → None¶

Overwrites the update on a real time source’s instances.

Note: it’s important to highlight that only the data parameter is used.

Parameters:: data – data to update the instances. The column names must correspond to source’s feature names.

class deepint.core.source.ExternalSourceInstances(source: Source)¶

Operates over a Deep Intelligence External Source’s instances.

Note: This class should not be instanced, and only be used within an deepint.core.source.Source

source¶: the source with which to operate with its instances

connection_url¶: url to update instances from external source.

classmethod build(source: Source) → ExternalSourceInstances¶

Builds a External source instances from an deepint.core.source.SourceInstances

This allows to use the External source instances extra funcionality.

Parameters:: source – original source.
Returns:: the source instances build from the given source and credentials.

update(*args, **kwargs) → None¶: Overwrites the update on a external source’s instances.