Skip to main content
Version: 0.17.23

DatabricksSQLDatasource

class great_expectations.datasource.fluent.DatabricksSQLDatasource(*, type: Literal['databricks_sql'] = 'databricks_sql', name: str, id: Optional[uuid.UUID] = None, assets: List[Union[great_expectations.datasource.fluent.sql_datasource.TableAsset, great_expectations.datasource.fluent.sql_datasource.QueryAsset][Union[great_expectations.datasource.fluent.sql_datasource.TableAsset, great_expectations.datasource.fluent.sql_datasource.QueryAsset]]] = [], connection_string: Union[great_expectations.datasource.fluent.config_str.ConfigStr, great_expectations.datasource.fluent.databricks_sql_datasource.DatabricksDsn], create_temp_table: bool = True, kwargs: Dict[str, Union[great_expectations.datasource.fluent.config_str.ConfigStr, Any]] = )#

Adds a DatabricksSQLDatasource to the data context.

Parameters:

  • name – The name of this DatabricksSQL datasource.

  • connection_string – The SQLAlchemy connection string used to connect to the postgres database. For example: “databricks://token:<token>@<host>:<port>/<database>?http_path=<http_path>&catalog=<catalog>&schema=<schema>””

  • assets – An optional dictionary whose keys are TableAsset or QueryAsset names and whose values are TableAsset or QueryAsset objects.

add_query_asset(name: str, query: str, order_by: Optional[SortersDefinition] = None, batch_metadata: Optional[BatchMetadata] = None) → QueryAsset#

Adds a query asset to this datasource.

Parameters:

  • name – The name of this table asset.

  • query – The SELECT query to selects the data to validate. It must begin with the “SELECT”.

  • order_by – A list of Sorters or Sorter strings.

  • batch_metadata – BatchMetadata we want to associate with this DataAsset and all batches derived from it.

Returns:

The query asset that is added to the datasource. The type of this object will match the necessary type for this datasource. eg, it could be a QueryAsset or a SqliteQueryAsset.

add_table_asset(name: str, table_name: str = '', schema_name: Optional[str] = None, order_by: Optional[SortersDefinition] = None, batch_metadata: Optional[BatchMetadata] = None) → TableAsset#

Adds a table asset to this datasource.

Parameters:

  • name – The name of this table asset.

  • table_name – The table where the data resides.

  • schema_name – The schema that holds the table.

  • order_by – A list of Sorters or Sorter strings.

  • batch_metadata – BatchMetadata we want to associate with this DataAsset and all batches derived from it.

Returns:

The table asset that is added to the datasource. The type of this object will match the necessary type for this datasource. eg, it could be a TableAsset or a SqliteTableAsset.