How to set up Great Expectations to work with general SQL databases
This guide will walk you through best practices for creating your GX Python environment and demonstrate how to locally install Great Expectations along with the necessary dependencies for working with SQL databases.
Prerequisites
- An installation of Python 3.8 to 3.11. To download and install Python, see Python downloads.
- The ability to install Python modules with pip
Steps
1. Check your Python version
You can check your version of Python by running:
python --version
GX currently supports Python versions 3.8 to 3.11
python
or
python3
Depending on your installation and configuration
of Python 3, you may find that executing Python
commands from the terminal by calling
python
doesn't work as desired.
If a command using python
does not
work, try using python3
.
Instead of:
python --version
Try:
python3 --version
If this produces the desired result, simply
replace python
with
python3
in our example terminal
commands.
If this does not work, you may need to look into your Python 3 installation or configuration.
2. Create a Python virtual environment
As a best practice, we recommend using a virtual environment to partition your GX installation from any other Python projects that may exist on the same system. This ensures that there will not be dependency conflicts between the GX installation and other Python projects.
Once we have confirmed that Python 3 is installed
locally, we can create a virtual environment with
venv
.
venv
?
We have chosen to use venv
for
virtual environments in this guide because it is
included with Python 3. You are not limited to
using venv
, and can just as easily
install Great Expectations into virtual
environments with tools such as
virtualenv
, pyenv
, etc.
We will create our virtual environment by running:
python -m venv my_venv
This command will create a new directory called
my_venv
. Our virtual environment will be
located in this directory.
In order to activate the virtual environment we will run:
source my_venv/bin/activate
my_venv
?
You can name your virtual environment anything you
like. Simply replace my_venv
in the
examples above with the name that you would like
to use.
3. Install GX with optional dependencies for SQL databases
To install Great Expectations with the optional dependencies needed to work with SQL databases we execute the following pip command from the terminal:
pip install 'great_expectations[sqlalchemy]'
The above pip instruction will install GX with basic SQL support through SqlAlchemy. However, certain SQL dialects require additional dependencies. Depending on the SQL database type you will be working with, you may wish to use one of the following installation commands, instead:
-
AWS Athena:
pip install 'great_expectations[athena]'
-
BigQuery:
pip install 'great_expectations[bigquery]'
-
MSSQL:
pip install 'great_expectations[mssql]'
-
PostgreSQL:
pip install 'great_expectations[postgresql]'
-
Redshift:
pip install 'great_expectations[redshift]'
-
Snowflake:
pip install 'great_expectations[snowflake]'
-
Trino:
pip install 'great_expectations[trino]'
4. Verify that GX has been installed correctly
You can verify that GX installed successfully with the CLI command:
great_expectations --version
The output you receive if GX was successfully installed will be:
great_expectations, version 0.17.23
5. Setting up credentials
Different SQL dialects have different requirements for connection strings and methods of configuring credentials. By default, GX allows you to define credentials as environment variables or as values in your Data Context. See Instantiate a Data Context.
There may also be third party utilities for setting up credentials of a given SQL database type. For more information on setting up credentials for a given source database, please reference the official documentation for that SQL dialect as well as our guide on how to set up credentials.
Next steps
Now that you have installed GX with the necessary dependencies for working with SQL databases, you are ready to initialize your Data ContextThe primary entry point for a Great Expectations deployment, with configurations and methods for all supporting components.. The Data Context will contain your configurations for GX components, as well as provide you with access to GX's Python API.
To create a Data Context, see Instantiate a Data Context.