Skip to main content
Version: 0.16.16

Create Expectations Icon Create Expectations: Overview

Setup Arrow Connect to Data Arrow Create Expectations Arrow Validate Data

Prerequisites​

Workflows​

The following image shows the four workflows you can follow to create ExpectationsA verifiable assertion about data.:

Where do Expectations come from?

The methodology for saving and testing Expectations is the same for all workflows. GX recommends using the interactive workflow or the Data Assistant workflow to create Expectations.

Create Expectations interactively​

In this workflow, you work in a Python interpreter or Jupyter Notebook. You use a ValidatorUsed to run an Expectation Suite against data. and call Expectations as methods on it to define them in an Expectation Suite, and when you have finished you save that Expectation Suite into your Expectation StoreA connector to store and retrieve information about collections of verifiable assertions about data.. For an overview of this workflow, see How to create and edit Expectations with instant feedback from a sample batch of data.

Create Expectations with Data Assistants​

In this workflow, you use a Data AssistantA utility that asks questions about your data, gathering information to describe what is observed, and then presents Metrics and proposes Expectations based on the answers. to generate Expectations based on the input data you provide. You can preview the Metrics that these Expectations are based on, and you can save the generated Expectations as an Expectation Suite in an Expectation Store.

As with creating Expectations interactively, you start with your Data Context. However, you work in a Python environment, so you need to load or create your Data Context as an instantiated object. Next, you create a Batch Request to specify the data you would like to ProfileThe act of generating Metrics and candidate Expectations from data. with your Data Assistant. Once you have a Batch RequestProvided to a Datasource in order to create a Batch. configured you will use it as the input for the run method of your Data Assistant, which can be accessed from your Data Context object. Once the Data Assistant has run, you will be able to review the results and save the generated Expectations to an empty Expectation Suite.

GX recommends using the Onboarding Data Assistant to create an Expectation Suite. See How to create an Expectation Suite with the Onboarding Data Assistant. After you've created the Expectation Suite, it's expected that you'll edit and update the Expectation Suite to better suit your specific use case. The Expectation Suite is not intended to be used without changes.

Manually define Expectations​

Advanced users can use a manual process to create Expectations. This workflow does not require source data to work against, but it does require knowledge of the configurations available for Expectations. To create Expectations manually, see how to create and edit expectations based on domain knowledge without inspecting data directly.

Create Expectations with custom methods​

Advanced users can use custom methods to generate Expectations that are based on source data system metadata. If you want to use custom methods to create Expectations, contact a member of the support team on Slack.

Test your Expectation Suite​

After you've created and saved your Expectation Suite, GX recommends that you test it by Validating it. You can Validate an Expectation with SimpleCheckpoint. An overview of the Validation process is provided here.

Edit a saved Expectation Suite​

See How to Edit an Expectation Suite

View Expectation Suite Expectations​

See View the Expectations in the Expectation Suite.

Next steps​