How to create and edit Expectations based on domain knowledge, without inspecting data directly

This guide shows how to create an Expectation SuiteA collection of verifiable assertions about data. without a sample BatchA selection of records from a Data Asset..

Here are some of the reasons why you may wish to do this:

1. You don't have a sample.
2. You don't currently have access to the data to make a sample.
3. You know exactly how you want your ExpectationsA verifiable assertion about data. to be configured.
4. You want to create Expectations parametrically (you can also do this in interactive mode).
5. You don't want to spend the time to validate against a sample.

If you have a use case we have not considered, please contact us on Slack.

Prerequisites: This how-to guide assumes you have:

• Completed the Getting Started Tutorial
• A working installation of Great Expectations
• Configured a Data Context.
• Have your Data ContextThe primary entry point for a Great Expectations deployment, with configurations and methods for all supporting components. configured to save Expectations to your filesystem (please see How to configure an Expectation store to use a filesystem) or another Expectation StoreA connector to store and retrieve information about collections of verifiable assertions about data. if you are in a hosted environment.

Steps

1. Use the CLI to generate a helper notebook

From the command line, use the CLICommand Line Interface to run:

great_expectations suite new

2. Create Expectation Configurations in the helper notebook

You are adding Expectation configurations to the suite. Since there is no sample Batch of data, no ValidationThe act of applying an Expectation Suite to a Batch. happens during this process. To illustrate how to do this, consider a hypothetical example. Suppose that you have a table with the columns account_id, user_id, transaction_id, transaction_type, and transaction_amt_usd. Then the following code snipped adds an Expectation that the columns of the actual table will appear in the order specified above:

# Create an Expectation
expectation_configuration = ExpectationConfiguration(
   # Name of expectation type being added
   expectation_type="expect_table_columns_to_match_ordered_list",
   # These are the arguments of the expectation
   # The keys allowed in the dictionary are Parameters and
   # Keyword Arguments of this Expectation Type
   kwargs={
      "column_list": [
         "account_id", "user_id", "transaction_id", "transaction_type", "transaction_amt_usd"
      ]
   },
   # This is how you can optionally add a comment about this expectation.
   # It will be rendered in Data Docs.
   # See this guide for details:
   # `How to add comments to Expectations and display them in Data Docs`.
   meta={
      "notes": {
         "format": "markdown",
         "content": "Some clever comment about this expectation. **Markdown** `Supported`"
      }
   }
)
# Add the Expectation to the suite
suite.add_expectation(expectation_configuration=expectation_configuration)

Here are a few more example expectations for this dataset:

expectation_configuration = ExpectationConfiguration(
   expectation_type="expect_column_values_to_be_in_set",
   kwargs={
      "column": "transaction_type",
      "value_set": ["purchase", "refund", "upgrade"]
   },
   # Note optional comments omitted
)
suite.add_expectation(expectation_configuration=expectation_configuration)

expectation_configuration = ExpectationConfiguration(
   expectation_type="expect_column_values_to_not_be_null",
   kwargs={
      "column": "account_id",
      "mostly": 1.0,
   },
   meta={
      "notes": {
         "format": "markdown",
         "content": "Some clever comment about this expectation. **Markdown** `Supported`"
      }
   }
)
suite.add_expectation(expectation_configuration=expectation_configuration)

expectation_configuration = ExpectationConfiguration(
   expectation_type="expect_column_values_to_not_be_null",
   kwargs={
      "column": "user_id",
      "mostly": 0.75,
   },
   meta={
      "notes": {
         "format": "markdown",
         "content": "Some clever comment about this expectation. **Markdown** `Supported`"
      }
   }
)
suite.add_expectation(expectation_configuration=expectation_configuration)

You can see all the available Expectations in the Expectation Gallery.

3. Save your Expectation Suite

Run the final cell in the helper notebook to save your Expectation Suite.

This will create a JSON file with your Expectation Suite in the StoreA connector to store and retrieve information about metadata in Great Expectations. you have configured, which you can then load and use for ValidationThe act of applying an Expectation Suite to a Batch..