Skip to main content
Version: 0.16.16

How to configure an Expectation Store to use a filesystem

In progress

The Great Expectations CLI is no longer the preferred method for implementing and configuring Great Expectations. This topic will be updated soon to reflect this change. For more information, see A fond farewell to the CLI.

By default, new ProfiledThe act of generating Metrics and candidate Expectations from data. ExpectationsA verifiable assertion about data. are stored as Expectation SuitesA collection of verifiable assertions about data. in JSON format in the expectations/ subdirectory of your great_expectations folder. Use the information provided here to configure a new storage location for Expectations on your filesystem.

Prerequisites​

1. Create a new folder for Expectations​

Run the following command to create a new folder for your Expectations and move your existing Expectations to the new folder: 

# in the great_expectations/ folder
mkdir shared_expectations
mv expectations/npi_expectations.json shared_expectations/

In this example, the name of the Expectation is npi_expectations and the path to the new storage location is /shared_expectations.

2. Identify your Data Context Expectations Store​

The configuration for your Expectations StoreA connector to store and retrieve information about metadata in Great Expectations. is available in your Data ContextThe primary entry point for a Great Expectations deployment, with configurations and methods for all supporting components.. Open great_expectations.ymland find the following entry: 

expectations_store_name: expectations_store

stores:
  expectations_store:
      class_name: ExpectationsStore
      store_backend:
          class_name: TupleFilesystemStoreBackend
          base_directory: expectations/

This configuration tells Great Expectations to look for Expectations in the expectations_store Store. The default base_directory for expectations_store is expectations/.

3. Update your configuration file to include a new Store for Expectations results​

In the following example, expectations_store_name is set to shared_expectations_filesystem_store, but it can be personalized. Also, base_directory is set to shared_expectations/, but you can set it to another path that is accessible by Great Expectations. 

expectations_store_name: shared_expectations_filesystem_store

stores:
  shared_expectations_filesystem_store:
      class_name: ExpectationsStore
      store_backend:
          class_name: TupleFilesystemStoreBackend
          base_directory: shared_expectations/

4. Confirm that the new Expectation Suites have been added​

If you copied your existing Expectation Suites to your filesystem, run the following Python command to confirm that Great Expectations can find them: 

import great_expectations as gx

context = gx.get_context()
context.list_expectation_suite_names()

A list of Expectation Suites you copied your filesystem is returned. Expectation Suites that weren't copied to the new Store aren't listed.

Version control systems​

GX recommends that you store Expectations in a version control system such as Git. The JSON format of Expectations allows for informative diff-statements and modification tracking. In the following example, the `expect_table_column_count_to_equal value changes from 333 to 331, and then to 330: 

git log -p npi_expectations.json

commit cbc127fb27095364c3c1fcbf6e7f078369b07455
  changed expect_table_column_count_to_equal to 331

diff --git a/great_expectations/expectations/npi_expectations.json b/great_expectations/expectations/npi_expectations.json

--- a/great_expectations/expectations/npi_expectations.json
+++ b/great_expectations/expectations/npi_expectations.json
@@ -17,7 +17,7 @@
   {
     "expectation_type": "expect_table_column_count_to_equal",
     "kwargs": {
-        "value": 333
+        "value": 331
     }
commit 05b3c8c1ed35d183bac1717d4877fe13bc574963
changed expect_table_column_count_to_equal to 333

diff --git a/great_expectations/expectations/npi_expectations.json b/great_expectations/expectations/npi_expectations.json
--- a/great_expectations/expectations/npi_expectations.json
+++ b/great_expectations/expectations/npi_expectations.json
   {
     "expectation_type": "expect_table_column_count_to_equal",
     "kwargs": {
-        "value": 330
+        "value": 333
     }