Create a Custom Set-Based Column Map Expectation
SetBasedColumnMapExpectations
are a sub-type of
ColumnMapExpectationA verifiable assertion about data.. They are evaluated for a single column and ask
whether each row in that column belongs to the
specified set.
Based on the result, they then calculate the
percentage of rows that gave a positive answer. If
that percentage meets a specified threshold (100% by
default), the Expectation considers that data valid.
This threshold is configured via the
mostly
parameter, which can be passed as
input to your Custom
SetBasedColumnMapExpectation
as a
float
between 0 and 1.
This guide will walk you through the process of
creating a Custom
SetBasedColumnMapExpectation
.
Prerequisites
Choose a name for your Expectation
First, decide on a name for your own Expectation. By
convention, all ColumnMapExpectations
,
including SetBasedColumnMapExpectations
,
start with expect_column_values_
.
Your Expectation will have two versions of the same
name: a CamelCaseName
and a
snake_case_name
. For example, this
tutorial will use:
-
ExpectColumnValuesToBeInSolfegeScaleSet
-
expect_column_values_to_be_in_solfege_scale_set
Copy and rename the template file
By convention, each Expectation is kept in its own python file, named with the snake_case version of the Expectation's name.
You can find the template file for a custom
SetBasedColumnMapExpectation
here. Download the file, place it in the appropriate
directory, and rename it to the appropriate name.
cp set_based_column_map_expectation_template.py /SOME_DIRECTORY/expect_column_values_to_be_in_solfege_scale_set.py
Storing Expectation files
During development, you don't need to store Expectation files in a specific location. Expectation files are self-contained and can be executed anywhere as long as GX is installed However, to use your new Expectation with other GX components, you'll need to make sure the file is stored one of the following locations:
-
If you're building a Custom ExpectationAn extension of the `Expectation` class, developed outside of the Great Expectations library. for personal use, you'll need to put it in the
great_expectations/plugins/expectations
folder of your GX deployment, and import your Custom Expectation from that directory whenever it will be used. When you instantiate the correspondingDataContext
, it will automatically make all PluginsExtends Great Expectations' components and/or functionality. in the directory available for use. -
If you're building a Custom Expectation to contribute to the open source project, you'll need to put it in the repo for the Great Expectations library itself. Most likely, this will be within a package within
contrib/
:great_expectations/contrib/SOME_PACKAGE/SOME_PACKAGE/expectations/
. To use these Expectations, you'll need to install the package.
For more information about Custom Expectations, see Use a Custom Expectation.
Generate a diagnostic checklist for your Expectation
Once you've copied and renamed the template file, you can execute it as follows.
python expect_column_values_to_be_in_solfege_scale_set.py
The template file is set up so that this will run the
Expectation's
print_diagnostic_checklist()
method. This
will run a diagnostic script on your new Expectation,
and return a checklist of steps to get it to full
production readiness.
Completeness checklist for ExpectColumnValuesToBeInSomeSet:
✔ Has a valid library_metadata object
Has a docstring, including a one-line short description that begins with "Expect" and ends with a period
Has at least one positive and negative example case, and all test cases pass
Has core logic and passes tests on at least one Execution Engine
Passes all linting checks
Has basic input validation and type checking
Has both Statement Renderers: prescriptive and diagnostic
Has core logic that passes tests for all applicable Execution Engines and SQL dialects
Has a robust suite of tests, as determined by a code owner
Has passed a manual review by a code owner for code standards and style guides
When in doubt, the next step to implement is the first one that doesn't have a ✔ next to it. This guide covers the first five steps on the checklist.
Change the Expectation class name and add a docstring
Let's start by updating your Expectation's name and docstring.
Replace the Expectation class name
class ExpectColumnValuesToBeInSomeSet(SetBasedColumnMapExpectation):
with your real Expectation class name, in upper camel case:
class ExpectColumnValuesToBeInSolfegeScaleSet(SetBasedColumnMapExpectation):
You can also go ahead and write a new one-line docstring, replacing
"""TODO: Add a docstring here"""
with something like:
"""Expect values in this column should be valid members of the Solfege scale: do, re, mi, etc."""
Make sure your one-line docstring begins with "Expect " and ends with a period. You'll also need to change the class name at the bottom of the file, by replacing this line:
ExpectColumnValuesToBeInSomeSet().print_diagnostic_checklist()
with this one:
ExpectColumnValuesToBeInSolfegeScaleSet().print_diagnostic_checklist()
Later, you can go back and write a more thorough docstring. See Expectation Docstring Formatting.
At this point you can re-run your diagnostic checklist. You should see something like this:
$ python expect_column_values_to_be_in_solfege_scale_set.py
Completeness checklist for ExpectColumnValuesToBeInSolfegeScaleSet:
✔ Has a valid library_metadata object
✔ Has a docstring, including a one-line short description that begins with "Expect" and ends with a period
Has at least one positive and negative example case, and all test cases pass
Has core logic and passes tests on at least one Execution Engine
Passes all linting checks
...
Congratulations! You're one step closer to implementing a Custom Expectation.
Add example cases
You're going to search for
examples = []
in your file, and replace
it with at least two test examples. These examples
serve the following purposes:
-
They provide test fixtures that Great Expectations can execute automatically with pytest.
-
They help users understand the logic of your Expectation by providing tidy examples of paired input and output. If you contribute your Expectation to open source, these examples will appear in the Gallery.
Your examples will look similar to this example:
examples = [
{
"data": {
"lowercase_solfege_scale": [
"do",
"re",
"mi",
"fa",
"so",
"la",
"ti",
"do",
],
"uppercase_solfege_scale": [
"DO",
"RE",
"MI",
"FA",
"SO",
"LA",
"TI",
"DO",
],
"mixed": ["do", "od", "re", "er", "mi", "im", "fa", "af"],
},
"only_for": ["pandas", "spark", "sqlite", "postgresql"],
"tests": [
{
"title": "positive_test_lowercase",
"exact_match_out": False,
"in": {"column": "lowercase_solfege_scale"},
"out": {
"success": True,
},
"include_in_gallery": True,
},
{
"title": "negative_test",
"exact_match_out": False,
"in": {"column": "mixed"},
"out": {
"success": False,
"unexpected_index_list": [1, 3, 5, 7],
},
"include_in_gallery": True,
},
{
"title": "postive_test_uppercase",
"exact_match_out": False,
"in": {"column": "uppercase_solfege_scale"},
"out": {
"success": True,
},
"include_in_gallery": True,
},
{
"title": "positive_test_with_mostly",
"exact_match_out": False,
"in": {"column": "mixed", "mostly": 0.4},
"out": {
"success": True,
"unexpected_index_list": [1, 3, 5, 7],
},
"include_in_gallery": True,
},
],
}
]
Here's a quick overview of how to create test
cases to populate examples
. The overall
structure is a list of dictionaries. Each dictionary
has two keys:
-
data
: defines the input data of the example as a table/data frame. In this example the table has columns namedlowercase_solfege_scale
,uppercase_solfege_scale
, andmixed
. All of these columns have 8 rows. (Note: if you define multiple columns, make sure that they have the same number of rows.) -
tests
: a list of test cases to validate against the data frame defined in the correspondingdata
.-
title
should be a descriptive name for the test case. Make sure to have no spaces. -
include_in_gallery
: This must be set toTrue
if you want this test case to be visible in the Gallery as an example. -
in
contains exactly the parameters that you want to pass in to the Expectation."in": {"column": "mixed", "mostly": .1}
in the example above is equivalent toexpect_column_values_to_be_in_solfege_scale_set(column="mixed", mostly=0.1)
-
out
is based on the Validation Result returned when executing the Expectation. -
exact_match_out
: if you setexact_match_out=False
, then you don’t need to include all the elements of the Validation Result object - only the ones that are important to test.
-
If you run your Expectation file again, you won't see any new checkmarks, as the logic for your Custom Expectation hasn't been implemented yet. However, you should see that the tests you've written are now being caught and reported in your checklist:
$ python expect_column_values_to_be_in_solfege_scale_set.py
Completeness checklist for ExpectColumnValuesToBeInSolfegeScaleSet:
✔ Has a valid library_metadata object
✔ Has a docstring, including a one-line short description that begins with "Expect" and ends with a period
...
Has core logic that passes tests for all applicable Execution Engines and SQL dialects
Only 0 / 2 tests for pandas are passing
Only 0 / 2 tests for spark are passing
Only 0 / 2 tests for sqlite are passing
Failing: basic_positive_test, basic_negative_test
...
Passes all linting checks
For more information on tests and example cases,
see our guide on
how to create example cases for a Custom
Expectation.
Define your set and connect it to your Expectation
This is the stage where you implement the actual business logic for your Expectation.
In the case of your Custom
SetBasedColumnMapExpectation
, Great
Expectations will handle the actual validation of your
data against your set.
To do this, we replace these:
set_ = []
set_camel_name = "SetName"
with something like this:
set_ = [
"do",
"re",
"mi",
"fa",
"so",
"la",
"ti",
"Do",
"Re",
"Mi",
"Fa",
"So",
"La",
"Ti",
"DO",
"RE",
"MI",
"FA",
"SO",
"LA",
"TI",
]
set_camel_name = "SolfegeScale"
For more detail when rendering your Custom Expectation, you can optionally specify the semantic name of the set you're validating.
For example:
set_semantic_name = None
becomes:
set_semantic_name = "the Solfege scale"
Great Expectations will use these values to tell your Custom Expectation to apply your specified set as a MetricA computed attribute of data such as the mean of a column. to be utilized in validating your data.
This is all that you need to define for now. The
SetBasedColumnMapExpectation
class has
built-in logic to handle all the machinery of data
validation, including standard parameters like
mostly
, generation of Validation Results,
etc.
Other parameters
Expectation Success Keys - A tuple consisting of values that must / could be provided by the user and defines how the Expectation evaluates success.
Expectation Default Kwarg Values (Optional) - Default values for success keys and the defined domain, among other values.
Running your diagnostic checklist at this point should return something like this:
$ python expect_column_values_to_be_in_solfege_scale_set.py
Completeness checklist for ExpectColumnValuesToBeInSolfegeScaleSet:
✔ Has a valid library_metadata object
✔ Has a docstring, including a one-line short description that begins with "Expect" and ends with a period
✔ Has at least one positive and negative example case, and all test cases pass
✔ Has core logic and passes tests on at least one Execution Engine
Passes all linting checks
...
Linting
Finally, we need to lint our now-functioning Custom
Expectation. Our CI system will test your code using
black
, and ruff
.
If you've set up your dev environment, these libraries will already be available to you, and can be invoked from your command line to automatically lint your code:
black <PATH/TO/YOUR/EXPECTATION.py>
ruff <PATH/TO/YOUR/EXPECTATION.py> --fix
If desired, you can automate this to happen at commit time. See our guidance on linting for more on this process.
Once this is done, running your diagnostic checklist should now reflect your Custom Expectation as meeting our linting requirements:
$ python expect_column_values_to_be_in_solfege_scale_set.py
Completeness checklist for ExpectColumnValuesToBeInSolfegeScaleSet:
✔ Has a valid library_metadata object
✔ Has a docstring, including a one-line short description that begins with "Expect" and ends with a period
✔ Has at least one positive and negative example case, and all test cases pass
✔ Has core logic and passes tests on at least one Execution Engine
✔ Passes all linting checks
...
If you've already built a
Custom Expectation
of a different type, you may notice that we
didn't explicitly implement a
_validate
method or Metric class
here. While we have to explicitly create these for
other types of Custom Expectations, the
SetBasedColumnMapExpectation
class
handles Metric creation and result validation
implicitly; no extra work needed!
Contribute (Optional)
This guide will leave you with a Custom Expectation sufficient for contribution to Great Expectations at an Experimental level.
If you plan to contribute your Expectation to the
public open source project, you should update the
library_metadata
object before submitting
your
Pull Request. For example:
library_metadata = {
"tags": ["set-based"], # Tags for this Expectation in the Gallery
"contributors": [ # Github handles for all contributors to this Expectation.
"@your_name_here", # Don't forget to add your github handle here!
],
}
would become
library_metadata = {
"tags": ["set-based"],
"contributors": ["@joegargery"],
}
This is particularly important because we want to make sure that you get credit for all your hard work!
For more information on our code standards and contribution, see our guide on Levels of Maturity for Expectations.
To view the full script used in this page, see it on GitHub: