Dagster Types#

The Dagster type system helps you describe what kind of values your ops accept and produce.

Relevant APIs#

Overview#

Each op input and output can be given a Dagster Type.

The type system:

• Is gradual and optional. Jobs can run without types specified explicitly, and specifying types in some places doesn't require that types be specified everywhere. Inputs and outputs default to the Any type.

• Happens at op execution time - each type defines a type_check_fn that knows how to check whether values match what it expects. When a type is specified for a op's input, then the type check occurs immediately before the op is executed. When a type is specified for a op's output, then the type check occurs immediately after the op is executed.

• Is complimentary to the PEP 484 Python type system. PEP 484 annotations enable static checks that verify variables and return values match particular Python types, while the Dagster type system enables runtime checks that include arbitrary validation logic.

DagsterTypes vs Python Types (mypy type-checking)#

You should not use manually-defined DagsterType objects in your python type annotations. The simple rule of thumb is to annotate op bodies just as you would a regular function, and if you wish to use a DagsterType alongside your annotations (to perform more complex validation than the default runtime typechecks), you should include this information in the relevant InputDefinition or OutputDefinition.

You can see an example of this pattern below.

Defining a Dagster Type#

The core API for defining Dagster types is DagsterType.

EvenDagsterType = DagsterType(
    name="EvenDagsterType",
    type_check_fn=lambda _, value: isinstance(value, int) and value % 2 is 0,
)

Once created, types can be attached to op InputDefinitions and OutputDefinitions.

@op(
    ins={"num": In(EvenDagsterType)},
    out=Out(EvenDagsterType),
)
def double_even(num):
    return num

The type system truly shines once the type check expresses richer behavior, such as column-level schema on a dataframe. For example, check out the Validating Pandas DataFrames with Dagster Types guide.

Dagster Types Inferred from Type Hints#

If a Python input or output has a PEP 484 type annotation, and a DagsterType is not provided on the corresponding input or output definition, then Dagster will automatically generate a DagsterType that corresponds to the annotated Python type.

In this example, the defined op will end up with a DagsterType named "MyClass" that:

• Shows up in Dagit in the representation of the op.
• Is checked at runtime on the value returned by the op.

class MyClass:
    pass


@op
def my_op() -> MyClass:
    return MyClass()

If the op in the above example returned an object that was not an instance of MyClass, Dagster would raise an error after executing the op.

The Nothing Type#

Dagster offers a special type called Nothing, which is used when you need to model a dependency between ops where Dagster is passing no data along the edge. See details in the Nothing dependencies example.

Testing a Dagster Type#

You can use check_dagster_type to test the type check function of a custom Dagster Type:

from dagster import check_dagster_type, Dict, Any


def test_dagster_type():

    assert check_dagster_type(Dict[Any, Any], {"foo": "bar"}).success

Examples#

Using Dagster Types with PEP 484 Type Annotations#

Dagster types peacefully coexist with Python type annotations. In this example, the inputs and outputs of the op compute function are integers, and the type check function for EvenDagsterType will be invoked at runtime to verify that they are even.

@op(
    ins={"num": In(EvenDagsterType)},
    out=Out(EvenDagsterType),
)
def double_even_with_annotations(num: int) -> int:
    return num

See it in action#

For more examples of the dagster type system, check out our Bollinger example