create_pull_request method on GithubClient that enables creating a pull request.create_ref method on GithubClient that enables creating a new branch.dlt.run(...).fetch_row_count() for both partitioned and non-partitioned assets. Thanks @kristianandre!PartitionedConfig objects can now return a RunConfig without causing a crash.AssetIn.__new__ typing for the dagster_type argument.fetch_row_count() method now works for databases returning uppercase column names. Thanks @kristianandre!GCSFileManager.read operations. Thanks @ollie-bell!JobDefinition, @job, and define_asset_job now take a run_tags parameter. If run_tags are defined, they will be attached to all runs of the job, and tags will not be. If run_tags is not set, then tags are attached to all runs of the job (status quo behavior). This change enables the separation of definition-level and run-level tags on jobs.DAGSTER_COMPUTE_LOG_TAIL_WAIT_AFTER_FINISH can now be used to pause before capturing logs (thanks @HynekBlaha!)kinds parameter is now available on AssetSpec.OutputContext now exposes the AssetSpec of the asset that is being stored as an output (thanks, @marijncv!)AutomationCondition on an asset, the label of this condition will now be shown in the sidebar on the Asset Details page.dagster definitions validate have been added.AssetSpec now has a with_io_manager_key method that returns an AssetSpec with the appropriate metadata entry to dictate the key for the IO manager used to load it. The deprecation warning for SourceAsset now references this method.max_runtime_seconds configuration option to run monitoring, allowing you to specify that any run in your Dagster deployment should terminate if it exceeds a certain runtime. Prevoiusly, jobs had to be individually tagged with a dagster/max_runtime tag in order to take advantage of this feature. Jobs and runs can still be tagged in order to override this value for an individual run.tags and a custom execution_fn on a ScheduleDefinition. Schedule tags are intended to annotate the definition and can be used to search and filter in the UI. They will not be attached to run requests emitted from the schedule if a custom execution_fn is provided. If no custom execution_fn is provided, then for back-compatibility the tags will also be automatically attached to run requests emitted from the schedule.SensorDefinition and all of its variants/decorators now accept a tags parameter. The tags annotate the definition and can be used to search and filter in the UI.dagster definitions validate command to Dagster CLI. This command validates if Dagster definitions are loadable.build_op_context in a unit test would sometimes raise a TypeError: signal handler must be signal.SIG_IGN, signal.SIG_DFL, or a callable object Exception on process shutdown.DefaultScheduleStatus.STOPPED / DefaultSensorStatus.STOPPED after performing a reset.--partition-range option to dagster asset materialize CLI. This option only works for assets with single-run Backfill Policies..without() method to AutomationCondition.eager(), AutomationCondition.on_cron(), and AutomationCondition.on_missing() which allows sub-conditions to be removed, e.g. AutomationCondition.eager().without(AutomationCondition.in_latest_time_window()).AutomationCondition.on_missing(), which materializes an asset partition as soon as all of its parent partitions are filled in.pyproject.toml can now load multiple Python modules as individual Code Locations. Thanks, @bdart!ADLS2PickleIOManager now accepts lease_duration configuration. Thanks, @0xfabioo!sling.replicate(...).fetch_row_count().AutomationCondition.any_downstream_condition() with downstream AutoMaterializePolicy objects.process_config_and_initialize did not properly handle processing nested resource config.DepConditionWrapperCondition instead of the key that they were evaluated against.wait_for_pod check now waits until all pods are available, rather than erroneously returning after the first pod becomes available. Thanks @easontm!AssetSpec constructor now raises an error if an invalid group name is provided, instead of an error being raised when constructing the Definitions object.dagster/relation_identifier metadata is now automatically attached to assets which are stored using a DbIOManager.DbtCliResource, the project_dir argument will now override the DBT_PROJECT_DIR environment variable if it exists in the local environment (thanks, @marijncv!).rows_loaded metadata (thanks, @kristianandre!).asset_selection=[] on RunRequest objects yielded from sensors using asset_selection would select all assets instead of none.assets_dbt_python example.database field set (thanks, @dargmuesli!)dbt.adapters.duckdb module when loading dbt assets without that package installed.DagsterInstance.wipe_asset_partitions.allow_missing_partitions as metadata on a downstream asset.object has no attribute '_base_path' error.statuses filter argument to the sensorsOrError GraphQL field was sometimes ignored when querying GraphQL for multiple sensors at the same time.CapturedLogManager and ComputeLogManager APIs into a single base class.PipesECSClient to allow Dagster to interface with ECS tasks.dbt subprocess to wait 25 seconds for the subprocess to cleanly terminate. Previously, it would only wait 2 seconds.sdf subprocess to wait 25 seconds for the subprocess to cleanly terminate. Previously, it would only wait 2 seconds.AutomationCondition using DagsterDltTranslator.get_automation_condition() (Thanks, aksestok!)dagsterDaemon.runRetries.retryOnAssetOrOpFailure to False in the Dagster Helm chart to prevent op retries and run retries from simultaneously firing on the same failure.recursive parameter (Thanks, chrishiste!)dagster/image label when image is provided from user_defined_k8s_config. (Thanks, @HynekBlaha!)custom_user_agent, was provided by defaultPipesGlueClient.run have been deprecated and will be removed in 1.9.0. The new params argument should be used instead.allow_missing_partitions configuration option.PartitionsDefinitions, there will no longer be an implicit asset job __ASSET_JOB_... for each PartitionsDefinition; there will just be one with all the assets. This reduces the time it takes to load code locations with assets with many different PartitionsDefinitions.OpenAIResource (Thanks, @chasleslr!)MultiPartitionMapping with @dbt_assets (Thanks, @arookieds!)AzureBlobComputeLogManager without a secret_key (Thanks, @ion-elgreco and @HynekBlaha!)AutomationCondition and associated static constructors.PipesCloudWatchMessageReaderBreaking Changes
The default sqlite and dagster-postgres implementations have been altered to extract the
event step_key field as a column, to enable faster per-step queries. You will need to run
dagster instance migrate to update the schema. You may optionally migrate your historical event
log data to extract the step_key using the migrate_event_log_data function. This will ensure
that your historical event log data will be captured in future step-key based views. This
event_log data migration can be invoked as follows:
from dagster.core.storage.event_log.migration import migrate_event_log_data from dagster import DagsterInstance migrate_event_log_data(instance=DagsterInstance.get())
We have made pipeline metadata serializable and persist that along with run information.
While there are no user-facing features to leverage this yet, it does require an instance
migration. Run dagster instance migrate. If you have already run the migration for the
event_log changes above, you do not need to run it again. Any unforeseen errors related to the
new snapshot_id in the runs table or the new snapshots table are related to this migration.
dagster-pandas ColumnTypeConstraint has been removed in favor of ColumnDTypeFnConstraint and
ColumnDTypeInSetConstraint.
New
FileManager machinery.PandasColumn constructors now support pandas 1.0 dtypes.env: to load from environment variables.Bugfix
dagit would not populate tags specified on the pipeline
definition.Failure was not displayed in the error modal in
dagit.dagstermill.get_context() outside of
the parameters cell of a dagstermill notebook could lead to unexpected behavior.Experimental
Schedule tab for scheduled, partitioned pipelines.
Includes views of run status, execution time, and materializations across partitions. The UI is
in flux and is currently optimized for daily schedules, but feedback is welcome.Breaking Changes
default_value in Field no longer accepts native instances of python enums. Instead
the underlying string representation in the config system must be used.default_value in Field no longer accepts callables.dagster_aws imports have been reorganized; you should now import resources from
dagster_aws.<AWS service name>. dagster_aws provides s3, emr, redshift, and cloudwatch
modules.dagster_aws S3 resource no longer attempts to model the underlying boto3 API, and you can
now just use any boto3 S3 API directly on a S3 resource, e.g.
context.resources.s3.list_objects_v2. (#2292)New
Playground view in dagit showing an interactive config mapInputDefinitiondagster pipeline launch to launch runs using a configured RunLauncherpdb utility to SolidExecutionContext to help with debugging, available within a solid
as context.pdbPresetDefinition.with_additional_config to allow for config overridesBugfix
@weekly partitioned schedule decoratordagstermill
--kernel
flag.dagster-dbt
dbt_solid now has a Nothing input to allow for sequencingdagster-k8s
get_celery_engine_config to select celery engine, leveraging Celery infrastructureDocumentation
New
Added the IntSource type, which lets integers be set from environment variables in config.
You may now set tags on pipeline definitions. These will resolve in the following cases:
execute_pipeline api will create a run with the union
of pipeline tags and RunConfig tags, with RunConfig tags taking precedence.Output materialization configs may now yield multiple Materializations, and the tutorial has been updated to reflect this.
We now export the SolidExecutionContext in the public API so that users can correctly type hint
solid compute functions.
Dagit
Bugfix
None.threads_per_worker on Dask distributed clusters.dagster-postgres
dagster-aws
s3_resource now exposes a list_objects_v2 method corresponding to the underlying boto3
API. (Thanks, @basilvetas!)redshift_resource to access Redshift databases.dagster-k8s
K8sRunLauncher config now includes the load_kubeconfig and kubeconfig_file options.Documentation
Dependencies
Community
We've added opt-in telemetry to Dagster so we can collect usage statistics in order to inform development priorities. Telemetry data will motivate projects such as adding features in frequently-used parts of the CLI and adding more examples in the docs in areas where users encounter more errors.
We will not see or store solid definitions (including generated context) or pipeline definitions (including modes and resources). We will not see or store any data that is processed within solids and pipelines.
If you'd like to opt in to telemetry, please add the following to $DAGSTER_HOME/dagster.yaml:
telemetry: enabled: true
Thanks to @basilvetas and @hspak for their contributions!
New
dagster_postgres.PostgresScheduleStorage on the instance.execute_pipeline_with_mode API to allow executing a pipeline in test with a specific
mode without having to specify RunConfig.--celery-base-priority to dagster pipeline backfill.@weekly schedule decorator.Deprecations
dagster-ge library has been removed from this release due to drift from the underlying
Great Expectations implementation.dagster-pandas
PandasColumn now includes an is_optional flag, replacing the previous
ColumnExistsConstraint.ignore_missing_values flag to PandasColumn in order to apply column
constraints only to the non-missing rows in a column.dagster-k8s
Documentation
New
dagster-k8s Helm chart.dagster-k8s Helm chart.Bugfix
SourceString.dagster schedule up would fail in certain scenarios
with the SystemCronScheduler.Pandas
Dagstermill
Docs
Experimental
RetryRequested exception, was added.
This API is experimental and likely to change.Other
runtime_type to dagster_type in definitions. The following are deprecated
and will be removed in a future version.InputDefinition.runtime_type is deprecated. Use InputDefinition.dagster_type instead.OutputDefinition.runtime_type is deprecated. Use OutputDefinition.dagster_type instead.CompositeSolidDefinition.all_runtime_types is deprecated. Use
CompositeSolidDefinition.all_dagster_types instead.SolidDefinition.all_runtime_types is deprecated. Use SolidDefinition.all_dagster_types
instead.PipelineDefinition.has_runtime_type is deprecated. Use PipelineDefinition.has_dagster_type
instead.PipelineDefinition.runtime_type_named is deprecated. Use
PipelineDefinition.dagster_type_named instead.PipelineDefinition.all_runtime_types is deprecated. Use
PipelineDefinition.all_dagster_types instead.