create_pull_request method on GithubClient that enables creating a pull request.create_ref method on GithubClient that enables creating a new branch.dlt.run(...).fetch_row_count() for both partitioned and non-partitioned assets. Thanks @kristianandre!PartitionedConfig objects can now return a RunConfig without causing a crash.AssetIn.__new__ typing for the dagster_type argument.fetch_row_count() method now works for databases returning uppercase column names. Thanks @kristianandre!GCSFileManager.read operations. Thanks @ollie-bell!JobDefinition, @job, and define_asset_job now take a run_tags parameter. If run_tags are defined, they will be attached to all runs of the job, and tags will not be. If run_tags is not set, then tags are attached to all runs of the job (status quo behavior). This change enables the separation of definition-level and run-level tags on jobs.DAGSTER_COMPUTE_LOG_TAIL_WAIT_AFTER_FINISH can now be used to pause before capturing logs (thanks @HynekBlaha!)kinds parameter is now available on AssetSpec.OutputContext now exposes the AssetSpec of the asset that is being stored as an output (thanks, @marijncv!)AutomationCondition on an asset, the label of this condition will now be shown in the sidebar on the Asset Details page.dagster definitions validate have been added.AssetSpec now has a with_io_manager_key method that returns an AssetSpec with the appropriate metadata entry to dictate the key for the IO manager used to load it. The deprecation warning for SourceAsset now references this method.max_runtime_seconds configuration option to run monitoring, allowing you to specify that any run in your Dagster deployment should terminate if it exceeds a certain runtime. Prevoiusly, jobs had to be individually tagged with a dagster/max_runtime tag in order to take advantage of this feature. Jobs and runs can still be tagged in order to override this value for an individual run.tags and a custom execution_fn on a ScheduleDefinition. Schedule tags are intended to annotate the definition and can be used to search and filter in the UI. They will not be attached to run requests emitted from the schedule if a custom execution_fn is provided. If no custom execution_fn is provided, then for back-compatibility the tags will also be automatically attached to run requests emitted from the schedule.SensorDefinition and all of its variants/decorators now accept a tags parameter. The tags annotate the definition and can be used to search and filter in the UI.dagster definitions validate command to Dagster CLI. This command validates if Dagster definitions are loadable.build_op_context in a unit test would sometimes raise a TypeError: signal handler must be signal.SIG_IGN, signal.SIG_DFL, or a callable object Exception on process shutdown.DefaultScheduleStatus.STOPPED / DefaultSensorStatus.STOPPED after performing a reset.--partition-range option to dagster asset materialize CLI. This option only works for assets with single-run Backfill Policies..without() method to AutomationCondition.eager(), AutomationCondition.on_cron(), and AutomationCondition.on_missing() which allows sub-conditions to be removed, e.g. AutomationCondition.eager().without(AutomationCondition.in_latest_time_window()).AutomationCondition.on_missing(), which materializes an asset partition as soon as all of its parent partitions are filled in.pyproject.toml can now load multiple Python modules as individual Code Locations. Thanks, @bdart!ADLS2PickleIOManager now accepts lease_duration configuration. Thanks, @0xfabioo!sling.replicate(...).fetch_row_count().AutomationCondition.any_downstream_condition() with downstream AutoMaterializePolicy objects.process_config_and_initialize did not properly handle processing nested resource config.DepConditionWrapperCondition instead of the key that they were evaluated against.wait_for_pod check now waits until all pods are available, rather than erroneously returning after the first pod becomes available. Thanks @easontm!AssetSpec constructor now raises an error if an invalid group name is provided, instead of an error being raised when constructing the Definitions object.dagster/relation_identifier metadata is now automatically attached to assets which are stored using a DbIOManager.DbtCliResource, the project_dir argument will now override the DBT_PROJECT_DIR environment variable if it exists in the local environment (thanks, @marijncv!).rows_loaded metadata (thanks, @kristianandre!).asset_selection=[] on RunRequest objects yielded from sensors using asset_selection would select all assets instead of none.assets_dbt_python example.database field set (thanks, @dargmuesli!)dbt.adapters.duckdb module when loading dbt assets without that package installed.DagsterInstance.wipe_asset_partitions.allow_missing_partitions as metadata on a downstream asset.object has no attribute '_base_path' error.statuses filter argument to the sensorsOrError GraphQL field was sometimes ignored when querying GraphQL for multiple sensors at the same time.CapturedLogManager and ComputeLogManager APIs into a single base class.PipesECSClient to allow Dagster to interface with ECS tasks.dbt subprocess to wait 25 seconds for the subprocess to cleanly terminate. Previously, it would only wait 2 seconds.sdf subprocess to wait 25 seconds for the subprocess to cleanly terminate. Previously, it would only wait 2 seconds.AutomationCondition using DagsterDltTranslator.get_automation_condition() (Thanks, aksestok!)dagsterDaemon.runRetries.retryOnAssetOrOpFailure to False in the Dagster Helm chart to prevent op retries and run retries from simultaneously firing on the same failure.recursive parameter (Thanks, chrishiste!)dagster/image label when image is provided from user_defined_k8s_config. (Thanks, @HynekBlaha!)custom_user_agent, was provided by defaultPipesGlueClient.run have been deprecated and will be removed in 1.9.0. The new params argument should be used instead.allow_missing_partitions configuration option.PartitionsDefinitions, there will no longer be an implicit asset job __ASSET_JOB_... for each PartitionsDefinition; there will just be one with all the assets. This reduces the time it takes to load code locations with assets with many different PartitionsDefinitions.OpenAIResource (Thanks, @chasleslr!)MultiPartitionMapping with @dbt_assets (Thanks, @arookieds!)AzureBlobComputeLogManager without a secret_key (Thanks, @ion-elgreco and @HynekBlaha!)AutomationCondition and associated static constructors.PipesCloudWatchMessageReaderNew
AssetMaterializations can now have type information attached as metadata. See the materializations tutorial for moreBugfixes
context['ts'] was not passed properlytask_acks_late: true that resulted in a 409 Conflict error from Kubernetes. The creation of a Kubernetes Job will now be aborted if another Job with the same name existsDocs
New
CeleryK8sRunLauncher supports termination of pipeline runs. This can be accessed via the
“Terminate” button in Dagit’s Pipeline Run view or via “Cancel” in Dagit’s All Runs page. This
will terminate the run master K8s Job along with all running step job K8s Jobs; steps that are
still in the Celery queue will not create K8s Jobs. The pipeline and all impacted steps will
be marked as failed. We recommend implementing resources as context managers and we will execute
the finally block upon termination.K8sRunLauncher supports termination of pipeline runs.AssetMaterialization events display the asset key in the Runs view.Bugfixes
DagsterInstance was leaving database connections open due to not being
garbage collected.Enum in resource config schemas resulted in an error.New
configured API makes it easy to create configured versions of resources.Materialization event type in favor of the new AssetMaterialization event type,
which requires the asset_key parameter. Solids yielding Materialization events will continue
to work as before, though the Materialization event will be removed in a future release.intermediate_storage_defs argument to ModeDefinition, which accepts a
list of IntermediateStorageDefinitions, e.g. s3_plus_default_intermediate_storage_defs.
As before, the default includes an in-memory intermediate and a local filesystem intermediate
storage.system_storage_defs argument to ModeDefinition in favor of
intermediate_storage_defs. system_storage_defs will be removed in 0.10.0 at the earliest.@intermediate_storage decorator, which makes it easy to define intermediate
storages.s3_file_manager and local_file_manager resources to replace the file managers
that previously lived inside system storages. The airline demo has been updated to include
an example of how to do this:
https://github.com/dagster-io/dagster/blob/0.8.8/examples/airline_demo/airline_demo/solids.py#L171.Bugfixes
default_value config on a field now works as expected. #2725Breaking Changes
dagster and dagit CLI commands no longer add the working directory to the
PYTHONPATH when resolving modules, which may break some imports. Explicitly installed python
packages can be specified in workspaces using the python_package workspace yaml config option.
The python_module config option is deprecated and will be removed in a future release.New
--path-prefix to the dagit CLI. #2073date_partition_range util function now accepts an optional inclusive boolean argument. By default, the function does not return include the partition for which the end time of the date range is greater than the current time. If inclusive=True, then the list of partitions returned will include the extra partition.MultiDependency or fan-in inputs will now only cause the solid step to skip if all of the
fanned-in inputs upstream outputs were skippedBugfixes
input_hydration_config argumentsalias on a solid output will produce a useful error message (thanks @iKintosh!)daily_schedule) for certain workspace.yaml formatsBreaking Changes
dagster-celery module has been broken apart to manage dependencies more coherently. There
are now three modules: dagster-celery, dagster-celery-k8s, and dagster-celery-docker.dagster-celery worker start command now takes a required -A parameter
which must point to the app.py file within the appropriate module. E.g if you are using the
celery_k8s_job_executor then you must use the -A dagster_celery_k8s.app option when using the
celery or dagster-celery cli tools. Similar for the celery_docker_executor:
-A dagster_celery_docker.app must be used.input_hydration_config and output_materialization_config decorators to
dagster_type_ and dagster_type_materializer respectively. Renamed DagsterType's
input_hydration_config and output_materialization_config arguments to loader and materializer respectively.New
New pipeline scoped runs tab in Dagit
Add the following Dask Job Queue clusters: moab, sge, lsf, slurm, oar (thanks @DavidKatz-il!)
K8s resource-requirements for run coordinator pods can be specified using the dagster-k8s/ resource_requirements tag on pipeline definitions:
@pipeline( tags={ 'dagster-k8s/resource_requirements': { 'requests': {'cpu': '250m', 'memory': '64Mi'}, 'limits': {'cpu': '500m', 'memory': '2560Mi'}, } }, ) def foo_bar_pipeline():
Added better error messaging in dagit for partition set and schedule configuration errors
An initial version of the CeleryDockerExecutor was added (thanks @mrdrprofuroboros!). The celery workers will launch tasks in docker containers.
Experimental: Great Expectations integration is currently under development in the new library dagster-ge. Example usage can be found here