[experimental] AutomationCondition.eager() will now only launch runs for missing partitions which become missing after the condition has been added to the asset. This avoids situations in which the eager policy kicks off a large amount of work when added to an asset with many missing historical static/dynamic partitions.
[experimental] Added a new AutomationCondition.asset_matches() condition, which can apply a condition against an arbitrary asset in the graph.
[experimental] Added the ability to specify multiple kinds for an asset with the kinds parameter.
[dagster-github] Added create_pull_request method on GithubClient that enables creating a pull request.
[dagster-github] Added create_ref method on GithubClient that enables creating a new branch.
[dagster-embedded-elt] dlt assets now generate column metadata for child tables.
[dagster-embedded-elt] dlt assets can now fetch row count metadata with dlt.run(...).fetch_row_count() for both partitioned and non-partitioned assets. Thanks @kristianandre!
[dagster-airbyte] relation identifier metadata is now attached to Airbyte assets.
[dagster-embedded-elt] relation identifier metadata is now attached to sling assets.
[dagster-embedded-elt] relation identifier metadata is now attached to dlt assets.
JobDefinition, @job, and define_asset_job now take a run_tags parameter. If run_tags are defined, they will be attached to all runs of the job, and tags will not be. If run_tags is not set, then tags are attached to all runs of the job (status quo behavior). This change enables the separation of definition-level and run-level tags on jobs.
Then env var DAGSTER_COMPUTE_LOG_TAIL_WAIT_AFTER_FINISH can now be used to pause before capturing logs (thanks @HynekBlaha!)
The kinds parameter is now available on AssetSpec.
OutputContext now exposes the AssetSpec of the asset that is being stored as an output (thanks, @marijncv!)
[experimental] Backfills are incorporated into the Runs page to improve observability and provide a more simplified UI. See the GitHub discussion for more details.
[ui] The updated navigation is now enabled for all users. You can revert to the legacy navigation via a feature flag. See GitHub discussion for more.
[ui] Improved performance for loading partition statuses of an asset job.
[dagster-docker] Run containers launched by the DockerRunLauncher now include dagster/job_name and dagster/run_id labels.
[dagster-aws] The ECS launcher now automatically retries transient ECS RunTask failures (like capacity placement failures).
Changed the log volume for global concurrency blocked runs in the run coordinator to be less spammy.
[ui] Asset checks are now visible in the run page header when launched from a schedule.
[ui] Fixed asset group outlines not rendering properly in Safari.
[ui] Reporting a materialization event now removes the asset from the asset health "Execution failures" list and returns the asset to a green / success state.
[ui] When setting an AutomationCondition on an asset, the label of this condition will now be shown in the sidebar on the Asset Details page.
[ui] Previously, filtering runs by Created date would include runs that had been updated after the lower bound of the requested time range. This has been updated so that only runs created after the lower bound will be included.
[ui] When using the new experimental navigation flag, added a fix for the automations page for code locations that have schedules but no sensors.
[ui] Fixed tag wrapping on asset column schema table.
[ui] Restored object counts on the code location list view.
[ui] Padding when displaying warnings on unsupported run coordinators has been corrected (thanks @hainenber!)
[dagster-k8s] Fixed an issue where run termination sometimes did not terminate all step processes when using the k8s_job_executor, if the termination was initiated while it was in the middle of launching a step pod.
AssetSpec now has a with_io_manager_key method that returns an AssetSpec with the appropriate metadata entry to dictate the key for the IO manager used to load it. The deprecation warning for SourceAsset now references this method.
Added a max_runtime_seconds configuration option to run monitoring, allowing you to specify that any run in your Dagster deployment should terminate if it exceeds a certain runtime. Prevoiusly, jobs had to be individually tagged with a dagster/max_runtime tag in order to take advantage of this feature. Jobs and runs can still be tagged in order to override this value for an individual run.
It is now possible to set both tags and a custom execution_fn on a ScheduleDefinition. Schedule tags are intended to annotate the definition and can be used to search and filter in the UI. They will not be attached to run requests emitted from the schedule if a custom execution_fn is provided. If no custom execution_fn is provided, then for back-compatibility the tags will also be automatically attached to run requests emitted from the schedule.
SensorDefinition and all of its variants/decorators now accept a tags parameter. The tags annotate the definition and can be used to search and filter in the UI.
Added the dagster definitions validate command to Dagster CLI. This command validates if Dagster definitions are loadable.
[dagster-databricks] Databricks Pipes now allow running tasks in existing clusters.
Fixed an issue where calling build_op_context in a unit test would sometimes raise a TypeError: signal handler must be signal.SIG_IGN, signal.SIG_DFL, or a callable object Exception on process shutdown.
[dagster-webserver] Fix an issue where the incorrect sensor/schedule state would appear when using DefaultScheduleStatus.STOPPED / DefaultSensorStatus.STOPPED after performing a reset.
Fixed an issue where users with Launcher permissions for a particular code location were not able to cancel backfills targeting only assets in that code location.
Fixed an issue preventing long-running alerts from being sent when there was a quick subsequent run.
Added --partition-range option to dagster asset materialize CLI. This option only works for assets with single-run Backfill Policies.
Added a new .without() method to AutomationCondition.eager(), AutomationCondition.on_cron(), and AutomationCondition.on_missing() which allows sub-conditions to be removed, e.g. AutomationCondition.eager().without(AutomationCondition.in_latest_time_window()).
Added AutomationCondition.on_missing(), which materializes an asset partition as soon as all of its parent partitions are filled in.
pyproject.toml can now load multiple Python modules as individual Code Locations. Thanks, @bdart!
[ui] If a code location has errors, a button will be shown to view the error on any page in the UI.
[dagster-adls2] The ADLS2PickleIOManager now accepts lease_duration configuration. Thanks, @0xfabioo!
[dagster-embedded-elt] Added an option to fetch row count metadata after running a Sling sync by calling sling.replicate(...).fetch_row_count().
[dagster-fivetran] The dagster-fivetran integration will now automatically pull and attach column schema metadata after each sync.
Fixed an issue which could cause errors when using AutomationCondition.any_downstream_condition() with downstream AutoMaterializePolicy objects.
Fixed an issue where process_config_and_initialize did not properly handle processing nested resource config.
[ui] Fixed an issue that would cause some AutomationCondition evaluations to be labeled DepConditionWrapperCondition instead of the key that they were evaluated against.
[dagster-webserver] Fixed an issue with code locations appearing in fluctuating incorrect state in deployments with multiple webserver processes.
[dagster-embedded-elt] Fixed an issue where Sling column lineage did not correctly resolve int the Dagster UI.
[dagster-k8s] The wait_for_pod check now waits until all pods are available, rather than erroneously returning after the first pod becomes available. Thanks @easontm!
The AssetSpec constructor now raises an error if an invalid group name is provided, instead of an error being raised when constructing the Definitions object.
dagster/relation_identifier metadata is now automatically attached to assets which are stored using a DbIOManager.
[ui] Streamlined the code location list view.
[ui] The “group by” selection on the Timeline Overview page is now part of the query parameters, meaning it will be retained when linked to directly or when navigating between pages.
[dagster-dbt] When instantiating DbtCliResource, the project_dir argument will now override the DBT_PROJECT_DIR environment variable if it exists in the local environment (thanks, @marijncv!).
[dagster-embedded-elt] dlt assets now generate rows_loaded metadata (thanks, @kristianandre!).
Fixed a bug where setting asset_selection=[] on RunRequest objects yielded from sensors using asset_selection would select all assets instead of none.
Fixed bug where the tick status filter for batch-fetched graphql sensors was not being respected.
[examples] Fixed missing assets in assets_dbt_python example.
[dagster-airbyte] Updated the op names generated for Airbyte assets to include the full connection ID, avoiding name collisions.
[dagster-dbt] Fixed issue causing dagster-dbt to be unable to load dbt projects where the adapter did not have a database field set (thanks, @dargmuesli!)
[dagster-dbt] Removed a warning about not being able to load the dbt.adapters.duckdb module when loading dbt assets without that package installed.
You may now wipe specific asset partitions directly from the execution context in user code by calling DagsterInstance.wipe_asset_partitions.
Dagster+ users with a "Viewer" role can now create private catalog views.
Fixed an issue where the default IOManager used by Dagster+ Serverless did not respect setting allow_missing_partitions as metadata on a downstream asset.
Fixed an issue where runs in Dagster+ Serverless that materialized partitioned assets would sometimes fail with an object has no attribute '_base_path' error.
[dagster-graphql] Fixed an issue where the statuses filter argument to the sensorsOrError GraphQL field was sometimes ignored when querying GraphQL for multiple sensors at the same time.
Updated multi-asset sensor definition to be less likely to timeout queries against the asset history storage.
Consolidated the CapturedLogManager and ComputeLogManager APIs into a single base class.
[ui] Added an option under user settings to clear client side indexeddb caches as an escape hatch for caching related bugs.
[dagster-aws, dagster-pipes] Added a new PipesECSClient to allow Dagster to interface with ECS tasks.
[dagster-dbt] Increased the default timeout when terminating a run that is running a dbt subprocess to wait 25 seconds for the subprocess to cleanly terminate. Previously, it would only wait 2 seconds.
[dagster-sdf] Increased the default timeout when terminating a run that is running an sdf subprocess to wait 25 seconds for the subprocess to cleanly terminate. Previously, it would only wait 2 seconds.
[dagster-sdf] Added support for caching and asset selection (Thanks, akbog!)
[dagster-dlt] Added support for AutomationCondition using DagsterDltTranslator.get_automation_condition() (Thanks, aksestok!)
[ui] Fixed a bug where in-progress runs from a backfill could not be terminated from the backfill UI.
[ui] Fixed a bug that caused an "Asset must be part of at least one job" error when clicking on an external asset in the asset graph UI
Fixed an issue where viewing run logs with the latest 5.0 release of the watchdog package raised an exception.
[ui] Fixed issue causing the “filter to group” action in the lineage graph to have no effect.
[ui] Fixed case sensitivity when searching for partitions in the launchpad.
[ui] Fixed a bug which would redirect to the events tab for an asset if you loaded the partitions tab directly.
[ui] Fixed issue causing runs to get skipped when paging through the runs list (Thanks, @HynekBlaha!)
[ui] Fixed a bug where the asset catalog list view for a particular group would show all assets.
[dagster-dbt] fix bug where empty newlines in raw dbt logs were not being handled correctly.
[dagster-k8s, dagster-celery-k8s] Correctly set dagster/image label when image is provided from user_defined_k8s_config. (Thanks, @HynekBlaha!)
[dagster-duckdb] Fixed an issue for DuckDB versions older than 1.0.0 where an unsupported configuration option, custom_user_agent, was provided by default
[dagster-k8s] Fixed an issue where Kubernetes Pipes failed to create a pod if the op name contained capital or non-alphanumeric containers.
[dagster-embedded-elt] Fixed an issue where dbt assets downstream of Sling were skipped
[dagser-aws]: Direct AWS API arguments in PipesGlueClient.run have been deprecated and will be removed in 1.9.0. The new params argument should be used instead.
The default io_manager on Serverless now supports the allow_missing_partitions configuration option.
Fixed a bug that caused an error when loading the launchpad for a partition, when using in Dagster+ with an agent with version below 1.8.2
1.8.3 (core) / 0.24.3 (libraries) (YANKED - This version of Dagster resulted in errors when trying to launch runs that target individual asset partitions)#
When different assets within a code location have different PartitionsDefinitions, there will no longer be an implicit asset job __ASSET_JOB_... for each PartitionsDefinition; there will just be one with all the assets. This reduces the time it takes to load code locations with assets with many different PartitionsDefinitions.
[ui] Fixed a collection of broken links pointing to renamed Declarative Automation pages.
[dagster-dbt] Fixed issue preventing usage of MultiPartitionMapping with @dbt_assets (Thanks, @arookieds!)
[dagster-azure] Fixed issue that would cause an error when configuring an AzureBlobComputeLogManager without a secret_key (Thanks, @ion-elgreco and @HynekBlaha!)
[ui] For users whose light/dark mode theme setting is set to match their system setting, the theme will update automatically when the system changes modes (e.g. based on time of day), with no page reload required.
[ui] We have introduced the typefaces Geist and Geist Mono as our new default fonts throughout the Dagster app, with the goal of improving legibility, consistency, and maintainability.
[ui][experimental] We have begun experimenting with a new navigation structure for the Dagster UI. The change can be enabled via User Settings.
[ui][experimental] Made performance improvements to the Concurrency settings page.
Fixed an issue where pressing the “Reset sensor status” button in the UI would also reset the sensor’s cursor.
Fixed a bug that caused input loading time not to be included in the reported step duration.
Pydantic warnings are no longer raised when importing Dagster with Pydantic 2.0+.
Fixed an issue which would cause incorrect behavior when auto-materializing partitioned assets based on updates to a parent asset in a different code location.
Fixed an issue which would cause every tick of the auto-materialize sensor to produce an evaluation for each asset, even if nothing had changed from the previous tick.
[dagster-dbt] Fixed a bug that could raise Duplicate check specs errors with singular tests ingested as asset checks.
[embedded-elt] resolved an issue where subset of resources were not recognized when using source.with_resources(...)
[ui] Fixed an issue where a sensor that targeted an invalid set of asset keys could cause the asset catalog to fail to load.
[ui] Fixed an issue in which runs in the Timeline that should have been considered overlapping were not correctly grouped together, leading to visual bugs.
[ui] On the asset overview page, job tags no longer render poorly when an asset appears in several jobs.
[ui] On the asset overview page, hovering over the timestamp tags in the metadata table explains where each entry originated.
[ui] Right clicking the background of the asset graph now consistently shows a context menu, and the lineage view supports vertical as well as horizontal layout.
[ui][experimental] A new Overview > Asset Health page provides visibility into failed and missing materializations, check warnings and check errors.
[ui] You can now share feedback with the Dagster team directly from the app. Open the Help menu in the top nav, then “Share feedback”. Bugs and feature requests are submitted directly to the Dagster team.
[ui] When editing a team, the list of team members is now virtualized, allowing for the UI to scale better for very large team sizes.
Performance improvements when loading large asset graphs in the Dagster UI.
@asset_check functions can now be invoked directly for unit testing.
dagster-embedded-elt dlt resource DagsterDltResource can now be used from @op definitions in addition to assets.
UPathIOManager.load_partitions has been added to assist with helping UpathIOManager subclasses deal with serialization formats which support partitioning. Thanks @danielgafni!
[dagster-polars] now supports other data types rather than only string for the partitioning columns. Also PolarsDeltaIOManager now supports MultiPartitionsDefinition with DeltaLake native partitioning. Metadata value "partition_by": {"dim_1": "col_1", "dim_2": "col_2"} should be specified to enable this feature. Thanks @danielgafni!
[dagster-airbyte] Auto materialization policies passed to load_assets_from_airbyte_instance and load_assets_from_airbyte_project will now be properly propagated to the created assets.
Fixed an issue where deleting a run that was intended to materialize a partitioned asset would sometimes leave the status of that asset as “Materializing” in the Dagster UI.
Fixed an issue with build_time_partition_freshness_checks where it would incorrectly intuit that an asset was not fresh in certain cases.
[dagster-k8s] Fix an error on transient ‘none’ responses for pod waiting reasons. Thanks @piotrmarczydlo!
[dagster-dbt] Failing to build column schema metadata will now result in a warning rather than an error.
Fixed an issue where incorrect asset keys would cause a backfill to fail loudly.
Fixed an issue where syncing unmaterialized assets could include source assets.
[dagster-polars] PolarsDeltaIOManager no longer supports loading natively partitioned DeltaLake tables as dictionaries. They should be loaded as a single pl.DataFrame/pl.LazyFrame instead.
Branch deployments now have Change Tracking. Assets in each branch deployment will be compared to the main deployment. New assets and changes to code version, dependencies, partitions definitions, tags, and metadata will be marked in the UI of the branch deployment.
Pagerduty alerting is now supported with Pro plans. See the documentation for more info.
Asset metadata is now included in the insights metrics for jobs materializing those assets.
Per-run Insights are now available on individual assets.
Previously, the before_storage_id / after_storage_id values in the AssetRecordsFilter class were ignored. This has been fixed.
Updated the output of dagster-cloud deployment alert-policies list to match the format of sync.
Fixed an issue where Dagster Cloud agents with many code locations would sometimes leave code servers running after the agent shut down.
[dagster-dbt][experimental] A new cli command dagster-dbt project prepare-for-deployment has been added in conjunction with DbtProject for managing the behavior of rebuilding the manifest during development and preparing a pre-built one for production.
Fixed an issue with duplicate asset check keys when loading checks from a package.
A bug with the new build_last_update_freshness_checks and build_time_partition_freshness_checks has been fixed where multi_asset checks passed in would not be executable.
[dagster-dbt] Fixed some issues with building column lineage for incremental models, models with implicit column aliases, and models with columns that have multiple dependencies on the same upstream column.
Major Changes since 1.6.0 (core) / 0.22.0 (libraries)#
Asset definitions can now have tags, via the tags argument on @asset, AssetSpec, and AssetOut. Tags are meant to be used for organizing, filtering, and searching for assets.
The Asset Details page has been revamped to include an “Overview” tab that centralizes the most important information about the asset – such as current status, description, and columns – in a single place.
Assets can now be assigned owners.
Asset checks are now considered generally available and will no longer raise experimental warnings when used.
Asset checks can now be marked blocking, which causes downstream assets in the same run to be skipped if the check fails with ERROR-level severity.
The new @multi_asset_check decorator enables defining a single op that executes multiple asset checks.
The new build_last_updated_freshness_checks and build_time_partition_freshness_checks APIs allow defining asset checks that error or warn when an asset is overdue for an update. Refer to the Freshness checks guide for more info.
The new build_column_schema_change_checks API allows defining asset checks that warn when an asset’s columns have changed since its latest materialization.
In the asset graph UI, the “Upstream data”, “Code version changed”, and “Upstream code version” statuses have been collapsed into a single “Unsynced” status. Clicking on “Unsynced” displays more detailed information.
I/O managers are now optional. This enhances flexibility for scenarios where they are not necessary. For guidance, see When to use I/O managers.
Assets with None or MaterializeResult return type annotations won't use I/O managers; dependencies for these assets can be set using the deps parameter in the @asset decorator.
[dagster-dbt] dbt tests are now pulled in as Dagster asset checks by default.
[dagster-dbt] dbt resource tags are now automatically pulled in as Dagster asset tags.
[dagster-snowflake][dagster-gcp] The dagster-snowflake and dagster-gcp packages now both expose a fetch_last_updated_timestamps API, which makes it straightforward to collect data freshness information in source asset observation functions.
Changes since 1.6.14 (core) / 0.22.14 (libraries)#
Metadata attached during asset or op execution can now be accessed in the I/O manager using OutputContext.output_metadata.
[experimental] Single-run backfills now support batched inserts of asset materialization events. This is a major performance improvement for large single-run backfills that have database writes as a bottleneck. The feature is off by default and can be enabled by setting the DAGSTER_EVENT_BATCH_SIZE environment variable in a code server to an integer (25 recommended, 50 max). It is only currently supported in Dagster Cloud and OSS deployments with a postgres backend.
[ui] The new Asset Details page is now enabled for new users by default. To turn this feature off, you can toggle the feature in the User Settings.
[ui] Queued runs now display a link to view all the potential reasons why a run might remain queued.
[ui] Starting a run status sensor with a stale cursor will now warn you in the UI that it will resume from the point that it was paused.
[asset-checks] Asset checks now support asset names that include ., which can occur when checks are ingested from dbt tests.
[dagster-dbt] The env var DBT_INDIRECT_SELECTION will no longer be set to empty when executing dbt tests as asset checks, unless specific asset checks are excluded. dagster-dbt will no longer explicitly select all dbt tests with the dbt cli, which had caused argument length issues.
[dagster-dbt] Singular tests with a single dependency are now ingested as asset checks.
[dagster-dbt] Singular tests with multiple dependencies must have the primary dependency must be specified using dbt meta.
{{
config(
meta={
'dagster': {
'ref': {
'name': <ref_name>,'package': ...# Optional, if included in the ref.'version': ...# Optional, if included in the ref.
},
}
}
)
}}
...
[dagster-dbt] Column lineage metadata can now be emitted when invoking dbt. See the documentation for details.
[experimental][dagster-embedded-elt] Add the data load tool (dlt) integration for easily building and integration dlt ingestion pipelines with Dagster.
[dagster-dbt][community-contribution] You can now specify a custom schedule name for schedules created with build_schedule_from_dbt_selection. Thanks @dragos-pop!
[helm][community-contribution] You can now specify a custom job namespace for your user code deployments. Thanks @tmatthews0020!
[dagster-polars][community-contribution] Column schema metadata is now integrated using the dagster-specific metadata key in dagster_polars. Thanks @danielgafni!
[dagster-datadog][community-contribution] Added datadog.api module to the DatadogClient resource, enabling direct access to API methods. Thanks @shivgupta!
Fixed a bug where run status sensors configured to monitor a specific job would trigger for jobs with the same name in other code locations.
Fixed a bug where multi-line asset check result descriptions were collapsed into a single line.
Fixed a bug that caused a value to show up under “Target materialization” in the asset check UI even when an asset had had observations but never been materialized.
Changed typehint of metadata argument on multi_asset and AssetSpec to Mapping[str, Any].
[dagster-snowflake-pandas] Fixed a bug introduced in 0.22.4 where column names were not using quote identifiers correctly. Column names will now be quoted.
[dagster-aws] Fixed an issue where a race condition where simultaneously materializing the same asset more than once would sometimes raise an Exception when using the s3_io_manager.
[ui] Fixed a bug where resizable panels could inadvertently be hidden and never recovered, for instance the right panel on the global asset graph.
[ui] Fixed a bug where opening a run with an op selection in the Launchpad could lose the op selection setting for the subsequently launched run. The op selection is now correctly preserved.
[community-contribution] Fixed dagster-polars tests by excluding Decimal types. Thanks @ion-elgreco!
[community-contribution] Fixed a bug where auto-materialize rule evaluation would error on FIPS-compliant machines. Thanks @jlloyd-widen!
[community-contribution] Fixed an issue where an excessive DeprecationWarning was being issued for a ScheduleDefinition passed into the Definitions object. Thanks @2Ryan09!
Creating a run with a custom non-UUID run_id was previously private and only used for testing. It will now raise an exception.
[community-contribution] Previously, calling get_partition_keys_in_range on a MultiPartitionsDefinition would erroneously return partition keys that were within the one-dimensional range of alphabetically-sorted partition keys for the definition. Now, this method returns the cartesian product of partition keys within each dimension’s range. Thanks, @mst!
Added AssetCheckExecutionContext to replace AssetExecutionContext as the type of the context param passed in to @asset_check functions. @asset_check was an experimental decorator.
[experimental] @classmethod decorators have been removed from dagster-embedded-slt.slingDagsterSlingTranslator
[dagster-dbt] @classmethod decorators have been removed from DagsterDbtTranslator.
[dagster-k8s] The default merge behavior when raw kubernetes config is supplied at multiple scopes (for example, at the instance level and for a particluar job) has been changed to be more consistent. Previously, configuration was merged shallowly by default, with fields replacing other fields instead of appending or merging. Now, it is merged deeply by default, with lists appended to each other and dictionaries merged, in order to be more consistent with how kubernetes configuration is combined in all other places. See the docs for more information, including how to restore the previous default merge behavior.
AssetSelection.keys() has been deprecated. Instead, you can now supply asset key arguments to AssetSelection.assets() .
Run tag keys with long lengths and certain characters are now deprecated. For consistency with asset tags, run tags keys are expected to only contain alpha-numeric characters, dashes, underscores, and periods. Run tag keys can also contain a prefix section, separated with a slash. The main section and prefix section of a run tag are limited to 63 characters.
AssetExecutionContext has been simplified. Op-related methods and methods with existing access paths have been marked deprecated. For a full list of deprecated methods see this GitHub Discussion.
The metadata property on InputContext and OutputContext has been deprecated and renamed to definition_metadata .
FreshnessPolicy is now deprecated. For monitoring freshness, use freshness checks instead. If you are using AutoMaterializePolicy.lazy(), FreshnessPolicy is still recommended, and will continue to be supported until an alternative is provided.
The Dagster Cloud agent will now monitor the code servers that it spins to detect whether they have stopped serving requests, and will automatically redeploy the code server if it has stopped responding for an extended period of time.
New additions and bugfixes in Insights:
Added per-metric cost estimation. Estimates can be added via the “Insights settings” button, and will appear in the table and chart for that metric.
Branch deployments are now included in the deployment filter control.
In the Deployments view, fixed deployment links in the data table.