<!---
  Licensed to the Apache Software Foundation (ASF) under one
  or more contributor license agreements.  See the NOTICE file
  distributed with this work for additional information
  regarding copyright ownership.  The ASF licenses this file
  to you under the Apache License, Version 2.0 (the
  "License"); you may not use this file except in compliance
  with the License.  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing,
  software distributed under the License is distributed on an
  "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
  KIND, either express or implied.  See the License for the
  specific language governing permissions and limitations
  under the License.
-->

# Upgrade Guides

## DataFusion 55.0.0

**Note:** DataFusion `55.0.0` has not been released yet. The information provided
in this section pertains to features and changes that have already been merged
to the main branch and are awaiting release in this version.

### `DataFrame::fill_null` now borrows its arguments

`DataFrame::fill_null` previously took its arguments by value:

```rust,ignore
// Before
pub fn fill_null(
    &self,
    value: ScalarValue,
    columns: Vec<String>,
) -> Result<DataFrame>
```

It now borrows them, matching the signature of the newly added
`DataFrame::fill_nan`:

```rust,ignore
// After
pub fn fill_null(
    &self,
    value: &ScalarValue,
    columns: &[&str],
) -> Result<DataFrame>
```

This lets callers pass a borrowed `ScalarValue` and slice literals (or
`&str` column names) without first allocating owned `String`s.

**Migration guide:**

Borrow the value and pass a slice of `&str` instead of an owned `Vec<String>`:

```rust,ignore
// Before
let df = df.fill_null(ScalarValue::from(0), vec!["a".to_owned(), "c".to_owned()])?;
let df = df.fill_null(ScalarValue::from(0), vec![])?;

// After
let df = df.fill_null(&ScalarValue::from(0), &["a", "c"])?;
let df = df.fill_null(&ScalarValue::from(0), &[])?;
```

### `FileScanConfig::partitioned_by_file_group` removed

`FileScanConfig::partitioned_by_file_group` and
`FileScanConfigBuilder::with_partitioned_by_file_group(...)` have been removed.
Use `FileScanConfig::output_partitioning` and
`FileScanConfigBuilder::with_output_partitioning(...)` instead.

**Who is affected:**

- Users who accessed `FileScanConfig::partitioned_by_file_group` directly.
- Users who called
  `FileScanConfigBuilder::with_partitioned_by_file_group(true)`.

**Migration guide:**

If your file groups are organized by table partition column values, declare hash
output partitioning over those partition columns:

```rust,ignore
use datafusion_datasource::file_scan_config::{
    FileScanConfigBuilder, output_partitioning_from_partition_fields,
};

let output_partitioning = output_partitioning_from_partition_fields(
    source.table_schema().table_schema(),
    source.table_schema().table_partition_cols(),
    file_groups.len(),
);

let config = FileScanConfigBuilder::new(object_store_url, source)
    .with_file_groups(file_groups)
    .with_output_partitioning(output_partitioning)
    .build();
```

`output_partitioning_from_partition_fields` returns
`Some(Partitioning::Hash(...))` when partition columns are present and `None`
otherwise. If you construct the partitioning manually, pass
`Some(Partitioning::Hash(partition_exprs, partition_count))` to
`with_output_partitioning(...)`.

### User `SpillFile` traits instead of [`RefCountedTempFile`]

Spill file APIs now use the `datafusion_execution::SpillFile` trait instead of
the concrete [`RefCountedTempFile`] type. [`DiskManager::create_tmp_file`] now
returns `Arc<dyn SpillFile>`.
This change was introduced in [PR #21882], which adds pluggable spill file
backends via `SpillFile` and `TempFileFactory`.

If your code matched on [`DiskManagerMode`], add a `DiskManagerMode::Custom(_)`
arm.

If your code wrote directly to a [`RefCountedTempFile`] or called
[`RefCountedTempFile::update_disk_usage`], open a spill writer instead:

```diff
- temp_file.inner().as_file().write_all(bytes)?;
- temp_file.update_disk_usage()?;
+ temp_file.open_writer()?.write_all(bytes)?;
```

Use `temp_file.size()` instead of [`RefCountedTempFile::current_disk_usage`].

[`diskmanager::create_tmp_file`]: https://docs.rs/datafusion-execution/latest/datafusion_execution/disk_manager/struct.DiskManager.html#method.create_tmp_file
[`diskmanagermode`]: https://docs.rs/datafusion-execution/latest/datafusion_execution/disk_manager/enum.DiskManagerMode.html
[`pr #21882`]: https://github.com/apache/datafusion/pull/21882
[`refcountedtempfile`]: https://docs.rs/datafusion-execution/latest/datafusion_execution/disk_manager/struct.RefCountedTempFile.html
[`refcountedtempfile::current_disk_usage`]: https://docs.rs/datafusion-execution/latest/datafusion_execution/disk_manager/struct.RefCountedTempFile.html#method.current_disk_usage
[`refcountedtempfile::update_disk_usage`]: https://docs.rs/datafusion-execution/latest/datafusion_execution/disk_manager/struct.RefCountedTempFile.html#method.update_disk_usage

### `Dialect::AVAILABLE` replaced by `Dialect::available()`

`datafusion_common::config::Dialect::AVAILABLE` has been removed. Use
`Dialect::available()` instead.

### `spill_record_batch_by_size` removed

`datafusion_physical_plan::spill::spill_record_batch_by_size` has been removed.
This function was deprecated in DataFusion `46.0.0`.

Use `datafusion_physical_plan::spill::SpillManager::spill_record_batch_by_size`
instead.

### Decimal scalar formatting uses human-readable values

Decimal scalar literals in `EXPLAIN` output, expression display strings, and
auto-generated column names now format the decimal value using its scale while
still showing the precision and scale. For example, a `Decimal128` literal with
stored value `1`, precision `1`, and scale `1` is now rendered as
`Decimal128(0.1,1,1)` instead of `Decimal128(Some(1),1,1)`. When formatting a
`ScalarValue` directly, it now appears as `0.1` instead of `Some(1),1,1`.

`NULL` decimal literals were previously shown as `Decimal128(None,10,2)`; they
will now appear as `Decimal128(NULL,10,2)`.

Query result values already used human-readable decimal formatting and are
unchanged.

### `Coercion` supports dictionary encoding preservation

`datafusion_expr_common::signature::Coercion` now supports optional dictionary
encoding preservation. When enabled for `TypeSignatureClass::Native(...)`
coercions, DataFusion coerces dictionary inputs to
`Dictionary(original_key_type, coerced_value_type)` instead of materializing them
to the coerced value type.

User-defined functions can opt in by setting dictionary encoding preservation on
the relevant coercion:

```rust
Coercion::new_exact(TypeSignatureClass::Native(logical_string()))
    .with_encoding_preservation(EncodingPreservation::dictionary())
```

This changes the coerced argument type passed to the function. If a function
derives its return type from that coerced argument type, code that checks exact
result types may need to update its expectations or add an explicit cast to
materialize the result.

### `GroupsAccumulator::merge_batch` no longer takes `opt_filter`

The `opt_filter` argument has been removed from
`datafusion_expr_common::groups_accumulator::GroupsAccumulator::merge_batch`:

```diff
  fn merge_batch(
      &mut self,
      values: &[ArrayRef],
      group_indices: &[usize],
-     opt_filter: Option<&BooleanArray>,
      total_num_groups: usize,
  ) -> Result<()>;
```

Aggregate `FILTER` clauses only apply to raw input rows during the partial
(update) phase, so by the time intermediate states are merged there is nothing
left to filter per row. In practice `opt_filter` was always `None` here, so
removing it makes the API self-explanatory and impossible to misuse.

**Who is affected:**

- Anyone with a custom `GroupsAccumulator` implementation.
- Anyone calling `merge_batch` directly.

**Migration guide:**

Drop the `opt_filter` argument from your `merge_batch` signature and from any
call sites:

```diff
  fn merge_batch(
      &mut self,
      values: &[ArrayRef],
      group_indices: &[usize],
-     opt_filter: Option<&BooleanArray>,
      total_num_groups: usize,
  ) -> Result<()> {
      // ...
  }
```

```diff
- acc.merge_batch(values, group_indices, None, total_num_groups)?;
+ acc.merge_batch(values, group_indices, total_num_groups)?;
```

If your implementation previously inspected `opt_filter` (for example asserting
it was `None`), that code can simply be deleted.

See [issue #22775](https://github.com/apache/datafusion/issues/22775) for details.

### `is_dynamic_physical_expr` is deprecated

`datafusion_physical_expr_common::physical_expr::is_dynamic_physical_expr` is
deprecated. It was a thin wrapper over `snapshot_generation(expr) != 0` used to
ask "does this predicate contain a dynamic filter?".

Prefer asking the question directly against the concrete type. For a one-off
check, downcast to `DynamicFilterPhysicalExpr`:

```rust
use datafusion_physical_expr::expressions::DynamicFilterPhysicalExpr;
use datafusion_common::tree_node::{TreeNode, TreeNodeRecursion};

let mut is_dynamic = false;
predicate.apply(|e| {
    if e.downcast_ref::<DynamicFilterPhysicalExpr>().is_some() {
        is_dynamic = true;
        Ok(TreeNodeRecursion::Stop)
    } else {
        Ok(TreeNodeRecursion::Continue)
    }
})?;
```

If you also need to know whether the dynamic filters can still change (and to be
notified when they do), use the new `DynamicFilterTracking` /
`DynamicFilterTracker` API in `datafusion_physical_expr`:

```rust
use datafusion_physical_expr::DynamicFilterTracking;

let tracking = DynamicFilterTracking::classify(&predicate);
if tracking.contains_dynamic_filter() {
    // worth re-evaluating the predicate at runtime
}
```

### `FilePruner::try_new` no longer builds a pruner for static predicates without statistics

`datafusion_pruning::FilePruner::try_new` now returns `None` when the predicate
is purely static _and_ the file carries no usable column statistics, because
such a pruner can never prune anything beyond what planning already did.
Previously it returned `Some` whenever a statistics struct was present (the
"is this worth pruning?" decision lived in the Parquet opener). Files with column
statistics, and predicates that carry a dynamic filter, are unaffected.

### `QueryPlanner` adds `Any` as a supertrait

To enable downcasting of `dyn QueryPlanner` to concrete query planner types (via
`is::<T>()` / `downcast_ref::<T>()`), the `QueryPlanner` trait now has `Any`
as a supertrait:

```diff
- pub trait QueryPlanner: Debug
+ pub trait QueryPlanner: Any + Debug
```

### `ExecutionPlan::partition_statistics` deprecated in favor of `statistics_with_args`

`ExecutionPlan::partition_statistics` is deprecated. A new method
`statistics_with_args` accepts a `StatisticsArgs` parameter that carries
the partition index and a shared cache for memoized child statistics lookups.

Existing implementations of `partition_statistics` continue to work unchanged.
The default `statistics_with_args` delegates to the deprecated method, so no
migration is required until the deprecated method is removed.

> **Warning:** The delegation is **one-way**: the default `statistics_with_args`
> calls `partition_statistics`, but the default `partition_statistics` does
> **not** call `statistics_with_args` — it returns `Statistics::new_unknown`.
> Nodes that override only `statistics_with_args` will silently return
> `Statistics::new_unknown` to any caller still using the deprecated
> `partition_statistics`.

**Who is affected:**

- Users who implement custom `ExecutionPlan` nodes (recommended to migrate)
- Users who call `partition_statistics` directly (recommended to switch to `statistics_with_args`)

**Migration guide:**

For **implementations**, override `statistics_with_args` instead of
`partition_statistics`. Leaf nodes that do not have children can ignore
the args.

Child statistics are looked up via `args.compute_child_statistics(child, partition)`.
Use `args.partition()` for partition-preserving operators, or `None` for
partition-merging operators that always need overall stats:

```rust,ignore
// Before:
fn partition_statistics(&self, partition: Option<usize>) -> Result<Arc<Statistics>> {
    let child_stats = self.input.partition_statistics(partition)?;
    // ... transform child_stats ...
}

// After (partition-preserving):
fn statistics_with_args(
    &self,
    args: &StatisticsArgs,
) -> Result<Arc<Statistics>> {
    let child_stats = args.compute_child_statistics(&self.input, args.partition())?;
    // ... transform child_stats ...
}

// After (partition-merging):
fn statistics_with_args(
    &self,
    args: &StatisticsArgs,
) -> Result<Arc<Statistics>> {
    let child_stats = args.compute_child_statistics(&self.input, None)?;
    // ... transform child_stats ...
}
```

For **callers**, create a `StatisticsArgs` and call `statistics_with_args`
directly. The cache is created automatically:

```rust,ignore
use datafusion_physical_plan::StatisticsArgs;

// Before:
let stats = plan.partition_statistics(None)?;

// After:
let stats = plan.statistics_with_args(&StatisticsArgs::new())?;
```

### `DdlStatement::CreateExternalTable` and `CreateFunction` are now boxed

The two largest variants of `datafusion_expr::DdlStatement` are now
`Box`ed:

```rust,ignore
// Before
pub enum DdlStatement {
    CreateExternalTable(CreateExternalTable),
    // ...
    CreateFunction(CreateFunction),
    // ...
}

// After
pub enum DdlStatement {
    CreateExternalTable(Box<CreateExternalTable>),
    // ...
    CreateFunction(Box<CreateFunction>),
    // ...
}
```

`CreateExternalTable` is 312 bytes and `CreateFunction` is 288 bytes, so
without boxing they forced the entire `LogicalPlan` enum to 320 bytes
even on SELECT-only query paths that never instantiate them. Boxing
shrinks `LogicalPlan` from 320 → 176 bytes (−45%), making every
`mem::take` / `mem::swap` / `Arc<LogicalPlan>` store on the planning
hot path move a smaller payload.

**Who is affected:**

- Users who construct `DdlStatement::CreateExternalTable(...)` or
  `DdlStatement::CreateFunction(...)` from an owned struct.
- Users who pattern-match these variants and destructure the inner
  struct in the same pattern (e.g.
  `DdlStatement::CreateExternalTable(CreateExternalTable { name, .. })`).
- Code that consumes the inner struct out of these variants (e.g. to
  pass `CreateExternalTable` by value to another function).

**Migration guide:**

When constructing the variants, wrap the inner struct in `Box::new`:

```rust,ignore
// Before
let stmt = DdlStatement::CreateFunction(CreateFunction { name, args, .. });

// After
let stmt = DdlStatement::CreateFunction(Box::new(CreateFunction {
    name,
    args,
    ..
}));
```

When pattern-matching, bind the boxed value and either access fields
through it (Rust auto-derefs the `Box`) or destructure via `.as_ref()`:

```rust,ignore
// Before
match ddl {
    DdlStatement::CreateExternalTable(CreateExternalTable {
        name, location, ..
    }) => { /* use name, location */ }
}

// After — access fields through the box
match ddl {
    DdlStatement::CreateExternalTable(ce) => {
        let name = &ce.name;
        let location = &ce.location;
        /* ... */
    }
}

// After — destructure the dereferenced struct
match ddl {
    DdlStatement::CreateExternalTable(ce) => {
        let CreateExternalTable { name, location, .. } = ce.as_ref();
        /* ... */
    }
}
```

When you need an owned `CreateExternalTable` / `CreateFunction` out of
the variant, dereference the box with `*`:

```rust,ignore
// Before
match plan {
    LogicalPlan::Ddl(DdlStatement::CreateExternalTable(cmd)) => Ok(cmd),
    _ => { /* ... */ }
}

// After
match plan {
    LogicalPlan::Ddl(DdlStatement::CreateExternalTable(cmd)) => Ok(*cmd),
    _ => { /* ... */ }
}
```

See [PR #22733](https://github.com/apache/datafusion/pull/22733) for
details, including the per-variant size breakdown and benchmark
results.

### `ListingOptions::target_partitions` and `collect_stat` removed

The `target_partitions` and `collect_stat` fields on
`datafusion_catalog_listing::ListingOptions`, their builder methods
(`with_target_partitions`, `with_collect_stat`), and the
`with_session_config_options` helper have been removed.

`ListingTable` now reads both values directly from the active `SessionConfig`
at scan time instead of from a copy snapshotted onto the table at construction
time.

**Who is affected:**

- Code that set `target_partitions` / `collect_stat` per table via
  `ListingOptions`, or read those public fields.
- Code that relied on a `ListingTable` freezing these values at construction
  time independently of the session config. The table now always reflects the
  current `SessionConfig`.

**Migration guide:**

Configure these on the `SessionConfig` instead:

```rust,ignore
// Before
let options = ListingOptions::new(format)
    .with_target_partitions(8)
    .with_collect_stat(true);

// After
let config = SessionConfig::new()
    .with_target_partitions(8)
    .with_collect_statistics(true);
```

See [PR #22969](https://github.com/apache/datafusion/pull/22969) for details.

### Spark map functions now reject duplicate keys by default

The Spark-compatibility map-construction functions (`map_from_arrays`,
`map_from_entries`, `str_to_map`) now raise `[DUPLICATED_MAP_KEY]` at runtime
when constructing a map that contains duplicate keys. This matches the default
of Spark's [`spark.sql.mapKeyDedupPolicy`](https://github.com/apache/spark/blob/v4.0.0/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala#L4502-L4511).

A new config option, `datafusion.spark.map_key_dedup_policy`, controls the
behavior:

- `EXCEPTION` (default): raise on any duplicate key.
- `LAST_WIN`: keep the last occurrence of each duplicate key. The key stays at
  its first-seen position with the value from its last occurrence (matching
  Spark's `ArrayBasedMapBuilder`).

**Who is affected:**

- Queries calling `map_from_arrays` or `str_to_map` on data that contains
  duplicate keys. Previously these functions either tolerated duplicates
  silently or raised a non-configurable error.

**Migration guide:**

To restore lenient duplicate-key handling, set the policy to `LAST_WIN`:

```sql
SET datafusion.spark.map_key_dedup_policy = 'LAST_WIN';
```

See [PR #21720](https://github.com/apache/datafusion/pull/21720) for details.

### Unify LRU memory-limiting caches into one generic cache

The caches `DefaultFileMetadataCache`, `DefaultListFilesCache` and `DefaultFileStatisticsCache`
are merged into one generic implementation `DefaultCache`. The corresponding traits are now
type aliases:

```diff
- pub trait FileStatisticsCache: CacheAccessor<TableScopedPath, CachedFileMetadata>
- pub trait ListFilesCache: CacheAccessor<TableScopedPath, CachedFileList>
- pub trait FileMetadataCache: CacheAccessor<Path, CachedFileMetadataEntry>
+ pub type FileStatisticsCache = dyn Cache<TableScopedPath, CachedFileMetadata>;
+ pub type ListFilesCache = dyn Cache<TableScopedPath, CachedFileList>;
+ pub type FileMetadataCache = dyn Cache<Path, CachedFileMetadataEntry>;
```

**Who is affected:**

- Users who introduced their own implementation of `FileMetadataCache`, `ListFilesCache` or `FileStatisticsCache`.

**Migration guide:**

Implement the newly introduced types for your custom cache implementation.

See [PR #22613](https://github.com/apache/datafusion/pull/22613) for details.

### `CachedFileMetadata` now validates file schema

The file-statistics cache remains keyed by `TableScopedPath`, but
`CachedFileMetadata` now stores a `SchemaFingerprint` of the `file_schema` used
to compute the cached statistics. Cache hits are valid only when both the file
metadata and schema fingerprint match.

**Who is affected:**

- Users constructing `CachedFileMetadata` values directly.

**Migration guide:**

- Pass `Arc::new(SchemaFingerprint::from_schema(file_schema))` to
  `CachedFileMetadata::new`.
- Pass the current schema fingerprint to `CachedFileMetadata::is_valid_for`.

See [PR #23201](https://github.com/apache/datafusion/pull/23201) for details.