Palantir is upgrading the Apache Spark version in Foundry transforms from 3.5.5 to 4.0.1 for all enrollments. The change is set out in its September 2026 announcements, dated September 22, 2026. Palantir says no action is required to enable the upgrade.

What changed

Palantir says transforms runtime versions "are managed automatically in the background through adjudication." It says the upgrade brings "many bug fixes, performance improvements, and new features," but gives no figures. Nor does it put a number on compute cost.

The entry says the version "is being upgraded" and gives no completion date or schedule. Palantir's October and November announcements carry no follow-up on the rollout.

Who it is for

The upgrade covers every team that runs transforms. Palantir flags one group in particular: "Workflows that use custom dependencies may still require updates to ensure those dependencies are compatible with Spark 4."

How it works

Palantir says Spark 4 introduces breaking changes and that "Foundry mitigates these changes in the transforms API layer." It says it tested the upgrade "to preserve existing workflow behavior." The announcement does not list which changes are mitigated, and it names no known regressions.

Palantir also warns that "the performance of existing pipelines may be affected and new bugs may occur." For high-risk jobs, it recommends "a temporary pin to control the new Spark version rollout" until the upgrade is confirmed issue-free. The announcement does not describe how the pin is set or how long it lasts.

Apache's Spark SQL migration guide says that since Spark 4.0, "spark.sql.ansi.enabled is on by default." Apache's PySpark upgrade guide says pandas API on Spark "will raise an exception" when ANSI mode is enabled. The guide also raises the minimum pandas version from 1.0.5 to 2.0.0 and the minimum PyArrow version from 4.0.0 to 11.0.0.

Palantir does not say whether these Spark-level changes apply in Foundry transforms. It also does not break out effects on Python, Java or SQL transforms.

Pipeline Builder has defaulted to "faster" pipelines since a November 3 change. Palantir's November announcements say these are powered by DataFusion rather than Spark. Palantir does not say how the upgrade affects Pipeline Builder pipelines that still run on Spark.

Why it matters

In Foundry News's view, Palantir's caveats point pipeline owners to where to look first. Those are jobs with custom dependencies, and jobs whose results depend on behavior that Apache's guides say changed in Spark 4.0. Our reading is that custom Python environments and pandas-on-Spark code fall into that group. Palantir's suggested path for such jobs is to pin them, test them, and lift the pin once they run cleanly.

Teams using Ontology SQL, which runs on a subset of the Spark SQL dialect, may also follow Palantir's planned Spark SQL work. Palantir does not link that work to this upgrade.

Three things are worth watching: a list of the changes Palantir mitigates, documentation on how the pin works, and confirmation that the rollout has finished.