1. All Inspections
  2. Spark
  3. Unresolved columns

Unresolved columns

Last modified: 03 December 2024

Reports attempts to access PySpark DataFrame columns that do not exist in the schema.

Example:

df = spark.createDataFrame(data=[(“John”, 10),
                                 (“Anakin”, 20),
                                 (“Luke”, 30)],
                           schema=[“name”, “age”])
df.select(“unknown_column”)

Locating this inspection

By ID

Can be used to locate inspection in e.g. Qodana configuration files, where you can quickly enable or disable it, or adjust its settings.

PySparkColumnCheck
Via Settings dialog

Path to the inspection settings via IntelliJ Platform IDE Settings dialog, when you need to adjust inspection settings directly from your IDE.

Settings or Preferences | Editor | Inspections | Spark

Inspection Details

By default bundled with:

IntelliJ IDEA 2024.3, Qodana for JVM 2024.3,

Can be installed with plugin:

Spark, 243.23126