Unresolved columns

Reports attempts to access PySpark DataFrame columns that do not exist in the schema.

Example:

df = spark.createDataFrame(data=[(“John”, 10),
                                 (“Anakin”, 20),
                                 (“Luke”, 30)],
                           schema=[“name”, “age”])
df.select(“unknown_column”)

Locating this inspection

By ID

Can be used to locate inspection in e.g. Qodana configuration files, where you can quickly enable or disable it, or adjust its settings.

PySparkColumnCheck

Via Settings dialog

Path to the inspection settings via IntelliJ Platform IDE Settings dialog, when you need to adjust inspection settings directly from your IDE.

Settings or Preferences | Editor | Inspections | Spark

Inspection Details
By default bundled with:	IntelliJ IDEA 2024.3, Qodana for JVM 2024.3,
Can be installed with plugin:	Spark, 243.23126

Last modified: 03 December 2024