Big Data Tools

The Big Data Tools is a set of plugins intended for data engineers.

It includes the following plugins:

Spark

Submit and monitor Spark jobs

Flink

Monitor Flink jobs

Kafka

Connect to Kafka brokers and Kafka Schema Registry, produce and consume data

Remote File Systems

Access remote storages

Big Data File Viewer

Preview Parquet, ORC, Avro, and CSV files (the plugin is installed automatically with the Remote File Systems plugin)

Zeppelin

Connect to Zeppelin, run code in notebooks, and preview output (development of the Zeppelin plugin is currently suspended)

Before PyCharm 2023.2, Big Data Tools was a single plugin, and none of its parts could be installed separately. Starting from 2023.2, you can install any of these tools as a separate plugin. You can also install the Big Data Tools plugin, which will automatically install all these six plugins.