Kafka

Produce data

1. Open the Kafka tool window: View | Tool Windows | Kafka.

2. Select a Kafka connection and click (Create Producer).

   This will open a producer in a new editor tab.

3. In the Topic list, select a topic to write messages to.

4. Under Key and Value, select the message key and value. Use to generate a random value based on the selected type.

   If you are connected to a Schema Registry, you can select Schema Registry to check the sent data against a selected schema.

5. Under Headers, provide any custom headers. If you have them in JSON or CSV format, you can paste them into this section.

6. Under Flow, you can control the record flow:

   • In Records at a time, enter a number if you want to send multiple records simultaneously.

   • Select Generate random keys and Generate random values if you want the record data to be randomly generated.

   • Set the Interval in milliseconds between sending records.

   • Provide Stop Conditions if you want the producer to stop sending messages when either a specified number of records is reached or a specified amount of time has elapsed.

7. Under Options, provide additional options:

   • Partition: specify a topic partition, to which the record must be sent. If not specified, the default logic is used: The producer takes the hash of the key modulo the number of partitions.

   • Compression: select the compression type for data generated by the producer: None, Gzip, Snappy, Lz4, or Zstd.

   • Idempotence: select if you want to ensure that exactly one copy of each message is written in the stream.

   • Acks: select Leader if you want the leader to write the record to its local log and respond without awaiting full acknowledgement from all followers. Select All for the leader to wait for the full set of in-sync replicas to acknowledge the record. Keep None for the producer in order not to wait for any acknowledgment from the server.

8. Click Produce.

You can then click any record in the Data tab to show its details. You can also click to enable statistics.

Consume data

1. Open the Kafka tool window: View | Tool Windows | Kafka.

2. Select a Kafka connection and click (Create Consumer).

   This will open a consumer in a new editor tab.

3. In the Topic list, select a topic to which you want to subscribe.

4. Under Key and Value, select the data types for the keys and values of records that you are going to consume.

5. Use Range and Filters to narrow down the data for consumption:

   • In the Start from list, select a period or offset from which you want to consume data. Select From the beginning to get all records from the topic.

   • In the Limit list, select when to stop receiving data, for example, when a certain number of records is reached in the topic.

   • Use Filter to filter records by substring in their keys, values, or headers.

6. Under Partitions enter a partition ID or a comma-separated list of IDs to get records from specific partitions only.

7. Click Start Consuming.

You can then click any record in the Data tab to show its details. You can also click to enable statistics.

Save a producer or consumer preset

If you often produce or consume data with the same keys, values, headers, or other parameters, you can save them as a preset. You can then reuse presets to quickly create a producer or a consumer.

1. In the Kafka tool window, click (Create Producer) or (Create Consumer).

2. Specify the needed parameters and, on top of the producer or consumer creation form, click (Save Preset).

The parameters are saved as a preset, which is available in the Presets tab. Click a preset to apply it.

Connect to Schema Registry

1. Create connection to a Kafka Broker using cloud providers, custom server, or properties.

2. If you use Confluent, you can paste both Broker and Schema Registry properties into the Configuration field.

   Otherwise, expand the Schema Registry section and select a provider: Confluent or Glue.

   • URL: enter the Schema Registry URL.

   • Configuration source: select the way to provide connection parameters:

     • Custom: select the authentication method and provide credentials.

       If you want to use SSL settings different from those of the Kafka Broker, clear the Use broker SSL settings checkbox and provide the path for the truststore.

     • Properties: paste provided configuration properties. Or you can enter properties manually using code completion and quick documentation that PyCharm provides.

   • Region: select the Schema Registry region.

   • AWS Authentication: select the authentication method:

     • Default credential providers chain: use the credentials from the default provider chain. For more information about the chain, refer to Using the Default Credential Provider Chain.

     • Profile from credentials file: select a profile from your credentials file.

     • Explicit access key and secret key: enter your credentials manually.

   • Registry name: enter the name of a Schema Registry to which you want to connect or click to select it from the list.

3. Once you fill in the settings, click Test connection to ensure that all configuration parameters are correct. Then click OK.

Create schema

1. Open the Kafka tool window: View | Tool Windows | Kafka.

2. Select Schema Registry and click (or press Alt+Insert).

3. In the Format list, select the schema format: Avro, Protobuf, or JSON.

4. In the Strategy list, select the naming strategy and, depending on the selected strategy, set up the name suffix or select a topic. Alternatively, select Custom name and enter any name.

You can preview schemas in a tree and raw view.

Compare schema versions

1. When connected to a Schema Registry, select a schema under Schema Registry.

2. Switch to Raw View and click Compare. The button is available if a schema has more than one version.

Delete a schema version

If a schema has more than one version, you can delete a particular version. Schema Registry supports two types of deletion: soft (when the schema metadata and ID are not removed from the registry after the version deletion) and hard (which removes all metadata, including schema IDs). The ability to choose depends on whether you use Confluent or AWS Glue Schema Registry:

• In Confluent Schema Registry, soft delete is used by default. You can choose to use a hard delete by selecting the Permanent deletion checkbox.

• AWS Glue Schema Registry always uses a hard delete.

1. Under Schema Registry, select a schema.

2. To the right of it, click and select Delete Version.