Run/Debug Configuration: TestNG

Last modified: 24 August 2023

The TestNG run/debug configuration is used to launch the tests that comply with the TestNG framework. The dialog consists of several tabs.

Common options

Name

Specify a name for the run configuration to quickly identify it among others when editing or running.

Allow multiple instances

Allow running multiple instances of this run configuration in parallel.

By default, it is disabled, and when you start this configuration while another instance is still running, IntelliJ IDEA suggests stopping the running instance and starting another one. This is helpful when a run configuration consumes a lot of resources and there is no good reason to run multiple instances.

Store as project file

Save the file with the run configuration settings to share it with other team members. The default location is .idea/runConfigurations. However, if you do not want to share the .idea directory, you can save the configuration to any other directory within the project.

By default, it is disabled, and IntelliJ IDEA stores run configuration settings in .idea/workspace.xml.

Configuration tab

The composition of this tab depends on the selected testing scope: package, project, and so on.

Item

Description

Test kind

  • All in package – run all tests in a package. In the Package field, specify the fully qualified name of the package.

  • Pattern – run the tests that conform to the specified pattern. In the Pattern field, enter one or more TestNG test classes.

  • Group – run a test group. In the Group field, specify the group you want to run.

  • Suite – run a test suite. In the Suite field, enter the corresponding testng.xml file.

    • In whole project – the IDE will look for the tests in all the modules

    • In single module – the IDE will look for the tests only in the module that is selected in the Use classpath of module field.

    • Across module dependencies – the same as the previous option plus the modules that depend on that module.

  • Class – run a test class. In the Class field, enter the fully qualified name of the test class you want to run.

  • Method – run a test method.

Output directory

The directory in which test reports will be generated.

JDK Settings

Item

Description

VM options

Specify the options to be passed to the Java virtual machine when launching the application, for example, -mx, -verbose, and so on.

When specifying JVM options, follow these rules:

  • Use spaces to separate individual options.

  • If the value of an option includes spaces, enclose either the value or the actual spaces with double quotes.

  • If an option includes double quotes as part of the value, escape the double quotes using backslashes.

  • You can pass environment variable values to custom Java properties.

-Xmx1024m -Dspaces="some arg" -Dmy.prop=\"quoted_value\" -Dfoo=${MY_ENV_VAR}

Use code completion in this field: start typing the name of a flag, and the IDE suggests a list of available command line options. This works for -XX: and -X options and some standard options that are not configured by IntelliJ IDEA automatically, like -ea, but not for -cp or –release.

The -classpath option specified in this field overrides the classpath of the module.

Test runner parameters

Arguments to be passed to the test runner.

Use the same rules as for specifying the VM options.

Working directory

Specify the working directory to be used for running the application. This directory is the starting point for all relative input and output paths. By default, the working directory is the project root.

Environment variables

Click the Browse button to open the Environment Variables dialog where you can create variables and specify their values.

Use module path

This checkbox is available for projects with JPMS modules (Java 9 and later). It allows you to select how to run your tests: on a class or on a module path.

JRE

The JRE to be used.

Shorten command line

Select a method that will be used to shorten the command line if the classpath gets too long, or you have many VM arguments that exceed your OS command line length limitation. The choice of option depends on the class loader implementation. Note that some frameworks do not support JAR manifest, while other frameworks with custom class loaders will not work well with the classpath.file option.

  • none: IntelliJ IDEA will not shorten a long classpath. If the command line exceeds the OS limitation, IntelliJ IDEA will be unable to run your application and will display a message suggesting you to specify the shortening method.

  • JAR manifest: IntelliJ IDEA will pass a long classpath via a temporary classpath.jar. The original classpath is defined in the manifest file as a class-path attribute in classpath.jar. Note that you will be able to preview the full command line if it was shortened using this method, not just the classpath of the temporary classpath.jar.

  • classpath.file: IntelliJ IDEA will write a long classpath into a text file.

  • @argFiles (java 9+): the command-line length limitations will be overcome using the @argument files option.

note

This setting is shared if you select to share your run/debug configuration, so the same method will be applied for your team members irrespective of their operating system.

Parameters

Item

Description

Parameters

Properties file

Specify the .properties file to be passed to TestNG.

Name - Value

Additional parameters as key - value pairs.

Listeners

App general add  App general remove

Use these icons to make up a list of listeners.

Code Coverage tab

Use this tab to configure code coverage monitoring options.

Item

Description

Choose coverage runner

Select the desired code coverage runner.

By default, IntelliJ IDEA uses its own coverage engine with the Sampling mode. You can also choose JaCoCo for calculating coverage.

Sampling

Select this option to measure code coverage with minimal slow-down.

Tracing

Select this option to collect accurate branch coverage. This mode is available for the IntelliJ IDEA code coverage runner only.

Track per test coverage

Select this checkbox to detect lines covered by one test and all tests covering line. If this checkbox is selected, Track per test coverage becomes available on the toolbar of the coverage statistic popup.

note

This option is only available for the Tracing mode of code coverage measurement for the testing run/debug configurations.

For more information, refer to Set coverage in run configurations.

Packages and classes to record code coverage data

Click the Add button and select the Add Class button Add Class or the Add Package button Add Package to specify classes and packages to be measured. You can also remove classes and packages from the list by selecting them in the list and clicking the remove the package button.

Enable coverage in test folders

If this checkbox is selected, the folders marked as test root_Test.png are included in the code coverage analysis.

Logs

The following options are related to logging the execution of this configuration. For more information, refer to View logs.

Item

Description

Specify logs to be shown in console

Specify which log files to display while running the application.

Click the Add button to add a new log. In the Edit Log Files Aliases dialog, configure the following:

  • Alias: The name of the tab where the log will be displayed.

  • Log File Location: Specify the path to the log file or an Ant pattern. If several files of a rolling log match the pattern, IntelliJ IDEA will display the most recent one.

  • Show all files coverable by pattern: Show all logs that match the pattern.

For logs in the table, you can configure the following options:

  • Is Active: Display the specified log file.

  • Skip Content: Do not display old log messages from previous runs.

Save console output to file

Save the console output to the specified location. Type the path manually or click the browse button and point to the desired location in the dialog that opens.

Show console when a message is printed to stdout

Activate the console when the application writes to the standard output stream.

Show console when a message is printed to stderr

Activate the console when the application writes to the standard error stream.

Before Launch

In this area, you can specify tasks to be performed before starting the selected run/debug configuration. The tasks are performed in the order they appear in the list.

Item

Description

Add before launch task

Enable this option to add one of the following available tasks:

  • Launch Web Browser: select this option to have a browser started. In the dialog that opens, select the type of the browser and provide the start URL. Also, specify if you want the browser be launched with JavaScript debugger.

  • Run External tool: run an external application. In the dialog that opens, select one or multiple applications you want to run. If it is not defined in IntelliJ IDEA yet, add its definition. For more information, refer to External tools.

  • Run Another Configuration: select to execute another run/debug configuration and wait until it finishes before starting the current configuration. If you want to run several configurations in parallel, use a compound run/debug configuration.

  • Run Remote External Tool: add a remote SSH external tool.

  • Run Gradle task: run a Gradle task. In the dialog that opens, specify the task and provide additional configuration if necessary.

  • Build: select to compile the specified module. The Build Module action will be executed.

    If an error occurs during compilation, IntelliJ IDEA won't attempt to start the run/debug configuration.

  • Build Project: select to compile the entire project. The Build Project action will be executed.

    If an error occurs during compilation, IntelliJ IDEA won't attempt to start the run/debug configuration.

  • Build, no error check: the same as the Build option, but IntelliJ IDEA will try to start the run/debug configuration irrespective of the compilation results.

  • Build Artifacts: select this option to build an artifact or artifacts. In the dialog that opens, select the artifact or artifacts that should be built.

  • Run Ant target: select this option to run an Ant target. In the dialog that opens, select the target to be run.

  • Run Maven Goal: select this option to run a Maven goal. In the dialog that opens, select the goal to be run.

  • Run Grunt task: select this option to run a Grunt task.

    In the Grunt task dialog that opens, specify the Gruntfile.js where the required task is defined, select the task to execute, and specify the arguments to pass to the Grunt tool.

    Specify the location of the Node.js interpreter, the parameters to pass to it, and the path to the grunt-cli package.

  • Run gulp task: select this option to run a Gulp task.

    In the Gulp task dialog that opens, specify the Gulpfile.js where the required task is defined, select the task to execute, and specify the arguments to pass to the Gulp tool.

    Specify the location of the Node.js interpreter, the parameters to pass to it, and the path to the gulp package.

  • Run npm script: select this option to execute an npm script.

    In the NPM Script dialog that opens, specify the npm run/debug configuration settings.

  • Compile TypeScript: select to run the built-in TypeScript compiler and thus make sure that all the changes you made to your TypeScript code are reflected in the generated JavaScript files. In the TypeScript Compile Settings dialog that opens, select or clear the Check errors checkbox to configure the behaviour of the compiler in case any errors are detected:

    • If the Check errors checkbox is selected, the compiler will show all the errors and the run configuration will not start.

    • If the Check errors checkbox is cleared, the compiler will show all the detected errors but the run configuration still will be launched.

  • Disconnect Data Source: select this option if you want to disrupt the connection to a data source before the run/debug configuration is run.

Open run/debug tool window when started

Depending on the type of configuration, open the Run, Debug, or Services tool window when you start this run configuration. If this option is disabled, you can open the tool window manually:

  • View | Tool Windows | Run or

  • View | Tool Windows | Debug or

  • View | Tool Windows | Services or

Show the run/debug configuration settings before start

Show the run configuration settings before actually starting it.

See also

Reference

Concepts

Language and Framework-Specific Guidelines