Skip to content

Latest commit

 

History

History
321 lines (228 loc) · 20 KB

CONTRIBUTING.md

File metadata and controls

321 lines (228 loc) · 20 KB
Raw

TestSpark

Table of contents

Description

In this document you can find the overall structure of TestSpark plugin. The classes are listed and their purpose is described. This section is intended for developers and contributors to TestSpark plugin.

Build project

Run gradle buildPlugin.

Including Test generation using Grazie (for JetBrains employees only!)

to include test generation using Grazie in the build process, you need to pass Space username and token as properties: gradle buildPlugin -Dspace.username=<USERNAME> -Dspace.pass=<TOKEN>.

<TOKEN> is generated by Space, which has access to Automatically generating unit tests maven packages.

Run IDE for UI tests

In IDEA, run the Run IDE for UI Tests configuration. Alternatively, use the following command: gradle runIdeForUiTests.

Including Test generation using Grazie (for JetBrains employees only!)

to include test generation using Grazie in the runIdeForUiTests process, you need to pass Space username and token as properties: gradle runIdeForUiTests -Dspace.username=<USERNAME> -Dspace.pass=<TOKEN>.

<TOKEN> is generated by Space, which has access to Automatically generating unit tests maven packages.

Language Support Documentation

The TestSpark plugin supports automatic test generation for various programming languages (currently Java and Kotlin) and aims to support even more programming languages in the future.

This document provides an overview of the existing implementation of Kotlin and Java support and guidelines for adding more programming languages.

How can I add support for a new programming language? In brief, you need to extend all the necessary interfaces with implementations specific to the new language. Below, you will find a detailed guide divided into six key components of the entire pipeline with the most important interfaces addressing this goal.

Key Components

1. PSI Parsers

The first step is to enable the collection of the appropriate information for the code under test. This part is responsible for working with the PSI (Program Structure Interface) generated by IntelliJ IDEA. It helps parse the part where the cursor is located, provides a choice of the code elements that are available for testing at cursor's position. Then find all the needed dependencies to make the prompt complete with all the necessary knowledge about the code under test.

This part is the most granular but complex at the same time.

The main reason for this is to include dependencies only for the languages we need. This avoids errors if the user does not have some languages that our plugin supports. For example, if we work with a Python project, we don't want to depend on Kotlin because it will cause an error if Kotlin isn't present.

Additionally, we want to incrementally add dependencies on other languages for faster startup. For example, we do not want to fetch the dependency on Java when we work with TypeScript. Other benefits include better organization, easier maintenance, and clearer separation of concerns. As a side-bonus, the addition of new languages will be easier.

Module Dependencies:

  • langwrappers: This is a foundational module for language extensions.
  • : Depends on the langwrappers module to implement the <Language>-specific PsiHelper and PsiHelperProvider.
  • src/: Depends on langwrappers because we want to use PsiHelper and other interfaces regardless of the current language. Depends on <Language>, to make plugin.xml aware of the implementations of the Extension Point.

Plugin Dependencies:

  • The main plugin.xml file declares the psiHelperProvider extension point using the com.intellij.lang.LanguageExtensionPoint class.
  • The language-specific modules extend this extension point to register their implementations.
  • When the project is opened, we load the EPs needed to work with the current project. Then, using the PsiHelperProvider interface, we can get the appropriate <language>PsiHelper class per file.

Implementation Details:

  • Common Module (langwrappers):

    • Contains the PsiHelper interface, which provides the necessary methods to interact with psiFile.
    • The PsiHelperProvider class includes a companion object to fetch the appropriate PsiHelper implementation based on the file's language.

  • Module:

    • Implements the <Language>PsiHelper and <Language>PsiHelperProvider classes, which provide -specific logic.
    • Declares the extension point in testspark-<Language>.xml.

To add new languages, create a separate module for this language and register its implementation as an extension of the psiHelperProvider EP. Then follow the template provided above.

2. Prompt Generation

When we know how to parse the code, we need to construct the prompt.

For each language, adjust the prompt that goes to the LLM. Ensure that the language, framework platform, and mocking framework are defined correctly in:

data class PromptConfiguration(
    val desiredLanguage: String,
    val desiredTestingPlatform: String,
    val desiredMockingFramework: String,
)

Additionally, check that all the dependencies (collected by PsiHelper for the current strategy) are passed properly. PromptGenerator and PromptBuilder are responsible for this job.

3. Parsing LLM Response

When the LLM response to our prompt is received, we have to parse it.

We want to retrieve test case, all the test functions and additional information like imports or supporting functions from the response.

The current structure of this part is located in:

  • kotlin/org/jetbrains/research/testspark/core/test
  • kotlin/org/jetbrains/research/testspark/tools

It can be more easily understood with the following diagram:

  • TestsAssembler: Assembler class for generating and organizing test cases from the LLM response.
  • TestSuiteParser: Extracts test cases from raw text and generates a test suite.
  • TestBodyPrinter: Generates the body of a test function as a string.

4. Compilation

Before showing the code to the user, it should be checked for compilation.

  • TestCompiler: Compiles a list of test cases and returns the compilation result.

Here one should specify the appropriate compilation strategy for each language. With all the dependencies and build paths.

5. UI Representation

Once the code generated by the LLM is checked for the compilation, it should be presented in the UI.

  • TestCaseDisplayService: Service responsible for the representation of all the UI components.
  • TestSuiteView: Interface specific for working with buttons.
  • TestClassCodeAnalyzer: Interface for retrieving information from test class code.
  • TestClassCodeGenerator: Interface for generating and formatting test class code.

6. Running and saving tests

We should be able to run all the tests in the UI and then save them to the desired folder.

  • TestPersistentStorage: Interface representing a contract for saving generated tests to a specified file system location.

For Kotlin and Java, the TestProcessor implementation also allows saving the JaCoCo report to see the code coverage of the test that will be saved.

Plugin Configuration File

The plugin configuration file is plugin.xml which can be found in src/main/resources/META-INF directory. All declarations (such as actions, services, listeners) are present in this file.

Classes

All the classes can be found in src/main/kotlin/org/jetbrains/research/testspark directory.

Actions

All the action classes can be found in actions directory.

  • common
    • GenerateTestsActionClassCommon This class contains all the logic related to generating tests for a class by all generators in the plugin.
    • GenerateTestsActionMethodCommon This class contains all the logic related to generating tests for a method by all generators in the plugin.
    • GenerateTestsActionLineCommon This class contains all the logic related to generating tests for a line by all generators in the plugin.
  • evosuite
    • GenerateTestsActionClassEvoSuite This class contains all the logic related to generating tests for a class by EvoSuite.
    • GenerateTestsActionMethodEvoSuite This class contains all the logic related to generating tests for a method by EvoSuite.
    • GenerateTestsActionLineEvoSuite This class contains all the logic related to generating tests for a line by EvoSuite.
  • llm
    • GenerateTestsActionClassLlm This class contains all the logic related to generating tests for a class by LLM.
    • GenerateTestsActionMethodLlm This class contains all the logic related to generating tests for a method by LLM.
    • GenerateTestsActionLineLlm This class contains all the logic related to generating tests for a line by LLM.
  • GenerateTestsUtils contains useful functions for GenerateTestsActionClass, GenerateTestsActionMethod and GenerateTestsActionLine classes, but one function are also used by StaticInvalidationService.

Coverage

All the classes related to the coverage visualisation can be found in coverage directory.

  • CoverageRenderer adds extra functionality to the default line marker and gutter which is used for the coverage visualisation. It adds tooltips that, when clicking on the gutter, show which tests cover a certain line or mutants (if any) that have been introduced on that line. It also highlights the corresponding tests in the tool window tab.

Editor

All the editor classes can be found in editor directory.

  • Workspace handles user workspace state and modifications of that state related to test generation. It also sets the event listeners that are triggered whenever the user changes the contents of the editor.

Validation

All the classes related to dynamic cache invalidation can be found in evosuite/validation directory.

  • TestCaseEditor.kt edits the test suite by visiting the test cases and setting the modified body if it has been modified. It also removes scaffolding.
  • ValidationResultListener.kt is a topic interface for sending and receiving results of test validation.
  • ValidationToolWindowFactory.kt creates the tabs and the UI of the tool window corresponding to dynamic test validation.
  • Validator.kt validates and calculates the coverage of an optionally edited set of test cases.

Helpers

All the helper classes can be found in helpers directory.

  • MethodDescriptorHelper contains helper functions for generating method descriptors. It is used by GenerateTestsActionMethod class.

Listeners

All the listener classes can be found in listeners directory.

  • TestGenerationResultListenerImpl is the implementation of TestGenerationResultListener topic interface. It notifies Workspace of the received test generation result and also puts the generated tests into the cache.
  • TelemetrySubmitListenerImplschedules potential submissions of the generated telemetry into a file, which is done every 5 minutes and when the project is closed.

Services

All the service classes can be found in services directory.

  • CoverageSelectionToggleListener is a topic interface for showing or hiding highlighting when the coverage is toggled for one test or many tests.
  • CoverageToolWindowDisplayService creates the "Coverage visualisation" tool window panel and the coverage table to display the test coverage data of the tests generated by EvoSuite.
  • CoverageVisualisationService visualises the coverage in the gutter and the editor (by colouring), injects the coverage data into the "Coverage visualisation" tool window tab.
  • ErrorMonitor class for handling error occurrences.
  • QuickAccessParametersService allows to load and get the state of the parameters in the "Parameters" tool window panel.
  • RunnerService is used to limit TestSpark to generate tests only once at a time.
  • SettingsApplicationService stores the application-level settings persistently. It uses SettingsApplicationState class for that.
  • SettingsProjectService stores the project-level settings persistently. It uses SettingsProjectState class for that.
  • StaticInvalidationService invalidates the cache statically.
  • TestCaseCachingService contains the data structure for caching the generated test cases and is responsible for adding, retrieving and removing (invalidating) the generated tests.
  • TestCaseDisplayService displays the tests generated by EvoSuite, in the "Generated tests" tool window panel.
  • TestSparkTelemetryService sends usage information to Intelligent Collaboration Tools Lab at JetBrains Research if the user has opted in.

Settings

All the classes related to TestSpark Settings/Preferences page can be found in settings directory.

  • SettingsApplicationState is responsible for storing the values of the EvoSuite Settings entries.
  • SettingsEvoSuiteComponent displays and captures the changes to the values of the entries in the EvoSuite page of the Settings dialog.
  • SettingsEvoSuiteConfigurable allows to configure some EvoSuite settings via the EvoSuite page in the Settings dialog, observes the changes and manages the UI and state.
  • SettingsPluginComponent displays and captures the changes to the values of the entries in the TestSpark main page of the Settings dialog.
  • SettingsPluginConfigurable allows to configure some Plugin settings via the Plugin page in the Settings dialog, observes the changes and manages the UI and state.
  • SettingsProjectState is responsible for storing the values of the Plugin Settings entries.

Tools

  • evosuite all the classes that interact with EvoSuite can be found in this directory.

    • error
      • EvoSuiteErrorManager this class represents the error manager for EvoSuite. It provides methods for handling and displaying errors and warnings encountered during EvoSuite execution.
    • generation
      • EvoSuiteProcessManager This class manages the execution of EvoSuite, a test generation tool.
      • ResultWatcher listens for the results of the generation process by EvoSuite. Used in conjunction with Runner.
    • EvoSuite
    • SettingsArguments is used for constructing the arguments and properties for EvoSuite.

  • llm all the classes that interact with Llm can be found in this directory.

    • error
      • LLMErrorManager LLMErrorManager is a class that handles error and warning messages for LLM (Live Logic Monitor).
    • generation
      • LLMProcessManager LLMProcessManager is a class that implements the ProcessManager interface and is responsible for generating tests using the LLM tool.
      • LLMRequestManager this class represents a manager for making requests to the LLM (Large Language Model).
      • PromptManager a class that manages prompts for generating unit tests.
      • TestCoverageCollector responsible for collecting test coverage data and generating a report
      • TestsAssembler assembler class for generating and organizing test cases.
    • Llm the Llm class represents a tool called "Llm" that is used to generate tests for Java code.
    • SettingsArguments is used for constructing the arguments and properties for Llm.

  • Manager Provides methods for generating tests using different tools.

  • Pipeline pipeline class represents a pipeline for running the test generation process.

  • ProjectBuilder builds the project before running EvoSuite and before validating the tests.

  • TestGenerationResultListener is a topic interface for sending and receiving test results produced by EvoSuite test generation process.

  • ToolUtils utils for working with tools

Tool Window

All the classes related to TestSpark Tool Window (on the right side) can be found in toolwindow directory.

  • QuickAccessParameters stores the main panel and the UI of the "Parameters" tool window tab.
  • QuickAccessParametersState is responsible for persisting the values of the parameters in the "Parameters" tool window tab.
  • TestSparkToolWindowFactory creates the tabs and the UI of the TestSpark tool window.

Bundles

  • TestSparkBundle is used to load EvoSuite messages in the code (from messages.TestSpark file in the recourses directory).
  • TestSparkDefaultsBundle is used to load the default values of the parameters in the code (from defaults/TestSpark.properties file in the resources directory).
  • TestSparkLabelsBundle is used to load the text of various UI labels in the code (from defaults/Labels.properties file in the recourses directory).
  • TestSparkToolTipsBundle is used to load the text of various tooltips in the code (from defaults/Tooltips.properties file in the resources directory).

Translations

The vast majority of labels, tooltip-texts and messages are saved in their own .properties files. This is practical if you wish to translate the plugin to a different language - changing these files should translate most of the plugin elements. The only omitted texts are those which require certain properties set to them (e.g. IntelliJ's .preferredSize). Those have to be translated in the code. There also exists a .properties file for default TestSpark configurations. It is not related to linguistics, but useful if you wish to change default values of the plugin.

Tests

The tests for TestSpark can be found in src/test directory.

  • resources directory contains dummy projects used for testing the plugin.
  • kotlin/org/jetbrains/research/testspark directory contains the actual tests.
    • helpers directory contains tests for the method descriptor helper (MethodDescriptorHelperTest).
    • runner directory contains tests for the settings arguments that are used when running EvoSuite (SettingsArgumentTest).
    • services directory contains tests for the coverage visualisation and caching service classes (CoverageVisualisationServiceTest, TestCaseCachingServicePropertyBasedTest, TestCaseCachingServiceTest).
    • settings directory contains unit tests for plugin and EvoSuite settings (SettingsEvoSuiteConfigurableTest, SettingsPluginConfigurableTest, TestSparkSettingsStateTest).
    • toolwindow directory contains unit tests for tool window tabs (QuickAccessParametersStateTest).
    • uiTest directory contains the UI tests.
      • customfixtures directory contains the custom fixtures that had to be created for testing.
      • pages directory has the frames and fixtures that are used for UI testing (IdeaFrame, QuickAccessParametersFixtures, SettingsFrame, WelcomeFrame).
      • utils directory contains utility files that are helpful for UI testing (RemoteRobotExtension, StepsLogger).
      • tests directory contains the actual UI tests.
      • CoverageVisualisationToolWindowTest contains the UI tests for Coverage Visualisation tab in the tool window.
      • PsiSelectionTest contains the UI tests for PSI element selection logic when generating tests for class, method and line.
      • QuickAccessParametersTest contains the UI tests for Quick Access Parameters tab in the tool window.
      • SettingsComponentTest contains the UI tests for the Settings page of the plugin (both the plugin settings page and EvoSuite settings page).