CodeRabbit Secure checks vulnerabilities with runtime verification

Turning a vulnerability finding into a reproducible test

Simply sending a request and examining the response is not enough to verify a suspected vulnerability. The application must be running, the required dependencies must be available, and the request must reach the code under investigation. In CodeRabbit's example, testing an endpoint that apparently allows one user to retrieve another user's document requires two authenticated users, the document owner and a running application.

CodeRabbit Secure generates a guide called the Reproducer Guide from Deep Scan findings. It summarizes the required preparations, the steps to follow and the observations that could support or challenge the finding. Developers can follow the guide in their own test environment or run it in a prepared CodeRabbit environment using Runtime Analysis.

Before creating the guide, the system checks again whether the previous finding still exists, since the code may have changed in the meantime. If the issue is no longer present or the verification result is inconclusive, guide generation stops. The run uses the code version recorded in the published guide. Pressing the Run button alone does not recheck the finding or automatically test the latest source.

Execution requirements and evidence

Using Runtime Analysis requires the repository to have a prepared CodeRabbit coding environment. This must include the dependencies, configuration and startup instructions needed to run the application. Teams can also enable automated verification. In that case, after every Deep Scan, the system creates guides for new findings at selected severity levels and runs them in the repository's prepared default environment.

Starting the application process does not by itself prove that the application is ready for testing. A service may be listening on the port while a required dependency is unavailable, or an authentication error may prevent the request from reaching the endpoint under investigation. For this reason, the result includes the Application ready and Target exercised indicators, showing whether CodeRabbit was able to establish that the application was ready and that the relevant code path was executed.

CodeRabbit recommends using a separate test environment and not using production credentials for tests. The agent may access variables configured in the selected environment, including secrets. These must not be printed in test scripts or commands, because the output may be retained as evidence.

Three possible outcomes with limited scope

Runtime Analysis returns one of three states. Reproduced means the run produced evidence supporting the finding under the conditions tested. With Not reproduced, the run did not reproduce the issue, but different results may still occur with other inputs or configurations. The Inconclusive state indicates that the system could not draw a well-founded conclusion. The result explains the reasons and limitations.

Every result applies to the code version and environment that were tested. In the document-access example, a 403 response proves that the authorization check is working only if the request reached the ownership check. A request rejected because of a missing session does not test this. Likewise, a “not reproduced” result does not automatically mean that the application is secure. If configuration problems prevented a meaningful test from taking place, the result should remain inconclusive.

The workflow separates the test plan, execution and evaluation. This allows developers to review the guide's assumptions before running it, perform the test in their own environment and retain the execution details and supporting evidence alongside the result.