Gatling vs JMeter vs The Grinder: Comparing Load Test Tools – Gatling vs JMeter vs The Grinder:负载测试工具的比较

1. Introduction

1.绪论

Choosing the right tool for the job can be daunting. In this tutorial, we’ll simplify this by comparing three web application load testing tools – Apache JMeter, Gatling, and The Grinder–against a simple REST API.

为工作选择正确的工具可能是令人生畏的。在本教程中，我们将通过比较三种网络应用程序负载测试工具–Apache JMeter、Gatling和The Grinder–与一个简单的REST API进行比较来简化这一问题。

2. Load Testing Tools

2.负载测试工具

First, let’s quickly review some background on each.

首先，让我们快速回顾一下各自的一些背景。

2.1. Gatling

2.1. 加特林

Gatling is a load testing tool that creates test scripts in Scala. Gatling’s recorder generates the Scala test scripts, a key feature for Gatling. Check out our Intro to Gatling tutorial for more information.

Gatling是一个负载测试工具，可在Scala中创建测试脚本。Gatling 的记录器可生成 Scala 测试脚本，这是 Gatling 的一项关键功能。请查看我们的 Intro to Gatling 教程以了解更多信息。

2.2. JMeter

2.2 JMeter

JMeter is a load testing tool by Apache. It provides a nice GUI that we use can for configuration. A unique feature called logic controllers gives great flexibility to set up tests in the GUI.

JMeter是Apache的一个负载测试工具。它提供了一个漂亮的GUI，我们可以用它来进行配置。一个名为逻辑控制器的独特功能为在GUI中设置测试提供了极大的灵活性。

Visit our Intro to JMeter tutorial for screenshots and more explanation.

请访问我们的Intro to JMeter教程，了解屏幕截图和更多解释。

2.3. The Grinder

2.3.研磨机

And our final tool, The Grinder, provides a more programming-based scripting engine than the other two and uses Jython. However, The Grinder 3 does have functionality for recording scripts.

而我们的最后一个工具，Grinder，提供了一个比其他两个更基于编程的脚本引擎，并使用Jython。然而，The Grinder 3确实具有记录脚本的功能。

The Grinder also differs from the other two tools by allowing for console and agent processes. This functionality provides the ability for an agent process so that the load tests can scale up across multiple servers.  It’s specifically advertised as a load test tool built for developers to find deadlocks and slowdowns.

Grinder还与其他两个工具不同，允许控制台和代理进程。该功能提供了代理进程的能力，从而使负载测试可以在多个服务器上进行扩展。 它被特别宣传为一个为开发人员建立的负载测试工具，以发现死锁和减速。

3. Test Case Setup

3.测试案例设置

Next, for our test, we need an API. Our API functionality includes:

接下来，对于我们的测试，我们需要一个API。我们的API功能包括。

• add/update a rewards record
• view one/all rewards record
• link a transaction to a customer rewards record
• view transactions for a customer rewards record

Our Scenario:

我们的方案：。

A store is having a nationwide sale with new and returning customers who need customer rewards accounts to get savings. The rewards API checks for customer rewards account by the customer id. If no rewards account exists, add it, then link to the transaction.

一家商店正在进行全国性的促销活动，新客户和老客户都需要客户奖励账户来获得优惠。奖励API通过客户ID检查客户奖励账户.如果不存在奖励账户，则添加它，然后链接到交易。

After this, we query the transactions.

在这之后，我们查询交易。

3.1. Our REST API

3.1.我们的REST API

Let’s get a quick highlight of the API by viewing some of the method stubs:

让我们通过查看一些方法的存根来快速了解API的亮点。

@PostMapping(path="/rewards/add")
public @ResponseBody RewardsAccount addRewardsAcount(@RequestBody RewardsAccount body)

@GetMapping(path="/rewards/find/{customerId}")
public @ResponseBody Optional<RewardsAccount> findCustomer(@PathVariable Integer customerId)

@PostMapping(path="/transactions/add")
public @ResponseBody Transaction addTransaction(@RequestBody Transaction transaction)

@GetMapping(path="/transactions/findAll/{rewardId}")
public @ResponseBody Iterable<Transaction> findTransactions(@PathVariable Integer rewardId)

Note some of the relationships such as querying for transactions by the reward id and getting the rewards account by customer id. These relationships force some logic and some response parsing for our test scenario creation.

请注意一些关系，如通过奖励ID查询交易和通过客户ID获得奖励账户。这些关系迫使一些逻辑和一些响应解析用于我们的测试场景创建。

The application under test also uses an H2 in-memory database for persistence.

被测试的应用程序也使用H2内存数据库进行持久化。

Luckily, our tools all handle it fairly well, some better than others.

幸运的是，我们的工具都处理得相当好，有些比其他的好。

3.2. Our Testing Plan

3.2.我们的测试计划

Next, we need test scripts.

接下来，我们需要测试脚本。

To get a fair comparison, we’ll perform the same automation steps for each tool:

为了得到一个公平的比较，我们将对每个工具执行相同的自动化步骤。

1. Generate random customer account ids
2. Post a transaction
3. Parse the response for the random customer id and transaction id
4. Query for a customer rewards account id with the customer id
5. Parse the response for the rewards account id
6. If no rewards account id exists then add one with a post
7. Post the same initial transaction with updated rewards id using the transaction id
8. Query for all transactions by rewards account id

Let’s take a closer look at Step 4 for each tool. And, make sure to check out the sample for all three completed scripts.

让我们仔细看看每个工具的步骤4。而且，请确保查看所有三个完成的脚本的样本。

3.3. Gatling

3.3. 加特林

For Gatling, familiarity with Scala adds a boon for developers since the Gatling API is robust and contains a lot of features.

对于Gatling来说，熟悉Scala对开发人员来说是一个福音，因为Gatling的API很强大，包含很多功能。

Gatling’s API takes a builder DSL approach, as we can see in its step 4:

Gatling的API采用了构建者DSL的方法，我们可以在其步骤4中看到。

.exec(http("get_reward")
  .get("/rewards/find/${custId}")
  .check(jsonPath("$.id").saveAs("rwdId")))

Of particular note is Gatling’s support for JSON Path when we need to read and verify an HTTP response. Here, we’ll pick up the reward id and save it to Gatling’s internal state.

特别值得注意的是，当我们需要读取和验证HTTP响应时，Gatling对JSON路径的支持。在这里，我们将拾取奖励ID并将其保存到Gatling的内部状态。

Also, Gatling’s expression language makes for easier dynamic request body Strings:

另外，Gatling的表达式语言使动态请求正文字符串：更容易。

.body(StringBody(
  """{ 
    "customerRewardsId":"${rwdId}",
    "customerId":"${custId}",
    "transactionDate":"${txtDate}" 
  }""")).asJson)

Lastly our configuration for this comparison. The 1000 runs set as a repeat of the entire scenario, atOnceUsers method sets the threads/users:

最后是我们这次比较的配置。1000次运行设置为整个方案的重复，atOnceUsers方法设置线程/用户。

val scn = scenario("RewardsScenario")
  .repeat(1000) {
  ...
  }
  setUp(
    scn.inject(atOnceUsers(100))
  ).protocols(httpProtocol)

The entire Scala script is viewable at our Github repo.

整个Scala脚本可在我们的Github repo中查看。。

3.4. JMeter

3.4 JMeter

JMeter generates an XML file after the GUI configuration. The file contains JMeter specific objects with set properties and their values, for example:

JMeter在GUI配置后会生成一个XML文件。该文件包含JMeter特定的对象，具有设定的属性和它们的值，例如：。

<HTTPSamplerProxy guiclass="HttpTestSampleGui" testclass="HTTPSamplerProxy" testname="Add Transaction" enabled="true">

<JSONPostProcessor guiclass="JSONPostProcessorGui" testclass="JSONPostProcessor" testname="Transaction Id Extractor" enabled="true">

Check out the testname attributes, they can be labeled as we recognize them matching the logical steps above. The ability to add children, variables and dependency steps gives JMeter flexibility as scripting provides. Furthermore, we even set the scope for our variables!

看看testname属性，它们可以按照我们认识到的与上面的逻辑步骤相匹配的标签。添加子代、变量和依赖性步骤的能力使JMeter具有脚本提供的灵活性。此外，我们甚至可以为我们的变量设置范围!

Our configuration for runs and users in JMeter uses ThreadGroups:

我们在JMeter中对运行和用户的配置使用ThreadGroups。

<stringProp name="ThreadGroup.num_threads">100</stringProp>

View the entire jmx file as a reference. While possible, writing tests in XML as .jmx files do not make sense with a full-featured GUI.

查看整个jmx文件作为参考。虽然有可能，但将XML中的测试写成.jmx文件在全功能的GUI中并没有意义。

3.5. The Grinder

3.5.磨床

Without the functional programming of Scala and GUI, our Jython script for The Grinder looks pretty basic. Add some system Java classes, and we have a lot fewer lines of code.

如果没有Scala和GUI的函数式编程，我们为Grinder编写的Jython脚本看起来很基本。加上一些系统的Java类，我们的代码行数就会减少很多。

customerId = str(random.nextInt());
result = request1.POST("http://localhost:8080/transactions/add",
  "{"'"customerRewardsId"'":null,"'"customerId"'":"+ customerId + ","'"transactionDate"'":null}")
txnId = parseJsonString(result.getText(), "id")

However, fewer lines of test setup code are balanced by the need for more string maintenance code such as parsing JSON strings. Also, the HTTPRequest API is slim on functionality.

然而，由于需要更多的字符串维护代码（如解析JSON字符串），测试设置代码的行数减少了，这也是一种平衡。另外，HTTPRequest API的功能也很有限。

With The Grinder, we define threads, processes, and runs values in an external properties file:

通过The Grinder，我们在一个外部属性文件中定义线程、进程和运行值。

grinder.threads = 100
grinder.processes = 1
grinder.runs = 1000

Our full Jython script for The Grinder will look like this.

我们为《磨床》编写的完整Jython脚本将看起来像这个。。

4. Test Runs

4.测试运行

4.1. Test Execution

4.1.测试执行

All three tools recommend using the command line for large load tests.

所有这三种工具都建议使用命令行进行大型负载测试。。

To run the tests, we’ll use Gatling open-source version 3.4.0 as a standalone tool, JMeter 5.3 and The Grinder version 3.

为了运行测试，我们将使用Gatling 开源版作为独立工具，JMeter 5.3和The Grinder version 3.

Gatling requires only that we have JAVA_HOME and GATLING_HOME set. To execute Gatling we use:

Gatling只要求我们设置了JAVA_HOME和GATLING_HOME。为了执行Gatling，我们使用。

./gatling.sh

in the GATLING_HOME/bin directory.

在GATLING_HOME/bin目录下。

JMeter needs a parameter to disable the GUI for the test as prompted when starting the GUI for configuration:

JMeter需要一个参数来禁用测试的GUI，因为在启动GUI进行配置时有提示。

./jmeter.sh -n -t TestPlan.jmx -l log.jtl

Like Gatling, The Grinder requires that we set JAVA_HOME and GRINDERPATH. However, it needs a couple more properties, too:

和Gatling一样，Grinder要求我们设置JAVA_HOME和GRINDERPATH。然而，它还需要几个属性。

export CLASSPATH=/home/lore/Documents/grinder-3/lib/grinder.jar:$CLASSPATH
export GRINDERPROPERTIES=/home/lore/Documents/grinder-3/examples/grinder.properties

As mentioned above, we provide a grinder.properties file for additional configuration such as threads, runs, processes, and console hosts.

如上所述，我们提供了一个grinder.properties文件，用于其他配置，如线程、运行、进程和控制台主机。

Finally, we bootstrap the console and agents with:

最后，我们用以下方法启动控制台和代理。

java -classpath $CLASSPATH net.grinder.Console

java -classpath $CLASSPATH net.grinder.Grinder $GRINDERPROPERTIES

4.2. Test Results

4.2.测试结果

Each of the tests ran 1000 runs with 100 users/threads. Let’s unpack some of the highlights:

每项测试在100个用户/线程中运行了1000次。让我们来解读其中的一些亮点。

The results show the 3 tools have similar speed, with Gatling slightly edging out the other 2, based on the mean throughput. 

结果显示，这3种工具的速度相似，根据平均吞吐量，Gatling略微领先于其他2种工具。

Each tool also provides additional information in a friendlier user interface.

每个工具还在一个更友好的用户界面上提供额外的信息。

Gatling will generate an HTML report at the end of the run, which contains multiple graphs and statistics, for the total run as well as for each request. Here’s a snippet of the test result report:

Gatling将在运行结束时生成一份HTML报告，其中包含多个图表和统计数据，包括总的运行情况以及每个请求的情况。以下是测试结果报告的一个片段。

When using JMeter, we can open the GUI after the test run and generate an HTML report based on the log file where we saved the results:

使用JMeter时，我们可以在测试运行后打开GUI，并根据保存结果的日志文件生成HTML报告。

The JMeter HTML report also contains a breakdown of the statistics per request.

JMeter的HTML报告还包含了每个请求的分类统计。

Finally, The Grinder Console records statistics for each agent and run:

最后，Grinder Console记录了每个代理和运行的统计数据：。

While The Grinder is high-speed, it comes at the cost of additional development time and less diversity of output data.

虽然研磨机是高速的，但它的代价是额外的开发时间和较少的输出数据的多样性。

5. Summary

5.摘要

Now it’s time to take an overall look at each of the load testing tools.

现在是时候对每个负载测试工具进行全面考察了。

Gatling:

加特林：

• Solid, polished load testing tool that outputs beautiful reports with Scala scripting
• Open Source and Enterprise support levels for the product

JMeter:

JMeter:。

• Robust API (through GUI) for test script development with no coding required
• Apache Foundation Support and great integration with Maven

The Grinder:

研磨机：

• Fast performance load testing tool for developers using Jython
• Cross-server scalability provides even more potential for large tests

Simply put, if speed and scalability is a need, then use The Grinder.

简单地说，如果速度和可扩展性是一种需要，那么就使用Grinder。

If great looking interactive graphs help show a performance gain to argue for a change, then use Gatling.

如果漂亮的交互式图表有助于显示性能的提高，以论证一个变化，那么就使用Gatling。

JMeter is the tool for complicated business logic or an integration layer with many message types. As part of the Apache Software Foundation, JMeter provides a mature product and a large community.

JMeter是用于复杂的商业逻辑或具有许多消息类型的集成层的工具。作为Apache软件基金会的一部分，JMeter提供了一个成熟的产品和一个庞大的社区。

6. Conclusion

6.结语

In conclusion, we see that the tools have comparable functionalities in some areas while shining in others. The right tool for the right job is colloquial wisdom that works in software development.

总之，我们看到，这些工具在某些方面具有可比性的功能，而在其他方面却大放异彩。正确的工具适用于正确的工作是在软件开发中起作用的俗语智慧。

Finally, the API and scripts can be found on Github.

最后，API和脚本可以在Github上找到。