Parsing HTML in Java with Jsoup – 用Jsoup在Java中解析HTML

最后修改: 2017年 1月 12日

中文/混合/英文(键盘快捷键：t)

1. Overview

1.概述

Jsoup is an open source Java library used mainly for extracting data from HTML. It also allows you to manipulate and output HTML. It has a steady development line, great documentation, and a fluent and flexible API. Jsoup can also be used to parse and build XML.

Jsoup是一个开源的Java库，主要用于从HTML提取数据。它还允许你操作和输出HTML。它拥有稳定的开发线、优秀的文档和流畅灵活的API。Jsoup还可以用来解析和构建XML。

In this tutorial, we’ll use the Spring Blog to illustrate a scraping exercise that demonstrates several features of jsoup:

在本教程中，我们将使用Spring Blog来说明一个刮削练习，以演示jsoup的几个功能。

  • Loading: fetching and parsing the HTML into a Document
  • Filtering: selecting the desired data into Elements and traversing it
  • Extracting: obtaining attributes, text, and HTML of nodes
  • Modifying: adding/editing/removing nodes and editing their attributes

2. Maven Dependency

2.Maven的依赖性

To make use of the jsoup library in your project, add the dependency to your pom.xml:

要在你的项目中使用jsoup库，请在你的pom.xml中添加该依赖关系。

<dependency>
    <groupId>org.jsoup</groupId>
    <artifactId>jsoup</artifactId>
    <version>1.10.2</version>
</dependency>

You can find the latest version of jsoup in the Maven Central repository.

你可以在Maven中心仓库找到最新版本的jsoup。

3. Jsoup at a Glance

3.Jsoup简介

Jsoup loads the page HTML and builds the corresponding DOM tree. This tree works the same way as the DOM in a browser, offering methods similar to jQuery and vanilla JavaScript to select, traverse, manipulate text/HTML/attributes and add/remove elements.

Jsoup加载页面的HTML并建立相应的DOM树。这个树的工作方式与浏览器中的DOM相同，提供类似于jQuery和vanilla JavaScript的方法来选择、遍历、操作文本/HTML/属性和添加/删除元素。

If you’re comfortable with client-side selectors and DOM traversing/manipulation, you’ll find jsoup very familiar. Check how easy it is to print the paragraphs of a page:

如果你对客户端选择器和DOM遍历/操作很熟悉，你会发现jsoup非常熟悉。看看打印一个页面的段落有多容易。

Document doc = Jsoup.connect("http://example.com").get();
doc.select("p").forEach(System.out::println);

Bear in mind that jsoup interprets HTML only — it does not interpret JavaScript. Therefore changes to the DOM that would normally take place after page loads in a JavaScript-enabled browser will not be seen in jsoup.

请记住，jsoup只解释HTML，它不解释JavaScript。因此，在支持JavaScript的浏览器中，通常在页面加载后发生的对DOM的改变不会在jsoup中看到。

4. Loading

4.装载

The loading phase comprises the fetching and parsing of the HTML into a Document. Jsoup guarantees the parsing of any HTML, from the most invalid to the totally validated ones, as a modern browser would do. It can be achieved by loading a String, an InputStream, a File or a URL.

加载阶段包括获取和解析HTML到一个文档。Jsoup保证解析任何HTML，从最无效的到完全有效的，就像现代浏览器会做的那样。它可以通过加载一个String、一个InputStream、一个File或一个URL来实现。

Let’s load a Document from the Spring Blog URL:

让我们从Spring博客的URL加载一个Document。

String blogUrl = "https://spring.io/blog";
Document doc = Jsoup.connect(blogUrl).get();

Notice the get method, it represents an HTTP GET call. You could also do an HTTP POST with the post method (or you could use a method which receives the HTTP method type as a parameter).

注意get方法，它代表一个HTTP GET调用。你也可以用post方法做一个HTTP POST（或者你可以使用一个method，它接收HTTP方法类型作为一个参数）。

If you need to detect abnormal status codes (e.g. 404), you should catch the HttpStatusException exception:

如果你需要检测异常的状态代码（例如404），你应该捕捉HttpStatusException异常。

try {
   Document doc404 = Jsoup.connect("https://spring.io/will-not-be-found").get();
} catch (HttpStatusException ex) {
   //...
}

Sometimes, the connection needs to be a bit more customized. Jsoup.connect(…) returns a Connection which allows you to set, among other things, the user agent, referrer, connection timeout, cookies, post data, and headers:

有时，连接需要更多的定制。Jsoup.connect(…)返回一个Connection，允许你设置用户代理、referrer、连接超时、cookies、post数据和头文件等。

Connection connection = Jsoup.connect(blogUrl);
connection.userAgent("Mozilla");
connection.timeout(5000);
connection.cookie("cookiename", "val234");
connection.cookie("cookiename", "val234");
connection.referrer("http://google.com");
connection.header("headersecurity", "xyz123");
Document docCustomConn = connection.get();

Since the connection follows a fluent interface, you can chain these methods before calling the desired HTTP method:

由于连接遵循一个流畅的接口，你可以在调用所需的HTTP方法之前连锁这些方法。

Document docCustomConn = Jsoup.connect(blogUrl)
  .userAgent("Mozilla")
  .timeout(5000)
  .cookie("cookiename", "val234")
  .cookie("anothercookie", "ilovejsoup")
  .referrer("http://google.com")
  .header("headersecurity", "xyz123")
  .get();

You can learn more about the Connection settings by browsing the corresponding Javadoc.

您可以通过浏览相应的Javadoc来了解关于Connection设置的更多信息。

5. Filtering

5.过滤

Now that we have the HTML converted into a Document, it’s time to navigate it and find what we are looking for. This is where the resemblance with jQuery/JavaScript is more evident, as its selectors and traversing methods are similar.

现在我们已经将HTML转换为Document，是时候对其进行导航并找到我们要找的东西了。这是与jQuery/JavaScript相似的地方，因为它的选择器和遍历方法都很相似。

5.1. Selecting

5.1.选择

The Document select method receives a String representing the selector, using the same selector syntax as in a CSS or JavaScript, and retrieves the matching list of Elements. This list can be empty but not null.

Document select方法接收代表选择器的String，使用与CSS或JavaScript中相同的选择器语法，并检索匹配的Elements的列表。这个列表可以是空的，但不能是null。

Let’s take a look at some selections using the select method:

让我们看一下使用select方法的一些选择。

Elements links = doc.select("a");
Elements sections = doc.select("section");
Elements logo = doc.select(".spring-logo--container");
Elements pagination = doc.select("#pagination_control");
Elements divsDescendant = doc.select("header div");
Elements divsDirect = doc.select("header > div");

You can also use more explicit methods inspired by the browser DOM instead of the generic select:

你也可以使用受浏览器DOM启发的更明确的方法，而不是通用的select。

Element pag = doc.getElementById("pagination_control");
Elements desktopOnly = doc.getElementsByClass("desktopOnly");

Since Element is a superclass of Document, you can learn more about working with the selection methods in the Document and Element Javadocs.

由于 Element 是 Document 的超类，您可以在 Document 和 Element Javadocs 中了解更多关于使用选择方法的工作。

5.2. Traversing

5.2.穿越

Traversing means navigating across the DOM tree. Jsoup provides methods that operate on the Document, on a set of Elements, or on a specific Element, allowing you to navigate to a node’s parents, siblings, or children.

遍历是指在DOM树上进行导航。 Jsoup提供了对Document、一组Elements或特定Element进行操作的方法，允许你导航到一个节点的父辈、兄弟姐妹或子辈。

Also, you can jump to the first, the last, and the nth (using a 0-based index) Element in a set of Elements:

此外，你可以跳到一组元素中的第一个、最后一个和第n个（使用基于0的索引）元素。

Element firstSection = sections.first();
Element lastSection = sections.last();
Element secondSection = sections.get(2);
Elements allParents = firstSection.parents();
Element parent = firstSection.parent();
Elements children = firstSection.children();
Elements siblings = firstSection.siblingElements();

You can also iterate through selections. In fact, anything of type Elements can be iterated:

你也可以通过选择进行迭代。事实上，任何类型的Elements都可以被迭代。

sections.forEach(el -> System.out.println("section: " + el));

You can make a selection restricted to a previous selection (sub-selection):

你可以将一个选择限制在之前的选择中（子选择）。

Elements sectionParagraphs = firstSection.select(".paragraph");

6. Extracting

6.提取

We now know how to reach specific elements, so it’s time to get their content — namely their attributes, HTML, or child text.

我们现在知道如何到达特定的元素，所以现在是时候获取它们的内容了–即它们的属性、HTML或子文本。

Take a look at this example that selects the first article from the blog and gets its date, its first section text, and finally, its inner and outer HTML:

看看这个例子，它选择了博客中的第一篇文章，并获得了它的日期、它的第一部分文本，最后是它的内部和外部HTML。

Element firstArticle = doc.select("article").first();
Element timeElement = firstArticle.select("time").first();
String dateTimeOfFirstArticle = timeElement.attr("datetime");
Element sectionDiv = firstArticle.select("section div").first();
String sectionDivText = sectionDiv.text();
String articleHtml = firstArticle.html();
String outerHtml = firstArticle.outerHtml();

Here are some tips to bear in mind when choosing and using selectors:

这里有一些选择和使用选择器时需要记住的提示。

  • Rely on “View Source” feature of your browser and not only on the page DOM as it might have changed (selecting at the browser console might yield different results than jsoup)
  • Know your selectors as there are a lot of them and it’s always good to have at least seen them before; mastering selectors takes time
  • Use a playground for selectors to experiment with them (paste a sample HTML there)
  • Be less dependent on page changes: aim for the smallest and least compromising selectors (e.g. prefer id. based)

7. Modifying

7.修改

Modifying encompasses setting attributes, text, and HTML of elements, as well as appending and removing elements. It is done to the DOM tree previously generated by jsoup – the Document.

修改包括设置元素的属性、文本和HTML，以及添加和删除元素。它是针对先前由jsoup生成的DOM树–Document进行的。

7.1. Setting Attributes and Inner Text/HTML

7.1.设置属性和内部文本/HTML

As in jQuery, the methods to set attributes, text, and HTML bear the same names but also receive the value to be set:

和jQuery一样，设置属性、文本和HTML的方法有相同的名称，但也会接收要设置的值。

  • attr() – sets an attribute’s values (it creates the attribute if it does not exist)
  • text() – sets element inner text, replacing content
  • html() – sets element inner HTML, replacing content

Let’s look at a quick example of these methods:

让我们来看看这些方法的一个快速例子。

timeElement.attr("datetime", "2016-12-16 15:19:54.3");
sectionDiv.text("foo bar");
firstArticle.select("h2").html("<div><span></span></div>");

7.2. Creating and Appending Elements

7.2.创建和添加元素

To add a new element, you need to build it first by instantiating Element. Once the Element has been built, you can append it to another Element using the appendChild method. The newly created and appended Element will be inserted at the end of the element where appendChild is called:

要添加一个新元素，你需要首先通过实例化Element来构建它。一旦Element被构建，你可以使用appendChild方法将其追加到另一个Element。新创建和追加的Element将被插入到调用appendChild的元素的末端。

Element link = new Element(Tag.valueOf("a"), "")
  .text("Checkout this amazing website!")
  .attr("href", "http://baeldung.com")
  .attr("target", "_blank");
firstArticle.appendChild(link);

7.3. Removing Elements

7.3.移除元素

To remove elements, you need to select them first and run the remove method.

要删除元素，你需要先选择它们并运行remove方法。

For example, let’s remove all <li> tags that contain the “navbar-link” class from Document, and all images from the first article:

例如，让我们从Document,中删除所有包含”navbar-link”类的<li>标签，以及第一篇文章中的所有图片。

doc.select("li.navbar-link").remove();
firstArticle.select("img").remove();

7.4. Converting the Modified Document to HTML

7.4.将修改后的文档转换为HTML

Finally, since we were changing the Document, we might want to check our work.

最后，由于我们正在改变Document，我们可能想检查一下我们的工作。

To do this, we can explore the Document DOM tree by selecting, traversing, and extracting using the presented methods, or we can simply extract its HTML as a String using the html() method:

要做到这一点，我们可以通过选择、遍历和使用提出的方法提取来探索Document DOM树，或者我们可以简单地使用html()方法将其HTML提取为String。

String docHtml = doc.html();

The String output is a tidy HTML.

String输出是一个整洁的HTML。

8. Conclusion

8.结论

Jsoup is a great library to scrape any page. If you’re using Java and don’t require browser-based scraping, it’s a library to take into account. It’s familiar and easy to use since it makes use of the knowledge you may have on front-end development and follows good practices and design patterns.

Jsoup是一个伟大的库，可以搜刮任何页面。如果你使用的是Java，并且不需要基于浏览器的搜刮，它是一个值得考虑的库。它很熟悉，也很容易使用，因为它利用了你可能拥有的前端开发知识，并遵循良好的实践和设计模式。

You can learn more about scraping web pages with jsoup by studying the jsoup API and reading the jsoup cookbook.

您可以通过学习jsoup API和阅读jsoup cookbook来了解更多关于使用jsoup刮取网页的信息。

The source code used in this tutorial can be found in the GitHub project.

本教程中使用的源代码可以在GitHub项目中找到。