Parsing HTML in Java with Jsoup – 用Jsoup在Java中解析HTML

最后修改: 2017年 1月 12日

1. Overview


Jsoup is an open source Java library used mainly for extracting data from HTML. It also allows you to manipulate and output HTML. It has a steady development line, great documentation, and a fluent and flexible API. Jsoup can also be used to parse and build XML.


In this tutorial, we’ll use the Spring Blog to illustrate a scraping exercise that demonstrates several features of jsoup:

在本教程中,我们将使用Spring Blog来说明一个刮削练习,以演示jsoup的几个功能。

  • Loading: fetching and parsing the HTML into a Document
  • Filtering: selecting the desired data into Elements and traversing it
  • Extracting: obtaining attributes, text, and HTML of nodes
  • Modifying: adding/editing/removing nodes and editing their attributes

2. Maven Dependency


To make use of the jsoup library in your project, add the dependency to your pom.xml:



You can find the latest version of jsoup in the Maven Central repository.


3. Jsoup at a Glance


Jsoup loads the page HTML and builds the corresponding DOM tree. This tree works the same way as the DOM in a browser, offering methods similar to jQuery and vanilla JavaScript to select, traverse, manipulate text/HTML/attributes and add/remove elements.

Jsoup加载页面的HTML并建立相应的DOM树。这个树的工作方式与浏览器中的DOM相同,提供类似于jQuery和vanilla JavaScript的方法来选择、遍历、操作文本/HTML/属性和添加/删除元素。

If you’re comfortable with client-side selectors and DOM traversing/manipulation, you’ll find jsoup very familiar. Check how easy it is to print the paragraphs of a page:


Document doc = Jsoup.connect("").get();"p").forEach(System.out::println);

Bear in mind that jsoup interprets HTML only — it does not interpret JavaScript. Therefore changes to the DOM that would normally take place after page loads in a JavaScript-enabled browser will not be seen in jsoup.


4. Loading


The loading phase comprises the fetching and parsing of the HTML into a Document. Jsoup guarantees the parsing of any HTML, from the most invalid to the totally validated ones, as a modern browser would do. It can be achieved by loading a String, an InputStream, a File or a URL.


Let’s load a Document from the Spring Blog URL:


String blogUrl = "";
Document doc = Jsoup.connect(blogUrl).get();

Notice the get method, it represents an HTTP GET call. You could also do an HTTP POST with the post method (or you could use a method which receives the HTTP method type as a parameter).

注意get方法,它代表一个HTTP GET调用。你也可以用post方法做一个HTTP POST(或者你可以使用一个method,它接收HTTP方法类型作为一个参数)。

If you need to detect abnormal status codes (e.g. 404), you should catch the HttpStatusException exception:


try {
   Document doc404 = Jsoup.connect("").get();
} catch (HttpStatusException ex) {

Sometimes, the connection needs to be a bit more customized. Jsoup.connect(…) returns a Connection which allows you to set, among other things, the user agent, referrer, connection timeout, cookies, post data, and headers:


Connection connection = Jsoup.connect(blogUrl);
connection.cookie("cookiename", "val234");
connection.cookie("cookiename", "val234");
connection.header("headersecurity", "xyz123");
Document docCustomConn = connection.get();

Since the connection follows a fluent interface, you can chain these methods before calling the desired HTTP method:


Document docCustomConn = Jsoup.connect(blogUrl)
  .cookie("cookiename", "val234")
  .cookie("anothercookie", "ilovejsoup")
  .header("headersecurity", "xyz123")

You can learn more about the Connection settings by browsing the corresponding Javadoc.


5. Filtering


Now that we have the HTML converted into a Document, it’s time to navigate it and find what we are looking for. This is where the resemblance with jQuery/JavaScript is more evident, as its selectors and traversing methods are similar.


5.1. Selecting


The Document select method receives a String representing the selector, using the same selector syntax as in a CSS or JavaScript, and retrieves the matching list of Elements. This list can be empty but not null.

Document select方法接收代表选择器的String,使用与CSS或JavaScript中相同的选择器语法,并检索匹配的Elements的列表。这个列表可以是空的,但不能是null

Let’s take a look at some selections using the select method:


Elements links ="a");
Elements sections ="section");
Elements logo =".spring-logo--container");
Elements pagination ="#pagination_control");
Elements divsDescendant ="header div");
Elements divsDirect ="header > div");

You can also use more explicit methods inspired by the browser DOM instead of the generic select:


Element pag = doc.getElementById("pagination_control");
Elements desktopOnly = doc.getElementsByClass("desktopOnly");

Since Element is a superclass of Document, you can learn more about working with the selection methods in the Document and Element Javadocs.

由于 ElementDocument 的超类,您可以在 DocumentElement Javadocs 中了解更多关于使用选择方法的工作。

5.2. Traversing


Traversing means navigating across the DOM tree. Jsoup provides methods that operate on the Document, on a set of Elements, or on a specific Element, allowing you to navigate to a node’s parents, siblings, or children.

遍历是指在DOM树上进行导航。 Jsoup提供了对Document、一组Elements或特定Element进行操作的方法,允许你导航到一个节点的父辈、兄弟姐妹或子辈。

Also, you can jump to the first, the last, and the nth (using a 0-based index) Element in a set of Elements:


Element firstSection = sections.first();
Element lastSection = sections.last();
Element secondSection = sections.get(2);
Elements allParents = firstSection.parents();
Element parent = firstSection.parent();
Elements children = firstSection.children();
Elements siblings = firstSection.siblingElements();

You can also iterate through selections. In fact, anything of type Elements can be iterated:


sections.forEach(el -> System.out.println("section: " + el));

You can make a selection restricted to a previous selection (sub-selection):


Elements sectionParagraphs =".paragraph");

6. Extracting


We now know how to reach specific elements, so it’s time to get their content — namely their attributes, HTML, or child text.


Take a look at this example that selects the first article from the blog and gets its date, its first section text, and finally, its inner and outer HTML:


Element firstArticle ="article").first();
Element timeElement ="time").first();
String dateTimeOfFirstArticle = timeElement.attr("datetime");
Element sectionDiv ="section div").first();
String sectionDivText = sectionDiv.text();
String articleHtml = firstArticle.html();
String outerHtml = firstArticle.outerHtml();

Here are some tips to bear in mind when choosing and using selectors:


  • Rely on “View Source” feature of your browser and not only on the page DOM as it might have changed (selecting at the browser console might yield different results than jsoup)
  • Know your selectors as there are a lot of them and it’s always good to have at least seen them before; mastering selectors takes time
  • Use a playground for selectors to experiment with them (paste a sample HTML there)
  • Be less dependent on page changes: aim for the smallest and least compromising selectors (e.g. prefer id. based)

7. Modifying


Modifying encompasses setting attributes, text, and HTML of elements, as well as appending and removing elements. It is done to the DOM tree previously generated by jsoup – the Document.


7.1. Setting Attributes and Inner Text/HTML


As in jQuery, the methods to set attributes, text, and HTML bear the same names but also receive the value to be set:


  • attr() – sets an attribute’s values (it creates the attribute if it does not exist)
  • text() – sets element inner text, replacing content
  • html() – sets element inner HTML, replacing content

Let’s look at a quick example of these methods:


timeElement.attr("datetime", "2016-12-16 15:19:54.3");
sectionDiv.text("foo bar");"h2").html("<div><span></span></div>");

7.2. Creating and Appending Elements


To add a new element, you need to build it first by instantiating Element. Once the Element has been built, you can append it to another Element using the appendChild method. The newly created and appended Element will be inserted at the end of the element where appendChild is called:


Element link = new Element(Tag.valueOf("a"), "")
  .text("Checkout this amazing website!")
  .attr("href", "")
  .attr("target", "_blank");

7.3. Removing Elements


To remove elements, you need to select them first and run the remove method.


For example, let’s remove all <li> tags that contain the “navbar-link” class from Document, and all images from the first article:


7.4. Converting the Modified Document to HTML


Finally, since we were changing the Document, we might want to check our work.


To do this, we can explore the Document DOM tree by selecting, traversing, and extracting using the presented methods, or we can simply extract its HTML as a String using the html() method:

要做到这一点,我们可以通过选择、遍历和使用提出的方法提取来探索Document DOM树,或者我们可以简单地使用html()方法将其HTML提取为String

String docHtml = doc.html();

The String output is a tidy HTML.


8. Conclusion


Jsoup is a great library to scrape any page. If you’re using Java and don’t require browser-based scraping, it’s a library to take into account. It’s familiar and easy to use since it makes use of the knowledge you may have on front-end development and follows good practices and design patterns.


You can learn more about scraping web pages with jsoup by studying the jsoup API and reading the jsoup cookbook.

您可以通过学习jsoup API和阅读jsoup cookbook来了解更多关于使用jsoup刮取网页的信息。

The source code used in this tutorial can be found in the GitHub project.