SpringBoot中如何实现使用Jsoup爬取网站数据

发布时间：2020-07-17 16:00:15 阅读：389 作者：小猪栏目：编程语言

前端开发者测试专用服务器限时活动，0元免费领，库存有限，领完即止！点击查看>>

这篇文章主要为大家展示了SpringBoot中如何实现使用Jsoup爬取网站数据，内容简而易懂，希望大家可以学习一下，学习完之后肯定会有收获的，下面让小编带大家一起来看看吧。

爬取数据

导入jar包

 <properties>
    <java.version>1.8</java.version>
    <elasticsearch.version>7.6.1</elasticsearch.version>
  </properties>
 
  <dependencies>
    <dependency>
      <groupId>org.jsoup</groupId>
      <artifactId>jsoup</artifactId>
      <version>1.10.2</version>
    </dependency>
    <dependency>
      <groupId>com.alibaba</groupId>
      <artifactId>fastjson</artifactId>
      <version>1.2.62</version>
    </dependency>
    <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-data-redis</artifactId>
    </dependency>
    <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-data-elasticsearch</artifactId>
    </dependency>
    <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-thymeleaf</artifactId>
    </dependency>
    <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-web</artifactId>
    </dependency>
 
    <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-devtools</artifactId>
      <scope>runtime</scope>
      <optional>true</optional>
    </dependency>
    <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-configuration-processor</artifactId>
      <optional>true</optional>
    </dependency>
    <dependency>
      <groupId>org.projectlombok</groupId>
      <artifactId>lombok</artifactId>
      <optional>true</optional>
    </dependency>
    <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-test</artifactId>
      <scope>test</scope>
      <exclusions>
        <exclusion>
          <groupId>org.junit.vintage</groupId>
          <artifactId>junit-vintage-engine</artifactId>
        </exclusion>
      </exclusions>
    </dependency>
  </dependencies>

新建实体类

@Data
@NoArgsConstructor
@AllArgsConstructor
public class Content {
  private String title;
  private String img;
  private String price;
}

编写爬虫工具类

public class HtmlParseUtil {
  public static void main(String[] args) throws Exception {
    new HtmlParseUtil().parseDDJJ("包").forEach(System.out::println);
  }
 
  public List<Content> parseDDJJ(String keywords) throws Exception {
    //爬取url地址
    String url = "https://search.xxxx.com/Search&#63;keyword="+keywords;
    //解析网页，30s内未爬取成功，打印错误
    Document document = Jsoup.parse(new URL(url),30000);
    //获取每一本书籍的id
    Element element = document.getElementById("DJ_goodsList");
    //获取所有的li标签
    Elements elements = element.getElementsByTag("li");
 
    ArrayList<Content> goodsList = new ArrayList<>();
 
    //遍历li标签的内容
    for (Element el : elements) {
      String img = el.getElementsByTag("img").eq(0).attr("src");
      String price = el.getElementsByClass("p-price").eq(0).text();
      String title = el.getElementsByClass("p-name").eq(0).text();
 
      Content content = new Content();
      content.setTitle(title);
      content.setPrice(price);
      content.setImg(img);
      goodsList.add(content);
    }
    return goodsList;
  }
}

以上就是关于SpringBoot中如何实现使用Jsoup爬取网站数据的内容，如果你们有学习到知识或者技能，可以把它分享出去让更多的人看到。

亿速云「云服务器」，即开即用、新一代英特尔至强铂金CPU、三副本存储NVMe SSD云盘，价格低至29元/月。点击查看>>

向AI问一下细节

SpringBoot中如何实现使用Jsoup爬取网站数据

猜你喜欢

最新资讯

相关推荐

开发者交流群：

相关标签