spring Boot手把手教学(9): 实战练习-爬虫获取天气信息并自动发邮件,出门不再忘记带伞


阅读 29

spring Boot手把手教学(9): 实战练习-爬虫获取天气信息并自动发邮件,出门不再忘记带伞

前情回顾:

  • Spring Boot手把手教学(1):如何生成优美的开篇banner.txt

  • Spring Boot手把手教学(2):使用yml多环境配置和创建多环境profile打包

  • Spring Boot手把手教学(3):从零配置logback日志

  • spring Boot手把手教学(4): mybatis-plus 代码生成器,自动帮你生成代码

  • spring Boot手把手教学(5): 定时任务

  • spring Boot手把手教学(6):发送邮件

  • spring Boot手把手教学(7): 抛弃try-catch, 如何优雅统一处理异常(包含404处理)

  • spring Boot手把手教学(8): 封装统一返回实体类

本篇文章,在前几篇文章的基础上,我们实战练习;

使用的知识点为:

  1. Controller
    Service
    serviceImpl
    entity
    mapper
    mapper xml
    
  2. 定时任务,定时获取最新天气数据和定时自动发送天气预报邮件

  3. 使用 jsoup
    写爬虫爬取天气预报数据,并存入mysql;

2、安装依赖

pom.xml

<!-- HTML解析器 -->
<dependency>
 <groupId>org.jsoup</groupId>
 <artifactId>jsoup</artifactId>
 <version>1.10.3</version>
</dependency>

<!-- 邮件 -->
<dependency>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-mail</artifactId>
</dependency>

<!-- thymeleaf 模板 -->
<dependency>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-thymeleaf</artifactId>
</dependency>

<dependency>
  <groupId>ognl</groupId>
  <artifactId>ognl</artifactId>
  <version>3.2.14</version>
</dependency>
复制代码

使用 jsoup
爬取页面,并做 HTML结构解析

3、mysql数据库建表

/*
 Navicat Premium Data Transfer

Source Server : local_mysql
Source Server Type : MySQL
Source Server Version : 80018
Source Host : localhost:3306
Source Schema : test

Target Server Type : MySQL
Target Server Version : 80018
File Encoding : 65001

Date: 19/06/2020 22:40:01
*/

SET NAMES utf8mb4;
SET FOREIGN_KEY_CHECKS = 0;

– Table structure for weather

weather<span>复制代码</span>

weather<span>复制代码</span>

id<span>复制代码</span>

name<span>复制代码</span>

status<span>复制代码</span>

date<span>复制代码</span>

max<span>复制代码</span>

min<span>复制代码</span>

create_time<span>复制代码</span>

id<span>复制代码</span>

date<span>复制代码</span>

date<span>复制代码</span>

SET FOREIGN_KEY_CHECKS = 1;

复制代码

<span>复制代码</span>

设置 id
为主键 PRIMARY KEY
date
为唯一约束 UNIQUE KEY
,这样的话,我每次定时爬取数据,存入数据库的时候,就能避免重复数据。

spring Boot手把手教学(9): 实战练习-爬虫获取天气信息并自动发邮件,出门不再忘记带伞

4、生成器生成代码

执行 com.scaffold.test.base.CodeGenerator
中的 main
方法:

spring Boot手把手教学(9): 实战练习-爬虫获取天气信息并自动发邮件,出门不再忘记带伞

如果有同学对生成器不知道怎么配置的,请前往前面的文章看一下;

spring Boot手把手教学(4): mybatis-plus 代码生成器,自动帮你生成代码

com.scaffold.test.entity.Weather:实体类

package com.scaffold.test.entity;

import com.baomidou.mybatisplus.annotation.IdType;
import com.baomidou.mybatisplus.annotation.TableId;
import lombok.Data;
import lombok.EqualsAndHashCode;

import java.io.Serializable;
import java.time.LocalDateTime;

/**
 * <p>
 * 实体类
 * </p>
 *
 * @author alex wong
 * @since 2020-06-18
 */
@Data
@EqualsAndHashCode(callSuper = false)
public class Weather implements Serializable {

    private static final long serialVersionUID = 1L;

    @TableId(value = "id", type = IdType.AUTO)
    private Integer id;

    private String name;

    private String status;

    private String date;

    // 最高气温
    private String max;

    // 最低气温
    private String min;

    private LocalDateTime createTime;

}

复制代码

com.scaffold.test.service.WeatherService: 服务类接口

package com.scaffold.test.service;

import com.baomidou.mybatisplus.extension.service.IService;
import com.scaffold.test.entity.Weather;
import org.jsoup.nodes.Document;

import java.util.List;

/**
 * <p>
 * 服务类
 * </p>
 *
 * @author alex wong
 * @since 2020-06-18
 */
public interface WeatherService extends IService<Weather> {

}

复制代码

com.scaffold.test.service.impl.WeatherServiceImpl: 服务实现类

package com.scaffold.test.service.impl;

import com.baomidou.mybatisplus.extension.service.impl.ServiceImpl;
import com.scaffold.test.entity.Weather;
import com.scaffold.test.mapper.WeatherMapper;
import com.scaffold.test.service.WeatherService;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;

import java.text.SimpleDateFormat;
import java.util.ArrayList;
import java.util.Date;
import java.util.List;

/**
 * <p>
 * 服务实现类
 * </p>
 *
 * @author alex wong
 * @since 2020-06-18
 */
@Service
public class WeatherServiceImpl extends ServiceImpl<WeatherMapper, Weather> implements WeatherService {

}

复制代码

com.scaffold.test.mapper.WeatherMapper: Mapper 接口

package com.scaffold.test.mapper;

import com.scaffold.test.entity.Weather;
import com.baomidou.mybatisplus.core.mapper.BaseMapper;

import java.util.List;

/**
 * <p>
 *  Mapper 接口
 * </p>
 *
 * @author alex wong
 * @since 2020-06-18
 */
public interface WeatherMapper extends BaseMapper<Weather> {

}

复制代码

src/main/resources/mapper/WeatherMapper.xml

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE mapper PUBLIC "-//mybatis.org//DTD Mapper 3.0//EN" "http://mybatis.org/dtd/mybatis-3-mapper.dtd">
<mapper namespace="com.scaffold.test.mapper.WeatherMapper">

    <!-- 通用查询映射结果 -->
    <resultMap id="BaseResultMap" type="com.scaffold.test.entity.Weather">
        <id column="id" property="id" />
        <result column="name" property="name" />
        <result column="status" property="status" />
        <result column="date" property="date" />
        <result column="max" property="max" />
        <result column="min" property="min" />
        <result column="create_time" property="createTime" />
    </resultMap>

    <!-- 通用查询结果列 -->
    <sql id="Base_Column_List">
        id, name, status, max, min, date
    </sql>

</mapper>

复制代码

com.scaffold.test.controller.WeatherController:

package com.scaffold.test.controller;

import com.scaffold.test.entity.Mail;
import com.scaffold.test.entity.Weather;
import com.scaffold.test.service.MailService;
import com.scaffold.test.service.WeatherService;
import com.scaffold.test.task.JobTask;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.scheduling.annotation.Async;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;
import org.thymeleaf.context.Context;
import org.thymeleaf.spring5.SpringTemplateEngine;

import javax.mail.MessagingException;
import java.text.SimpleDateFormat;
import java.util.Date;
import java.util.List;

@RestController
@RequestMapping("/weather")
public class WeatherController {

}
复制代码

5、制作爬虫

制作爬虫我们使用 Jsoup 获取远程URL地址的页面结构,并进行HTML解析;

这需要你对javascript有所了解,方法类似;

com.scaffold.test.controller.WeatherController

    // 获取七日天气数据
  @GetMapping("/weather/get")
    public void getDataFromHtml() {
        String url = "http://www.weather.com.cn/weather/101020100.shtml";
        try {
            Document document = Jsoup.connect(url).get();
            weatherService.getWeekList(document);
        } catch (Exception e) {
            log.error(e.getMessage());
        }
    }
复制代码

我们获取数据的来源是中国天气预报网站,如图。

spring Boot手把手教学(9): 实战练习-爬虫获取天气信息并自动发邮件,出门不再忘记带伞

获取完 document 之后,html解析是放在service里面实现的

com.scaffold.test.service.WeatherService

package com.scaffold.test.service;

import com.baomidou.mybatisplus.extension.service.IService;
import com.scaffold.test.entity.Weather;
import org.jsoup.nodes.Document;

import java.util.List;

/**
 * <p>
 * 服务类
 * </p>
 *
 * @author alex wong
 * @since 2020-06-18
 */
public interface WeatherService extends IService<Weather> {

    // HTML解析获取七日天气预报数据
    List<Weather> getWeekList(Document document);
}

复制代码

com.scaffold.test.service.impl.WeatherServiceImpl

HTML 解析需要你去网站查看页面HTML DOM结构,并查找获取你想要的数据的过程;

spring Boot手把手教学(9): 实战练习-爬虫获取天气信息并自动发邮件,出门不再忘记带伞
package com.scaffold.test.service.impl;

import com.baomidou.mybatisplus.extension.service.impl.ServiceImpl;
import com.scaffold.test.entity.Weather;
import com.scaffold.test.mapper.WeatherMapper;
import com.scaffold.test.service.WeatherService;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;

import java.text.SimpleDateFormat;
import java.util.ArrayList;
import java.util.Date;
import java.util.List;

/**
 * <p>
 * 服务实现类
 * </p>
 *
 * @author alex wong
 * @since 2020-06-18
 */
@Service
public class WeatherServiceImpl extends ServiceImpl<WeatherMapper, Weather> implements WeatherService {

    @Autowired
    private WeatherMapper weatherMapper;

    @Override
    public List<Weather> getWeekList(Document document) {

        ArrayList<Weather> weathers = new ArrayList<>();

        // 数据提取
        Elements sevenBox = document.getElementById("7d").getElementsByClass("t");
        Elements liColumns = sevenBox.get(0).getElementsByTag("li");
        for (Element column : liColumns) {
            Weather weather = new Weather();
            // 获取 name
            String name = column.getElementsByTag("h1").text().split("(")[1].split(")")[0];
            String day = column.getElementsByTag("h1").text().split("//D{1,2}")[0];
            String date = getDate(day);
            // 今日天气状态
            String status = column.getElementsByClass("wea").text();
            // 温度
            String tem = column.getElementsByClass("tem").text();
            String[] temArr = tem.split("//D{1,2}");
            // 最高气温
            String maxTem;
            // 最低气温
            String minTem;
            if (temArr.length == 2) {
                maxTem = tem.split("//D{1,2}")[0];
                minTem = tem.split("//D{1,2}")[1];
            } else {
                maxTem = tem.split("//D{1,2}")[0];
                minTem = tem.split("//D{1,2}")[0];
            }
            weather.setName(name);
            weather.setStatus(status);
            weather.setDate(date);
            weather.setMax(maxTem);
            weather.setMin(minTem);
            weathers.add(weather);
        }

        // 插入数据库
        for (Weather weather : weathers) {
            weatherMapper.insertWeather(weather);
        }

        return weathers;
    }

    // 获取对应日期
    public String getDate(String day) {
        Date date = new Date();
        //格式日期
        SimpleDateFormat dateFormat = new SimpleDateFormat("yyyy-MM-dd");
        date.setDate(Integer.parseInt(day));
        return dateFormat.format(date);
    }
}
复制代码

com.scaffold.test.mapper.WeatherMapper: mapper接口, 数据插入

package com.scaffold.test.mapper;

import com.scaffold.test.entity.Weather;
import com.baomidou.mybatisplus.core.mapper.BaseMapper;

import java.util.List;

/**
 * <p>
 *  Mapper 接口
 * </p>
 *
 * @author alex wong
 * @since 2020-06-18
 */
public interface WeatherMapper extends BaseMapper<Weather> {

    void insertWeather(Weather weather);
}
复制代码

Mapper XML

 <insert id="insertWeather">
   insert ignore into weather
   (name, status, date, max, min)
   values
   (#{name}, #{status}, #{date}, #{max}, #{min})
 </insert>
复制代码

接下来我们用POSTMAN,发送请求,手动调取数据

spring Boot手把手教学(9): 实战练习-爬虫获取天气信息并自动发邮件,出门不再忘记带伞
spring Boot手把手教学(9): 实战练习-爬虫获取天气信息并自动发邮件,出门不再忘记带伞

接下来,我们需要用定时任务定时获取天气实时信息,这样我们获取的数据才会有效和使用,一切源于实际应用,没有什么用也没必要做;

// 定时获取七日天气数据
@Async
@Scheduled(fixedRate = 60*60*1000)
public void getDataFromHtml() {
  String url = "http://www.weather.com.cn/weather/101020100.shtml";
  try {
    Document document = Jsoup.connect(url).get();
    weatherService.getWeekList(document);
  } catch (Exception e) {
    log.error(e.getMessage());
  }
}
复制代码

我们这里是设置 一小时
获取一次天气信息;

spring Boot手把手教学(9): 实战练习-爬虫获取天气信息并自动发邮件,出门不再忘记带伞

这样的话,我们就定时获取到数据了;

如果细心一点会发现有个问题,虽然这是按天为维度进行的数据统计,但是天气气象台会定时更新当天的数据,比如7:30更新数据,之后在当天的一个时间点还是会更新,而我们的SQL语句用的是insert into(加ignore, 是为了忽略警告,如图返回Updates: 0), 意味着只要当天date是存在的,就不会再次更新数据;

因此我们需要使用 mysql replace into , 这是 insert into 的增加版

`Mysql提供了``replace into``insert into` 的功能一样, 不同点在于:

  `如果发现表中已经有此行数据(根据主键或者唯一索引判断)则先删除此行数据,然后插入新的数据。否则,直接插入新数据`  
  不过有点要注意,`操作的表必须要有主键或者是唯一索引。因为要根据主键或者是唯一索引判断是否有重复数据;如果没有的话,replace into 会直接插入数据`

复制代码
spring Boot手把手教学(9): 实战练习-爬虫获取天气信息并自动发邮件,出门不再忘记带伞
  <insert id="insertWeather">
    --  insert ignore into weather
    replace into weather
    (name, status, date, max, min)
    values
    (#{name}, #{status}, #{date}, #{max}, #{min})
   </insert>
复制代码
spring Boot手把手教学(9): 实战练习-爬虫获取天气信息并自动发邮件,出门不再忘记带伞

id会递增,所以保证你的字段长度是要满足的;

spring Boot手把手教学(9): 实战练习-爬虫获取天气信息并自动发邮件,出门不再忘记带伞

其实还有很多要做,比如历史数据如何处理,是逻辑删除还是物理删除。我这里获取七日数据只做了数据日期过滤,取当天日期之后的数据;

6、发邮件

获取七日数据: src/main/resources/mapper/WeatherMapper.xml

<select id="selectAll" resultType="com.scaffold.test.entity.Weather">
  select
  <include refid="Base_Column_List"></include>
  from weather
  WHERE STR_TO_DATE(date, '%Y-%m-%d') >= CURRENT_DATE
</select>
复制代码

com.scaffold.test.service.WeatherService: 增加方法

public interface WeatherService extends IService<Weather> {

    // 获取数据
    List<Weather> selectAll();
}
复制代码

com.scaffold.test.service.impl.WeatherServiceImpl

@Service
public class WeatherServiceImpl extends ServiceImpl<WeatherMapper, Weather> implements WeatherService {

    @Override
    public List<Weather> selectAll() {
        return weatherMapper.selectAll();
    }

}
复制代码

com.scaffold.test.controller.WeatherController

    
    @Autowired
    private SpringTemplateEngine templateEngine;
 
  // 定时发送邮件
    @Async
    @GetMapping("post")
    public void sendMail() throws MessagingException {
        Context context = new Context();
        // 获取七日天气
        List<Weather> weathers = weatherService.selectAll();
        context.setVariable("resultList", weathers);
        String emailTemplate = templateEngine.process("weatherTemplate", context);

        //获取当前时间
        SimpleDateFormat simpleDateFormat = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");
        String currentTime = simpleDateFormat.format(new Date());

        // 邮件发送, 多人接收
    String[] addressList = {"****@qq.com", "***@qq.com"};
        for (String address : addressList) {
            Mail mail = new Mail();
            mail.setTo(address);
            mail.setSubject("天气预报小助手-" + currentTime);
            mail.setContent(emailTemplate);
            mailService.sendHtmlMail(mail);
        }
    }
复制代码

邮件模版:src/main/resources/templates/weatherTemplate.html

<!DOCTYPE html>
<html lang="en"
      xmlns="http://www.w3.org/1999/xhtml"
      xmlns:th="http://www.thymeleaf.org">
<head>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
    <meta name="viewport" content="width=device-width, initial-scale=1.0"/>
    <title>天气预报</title>
    <style>
        .title {
            text-align: center;
        }

        .table {
            width: 100%;
            border-collapse: collapse;
            border: 1px solid #eee;
            margin: 0 auto;
        }
        
        .table thead {
            background: #40B8FA;
            color: #fff;
        }

        .table td, .table th {
            height: 40px;
            line-height: 40px;
            text-align: center;
            white-space: nowrap;
        }


    </style>
</head>
<body>
<div class="title">
    <h1>
        上海最近7日天气预报
    </h1>
</div>
<table class="table">
    <thead>
    <tr>
        <th>日期</th>
        <th>天气状态</th>
        <th>最高气温</th>
        <th>最低气温</th>
    </tr>
    </thead>
    <tbody>
    <tr th:each="entries,stat:${resultList}">
        <td th:text="${entries['date']}"></td>
        <td th:text="${entries['status']}"></td>
        <td th:text="${entries['max']}"></td>
        <td th:text="${entries['min']}"></td>
    </tr>
    </tbody>

</table>
</body>
</html>
复制代码

测试发邮件 spring Boot手把手教学(9): 实战练习-爬虫获取天气信息并自动发邮件,出门不再忘记带伞
spring Boot手把手教学(9): 实战练习-爬虫获取天气信息并自动发邮件,出门不再忘记带伞

定时发送

    // 定时发送邮件
    @Async
//  @GetMapping("post")
    @Scheduled(fixedRate = 60*1000)
    public void sendMail() throws MessagingException {
        Context context = new Context();
        // 获取七日天气
        List<Weather> weathers = weatherService.selectAll();
        context.setVariable("resultList", weathers);
        String emailTemplate = templateEngine.process("weatherTemplate", context);

        //获取当前时间
        SimpleDateFormat simpleDateFormat = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");
        String currentTime = simpleDateFormat.format(new Date());

        // 邮件发送, 多人接收
        String[] addressList = {"***@qq.com", "***@qq.com"};
        for (String address : addressList) {
            Mail mail = new Mail();
            mail.setTo(address);
            mail.setSubject("天气预报小助手-" + currentTime);
            mail.setContent(emailTemplate);
            mailService.sendHtmlMail(mail);
        }
    }
复制代码

测试代码:一分钟发一次邮件

spring Boot手把手教学(9): 实战练习-爬虫获取天气信息并自动发邮件,出门不再忘记带伞

7、完整代码

com.scaffold.test.entity.Weather

package com.scaffold.test.entity;

import com.baomidou.mybatisplus.annotation.IdType;
import com.baomidou.mybatisplus.annotation.TableId;
import lombok.Data;
import lombok.EqualsAndHashCode;

import java.io.Serializable;
import java.time.LocalDateTime;

/**
 * <p>
 *
 * </p>
 *
 * @author alex wong
 * @since 2020-06-18
 */
@Data
@EqualsAndHashCode(callSuper = false)
public class Weather implements Serializable {

    private static final long serialVersionUID = 1L;

    @TableId(value = "id", type = IdType.AUTO)
    private Integer id;

    private String name;

    private String status;

    private String date;

    // 最高气温
    private String max;

    // 最低气温
    private String min;

    private LocalDateTime createTime;


}
复制代码

com.scaffold.test.controller.WeatherController

package com.scaffold.test.controller;


import com.scaffold.test.entity.Mail;
import com.scaffold.test.entity.Weather;
import com.scaffold.test.service.MailService;
import com.scaffold.test.service.WeatherService;
import com.scaffold.test.task.JobTask;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.scheduling.annotation.Async;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;
import org.thymeleaf.context.Context;
import org.thymeleaf.spring5.SpringTemplateEngine;

import javax.mail.MessagingException;
import java.text.SimpleDateFormat;
import java.util.Date;
import java.util.List;

@RestController
@RequestMapping("/weather")
public class WeatherController {

    private static final Logger log = LoggerFactory.getLogger(JobTask.class);

    @Autowired
    private WeatherService weatherService;

    @Autowired
    private MailService mailService;

    @Autowired
    private SpringTemplateEngine templateEngine;

    // 定时获取七日天气数据
    @Async
    @Scheduled(fixedRate = 60*60*1000)
//    @GetMapping("/get")
    public void getDataFromHtml() {
        String url = "http://www.weather.com.cn/weather/101020100.shtml";
        log.info("-------定时获取七日天气数据--------");
        try {
            Document document = Jsoup.connect(url).get();
            weatherService.getWeekList(document);
        } catch (Exception e) {
            log.error(e.getMessage());
        }
    }

    // 定时发送邮件
    @Async
//    @GetMapping("post")
    @Scheduled(fixedRate = 60*1000)
    public void sendMail() throws MessagingException {
        Context context = new Context();
        // 获取七日天气
        List<Weather> weathers = weatherService.selectAll();
        context.setVariable("resultList", weathers);
        String emailTemplate = templateEngine.process("weatherTemplate", context);

        //获取当前时间
        SimpleDateFormat simpleDateFormat = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");
        String currentTime = simpleDateFormat.format(new Date());

        // 邮件发送, 多人接收
        String[] addressList = {"***@qq.com", "****@qq.com"};
        for (String address : addressList) {
            Mail mail = new Mail();
            mail.setTo(address);
            mail.setSubject("天气预报小助手-" + currentTime);
            mail.setContent(emailTemplate);
            mailService.sendHtmlMail(mail);
        }
    }
}
复制代码

com.scaffold.test.service.WeatherService

package com.scaffold.test.service;

import com.baomidou.mybatisplus.extension.service.IService;
import com.scaffold.test.entity.Weather;
import org.jsoup.nodes.Document;

import java.util.List;

/**
 * <p>
 * 服务类
 * </p>
 *
 * @author alex wong
 * @since 2020-06-18
 */
public interface WeatherService extends IService<Weather> {

    // HTML解析获取七日天气预报数据
    List<Weather> getWeekList(Document document);

    // 获取数据
    List<Weather> selectAll();
}

复制代码

com.scaffold.test.service.impl.WeatherServiceImpl

package com.scaffold.test.service.impl;

import com.baomidou.mybatisplus.extension.service.impl.ServiceImpl;
import com.scaffold.test.entity.Weather;
import com.scaffold.test.mapper.WeatherMapper;
import com.scaffold.test.service.WeatherService;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;

import java.text.SimpleDateFormat;
import java.util.ArrayList;
import java.util.Date;
import java.util.List;

/**
 * <p>
 * 服务实现类
 * </p>
 *
 * @author alex wong
 * @since 2020-06-18
 */
@Service
public class WeatherServiceImpl extends ServiceImpl<WeatherMapper, Weather> implements WeatherService {

    @Autowired
    private WeatherMapper weatherMapper;

    @Override
    public List<Weather> getWeekList(Document document) {

        ArrayList<Weather> weathers = new ArrayList<>();

        // 数据提取
        Elements sevenBox = document.getElementById("7d").getElementsByClass("t");
        Elements liColumns = sevenBox.get(0).getElementsByTag("li");
        for (Element column : liColumns) {
            Weather weather = new Weather();
            // 获取 name
            String name = column.getElementsByTag("h1").text().split("(")[1].split(")")[0];
            String day = column.getElementsByTag("h1").text().split("//D{1,2}")[0];
            String date = getDate(day);
            // 今日天气状态
            String status = column.getElementsByClass("wea").text();
            // 温度
            String tem = column.getElementsByClass("tem").text();
            String[] temArr = tem.split("//D{1,2}");
            // 最高气温
            String maxTem;
            // 最低气温
            String minTem;
            if (temArr.length == 2) {
                maxTem = tem.split("//D{1,2}")[0];
                minTem = tem.split("//D{1,2}")[1];
            } else {
                maxTem = tem.split("//D{1,2}")[0];
                minTem = tem.split("//D{1,2}")[0];
            }
            weather.setName(name);
            weather.setStatus(status);
            weather.setDate(date);
            weather.setMax(maxTem);
            weather.setMin(minTem);
            weathers.add(weather);
        }

        // 插入数据库
        for (Weather weather : weathers) {
            weatherMapper.insertWeather(weather);
        }

        return weathers;
    }

    @Override
    public List<Weather> selectAll() {
        return weatherMapper.selectAll();
    }

    // 获取对应日期
    public String getDate(String day) {
        Date date = new Date();
        //格式日期
        SimpleDateFormat dateFormat = new SimpleDateFormat("yyyy-MM-dd");
        date.setDate(Integer.parseInt(day));
        return dateFormat.format(date);
    }
}

复制代码

com.scaffold.test.mapper.WeatherMapper

package com.scaffold.test.mapper;

import com.scaffold.test.entity.Weather;
import com.baomidou.mybatisplus.core.mapper.BaseMapper;

import java.util.List;

/**
 * <p>
 *  Mapper 接口
 * </p>
 *
 * @author alex wong
 * @since 2020-06-18
 */
public interface WeatherMapper extends BaseMapper<Weather> {

    void insertWeather(Weather weather);

    List<Weather> selectAll();
}

复制代码

src/main/resources/mapper/WeatherMapper.xml

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE mapper PUBLIC "-//mybatis.org//DTD Mapper 3.0//EN" "http://mybatis.org/dtd/mybatis-3-mapper.dtd">
<mapper namespace="com.scaffold.test.mapper.WeatherMapper">

    <!-- 通用查询映射结果 -->
    <resultMap id="BaseResultMap" type="com.scaffold.test.entity.Weather">
        <id column="id" property="id" />
        <result column="name" property="name" />
        <result column="status" property="status" />
        <result column="date" property="date" />
        <result column="max" property="max" />
        <result column="min" property="min" />
        <result column="create_time" property="createTime" />
    </resultMap>

    <!-- 通用查询结果列 -->
    <sql id="Base_Column_List">
        id, name, status, max, min, date
    </sql>

    <insert id="insertWeather">
--         insert ignore into weather
        replace into weather
        (name, status, date, max, min)
        values
        (#{name}, #{status}, #{date}, #{max}, #{min})

    </insert>
    <select id="selectAll" resultType="com.scaffold.test.entity.Weather">
        select
        <include refid="Base_Column_List"></include>
        from weather
        WHERE STR_TO_DATE(date, '%Y-%m-%d') >= CURRENT_DATE
    </select>

</mapper>

复制代码

以上代码是基于之前文章的知识点实现的,如果有不清楚的同学,请前往前面的文章查看,这里不能一一详细说明;

码字不易,喜欢的点个赞。

本文使用 mdnice
排版

原文 

https://juejin.im/post/5eed852fe51d4573f22eb56d

本站部分文章源于互联网,本着传播知识、有益学习和研究的目的进行的转载,为网友免费提供。如有著作权人或出版方提出异议,本站将立即删除。如果您对文章转载有任何疑问请告之我们,以便我们及时纠正。

PS:推荐一个微信公众号: askHarries 或者qq群:474807195,里面会分享一些资深架构师录制的视频录像:有Spring,MyBatis,Netty源码分析,高并发、高性能、分布式、微服务架构的原理,JVM性能优化这些成为架构师必备的知识体系。还能领取免费的学习资源,目前受益良多

转载请注明原文出处:Harries Blog™ » spring Boot手把手教学(9): 实战练习-爬虫获取天气信息并自动发邮件,出门不再忘记带伞

赞 (0)
分享到:更多 ()

评论 0

  • 昵称 (必填)
  • 邮箱 (必填)
  • 网址