转载

Littleproxy的使用

LittleProxy是一个用Java编写的高性能HTTP代理，它基于Netty事件的网络库之上。它非常稳定，性能良好，并且易于集成到的项目中。

项目页面： https://github.com/adamfisk/LittleProxy

这里介绍几个简单的应用，其它复杂的应用都是可以基于这几个应用进行改造。

按域名或者url进行拦截和过滤
修改http头，修改请求参数
修改返回Response数据
中间人代理，截取https的数据

前置知识

因为代理库是基于netty事件驱动，所以需要对netty的原理有所了解

因为是对http协议进行处理，所以需要了解 io.netty.handler.codec.http 包下的类。

因为效率，大部分数据是由 ByteBuf 进行管理的，所以需要了解 ByteBuf 相关操作。

io.netty.handler.codec.http 包的相关介绍

主要接口图：

HttpObject
- httpContent(http协议体的抽象，比如POST数据的体，和响应数据的体)
  - LastHttpContent
- HttpMessage(http协议头的抽象，包含请求头和响应头)
  - FullHttpMessage(也继承于LastHttpContent)
  - HttpRequest
    - FullHttpRequest(也继承于FullHttpMessage)
  - HttpResponse
    - FullHttpResponse(也继承于FullHttpMessage)

主要类：

类主要是对上面接口的实现

DefaultHttpObject
- DefautlHttpContent
  - DefaultLastHttpContent
- DefaultHttpMessage
  - DefaultHttpRequest
    - DefaultFullHttpRequest
  - DefaultHttpResponse
    - DefaultFullHttpResponse

更多可以参考API文档 https://netty.io/4.1/api/index.html

辅助类 io.netty.handler.codec.http.HttpHeaders.Names

io.netty.buffer.ByteBuf 的相关使用

主要使用是 Unpooled 和 ByteBufUtil

Unpooled.wrappedBuffe
toString(Charset.forName("UTF-8")
ByteBufUtil.prettyHexDump(buf);

基本流程代码

示例代码

public static void main(String[] args) {
  HttpProxyServer server = DefaultHttpProxyServer.bootstrap().withPort(8181)
      .withFiltersSource(new HttpFiltersSourceAdapter() {
        @Override
        public HttpFilters filterRequest(HttpRequest req, ChannelHandlerContext ct) {
          return new HttpFiltersAdapter(req) {

            @Override
            public HttpResponse clientToProxyRequest(HttpObject httpObject) {
              System.out.println("1-" + httpObject);
              return super.clientToProxyRequest(httpObject);
            }

            @Override
            public HttpResponse proxyToServerRequest(HttpObject httpObject) {
              System.out.println("2-" + httpObject);
              return super.proxyToServerRequest(httpObject);
            }

            @Override
            public HttpObject serverToProxyResponse(HttpObject httpObject) {
              System.out.println("3-" + httpObject);
              return super.serverToProxyResponse(httpObject);
            }

            @Override
            public HttpObject proxyToClientResponse(HttpObject httpObject) {
              System.out.println("4-" + httpObject);
              return super.proxyToClientResponse(httpObject);
            }
          };
        }
      }).start();
}

代码分析：

启动代理类
实现 HttpFiltersSourceAdapter 的 filterRequest 函数
实现 HttpFiltersAdapter 的4个关键性函数，并打印日志

HttpFiltersAdapter 分别是：

clientToProxyRequest(默认返回null，表示不拦截，若返回数据，则不再经过P2S和S2P。这里可以修改数据)
proxyToServerRequest(这里的原理与上面一条一样，基本原封不动)
serverToProxyResponse(这里默认返回传入参数，可以做一定的修改)
proxyToClientResponse(与上面一条类似)

这个流程符合普通代理的流程。

请求数据 C -> P -> S，

响应数据 S -> P -> C

预期代码输出会是 1,2,3,4 按顺序执行

但实际运行结果(省略若干非关键性信息)：

1-DefaultHttpRequest(decodeResult: success, version: HTTP/1.1)
2-DefaultHttpRequest(decodeResult: success, version: HTTP/1.1)
1-EmptyLastHttpContent
2-EmptyLastHttpContent
3-DefaultHttpResponse(decodeResult: success, version: HTTP/1.1)
4-DefaultHttpResponse(decodeResult: success, version: HTTP/1.1)
3-DefaultHttpContent(data: SlicedAbstractByteBuf(ridx: 0, widx: 624, cap: 624/624, ), )
4-DefaultHttpContent(data: SlicedAbstractByteBuf(ridx: 0, widx: 624, cap: 624/624, : ), )
3-DefaultHttpContent(data: SlicedAbstractByteBuf(ridx: 0, widx: 1024, cap: 1024/1024, : , )
4-DefaultHttpContent(data: SlicedAbstractByteBuf(ridx: 0, widx: 1024, cap: 1024/1024, : ), )
3-DefaultLastHttpContent(data: SlicedAbstractByteBuf(ridx: 0, widx: 733, cap: 733/733, : ), )
4-DefaultLastHttpContent(data: SlicedAbstractByteBuf(ridx: 0, widx: 733, cap: 733/733, : ), )

可以看出：

Last-xx

修改请求参数

比如这里实现了把每次百度搜索的关键字加一个前缀的功能。

主要原理是修改 DefaultHttpRequest 的url中所带的参数（只能修改GET方式的参数)

如果需要修改POST的内容，同样的原理，不过是要修改Request的内容体。

@Override
public HttpResponse proxyToServerRequest(HttpObject httpObject) {
  if(httpObject instanceof DefaultHttpRequest )
  {
    DefaultHttpRequest dhr = (DefaultHttpRequest)httpObject;
    String url = dhr.getUri();
    String host = dhr.headers().get(HttpHeaders.Names.HOST);
    String method = dhr.getMethod().toString();
    if(method.equals("GET") && host.equals("www.baidu.com"))
    {
      try {
        dhr.setUri(replaceParam(url));
      } catch (Exception e) {
        e.printStackTrace();
      }
    }
  }
  return null;
}

replaceParam函数就是把搜索的关键字提取出来，并添加前缀，然后拼接成新url。

static public String replaceParam(String url) throws Exception
{
  String add_str = "你好 ";
  String paramKey = "&wd=";
  int wd_start = url.indexOf(paramKey);
  int wd_end = -1;
  if(wd_start != -1)
  {
    wd_end = url.indexOf("&",wd_start+paramKey.length());
  }
  if(wd_end !=-1)
  {
    String key = url.substring(wd_start+paramKey.length(), wd_end);
    String new_key = URLEncoder.encode(add_str,"UTF-8") + key;
    String new_url = url.substring(0,wd_start+paramKey.length())
        + new_key + url.substring(wd_end,url.length());
    return new_url;
  }
  return url;
}

拦截指定域名或者URL

按上面基础代码重写clientToProxyRequest或者proxyToServerRequest。

如果是指定域名，如 hm.baidu.com 就返回一个空的response。这个请求就不会继续请求服务端。

如果是多个域名，使用set来存储。如果是需要按后缀，可以用后缀树。

@Override
public HttpResponse proxyToServerRequest(HttpObject httpObject) {
  if(httpObject instanceof DefaultHttpRequest )
  {
    DefaultHttpRequest dhr = (DefaultHttpRequest)httpObject;
    String url = dhr.getUri();
    String host = dhr.headers().get(HttpHeaders.Names.HOST);
    String method = dhr.getMethod().toString();
    if("hm.baidu.com".endsWith(host) && !method.equals("CONNECT"))
    {
      return new DefaultFullHttpResponse(HttpVersion.HTTP_1_1, HttpResponseStatus.OK);
    }
    if(!method.equals("CONNECT"))
    {
      System.out.println(method+ " http://"+host+url);
    }  
  }
  return null;
}

修改返回内容

修改内容会涉及几个很麻烦的事

Transfer-Encoding: chunked

对于压缩

简单的做法就是修改请求报文，让请求头不支持压缩算法，服务器就不会对内容进行压缩。

复杂的办法就是记录响应头，老实进行解压。

解码之后再修改内容，内容修改好之后，再进行压缩。

对于chunked

没有什么好的办法，在Response中去掉标识，然后按次拼接，服务器来的块，拼接好，修改好后，一次返回给客户端。

代码很长就不贴出来了。

但写 proxyToClientResponse 函数中拼报文时，有几个注意事项：

不能直接返回null（客户端会报错），要返回 return new DefaultHttpContent(Unpooled.EMPTY_BUFFER); 一个空的response。
httpObject的类型，在非chunked是几个 DefaultHttpContent ，最后一个 DefaultLastHttpContent ,判断语句Lastxx要写在前面，因为后面是前面的子类（先判断范围小的，再判断范围大的）。
chunked的方式下是几个 DefaultHttpContent ,最后一个 LastHttpContent ,写法同上。
一个请求会对应 HttpFiltersAdapter 一个实例，状代码可以写成类成员变量。

中间人代理

中间人代理可以在授信设备安装证书后，截取https流量。

littleproxy实现中间人的方式很简单，实现 MitmManager 接口，在启动类中调用 withManInTheMiddle 方法。

MitmManager 接口要求返回 SSLEngine 对象,实现 SslEngineSource 接口。

SSLEngine 对象是要通过 SSLContext 调用 createSSLEngine

而 SSLContext 的初始化，需要证书文件，又涉及CA认证签名体系。

然后https流量会先进行解包，和普通http一样，可以通过上面的手段进行捕获，然后再用自己的证书进行签名

目前使用openssl实现了一个版本。

启动器

public static void main(String[] args) {

  HttpProxyServer server = DefaultHttpProxyServer.bootstrap().withPort(8181).withTransparent(true)
      .withManInTheMiddle(new MitmManager() {
        private HashMap<String, SslEngineSource> sslEngineSources = new HashMap<String, SslEngineSource>();
        @Override
        public SSLEngine serverSslEngine(String peerHost, int peerPort) {
          if (!sslEngineSources.containsKey(peerHost)) {
            sslEngineSources.put(peerHost, new FclSslEngineSource(peerHost, peerPort));
          }
          return sslEngineSources.get(peerHost).newSslEngine();
        }
        @Override
        public SSLEngine serverSslEngine() {
          return null;
        }
        @Override
        public SSLEngine clientSslEngineFor(HttpRequest httpRequest, SSLSession serverSslSession) {
          return sslEngineSources.get(serverSslSession.getPeerHost()).newSslEngine();
        }

      }).withFiltersSource(new HttpFiltersSourceAdapter() {
        @Override
        public HttpFilters filterRequest(HttpRequest req, ChannelHandlerContext ct) {
          return new HttpFiltersAdapter(req) {
            @Override
            public HttpResponse proxyToServerRequest(HttpObject httpObject) {
              if (httpObject instanceof DefaultHttpRequest) {
                DefaultHttpRequest dhr = (DefaultHttpRequest) httpObject;
                String url = dhr.getUri();
                String method = dhr.getMethod().toString();
                String host = dhr.headers().get(Names.HOST);
                System.out.println(method + " " + host + url);
              }
              return super.proxyToServerRequest(httpObject);
            }

          };
        }
      }).start();
}

SslEngineSource实现类

public class FclSslEngineSource implements SslEngineSource {

  private String host;
  private int port;
  private SSLContext sslContext;

  private final File keyStoreFile;// 当前域名的JKS文件

  private String dir = "cert/";// 证书目录文件

  private static final String PASSWORD = "123123";
  private static final String PROTOCOL = "TLS";

  public static String CA_KEY = "MITM_CA.key";
  public static String CA_CRT = "MITM_CA.crt";

  
  public FclSslEngineSource(String peerHost, int peerPort) {
    this.host = peerHost;
    this.port = peerPort;
    this.keyStoreFile = new File(dir + host + ".jks");
    initCA();
    initializeKeyStore();
    initializeSSLContext();
  }

  @Override
  public SSLEngine newSslEngine() {
    SSLEngine sslengine = sslContext.createSSLEngine(host, port);
    return sslengine;
  }

  @Override
  public SSLEngine newSslEngine(String peerHost, int peerPort) {
    SSLEngine sslengine = sslContext.createSSLEngine(host, port);
    return sslengine;
  }

  public void initCA() {
    if (!new File(CA_CRT).exists()) {
      // 如果不存在，就创建证书
      // 生成证书
      nativeCall("openssl", "genrsa", "-out", CA_KEY, "2048");
      // 生成CA证书
      nativeCall("openssl", "req", "-x509", "-new", "-nodes", "-key", CA_KEY, "-subj", "/"/CN=NOT_TRUST_CA/"",
          "-days", "365", "-out", CA_CRT);
    }
  }

  private void initializeKeyStore() {
    // 存在证书就不用再生成了
    if (keyStoreFile.isFile()) {
      return;
    }

    // 生成站点key
    nativeCall("openssl", "genrsa", "-out", dir + host + ".key", "2048");
    // 生成待签名证书
    nativeCall("openssl", "req", "-new", "-key", dir + host + ".key", "-subj", "/"/CN=" + host + "/"", "-out",
        dir + host + ".csr");
    // 用ca进行签名
    nativeCall("openssl", "x509", "-req", "-days", "30", "-in", dir + host + ".csr", "-CA", CA_CRT, "-CAkey",
        CA_KEY, "-CAcreateserial", "-out", dir + host + ".crt");
    // 把crt导成p12
    nativeCall("openssl", "pkcs12", "-export", "-clcerts", "-password", "pass:" + PASSWORD, "-in",
        dir + host + ".crt", "-inkey", dir + host + ".key", "-out", dir + host + ".p12");
    // 把p12导成jks
    nativeCall("keytool", "-importkeystore", "-srckeystore", dir + host + ".p12", "-srcstoretype", "pkcs12",
        "-destkeystore", dir + host + ".jks", "-deststoretype", "jks", "-srcstorepass", PASSWORD,
        "-deststorepass", PASSWORD);
    ;

  }

  private void initializeSSLContext() {
    String algorithm = Security.getProperty("ssl.KeyManagerFactory.algorithm");
    algorithm = algorithm == null ? "SunX509" : algorithm;
    try {
      final KeyStore ks = KeyStore.getInstance("JKS");
      ks.load(new FileInputStream(keyStoreFile), PASSWORD.toCharArray());

      // Set up key manager factory to use our key store
      final KeyManagerFactory kmf = KeyManagerFactory.getInstance(algorithm);
      kmf.init(ks, PASSWORD.toCharArray());

      TrustManager[] trustManagers = new TrustManager[] { new X509TrustManager() {
        // TrustManager that trusts all servers
        @Override
        public void checkClientTrusted(X509Certificate[] arg0, String arg1) throws CertificateException {
        }

        @Override
        public void checkServerTrusted(X509Certificate[] arg0, String arg1) throws CertificateException {
        }

        @Override
        public X509Certificate[] getAcceptedIssuers() {
          return null;
        }
      } };

      KeyManager[] keyManagers = kmf.getKeyManagers();

      // Initialize the SSLContext to work with our key managers.
      sslContext = SSLContext.getInstance(PROTOCOL);
      sslContext.init(keyManagers, trustManagers, null);
    } catch (final Exception e) {
      throw new Error("Failed to initialize the server-side SSLContext", e);
    }

  }

  private String nativeCall(final String... commands) {
    final ProcessBuilder pb = new ProcessBuilder(commands);
    try {
      final Process process = pb.start();
      final InputStream is = process.getInputStream();
      return IOUtils.toString(is);
    } catch (final IOException e) {
      e.printStackTrace(System.out);
      return "";
    }
  }
}