ElasticSearch

es是一个开源的可拓展的全文检索引擎服务器,它可以近乎实时的存储,检索数据;本身拓展性好 ,可以拓展到上百台服务器,处理pb级别的数据.es使用java开发并使用lucene作为其核心来实现索引和搜索的功能, 但是它通过简单的RestfulAPI和javaAPI来隐藏lucene的复杂性,从而让全文检索变得简单 **全文检索**:全文检索是利用倒排索引技术对需要搜索的数据进行处理,然后提供快速匹配的技术. 其实全文检索还有另外一种专业定义,先创建索引然后对索引进行所搜的过程,就是全文检索

github地址: https://github.com/lishanglei/elasticsearch.git

es,kibana,ik分词器可以从百度网盘下载:

https://pan.baidu.com/s/1Ozx4P-jZtxNVfXy8fBP1bA**

提取码: xli6

1.ES的安装

1.1创建用户(es的安装不允许使用root用户)

useradd elastic passwd elastic

1.2(root用户)为该用户增加权限

vim /etc/security/limits.conf #可打开的文件描述符的最大数(软限制) *soft nofile 65536 #可打开的文件描述符的最大数(硬限制) *hard nofile 131072 #单个用户可用的最大进程数(软限制) *soft nproc 4096 #单个用户可用的最大进程数量(硬限制) *hard nproc 4096

1.3(root用户)修改es用户的最大虚拟内存,不能小于262144

vim /etc/sysctl.conf **增加内容:** vm.max_map_count=262144 **执行命令:** sysctl -p

1.4(root)增加elastic用户对jvm.options的文件权限

chown -R elastic:elastic /usr/local/elastic/elastic/config/jvm.options

1.5切换用户

su - elastic

1.6修改elasticsearch.yml文件

pah.data: /usr/local/es/es/data path.logs: /usr/local/es/es/logs network.host: 0.0.0.0

1.7启动

cd bin ./elasticsearch -d

1.8 验证

在浏览器输入 127.0.0.1:9200

2.kibana安装

2.1下载

wget https://artifacts.elastic.co/downloads/kibana/kibana-6.8.8-linux-x86_64.tar.gz

2.2解压

tar zxvf tar zxvf kibana-6.8.8-linux-x86_64.tar.gz

2.3重命名

mv kibana-6.8.8-linux-x86_64 kibana

2.4修改配置

cd config vim kibana.yml server.host: "0.0.0.0" il8n.locale: "zh-CN"

2.5启动

./kibana

2.6验证

127.0.0.1:5601

3.IK分词器

将ik分词器解压到elasticsearch目录下的plugin目录下,并命名文件夹名为 analysis-ik

4.ElasticSearch相关概念

ElasticSearch是文件存储,是一种面向文档型数据库,一条数据在这里就是一个文档,用JSON作为文档序列化的格式 - 索引库indexs ----------------------------database数据库 - 类型type--------------------------------table数据表 - 文档document----------------------------row行 - 字段field--------------------------------columns列 - 映射配置mappings-------------------------表结构

一些简单的命令

创建索引库

PUT /索引库名称

查看索引库

GET /索引库名称

删除索引库

DELETE /索引库名称

创建映射字段

PUT /索引库名称/_mapping/类型名称(表名){ "properties":{ ​ "title":{ "type":"text", //string类型 ​ "analyzer":"ik_max_word" //使用分词器 ​ }, ​ "subtitle":{ "type":"text", "analyzer":"ik_max_word" ​ }, ​ "images":{ "type":"keyword", //类型为关键词 不进行分词 "index":"false" //不创建索引 默认为ture ​ }, ​ "price":{ ​ "type":"float" ​ } }

查看表映射

GET /索引库名称/-mapping/类型名称

一次性创建索引库

PUT /heima{ ​ "settings":{}, ​ "mappings":{ ​ "goods":{ ​ "properties":{ ​ "title":{ ​ "tyepe":"test", ​ "analyzer":"ik_max_word" ​ } ​ } ​ } ​ } }

新增文档并随即生成id(新增一条数据)

5.java整合ElasticSearch

5.1依赖

<dependencies> <!--rest-high-level-client--> <dependency> <groupId>org.elasticsearch.client</groupId> <artifactId>elasticsearch-rest-high-level-client</artifactId> <version>6.8.8</version> </dependency> <dependency> <groupId>com.alibaba</groupId> <artifactId>fastjson</artifactId> <version>1.2.58</version> </dependency> <!--es依赖--> <dependency> <groupId>org.elasticsearch</groupId> <artifactId>elasticsearch</artifactId> <version>6.8.8</version> </dependency> <dependency> <groupId>org.projectlombok</groupId> <artifactId>lombok</artifactId> <version>1.18.8</version> <scope>provided</scope> </dependency> </dependencies>

5.2实体类

@Data @AllArgsConstructor @NoArgsConstructor public class Product { private Long id; //标题 private String title; //分类 private String category; //品牌 private String brand; //价格 private Double price; //图片地址 private String images; }

5.3测试实力类

package com.learn.es; import com.alibaba.fastjson.JSON; import com.learn.es.pojo.Product; import org.apache.http.HttpHost; import org.elasticsearch.action.bulk.BulkRequest; import org.elasticsearch.action.bulk.BulkResponse; import org.elasticsearch.action.delete.DeleteRequest; import org.elasticsearch.action.delete.DeleteResponse; import org.elasticsearch.action.get.GetRequest; import org.elasticsearch.action.get.GetResponse; import org.elasticsearch.action.index.IndexRequest; import org.elasticsearch.action.index.IndexResponse; import org.elasticsearch.action.search.SearchRequest; import org.elasticsearch.action.search.SearchResponse; import org.elasticsearch.action.update.UpdateRequest; import org.elasticsearch.action.update.UpdateResponse; import org.elasticsearch.client.RequestOptions; import org.elasticsearch.client.RestClient; import org.elasticsearch.client.RestHighLevelClient; import org.elasticsearch.common.text.Text; import org.elasticsearch.common.xcontent.XContentType; import org.elasticsearch.index.query.QueryBuilder; import org.elasticsearch.index.query.QueryBuilders; import org.elasticsearch.search.SearchHit; import org.elasticsearch.search.builder.SearchSourceBuilder; import org.elasticsearch.search.fetch.subphase.highlight.HighlightBuilder; import org.elasticsearch.search.fetch.subphase.highlight.HighlightField; import org.elasticsearch.search.sort.SortOrder; import org.junit.After; import org.junit.Before; import org.junit.Test; import java.io.IOException; import java.util.Map; public class Test01 { private RestHighLevelClient client; private RequestOptions requestOptions = RequestOptions.DEFAULT; /** * 初始化连接 */ @Before public void init() { client = new RestHighLevelClient( RestClient.builder( new HttpHost("192.168.5.62", 9200, "http") ) ); } /** * 关闭连接 */ @After public void close() { if (client != null) { try { client.close(); } catch (IOException e) { e.printStackTrace(); } } } /** * 新增Doc */ @Test public void addDoc() { //构造请求数据 Product product = new Product(2L, "大米手机", "手机", "小米", 2699.00, "http://images.leyou.com/123132.jpg"); //转换为json格式数据 String json = JSON.toJSONString(product); //准备请求对象 IndexRequest indexRequest = new IndexRequest("heima", "product", "1"); //将请求体封装到请求对象 indexRequest.source(json, XContentType.JSON); //发送请i去 try { IndexResponse index = client.index(indexRequest, RequestOptions.DEFAULT); System.out.println("响应结果: " + JSON.toJSONString(index)); } catch (IOException e) { e.printStackTrace(); } } /** * 查询doc */ @Test public void getDoc() { //声明id参数 String id = "1"; //构建请求对象 GetRequest getRequest = new GetRequest("heima", "product", id); //发送请求 try { GetResponse getResponse = client.get(getRequest, RequestOptions.DEFAULT); System.out.println("响应结果: " + JSON.toJSONString(getResponse)); } catch (IOException e) { e.printStackTrace(); } } /** * 根据id修改doc */ @Test public void updateDoc() throws IOException { //构建修改请求对象 UpdateRequest updateRequest = new UpdateRequest("heima", "product", "1"); //构建修改数据 Product product = new Product(1L, "超米手机", "手机", "小米", 2899.00, "http://images.leyou.com/123132.jpg"); String json = JSON.toJSONString(product); //封装请求体至请求对象 updateRequest.doc(json, XContentType.JSON); //发送请求 UpdateResponse response = client.update(updateRequest, RequestOptions.DEFAULT); System.out.println("响应结果:" + JSON.toJSONString(response)); } /** * 根据id删除doc */ @Test public void deleteDoc() throws IOException { //构建删除对象 DeleteRequest deleteRequest = new DeleteRequest("heima", "product", "3"); //调用方法进行数据通信 DeleteResponse delete = client.delete(deleteRequest, requestOptions); System.out.println("响应结果: " + delete); } /** * 批量新增doc */ @Test public void bulkAddDoc() throws IOException { //构建批量新增对象 BulkRequest bulkRequest = new BulkRequest(); for (int i = 3; i < 9; i++) { //构造请求数据 Product product = new Product(new Long(i), "大米手机" + i, "手机", "小米", 2699.00 + i, "http://images.leyou.com/123132.jpg"); //转换为json格式数据 String json = JSON.toJSONString(product); //准备请求对象 IndexRequest indexRequest = new IndexRequest("heima", "product", String.valueOf(i)); indexRequest.source(json, XContentType.JSON); //添加数据 bulkRequest.add(indexRequest); } //发送数据 BulkResponse bulk = client.bulk(bulkRequest, RequestOptions.DEFAULT); System.out.println("响应结果:" + bulk); } /** * match查询doc */ @Test public void matchDoc() throws IOException { //1.构建SearchRequest请求对象,指定索引库 SearchRequest searchRequest = new SearchRequest("heima"); //2.构建SearchSourceBuilder查询对象 SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder(); //3.构建QueryBuilder对象指定查询方式和查询条件 QueryBuilder queryBuilder = QueryBuilders.matchQuery("title", "大米"); //4.将queryBuilder对象设置到SearchSourceBuilder中 searchSourceBuilder.query(queryBuilder); //5.将SearchSourceBuilder查询对象封装到请求对象SearchRequest中 searchRequest.source(searchSourceBuilder); //6.调用方法进行数据通信 SearchResponse response = client.search(searchRequest, RequestOptions.DEFAULT); //7.输出结果 System.out.println("响应结果: " + response); SearchHit[] hits = response.getHits().getHits(); for (int i = 0; i < hits.length; i++) { String sourceAsString = hits[i].getSourceAsString(); System.out.println("查询结果:" + sourceAsString); } } /** * match All查询 */ @Test public void matchAllDoc() throws IOException { //1.构建SearchRequest请求对象,指定索引库 SearchRequest searchRequest = new SearchRequest("heima"); //2.构建SearchSourceBuilder查询对象 SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder(); //3.构建QueryBuilder对象指定查询方式和查询条件 QueryBuilder queryBuilder = QueryBuilders.matchAllQuery(); //4.将queryBuilder对象设置到SearchSourceBuilder中 searchSourceBuilder.query(queryBuilder); //5.将SearchSourceBuilder查询对象封装到请求对象SearchRequest中 searchRequest.source(searchSourceBuilder); //6.调用方法进行数据通信 SearchResponse response = client.search(searchRequest, RequestOptions.DEFAULT); //7.输出结果 System.out.println("响应结果: " + response); SearchHit[] hits = response.getHits().getHits(); for (int i = 0; i < hits.length; i++) { String sourceAsString = hits[i].getSourceAsString(); System.out.println("查询结果:" + sourceAsString); } } /** * source过滤(作业) */ @Test public void sourceSearch() throws IOException { //1.构建SearchRequest请求对象,指定索引库 SearchRequest searchRequest = new SearchRequest("heima"); //2.构建SearchSourceBuilder查询对象 SearchSourceBuilder sourceBuilder = new SearchSourceBuilder(); //3.构建QueryBuilder对象指定查询方式和查询条件 QueryBuilder queryBuilder = QueryBuilders.matchAllQuery(); //4.将queryBuilder对象设置到SearchSourceBuilder中 sourceBuilder.query(queryBuilder); //使用fetchSource实现过滤 sourceBuilder.fetchSource(new String[]{"id", "title", "price"}, null); //5.将SearchSourceBuilder查询对象封装到请求对象SearchRequest中 searchRequest.source(sourceBuilder); //6.调用方法进行数据通信 SearchResponse response = client.search(searchRequest, RequestOptions.DEFAULT); //7.输出结果 System.out.println("响应结果: " + response); SearchHit[] hits = response.getHits().getHits(); for (int i = 0; i < hits.length; i++) { String sourceAsString = hits[i].getSourceAsString(); System.out.println("查询结果:" + sourceAsString); } } /** * 排序 */ @Test public void orderDoc() throws IOException { //1.构建SearchRequest请求对象,指定索引库 SearchRequest searchRequest = new SearchRequest("heima"); //2.构建SearchSourceBuilder查询对象 SearchSourceBuilder sourceBuilder = new SearchSourceBuilder(); //3.构建QueryBuilder对象指定查询方式和查询条件 QueryBuilder queryBuilder = QueryBuilders.matchAllQuery(); //4.将queryBuilder对象设置到SearchSourceBuilder中 sourceBuilder.query(queryBuilder); /** * 通过sort方法指定排序规则 * args1: 排序字段 * args2: 升序还是降序,默认升序 * * 默认不能使用text字段排序( 设置"fielddata":true) */ sourceBuilder.sort("id", SortOrder.DESC); sourceBuilder.sort("price"); //5.将SearchSourceBuilder查询对象封装到请求对象SearchRequest中 searchRequest.source(sourceBuilder); //6.调用方法进行数据通信 SearchResponse response = client.search(searchRequest, RequestOptions.DEFAULT); //7.输出结果 System.out.println("响应结果: " + response); SearchHit[] hits = response.getHits().getHits(); for (int i = 0; i < hits.length; i++) { String sourceAsString = hits[i].getSourceAsString(); System.out.println("查询结果:" + sourceAsString); } } /** * 分页查询 * * @throws IOException */ @Test public void limitDoc() throws IOException { //1.构建SearchRequest请求对象,指定索引库 SearchRequest searchRequest = new SearchRequest("heima"); //2.构建SearchSourceBuilder查询对象 SearchSourceBuilder sourceBuilder = new SearchSourceBuilder(); //3.构建QueryBuilder对象指定查询方式和查询条件 QueryBuilder queryBuilder = QueryBuilders.matchAllQuery(); //4.将queryBuilder对象设置到SearchSourceBuilder中 sourceBuilder.query(queryBuilder); /** * 通过sort方法指定排序规则 * args1: 排序字段 * args2: 升序还是降序,默认升序 * * 默认不能使用text字段排序( 设置"fielddata":true) */ //sourceBuilder.sort("id", SortOrder.DESC); sourceBuilder.sort("price"); /** * 分页 :查询第一页,每页两条 * int start =(pageNum-1)*pageSize */ sourceBuilder.from(1); sourceBuilder.size(2); //分两页 //5.将SearchSourceBuilder查询对象封装到请求对象SearchRequest中 searchRequest.source(sourceBuilder); //6.调用方法进行数据通信 SearchResponse response = client.search(searchRequest, RequestOptions.DEFAULT); //7.输出结果 System.out.println("响应结果: " + response); SearchHit[] hits = response.getHits().getHits(); for (int i = 0; i < hits.length; i++) { String sourceAsString = hits[i].getSourceAsString(); System.out.println("查询结果:" + sourceAsString); } } /** * 高亮查询 * * @throws IOException */ @Test public void highLightDoc() throws IOException { //1.构建SearchRequest请求对象,指定索引库 SearchRequest searchRequest = new SearchRequest("heima"); //2.构建SearchSourceBuilder查询对象 SearchSourceBuilder sourceBuilder = new SearchSourceBuilder(); //3.构建QueryBuilder对象指定查询方式和查询条件 高亮使用matchquery,不支持matchquery QueryBuilder queryBuilder = QueryBuilders.matchQuery("title", "小米"); //4.将queryBuilder对象设置到SearchSourceBuilder中 sourceBuilder.query(queryBuilder); HighlightBuilder highlightBuilder = new HighlightBuilder(); highlightBuilder.preTags("<font color='red'>"); highlightBuilder.postTags("</font>"); highlightBuilder.field("title"); sourceBuilder.highlighter(highlightBuilder); /** * 通过sort方法指定排序规则 * args1: 排序字段 * args2: 升序还是降序,默认升序 * * 默认不能使用text字段排序( 设置"fielddata":true) */ //sourceBuilder.sort("id", SortOrder.DESC); sourceBuilder.sort("price"); /** * 分页 :查询第一页,每页两条 * int start =(pageNum-1)*pageSize */ //sourceBuilder.from(1); //sourceBuilder.size(2); //分两页 //5.将SearchSourceBuilder查询对象封装到请求对象SearchRequest中 searchRequest.source(sourceBuilder); //6.调用方法进行数据通信 SearchResponse response = client.search(searchRequest, RequestOptions.DEFAULT); //7.输出结果 //System.out.println("响应结果: " + response); SearchHit[] hits = response.getHits().getHits(); for (int i = 0; i < hits.length; i++) { String sourceAsString = hits[i].getSourceAsString(); //System.out.println("查询结果:" + sourceAsString); Map<String, HighlightField> highlightFields = hits[i].getHighlightFields(); HighlightField title = highlightFields.get("title"); Text[] texts = title.getFragments(); for (Text fragment : texts) { System.out.println("title:" + fragment); } } } }