将ip转换为地理位置存入elasticsearch

分手后的思念是犯贱 2023-05-30 06:11 8阅读 0赞

有些需要将一些数据基于地址位置进行分析，比如哪些区域比较活跃，在什么时间范围内活跃，但是后端仅能获取ip地址，因此需要将ip地址转换为地理位置，幸运的是我们有开源的工具可以使用，[maxmind][]/**[GeoIP2-java][]**使用GeoLite2-City.mmdb库就可以由ip分析得到对应的经纬度，下面给出具体操作步骤：

1、从[https://dev.maxmind.com/geoip/geoip2/geolite2/][https_dev.maxmind.com_geoip_geoip2_geolite2]下载免费的GeoLite2-City库，但是准确度不如收费的geoip2-city库

2、创建elasticsearch索引

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2pvaG4xMzM3_size_16_color_FFFFFF_t_70][]

3、更新映射

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2pvaG4xMzM3_size_16_color_FFFFFF_t_70 1][]

4、将数据插入到elasticsearch,ip解析经纬度参考[https://github.com/maxmind/GeoIP2-java][GeoIP2-java]，

//使用RestHighLevelClient BulkRequest批量插入数据
         @Test
    	public void createSubBankIndex(){
        	List<LogDetail>  details = logDetailMapper.getLogDetails();
        	
        	//bulkRequest.timeout("2m");
    		DatabaseReader reader = null;
    		try {
    			Resource resource = new ClassPathResource("GeoLite2-City.mmdb");
    			InputStream file = resource.getInputStream();				
    			reader = new DatabaseReader.Builder(file).build();
    		} catch (IOException e1) {
    			// TODO Auto-generated catch block
    			e1.printStackTrace();
    		}
    		if(null == reader)
    			return;
        	for(int i=0,len=details.size();i<len;){
        		int sz = len - i > 2000?2000: len - i;
        		List<LogDetail> subBanks = details.subList(i, sz+i);
        		i += sz;
            	BulkRequest bulkRequest = new BulkRequest("log_detail1104", "log_detail");
            	bulkRequest.timeout(TimeValue.timeValueMinutes(2));   
            	IndexRequest indexRequest = new IndexRequest("log_detail1104", "log_detail");
            	GeoLocation loc = new GeoLocation();
        		for(LogDetail b:subBanks){
        			GeoPoint gp = ip2Geo(reader,b.getIpAddress());
        			loc.setLat(gp.getLat());
        			loc.setLon(gp.getLon());
        			b.setLocation(loc);
        			//indexRequest.timeout(TimeValue.timeValueMinutes(2)); 
        			indexRequest.source(JSON.toJSONString(b), XContentType.JSON);
        			bulkRequest.add(indexRequest);			
        		}
        		
        		try {
        			highLevelClient.bulk(bulkRequest, RequestOptions.DEFAULT);
        			//highLevelClient.index(indexRequest, RequestOptions.DEFAULT);
        		} catch (IOException e) {
        			// TODO Auto-generated catch block
        			e.printStackTrace();
        		}	    		
        	}	
    	}
    
    
        //将ip转换为地理位置信息
    	private GeoPoint ip2Geo(DatabaseReader reader,String ip){
    		try {
    			InetAddress ipAddress = InetAddress.getByName(ip);
    
    			// Replace "city" with the appropriate method for your database, e.g.,
    			// "country".
    			CityResponse response = reader.city(ipAddress);		
    
    
    			Country country = response.getCountry();
    			//System.out.println(country.getIsoCode());            // 'US'
    			//System.out.println(country.getName());               // 'United States'
    			System.out.println(country.getNames().get("zh-CN")); // '美国'
    
    			Subdivision subdivision = response.getMostSpecificSubdivision();
    			System.out.println(subdivision.getName());    // 'Minnesota'
    			//System.out.println(subdivision.getIsoCode()); // 'MN'
    
    			City city = response.getCity();
    			System.out.println(city.getName()); // 'Minneapolis'
    
    			Postal postal = response.getPostal();
    			//System.out.println(postal.getCode()); // '55455'
    
    			Location location = response.getLocation();
    			System.out.println(location.getLatitude());  // 44.9733
    			System.out.println(location.getLongitude()); // -93.2323	
    			
    			GeoPoint gp = new GeoPoint(location.getLatitude(),location.getLongitude());
    			return gp;
    		} catch (Exception e) {
    			// TODO Auto-generated catch block
    			e.printStackTrace();
    		}
    		return null;
    	}

5、运行后效果图

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2pvaG4xMzM3_size_16_color_FFFFFF_t_70 2][]

[maxmind]: https://github.com/maxmind
[GeoIP2-java]: https://github.com/maxmind/GeoIP2-java
[https_dev.maxmind.com_geoip_geoip2_geolite2]: https://dev.maxmind.com/geoip/geoip2/geolite2/
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2pvaG4xMzM3_size_16_color_FFFFFF_t_70]: https://img-blog.csdnimg.cn/20191107183650853.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2pvaG4xMzM3,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2pvaG4xMzM3_size_16_color_FFFFFF_t_70 1]: https://img-blog.csdnimg.cn/20191107183715969.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2pvaG4xMzM3,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2pvaG4xMzM3_size_16_color_FFFFFF_t_70 2]: https://img-blog.csdnimg.cn/20191108143945278.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2pvaG4xMzM3,size_16,color_FFFFFF,t_70