行业知识
基于MapReduce的汽车数据清洗与统计案例
2026-11-14 15:50:43
来源：互联网转载
数据简介
ecar168.csv（���车销售数据表）：
（）| 字段 | 数据类型 | 字段说明 |
|---|---|---|
| ranking | String | 排名 |
| manufacturer | String | 厂商 |
| vehicle_type | String | 车型 |
| monthly_sales_volume | String | 月销量 |
| accumulated_this_year | String | 本年累计 |
| last_month | String | 上月 |
| chain_ratio | String | 环比 |
| corresponding_period_of_last_yea | String | 去年同期 |
| year_on_year | String | 同比 |
| series | String | 系列 |
| month | String | 统计时间 |
| url_href | String | 数据来源 url |
| url_title | String | 数据来源 标题 |
| url_car_detail | String | 汽车详情 url |
"1,东风日产,日产轩逸,56201,421871,57525,-2.30%,48483,15.92%,全部,2020年10月,http://www.ecar168.cn/xiaoliang/phb_cx/37.htm,2020年10月全部车型销量排行榜,http://data.ecar168.cn/car/216/""2,长城汽车,哈弗H6,52734,266231,40475,30.29%,40623,29.81%,全部,2020年10月,http://www.ecar168.cn/xiaoliang/phb_cx/37.htm,2020年10月全部车型销量排行榜,http://data.ecar168.cn/car/658/""3,上汽大众,大众朗逸,40984,332685,39595,3.51%,40608,0.93%,全部,2020年10月,http://www.ecar168.cn/xiaoliang/phb_cx/37.htm,2020年10月全部车型销量排行榜,http://data.ecar168.cn/car/465/""4,一汽大众,大众宝来,37944,251470,37959,-0.04%,36799,3.11%,全部,2020年10月,http://www.ecar168.cn/xiaoliang/phb_cx/37.htm,2020年10月全部车型销量排行榜,http://data.ecar168.cn/car/477/""5,一汽大众,大众速腾,36380,238334,33219,9.52%,36015,1.01%,全部,2020年10月,http://www.ecar168.cn/xiaoliang/phb_cx/37.htm,2020年10月全部车型销量排行榜,http://data.ecar168.cn/car/203/"
ecar168_car_detail.csv（汽车信息表）:
| 字段 | 数据类型 | 字段说明 |
|---|---|---|
| manufacturer | String | 厂商 |
| vehicle_type | String | 车型 |
| series | String | 系列 |
| url_href | String | 数据来源url |
| url_title | String | 数据来源 标题 |
| url_car_detail | String | 汽车详情url/综述链接 |
| price | String | 价格 |
| url_sales | String | 销量链接 |
"大众速腾,http://data.ecar168.cn/car/203/,13.18-19.68万,紧凑型,2019,1.2L、1.4L,手动、自动,5.4-5.6L（综合）""大众宝来,http://data.ecar168.cn/car/477/,9.88-15.6万,紧凑型,2019,1.4L、1.5L,手动、自动,5.6-5.9L（综合）""大众朗逸,http://data.ecar168.cn/car/465/,9.99-16.59万,紧凑型,2020,1.2L、1.4L、1.5L,手动、自动,5.1-5.7L（综合）""哈弗H6,http://data.ecar168.cn/car/658/,9.8-14.1万,SUV,2020,1.5L、2.0L,手动、自动,6.6-6.9L（综合）""长安CS75,http://data.ecar168.cn/car/1133/,9.58-14.68万,SUV,2019,1.5L,手动、自动,6.5-7.0L（综合）"
一、数据去重
在master节点上操作，编写mapreduce代码对ecar168.csv文本文件中的数据去重处理，并把处理完成的数据输出到hdfs上的/output/DupRemoval/路径下。（1分）
1.Mapper类
package com.saddam.bigdata.platform.DupRemoval;import org.apache.hadoop.io.LongWritable;import org.apache.hadoop.io.Text;import org.apache.hadoop.mapreduce.Mapper;import java.io.IOException;public class DupRemovalMapper extends Mapper { Text outKey=new Text(); Text outValue=new Text(); /** * @param key--->输入的key:偏移量；输出的key:一整行内容 * @param value->输入的value:一整行内容；输出的value：一整行内容 * @param context：1,东风日产,日产轩逸,56201,421871,57525,-2.30%,48483,15.92%,全部,2020年 10月,http://www.ecar168.cn/xiaoliang/phb_cx/37.htm,2020年10月全部车型 销量排行榜,http://data.ecar168.cn/car/216/ * @throws IOException * @throws InterruptedException */ @Override protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException { //补全代码 String line=value.toString();//获取一行 outKey.set(line); outValue.set(line); context.write(outKey,outValue); }} 2.Reducer类
package com.saddam.bigdata.platform.DupRemoval;import org.apache.hadoop.io.NullWritable;import org.apache.hadoop.io.Text;import org.apache.hadoop.mapreduce.Reducer;import java.io.IOException;public class DupRemovalReducer extends Reducer { @Override protected void reduce(Text key, Iterable values, Context context) throws IOException, InterruptedException { context.write(null,key); }} 3.Driver类
package com.saddam.bigdata.platform.DupRemoval;import org.apache.hadoop.conf.Configuration;import org.apache.hadoop.fs.Path;import org.apache.hadoop.io.NullWritable;import org.apache.hadoop.io.Text;import org.apache.hadoop.mapreduce.Job;import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;public class DupRemovalMapReduce { public static void main(String[] args) throws Exception { Configuration conf = new Configuration(); Job job = Job.getInstance(conf, "DupaRemoval"); job.setJarByClass(DupRemovalMapReduce.class); job.setMapperClass(DupRemovalMapper.class); job.setReducerClass(DupRemovalReducer.class); job.setMapOutputKeyClass(Text.class); job.setMapOutputValueClass(Text.class); job.setOutputKeyClass(NullWritable.class); job.setOutputValueClass(Text.class); //补全代码 String inputPath ="D:\Maven_java content root\Maven_web\hadoop-train\mapreduce_data\input\car_data.csv"; String outputPath ="D:\Maven_java content root\Maven_web\hadoop-train\mapreduce_data\output\output_car_dup333333"; FileInputFormat.setInputPaths(job,new Path(inputPath)); FileOutputFormat.setOutputPath(job,new Path(outputPath)); boolean b = job.waitForCompletion(true); if (!b) { System.out.println("wordcount task fail!"); } }} 二、数据筛选
在master节点上操作，编写 mapreduce 代码对 hdfs 上的/output/DupRemoval/part-r-00000数据中month字段统一清洗为yyyy-mm（例：2012-04）格式，并保留month（清洗后）、manufacturer 、vehicle_type、monthly_sales_volume、series 、url_car_detail 6个字段输出到hdfs上的/output/dataclean/路径下。（2分）
1.Mapper类
package com.saddam.bigdata.platform.CarData_3000.Clean3000;import org.apache.hadoop.io.LongWritable;import org.apache.hadoop.io.Text;import org.apache.hadoop.mapreduce.Mapper;import java.io.IOException;public class Clean3Mapper extends Mapper { Text outKey = new Text(); Text outValue = new Text(); @Override protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException { String[] word = value.toString().split(","); try { //2020年10月 String word10 = word[10].replaceAll("年", "-"); //word10:2020-10月 word10=word10.replaceAll("月",""); //word10:2020-10 String year=word10.substring(0,5); String month=word10.substring(5); if (month.length()==1){ month="0"+month; word10=year+month; } outKey.set(word10); outValue.set(word10 + "," + word[1] + "," + word[2] + "," + word[3] + "," + word[9] + "," + word[13]); context.write(outKey, outValue); } catch (Exception e) { } }} 2.Reducer类
package com.saddam.bigdata.platform.CarData_3000.Clean3000;import org.apache.hadoop.io.NullWritable;import org.apache.hadoop.io.Text;import org.apache.hadoop.mapreduce.Reducer;import java.io.IOException;public class Clean3Reducer extends Reducer { @Override protected void reduce(Text key, Iterable values, Context context) throws IOException, InterruptedException { for (Text value : values) { context.write(null,value); } }} 3.Driver类
package com.saddam.bigdata.platform.CarData_3000.Clean3000;import org.apache.hadoop.conf.Configuration;import org.apache.hadoop.fs.Path;import org.apache.hadoop.io.NullWritable;import org.apache.hadoop.io.Text;import org.apache.hadoop.mapreduce.Job;import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;public class Clean3Driver { public static void main(String[] args) throws Exception { Configuration conf = new Configuration(); Job job = Job.getInstance(conf, "carclean"); job.setJarByClass(Clean3Driver.class); job.setMapperClass(Clean3Mapper.class); job.setReducerClass(Clean3Reducer.class); job.setMapOutputKeyClass(Text.class); job.setMapOutputValueClass(Text.class); job.setOutputKeyClass(NullWritable.class); job.setOutputValueClass(Text.class); String inputPath="D:\Maven_java content root\Maven_web\hadoop-train\mapreduce_data\output\Car_Data\3000_dup\part-r-00000"; String outputPath="D:\Maven_java content root\Maven_web\hadoop-train\mapreduce_data\output\Car_Data\3000_clean"; FileInputFormat.setInputPaths(job,new Path(inputPath)); FileOutputFormat.setOutputPath(job,new Path(outputPath)); job.waitForCompletion(true); }} 三、数据整合
在master节点上操作，编写 mapreduce 代码实现以下要求的操作：（3分）oil_consumption字段清洗掉多余内容只保留数值，并计算平均值，保留2位小数，生成为oil_consumption_average新字段。transmission_case字段中的"手动 自动"类别替换为"全动"。整合hdfs上的/data/ecar168_car_detail.csv和/output/carjoindetail/part-r-00000数据，要求如下：以/data/ecar168_car_detail.csv数据为基准，整合/output/carjoindetail/part-r-00000的数据。按顺序输出date,manufacturer,vehicle_type,monthly_sales_volume,series,url_car_detail,price,car_type,displacement,transmission_case,oil_consumption_average字段。输出路径为：hdfs://master:9000/output/carjoindetail/
1.Mapper类
package com.saddam.bigdata.platform.CarData_3000.Join3000;import org.apache.hadoop.io.LongWritable;import org.apache.hadoop.io.MapWritable;import org.apache.hadoop.io.Text;import org.apache.hadoop.mapreduce.Mapper;import org.apache.hadoop.mapreduce.lib.input.FileSplit;import java.io.IOException;public class Join3Mapper extends Mapper { //创建对象 MapWritable mapWritable=new MapWritable(); @Override protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException { //获取当前处理的切片 FileSplit inputSplit=(FileSplit)context.getInputSplit(); //获取切片所属的文件名称（文件名称） String filename=inputSplit.getPath().getName(); //切割处理读取的每一行数据 String line=value.toString(); String[] words=line.split(","); //将Key的值赋空 Text TextKey=null; //判断文本名 if (filename.equals("part-r-00000")){//以/data/ecar168_car_detail.csv数据为基准，整合/output/carjoindetail/part-r-00000的数据。 //details:大众速腾,http://data.ecar168.cn/car/203/,13.18-19.68万,紧凑型,2019,1.2L、1.4L,手动、自动,5.4-5.6L（综合） //part-r-00000:2019-11,华晨中华,中华H530,84,全部,http://data.ecar168.cn/car/778/ TextKey=new Text(words[2]+words[5]); Text month=new Text(words[0]);//获取日期 Text manufacturer=new Text(words[1]);//获取厂商 Text vehicle_type=new Text(words[2]);//获取车型 Text monthly_sales_volume=new Text(words[3]);//获取月销量 Text series=new Text(words[4]);//获取系列 Text url_car_detail=new Text(words[5]);//获取汽车详情url mapWritable.put(new Text("flag"),new Text("car")); mapWritable.put(new Text("month"),month); mapWritable.put(new Text("manufacturer"),manufacturer); mapWritable.put(new Text("vehicle_type"),vehicle_type); mapWritable.put(new Text("monthly_sales_volume"),monthly_sales_volume); mapWritable.put(new Text("series"),series); mapWritable.put(new Text("url_car_detail"),url_car_detail); context.write(TextKey,mapWritable); }else { TextKey=new Text(words[0]+words[1]); Text vehicle_type=new Text(words[0]); Text url_href=new Text(words[1]); Text price=new Text(words[2]); Text car_type=new Text(words[3]); Text time_to_market=new Text(words[4]); Text displacement=new Text(words[5]); Text transmission_case=new Text(words[6]); //处理油量 String oil_consumption_str=words[7];//5.4-5.6L（综合） oil_consumption_str=oil_consumption_str.replaceAll("（综合）","");//5.4-5.6L oil_consumption_str=oil_consumption_str.replaceAll("L","");//5.4-5.6 /** * indexOf() 方法有以下四种形式： * * public int indexOf(int ch): 返回指定字符在字符串中第一次出现处的索引，如果此字符串中没有这样的字符，则返回 -1。 * * public int indexOf(int ch, int fromIndex): 返回从 fromIndex 位置开始查找指定字符在字符串中第一次出现处的索引，如果此字符串中没有这样的字符，则返回 -1。 * * int indexOf(String str): 返回指定字符在字符串中第一次出现处的索引，如果此字符串中没有这样的字符，则返回 -1。 * * int indexOf(String str, int fromIndex): 返回从 fromIndex 位置开始查找指定字符在字符串中第一次出现处的索引，如果此字符串中没有这样的字符，则返回 -1。 */ if (oil_consumption_str.indexOf("-")>0){ String[] oil_consumption_=oil_consumption_str.split("-"); float oil_consumption_0=Float.parseFloat(oil_consumption_[0]); float oil_consumption_1=Float.parseFloat(oil_consumption_[1]); oil_consumption_str=String.valueOf((oil_consumption_0+oil_consumption_1)/2).substring(0,3); } Text oil_consumption_average=new Text(oil_consumption_str); mapWritable.put(new Text("flag"),new Text("detail")); mapWritable.put(new Text("vehicle_type"),vehicle_type); mapWritable.put(new Text("url_href"),url_href); mapWritable.put(new Text("price"),price); mapWritable.put(new Text("car_type"),car_type); mapWritable.put(new Text("time_to_market"),time_to_market); mapWritable.put(new Text("displacement"),displacement); mapWritable.put(new Text("time_to_market"),time_to_market); mapWritable.put(new Text("transmission_case"),transmission_case); mapWritable.put(new Text("oil_consumption_average"),oil_consumption_average); context.write(TextKey,mapWritable); } }} 2.Reducer类
package com.saddam.bigdata.platform.CarData_3000.Join3000;import org.apache.hadoop.io.MapWritable;import org.apache.hadoop.io.NullWritable;import org.apache.hadoop.io.Text;import org.apache.hadoop.mapreduce.Reducer;import java.io.IOException;import java.util.ArrayList;import java.util.HashMap;import java.util.List;import java.util.Map;public class Join3Reducer extends Reducer { @Override protected void reduce(Text key, Iterable values, Context context) throws IOException, InterruptedException { boolean blag=false; List list=new ArrayList(); Map map=new HashMap(); for (MapWritable mapWritable:values){ String flag=mapWritable.get(new Text("flag")).toString(); if (flag.equals("car")){ Map car=new HashMap(); car.put("month",mapWritable.get(new Text("month")).toString()); car.put("manufacturer",mapWritable.get(new Text("manufacturer")).toString()); car.put("vehicle_type",mapWritable.get(new Text("vehicle_type")).toString()); car.put("monthly_sales_volume",mapWritable.get(new Text("monthly_sales_volume")).toString()); car.put("series",mapWritable.get(new Text("series")).toString()); car.put("url_car_detail",mapWritable.get(new Text("url_car_detail")).toString()); list.add(car); } if (flag.equals("detail")){ blag=true; map.put("vehicle_type",mapWritable.get(new Text("vehicle_type")).toString()); map.put("url_href",mapWritable.get(new Text("url_href")).toString()); map.put("price",mapWritable.get(new Text("price")).toString()); map.put("car_type",mapWritable.get(new Text("car_type")).toString()); map.put("time_to_market",mapWritable.get(new Text("time_to_market")).toString()); map.put("displacement",mapWritable.get(new Text("displacement")).toString()); map.put("time_to_market",mapWritable.get(new Text("time_to_market")).toString()); map.put("transmission_case",mapWritable.get(new Text("transmission_case")).toString()); map.put("oil_consumption_average",mapWritable.get(new Text("oil_consumption_average")).toString()); } } try { if (blag){ String price=String.valueOf(map.get("price")).replaceAll(" ",""); String transmission_case=String.valueOf(map.get("transmission_case")); if ("手动、自动".equals(transmission_case)){ transmission_case="全动"; } String outputmap=price+","+map.get("car_type")+","+map.get("displacement")+","+transmission_case+","+map.get("oil_consumption_average"); for (Object obj:list){ Map mapobj=(Map)obj; String output=mapobj.get("month")+","+mapobj.get("manufacturer")+","+mapobj.get("vehicle_type")+","+mapobj.get("monthly_sales_volume")+","+mapobj.get("series")+","+mapobj.get("url_car_detail")+","+outputmap; context.write(null,new Text(output)); } } }catch (Exception e){ } } } 3.Driver类
package com.saddam.bigdata.platform.CarData_3000.Join3000;import org.apache.hadoop.conf.Configuration;import org.apache.hadoop.fs.Path;import org.apache.hadoop.io.MapWritable;import org.apache.hadoop.io.NullWritable;import org.apache.hadoop.io.Text;import org.apache.hadoop.mapreduce.Job;import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;public class Join3Driver { public static void main(String[] args) throws Exception { Configuration conf = new Configuration(); Job job = Job.getInstance(conf, "carjoindetail"); job.setJarByClass(Join3Driver.class); job.setMapperClass(Join3Mapper.class); job.setReducerClass(Join3Reducer.class); job.setMapOutputKeyClass(Text.class); job.setMapOutputValueClass(MapWritable.class); job.setOutputKeyClass(NullWritable.class); job.setOutputValueClass(Text.class); //根据题目要求编写代码 String inputPath1 ="D:\Maven_java content root\Maven_web\hadoop-train\mapreduce_data\output\Car_Data\output_car_clean\part-r-00000"; String inputPath2 ="D:\Maven_java content root\Maven_web\hadoop-train\mapreduce_data\input\ecar168_car_detail.txt"; String outputPath ="D:\Maven_java content root\Maven_web\hadoop-train\mapreduce_data\output\Car_Data\output_car3Join"; //同时加载2个数据源 FileInputFormat.setInputPaths(job,new Path(inputPath1),new Path(inputPath2)); FileOutputFormat.setOutputPath(job,new Path(outputPath)); boolean b = job.waitForCompletion(true); if (!b) { System.out.println("wordcount task fail!"); } }} 数据筛选结果
2007-08,哈飞汽车,赛豹V,287,全部,https://www.16888.com/125890/2007-08,天津一汽,威姿,145,全部,https://www.16888.com/57968/2007-09,天津一汽,威姿,287,全部,https://www.16888.com/57968/2007-09,哈飞汽车,赛豹V,330,全部,https://www.16888.com/125890/2007-10,哈飞汽车,赛豹V,576,全部,https://www.16888.com/125890/2007-10,广汽三菱,欧蓝德,2,全部,https://www.16888.com/127531/2007-10,天津一汽,威姿,203,全部,https://www.16888.com/57968/2007-11,哈飞汽车,赛豹V,439,全部,https://www.16888.com/125890/2007-11,广汽三菱,欧蓝德,87,全部,https://www.16888.com/127531/2007-11,天津一汽,威姿,206,全部,https://www.16888.com/57968/2007-12,广汽三菱,欧蓝德,7,全部,https://www.16888.com/127531/2007-12,华晨中华,中华酷宝,182,全部,https://www.16888.com/58080/2007-12,天津一汽,威姿,80,全部,https://www.16888.com/57968/2007-12,哈飞汽车,赛豹V,387,全部,https://www.16888.com/125890/2008-01,华晨中华,中华酷宝,487,全部,https://www.16888.com/58080/2008-01,天津一汽,威姿,80,全部,https://www.16888.com/57968/2008-01,哈飞汽车,赛豹V,540,全部,https://www.16888.com/125890/2008-02,华晨中华,中华酷宝,132,全部,https://www.16888.com/58080/2008-02,天津一汽,威姿,363,全部,https://www.16888.com/57968/2008-02,哈飞汽车,赛豹V,465,全部,https://www.16888.com/125890/
数据整合结果
2020-10,长安标致雪铁龙,DS 4S,0,全部,http://data.ecar168.cn/car/1498/,17.19-22.99万,紧凑型,1.6L,自动,6.32020-10,长安标致雪铁龙,DS 6,0,全部,http://data.ecar168.cn/car/1298/,20.69-27.29万,SUV,1.6L,自动,6.72020-10,长安标致雪铁龙,DS 7,5,全部,http://data.ecar168.cn/car/1864/,20.89-31.99万,SUV,1.6L,自动,6.22020-10,广汽菲克,Jeep大指挥官PHEV,30,美系,http://data.ecar168.cn/car/2222/,30.98-33.68万,SUV,2.0L,自动,1.62020-10,广汽菲克,Jeep大指挥官PHEV,30,全部,http://data.ecar168.cn/car/2222/,30.98-33.68万,SUV,2.0L,自动,1.62020-10,广汽菲克,Jeep指南者,2092,美系,http://data.ecar168.cn/car/1576/,15.58-22.98万,SUV,1.3L,自动,6.82020-04,广汽菲克,Jeep指南者,2020,美系,http://data.ecar168.cn/car/1576/,15.58-22.98万,SUV,1.3L,自动,6.82020-10,广汽菲克,Jeep指南者,2092,全部,http://data.ecar168.cn/car/1576/,15.58-22.98万,SUV,1.3L,自动,6.82020-04,广汽菲克,Jeep指南者,2020,全部,http://data.ecar168.cn/car/1576/,15.58-22.98万,SUV,1.3L,自动,6.82020-04,广汽菲克,Jeep自由光,510,美系,http://data.ecar168.cn/car/1451/,17.98-31.98万,SUV,2.0L,自动,8.52020-10,广汽菲克,Jeep自由光,1259,美系,http://data.ecar168.cn/car/1451/,17.98-31.98万,SUV,2.0L,自动,8.52020-10,广汽菲克,Jeep自由光,1259,全部,http://data.ecar168.cn/car/1451/,17.98-31.98万,SUV,2.0L,自动,8.5（）
