hadoop的wordcount java举例分析

本篇内容介绍了“hadoop的wordcount java举例分析”的有关知识，在实际案例的操作过程中，不少人都会遇到这样的困境，接下来就让小编带领大家学习一下如何处理这些情况吧！希望大家仔细阅读，能够学有所成！

1.导入hadoop需要用到的包

hadoop-2.4.2/share/hadoop/mapreduce/*.jar
hadoop-2.4.2/share/hadoop/mapreduce/lib/*.jar

hadoop-2.4.2/share/hadoop/common/*.jar
hadoop-2.4.2/share/hadoop/common/lib/*.jar

2.编写java程序

package demo;

import java.io.IOException;
import java.util.*;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.*;
import org.apache.hadoop.mapred.*;

public class WordCount {

 public static class Map extends MapReduceBase implements Mapper{
  private final static IntWritable one=new IntWritable(1);
  private Text word=new Text();
  @Override
  public void map(LongWritable key,  Text value,
    OutputCollector output, Reporter reporter)
    throws IOException {
   // TODO Auto-generated method stub
   String line=value.toString();
   StringTokenizer tokenizer=new StringTokenizer(line);
   while (tokenizer.hasMoreTokens()){
    word.set(tokenizer.nextToken());
    output.collect(word,one);
    }
   }
   
  }
 //旧版本
 public static class Reduce extends MapReduceBase implements Reducer{

  @Override
  public void reduce(Text key, Iterator values,
    OutputCollector output, Reporter reporter)
    throws IOException {
   // TODO Auto-generated method stub
    int sum=0;
    while(values.hasNext()){
     sum+=values.next().get();
    }
    output.collect(key, new IntWritable(sum)); 
  }
 }

 
 public static void main(String[] args) throws Exception{
  // TODO Auto-generated method stub
  //System.setProperty("HADOOP_USER_NAME","root");  
  JobConf conf=new JobConf(WordCount.class);
  //conf.set("fs.defaultFS","hdfs://192.168.1.120:9000");
  conf.setJobName("wordcount");
  conf.setOutputKeyClass(Text.class);
  conf.setOutputValueClass(IntWritable.class);

  conf.setMapperClass(Map.class);
  conf.setReducerClass(Reduce.class);
  
  conf.setInputFormat(TextInputFormat.class);
  conf.setOutputFormat(TextOutputFormat.class);
  
  FileInputFormat.setInputPaths(conf,new Path(args[0]));
  FileOutputFormat.setOutputPath(conf,new Path(args[1]));

  JobClient.runJob(conf);
 }

}

3.导出为jar文件

4.上传到linux系统中。

5.新建input目录，如果有output目录，先删除

6.上传jar包后，到jar包的目录下，执行

    hadoop jar WordCount.jar demo.WordCount /input/* /output/

7.如果执行时不带“/”，会在hadoop目录中新建/user/root下新建两个文件夹，会提示

      Exception in thread "main" org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: file:  /input  Exception in thread "main" org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist: hdfs://master:9000/user/root/input

  只需要在执行的时候带上“/”就行。

8.获取分离后的文件

hadoop fs -get /output/* output/

“hadoop的wordcount java举例分析”的内容就介绍到这里了，感谢大家的阅读。如果想了解更多行业相关的知识可以关注亿速云网站，小编将为大家输出更多高质量的实用文章！