首先这个错误,一定是发生了Jar冲突
Jar冲突指的是相同的类,在不同的Jar中出现,这就早晨了 Jar冲突
一、错误信息
06-08 14:18:26 [ERROR][job.RunnableJobRunner(67)] java.lang.NoSuchMethodError: org.apache.hadoop.hbase.client.Scan.setCacheBlocks(Z)V at cc.blueview.data.job.task.kol.FansAnalyeJob.fansBase(FansAnalyeJob.java:403) at cc.blueview.data.job.task.kol.FansAnalyeJob.runJob(FansAnalyeJob.java:143) at cc.blueview.data.job.task.kol.FansAnalyeJob.call(FansAnalyeJob.java:935) at cc.blueview.dmp.schedule.job.RunnableJob.start(RunnableJob.java:31) at cc.blueview.dmp.schedule.job.RunnableJobRunner.call(RunnableJobRunner.java:63) at cc.blueview.dmp.lego.task.LegoThreadPoolScheduler$2.run(LegoThreadPoolScheduler.java:190) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748)06-08 14:18:26 [INFO][listener.JobStatusListener(104)] stop job, jobId: kolFansAnalyeJob Stop Status: FAILED06-08 14:18:26 [ERROR][support.TaskUtils$LoggingErrorHandler(95)] Unexpected error occurred in scheduled task.java.lang.IllegalStateException: java.lang.IllegalStateException: java.lang.NoSuchMethodError: org.apache.hadoop.hbase.client.Scan.setCacheBlocks(Z)V at cc.blueview.dmp.lego.task.LegoThreadPoolScheduler$2.run(LegoThreadPoolScheduler.java:192) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748)Caused by: java.lang.IllegalStateException: java.lang.NoSuchMethodError: org.apache.hadoop.hbase.client.Scan.setCacheBlocks(Z)V at cc.blueview.dmp.schedule.job.RunnableJobRunner.call(RunnableJobRunner.java:80) at cc.blueview.dmp.lego.task.LegoThreadPoolScheduler$2.run(LegoThreadPoolScheduler.java:190) ... 8 moreCaused by: java.lang.NoSuchMethodError: org.apache.hadoop.hbase.client.Scan.setCacheBlocks(Z)V at cc.blueview.data.job.task.kol.FansAnalyeJob.fansBase(FansAnalyeJob.java:403) at cc.blueview.data.job.task.kol.FansAnalyeJob.runJob(FansAnalyeJob.java:143) at cc.blueview.data.job.task.kol.FansAnalyeJob.call(FansAnalyeJob.java:935) at cc.blueview.dmp.schedule.job.RunnableJob.start(RunnableJob.java:31) at cc.blueview.dmp.schedule.job.RunnableJobRunner.call(RunnableJobRunner.java:63) ... 9 more
二、解决方案
沿着线索找下去,发现,spark安装目录的lib下面有个spark-examples-1.4.1-hadoop2.2.0.jar,里面有
org.apache.hadoop.hbase.client.Scan,这个类使用的hbase-client-0.98.7,而在spark的driver端,同样
有hbase-client-1.2.5,里面有org.apache.hadoop.hbase.client.Scan,这样就造成了冲突