Comments for My experience while making AjaxTrend.com http://ajaxtrend.wordpress.com AjaxTrend.com : making Tue, 18 Nov 2008 07:42:36 +0000 http://wordpress.com/ hourly 1 Comment on How to merge nutch indexes v 0.9 by Kimvais http://ajaxtrend.wordpress.com/2007/11/29/how-to-merge-nutch-indexes-v-09/#comment-12 Kimvais Tue, 18 Nov 2008 07:42:36 +0000 http://ajaxtrend.wordpress.com/2007/11/29/how-to-merge-nutch-indexes-v-09/#comment-12 I get an error when: $ bin/nutch index new/indexes new/linkdb/ new/crawldb/ new/segments/* Indexer: starting Indexer: linkdb: new/crawldb Indexer: adding segment: new/segments/20081118093959 Indexer: org.apache.hadoop.mapred.InvalidInputException: Input path doesnt exist : /opt/nutch-0.9/new/segments/20081118093959/crawl_fetch Input path doesnt exist : /opt/nutch-0.9/new/segments/20081118093959/parse_data Input path doesnt exist : /opt/nutch-0.9/new/segments/20081118093959/parse_text at org.apache.hadoop.mapred.InputFormatBase.validateInput(InputFormatBase.java:138) at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:326) at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:543) at org.apache.nutch.indexer.Indexer.index(Indexer.java:273) at org.apache.nutch.indexer.Indexer.run(Indexer.java:295) at org.apache.hadoop.util.ToolBase.doMain(ToolBase.java:189) at org.apache.nutch.indexer.Indexer.main(Indexer.java:278) I get an error when:

$ bin/nutch index new/indexes new/linkdb/ new/crawldb/ new/segments/*
Indexer: starting
Indexer: linkdb: new/crawldb
Indexer: adding segment: new/segments/20081118093959
Indexer: org.apache.hadoop.mapred.InvalidInputException: Input path doesnt exist : /opt/nutch-0.9/new/segments/20081118093959/crawl_fetch
Input path doesnt exist : /opt/nutch-0.9/new/segments/20081118093959/parse_data
Input path doesnt exist : /opt/nutch-0.9/new/segments/20081118093959/parse_text
at org.apache.hadoop.mapred.InputFormatBase.validateInput(InputFormatBase.java:138)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:326)
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:543)
at org.apache.nutch.indexer.Indexer.index(Indexer.java:273)
at org.apache.nutch.indexer.Indexer.run(Indexer.java:295)
at org.apache.hadoop.util.ToolBase.doMain(ToolBase.java:189)
at org.apache.nutch.indexer.Indexer.main(Indexer.java:278)

]]> Comment on How to merge nutch indexes v 0.9 by tigertail http://ajaxtrend.wordpress.com/2007/11/29/how-to-merge-nutch-indexes-v-09/#comment-11 tigertail Thu, 28 Aug 2008 20:19:32 +0000 http://ajaxtrend.wordpress.com/2007/11/29/how-to-merge-nutch-indexes-v-09/#comment-11 Good stuff for index merging. The question is at the last step, we have index all fetched URLs again. If we have two large crawl1 and crawl2, it takes long time. Is there anyway to avoid index again? I tried nutch merge mergeaall/index crawl1/indexes crawl2/indexes It seems to work at the first glance, because we can get some search results. But when i click cached to see the detailed content for 1 URL, it returns error. Good stuff for index merging. The question is at the last step, we have index all fetched URLs again. If we have two large crawl1 and crawl2, it takes long time. Is there anyway to avoid index again? I tried

nutch merge mergeaall/index crawl1/indexes crawl2/indexes

It seems to work at the first glance, because we can get some search results. But when i click cached to see the detailed content for 1 URL, it returns error.

]]>