Error Setting WEBHDFS endpoint for exporting tables
Apr 17, 2015
I have been trying to simulate this example (https://github.com/VoltDB/app-bank-offers). Example works fine with no export or export to CSV files.
However with the instructions given in the example, it does not work well for hadoop export.
With the endpoint property set to
http://localhost:50070/webhdfs/v1.0/user/root/%t/data%p.%g.%t.csv
I get the following error:
ERROR: Export processor failed its configuration check
java.lang.IllegalArgumentException: could not expand endpoint http://localhost:50070/webhdfs/v1.0/%t/data%p.%t.csv
at org.voltdb.exportclient.HttpExportClient.configure(HttpExportClient.java:245)
at org.voltdb.export.processors.GuestProcessor.checkProcessorConfig(GuestProcessor.java:100)
at org.voltdb.utils.CatalogUtil.checkExportProcessorConfiguration(CatalogUtil.java:1079)
at org.voltdb.utils.CatalogUtil.setExportInfo(CatalogUtil.java:1143)
at org.voltdb.utils.CatalogUtil.compileDeployment(CatalogUtil.java:568)
at org.voltdb.Inits$LoadCatalog.run(Inits.java:378)
at org.voltdb.Inits$InitializerWorker.run(Inits.java:108)
at org.voltdb.Inits.doInitializationWork(Inits.java:188)
at org.voltdb.RealVoltDB.initialize(RealVoltDB.java:620)
at org.voltdb.VoltDB.initialize(VoltDB.java:897)
at org.voltdb.VoltDB.main(VoltDB.java:881)
Caused by: java.lang.IllegalArgumentException: hdfs endpoint "http://localhost:50070/webhdfs/v1.0/%t/data%p.%t.csv" must contain the (%t)able, the (%p)artition, and the (%g) generation conversions
at com.google_voltpatches.common.base.Preconditions.checkArgument(Preconditions.java:122)
at org.voltdb.exportclient.decode.EndpointExpander.verifyForHdfsUse(EndpointExpander.java:246)
at org.voltdb.exportclient.HttpExportClient.configure(HttpExportClient.java:239)
So, I appended %g to the file name, and then I get another error
ERROR: Notification request failed with HTTP/1.1 405 HTTP method PUT is not supported by this URL
ERROR: Unable to prime http export client to http://localhost:50070/webhdfs/v1.0/OFFERS_GIVEN_EXP/data4.eo08fdyr1atb.OFFERS_GIVEN_EXP.csv org.voltdb.exportclient.HttpExportClient$PathHandlingException: hdfs makePath returned false for http://localhost:50070/webhdfs/v1.0/OFFERS_GIVEN_EXP/data4.eo08fdyr1atb.OFFERS_GIVEN_EXP.csv
at org.voltdb.exportclient.HttpExportClient$HttpExportDecoder.processRow(HttpExportClient.java:834)
at org.voltdb.export.processors.GuestProcessor$2.run(GuestProcessor.java:221)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.voltcore.utils.CoreUtils$7$1.run(CoreUtils.java:735)
at java.lang.Thread.run(Thread.java:745)
I also tried the following URLs
which returned the same error as Notification request failed with HTTP/1.1 405 HTTP method PUT is not supported by this URL
What might I possibly be doing wrong?
However with the instructions given in the example, it does not work well for hadoop export.
With the endpoint property set to
http://localhost:50070/webhdfs/v1.0/user/root/%t/data%p.%g.%t.csv
I get the following error:
ERROR: Export processor failed its configuration check
java.lang.IllegalArgumentException: could not expand endpoint http://localhost:50070/webhdfs/v1.0/%t/data%p.%t.csv
at org.voltdb.exportclient.HttpExportClient.configure(HttpExportClient.java:245)
at org.voltdb.export.processors.GuestProcessor.checkProcessorConfig(GuestProcessor.java:100)
at org.voltdb.utils.CatalogUtil.checkExportProcessorConfiguration(CatalogUtil.java:1079)
at org.voltdb.utils.CatalogUtil.setExportInfo(CatalogUtil.java:1143)
at org.voltdb.utils.CatalogUtil.compileDeployment(CatalogUtil.java:568)
at org.voltdb.Inits$LoadCatalog.run(Inits.java:378)
at org.voltdb.Inits$InitializerWorker.run(Inits.java:108)
at org.voltdb.Inits.doInitializationWork(Inits.java:188)
at org.voltdb.RealVoltDB.initialize(RealVoltDB.java:620)
at org.voltdb.VoltDB.initialize(VoltDB.java:897)
at org.voltdb.VoltDB.main(VoltDB.java:881)
Caused by: java.lang.IllegalArgumentException: hdfs endpoint "http://localhost:50070/webhdfs/v1.0/%t/data%p.%t.csv" must contain the (%t)able, the (%p)artition, and the (%g) generation conversions
at com.google_voltpatches.common.base.Preconditions.checkArgument(Preconditions.java:122)
at org.voltdb.exportclient.decode.EndpointExpander.verifyForHdfsUse(EndpointExpander.java:246)
at org.voltdb.exportclient.HttpExportClient.configure(HttpExportClient.java:239)
So, I appended %g to the file name, and then I get another error
ERROR: Notification request failed with HTTP/1.1 405 HTTP method PUT is not supported by this URL
ERROR: Unable to prime http export client to http://localhost:50070/webhdfs/v1.0/OFFERS_GIVEN_EXP/data4.eo08fdyr1atb.OFFERS_GIVEN_EXP.csv org.voltdb.exportclient.HttpExportClient$PathHandlingException: hdfs makePath returned false for http://localhost:50070/webhdfs/v1.0/OFFERS_GIVEN_EXP/data4.eo08fdyr1atb.OFFERS_GIVEN_EXP.csv
at org.voltdb.exportclient.HttpExportClient$HttpExportDecoder.processRow(HttpExportClient.java:834)
at org.voltdb.export.processors.GuestProcessor$2.run(GuestProcessor.java:221)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.voltcore.utils.CoreUtils$7$1.run(CoreUtils.java:735)
at java.lang.Thread.run(Thread.java:745)
I also tried the following URLs
- http://localhost:50070/webhdfs/v1.0/user/root/%t/data%p.%g.%t.csv
- http://localhost:50070/webhdfs/v1.0/%t/data%p.%g.%t.csv?user.name=root
which returned the same error as Notification request failed with HTTP/1.1 405 HTTP method PUT is not supported by this URL
What might I possibly be doing wrong?