Announcement

Collapse
No announcement yet.

Error Setting WEBHDFS endpoint for exporting tables

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • Error Setting WEBHDFS endpoint for exporting tables

    I have been trying to simulate this example (https://github.com/VoltDB/app-bank-offers). Example works fine with no export or export to CSV files.

    However with the instructions given in the example, it does not work well for hadoop export.
    With the endpoint property set to

    http://localhost:50070/webhdfs/v1.0/...ta%p.%g.%t.csv

    I get the following error:

    ERROR: Export processor failed its configuration check
    java.lang.IllegalArgumentException: could not expand endpoint http://localhost:50070/webhdfs/v1.0/%t/data%p.%t.csv

    at org.voltdb.exportclient.HttpExportClient.configure (HttpExportClient.java:245)
    at org.voltdb.export.processors.GuestProcessor.checkP rocessorConfig(GuestProcessor.java:100)
    at org.voltdb.utils.CatalogUtil.checkExportProcessorC onfiguration(CatalogUtil.java:1079)
    at org.voltdb.utils.CatalogUtil.setExportInfo(Catalog Util.java:1143)
    at org.voltdb.utils.CatalogUtil.compileDeployment(Cat alogUtil.java:568)
    at org.voltdb.Inits$LoadCatalog.run(Inits.java:378)
    at org.voltdb.Inits$InitializerWorker.run(Inits.java: 108)
    at org.voltdb.Inits.doInitializationWork(Inits.java:1 88)
    at org.voltdb.RealVoltDB.initialize(RealVoltDB.java:6 20)
    at org.voltdb.VoltDB.initialize(VoltDB.java:897)
    at org.voltdb.VoltDB.main(VoltDB.java:881)
    Caused by: java.lang.IllegalArgumentException: hdfs endpoint "http://localhost:50070/webhdfs/v1.0/%t/data%p.%t.csv" must contain the (%t)able, the (%p)artition, and the (%g) generation conversions
    at com.google_voltpatches.common.base.Preconditions.c heckArgument(Preconditions.java:122)
    at org.voltdb.exportclient.decode.EndpointExpander.ve rifyForHdfsUse(EndpointExpander.java:246)
    at org.voltdb.exportclient.HttpExportClient.configure (HttpExportClient.java:239)


    So, I appended %g to the file name, and then I get another error

    ERROR: Notification request failed with HTTP/1.1 405 HTTP method PUT is not supported by this URL
    ERROR: Unable to prime http export client to http://localhost:50070/webhdfs/v1.0/..._GIVEN_EXP.csv org.voltdb.exportclient.HttpExportClient$PathHandl ingException: hdfs makePath returned false for http://localhost:50070/webhdfs/v1.0/..._GIVEN_EXP.csv
    at org.voltdb.exportclient.HttpExportClient$HttpExpor tDecoder.processRow(HttpExportClient.java:834)
    at org.voltdb.export.processors.GuestProcessor$2.run( GuestProcessor.java:221)
    at java.util.concurrent.ThreadPoolExecutor.runWorker( ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:617)
    at org.voltcore.utils.CoreUtils$7$1.run(CoreUtils.jav a:735)
    at java.lang.Thread.run(Thread.java:745)


    I also tried the following URLs



    which returned the same error as Notification request failed with HTTP/1.1 405 HTTP method PUT is not supported by this URL

    What might I possibly be doing wrong?

  • #2
    Hi interntest,

    I just updated the app-bank-offers to provide two example deployment.xml files for the Hortonworks sandbox, and Cloudera quickstart. You were correct to add %g, that was missing from the endpoint and is required.

    I think you need to set up a hostname alias in your /etc/hosts file for your hadoop instance. Is it one of these VM-based hadoop instances, or can you describe your Hadoop installation?

    The REST interface of WebHDFS involves some redirects. So you may be able to make the initial connection on localhost:50070, but then it redirects you to another page for part of the actions, and that may be on quickstart.cloudera:50070 or sandbox.hortonworks.com:50070, which your host might not recognize.

    If you contact me by email at support@voltdb.com, I can send you some word document instructions for integrating with Cloudera or Hortonworks.

    Best regards,
    Ben

    Comment


    • #3
      Thanks Ben.

      Your post was really helpful and the updated deployment.xml files did the trick.
      Although I could see my files HDFS explorer (http://127.0.0.1:50070/explorer.html), I continuously get the following error on my terminal

      ERROR: Notification request failed with HTTP/1.1 403 Forbidden
      ERROR: Unable to complete request to "POST http://hdp.voltdb1.com:50070/webhdfs...user.name=root HTTP/1.1". Reason:

      org.voltdb.exportclient.ExportDecoderBase$RestartB lockException: requeing on failed response check: FAIL
      at org.voltdb.exportclient.HttpExportClient$HttpExpor tDecoder.onBlockCompletion(HttpExportClient.java:9 28)
      at org.voltdb.export.processors.GuestProcessor$2.run( GuestProcessor.java:223)
      at java.util.concurrent.ThreadPoolExecutor.runWorker( ThreadPoolExecutor.java:1142)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:617)
      at org.voltcore.utils.CoreUtils$7$1.run(CoreUtils.jav a:735)
      at java.lang.Thread.run(Thread.java:745)

      What is wrong here despite my data being exported successfully to HDFS ?

      Comment


      • #4
        This is showing that VoltDB is getting a 403 response code from WebHDFS. Usually a 403 error means the user doesn't have permission to do the action. You might check the permissions to the folder in HDFS.

        Comment


        • #5
          Thanks Ben
          I had to change ownership(chown) of the folder in HDFS and the error was gone. :)

          Comment

          Working...
          X