Friday, February 15, 2013

new MaxL "export outline" XML to delimited text parser - orders of magnitude faster

The Java code in my last post works fine for XML containing less than 70,000 members, which is usually sufficient for BSO cubes. ASO cubes, however, can contain a LOT more members. Dan Pressman contacted me earlier in the week to let me know that his XML dimension export file containing 4 million members in almost 90 million lines of XML was taking over 17 hours to run (when it wasn't crashing from running out of memory).

So I rewrote the code to make it more efficient. The same XML above with 90 million lines now takes about 3.5 minutes to run and only uses 75MB of RAM. The functionality is even improved, in that member attributes are now also output. A header line is also prepended, which, due to Java I/O, adds about a minute to the process.

I recommend using this new code for all files instead of the code in the previous post. You can find the two files needed in my GitHub essbase-parse-export-outline-xml-to-text repository.

I also created a very simple Java GUI that allows you to choose the input file, output file, and delimiter:

You can use the XMLOutlineParser.jar file from the above GitHub repository to launch the GUI. If you have a fairly recent Java jre or jdk installed (version 6 or higher), you should be able to save the jar file to your local drive, then double-click on it to launch the GUI.

P.S. As you can see in the comments, some people have experienced issues when trying to save the output file on Windows. The following steps are the workaround to this permissions issue:
  1. Go to Start -> Programs -> Accessories
  2. right click on Command Prompt and select "Run as administrator"
  3. cd to the directory where you saved essbase-parse-export-outline-xml-to-text
  4. run "java -jar XMLOutlineParser.jar"
You can avoid this issue entirely by not selecting an output location at the root of the C directory (e.g. C:\test). Rather use one such as "C:\Documents and Settings\harry\outputFile.txt".

Update: I've added the ability to pass in command line arguments, making XMLOutlineParser suitable for scripting in batch and shell scripts. Passing in arguments will prevent the GUI from running. 
  • run java -jar XMLOutlineParser.jar "input file: path & name" "output file: path & name" "field separator to use in output file"
  • -------e.g. java -jar XMLOutlineParser.jar "/Users/harry/year.xml" "/Users/harry/Documents/year.txt" "!"