Krishna's Creations...: XSLT

Showing posts with label XSLT. Show all posts

Saturday, April 5, 2014

Extracting Data from XML files using XSL and XPath...8

Comments in HTML file

Sometimes we might need to have comments in HTML file that we generate from the XML file. This could be needed if we require opening the HTML file and edit it. Comments can then help us understand the contents of the HTML file. But we cannot add comments using <!—and --> since that allows the XML Starlet interpreter to skip anything within that. XSL has a comment tag for that. Hence, if we need to add a comment in the file that is generated, we can use the construct:
<xsl:comment> --------------------------- </xsl:comment>
An example is given below:

In the above code snippet, we are doing something to get data for entity ‘DC10’. This is being put in some table column (indicated by <td> tag). Note the comment at the top. The output HTML will look something like this:

The ‘./DC10’ in the XSL code generated value as 536 in HTML.

Important points to remember

1. The understanding of XML structure is very important; unless you understand it well don’t start coding. Draw a hierarchy diagram of XML tags to understand the XML structure. The best way to know the XML structure is to go through the file manually in Notepad++. Besides this, there is a command that can be used:

xml el table.xml

The above command when run through in the XML starlet folder on the command prompt, will give the structure of table.xml file. The output can look somewhat like below:

I have marked the structure of one the files on which I ran the command. The ‘el’ command is XML Starlet command (just as ‘tr’ is).
2. If one understands the XML structure, data can be handled and extracted from almost any XML.
3. Basic DOS commands must be understood by the developer. It will be better if you can prepare a batch file within the XML Starlet folder. Batch file can contain code something like this:

del Transition_Flag_DC23_TPData.csv
xml tr ALL_Test_CasesDC2-3Transition.xsl test.xml >> Transition_Flag_DC23_TPData.csv

This code deletes the old csv file. After that it runs the xml transform command on the ‘test.xml’ using the ‘ALL_Test_CasesDC2-3Transition.xsl’file producing the output in new ‘Transition_Flag_DC23_TPData.csv’ file.
4. It will also be better if the XSLT code and XML code be present in the XML Starlet folder. In that case the running of the batch file will be easy and it will operate in that folder, producing the desired output file.
5. Sometimes the XML Starlet refuses to parse the XML file stating that it contains and error. In that case you will have to edit the XML file in Notepad++, such that any tags causing the problem are either deleted or edited. In case of XML starlet erred out because of XSLT, code editing is required.

Advantages and Disadvantages of XML and XSLT
Advantages
1. XSLT applies user defined transformations to an XML document and the output can be HTML, XML, or any other structured document. So it is easy to merge XML data into presentation.
2. XPath is used by XSLT to locate elements/attribute within an XML document. So it is more convenient way to traverse an XML document rather than a traditional way, by using scripting language.
3. By separating data (XML document) from the presentation (XSLT), it is very easy to change the output format in any time easily without touching the code-behind.
4. The output is generated at a fast pace.
5. XML is platform independent.

Disadvantages
1. It is difficult to implement complicate business rules in XSLT.
2. Changing variable value in looping is difficult in XSLT.
3. Development and maintenance of code in XSLT is difficult as it is different from traditional programming languages such as C, C++ and Java.
4. Using XSLT have performance penalty in some cases as its engine (e.g. XML starlet) doesn’t optimize code by using caching technique like traditional compiler.
5. When processing XML files, XSLT must load the entire document into memory. With Xalan, this consumes roughly 10x the size of the input file (saxon has an alternative DOM implementation that uses less memory). If any of your input datasets grow beyond a couple of hundred megabytes, your XSLT processor might just flake out. This has been taken care of in the XSLT 3.0.

Important resources/links

1. http://www.xsltfunctions.com/xsl/
2. http://stackoverflow.com/ is a website where users can ask questions related to XSLT.
3. http://www.w3schools.com/xsl/
4. XSLT Cookbook is a good book to refer for XSLT coding.

Extracting Data from XML files using XSL and XPath...6

Removing the white space from the XML file

Sometimes when we try to shorten an XML file (see the above discussion on that), the file contains a lot of white space because of removed tags. New XML file can look something like this:

Such white space can be removed by adding the following template construct:

Just add in your code before the </stylesheet> tag. The new XML file looks as follows:

If not all, then at least some white space will go. See this link.

Conditional reports (based on group)

In point 4, I stated that we needed an exception for one of the groups. Extending that idea, we can actually segregate the reports pertaining to a single group also. The code will run as follows:

Notice that since we are taking counts at the testplan level, we are using ‘../’ 2 times. The above code can also be written as:

Notice that now we have just one ‘../’ because the condition is at the repository level. One can very well imagine that if we are taking all testcases, we can actually restrict the output by putting conditions at several levels (datasource, repository, testplan, build, platform etc. depending on the XML structure) E.g.

In the above code, the restrictive conditions at the datasource level, repository level, testplan level, build level and platform level can help us limit the number of testcases that appear in our output file. The above is essentially an advanced XPath expression, because we have introduced additional conditions at different navigation levels.

A typical problem

Recently, I faced a problem when I had to access two parts of an XML file. Actually, I had put condition on the 2^nd part of the XML, which was not possible because there were two for-each loops involved. The XML Structure was something like.

The platforms, testplans, and execution_testplan were all at the same level. The outer ‘for-each’ loop accesses all the ‘execution_build’ tags. The inner ‘for-each’ loop accesses (based on the conditions), the details of ‘testplan’ tags (within ‘testplans’). To see how we can access the ‘testplan’, please see ‘Handling XML with non-regular structure’.

A condition needed to put on the Phase (a custom attribute within the testplan tag) value for a testplan. That looked impossible, since the Phase was in the inner loop where if the condition is true, will give blank data values for data elements within inner element, but will give some values for the execution_build tag. Putting the condition was necessary because we needed to segregate the report by Phase value.

The problem can be tackled easily. I created a new intermediate XML file from the original file. The new intermediate file contained all data values form inner and outer loop. Since it had no conditions, it contained all the data. The code for creating the intermediate XML file looked as follows:

The xsl:template match tag creates a template with the rules defined within the tag and applies to all execution_build tags. As you can also see the xsl:output method is ‘xml’ because the output file will be an XML file to be used by our next XSL code that will use this XML file for generating the final output file. Also, since we are generating the XML file, we can add our own tags. For example <Instance> tag has been added and it gives the value of instance with respect to the current execution_build (note the usage of ../ a number of times). Also, see how a tag is given value using the ‘xsl:value-of’ tag within a tag. The lower part of the code accessed the details of the execution_build.

The xsl:copy will then define the copying of all other tags within execution_build. The xsl:apply-templates then applies that template to the XML file to create a new XML file. Also see that platforms and testplans tags and their contents are all ignored (see ‘Handling a big XML file’ for details).

The XML file that was created had the data as follows:

As you can see the execution_build contains all the details that we need in our output file. Hence the XSL code to generate the output file looked as follows:

Note that for an execution_build, this data is contained just above it in the XML file. Also, it is parent to the execution_build, hence the usage of ../. This can be changed if required. Now putting a condition became fairly easy because the Phase value lies close (in terms of hierarchy) to execution_build.

Friday, April 4, 2014

Extracting Data from XML files using XSL and XPath...5

Accessing another XML file outside of current file

If we need to access a second XML file and get values from that file, it is possible to do that in XSLT.

The above tag will access the ‘server’ node of 'lt00tm400000001_life_Life_Repository_TestSuites.xml' file and store that value in ‘TCVersion’ variable. Hence, moving forward we can access any node in that file using this variable, as below:

Note that ‘//’ has been used to get directly to testcase_version.

Tackling the comma ‘,’

Sometimes there are values that can have commas. In that case, for our convenience we can use a construct that detects a comma (‘,’) in a value and replaces it with a space (‘ ‘) or any other separator (such as a ‘;‘or a ‘|’). Example:

In the above code snippet, we have used the xsl:choose-when-otherwise construct. If the testcase_name contains a comma, it will replaced by a space, ‘otherwise’ we will take the testcase_name as is.

Notice the ‘contains’ and ‘translate‘functions used. ‘contains’ will detect the presence of comma in ‘testcase_name’ and return true or false. If true, the translate function will replace comma with a space, else ‘testcase_name’ will be selected as is. The above code can be shortened to:

Just by using this, we can accomplish what we want. If case a testcase_name contains a ‘,’, it will get replaced by a ‘ ‘.This statement takes care of appearance of all ‘,’s coming in testcase_name.

More on Substring

There is another function that helps us in getting a part of a string. This was particularly helpful when the defect_id in the XML file was something like – ‘XXXXX_YYYYYY_12345’. Out of such a string the only useful part is ‘12345’. This can be accomplished by using a code as:

In the ‘Defect’ variable, ‘12345’ will be stored. Note that the substring-after function is used twice to remove the two underscores. First the inner bracket is executed. Hence, we get ‘YYYYYY_12345’ after first substring-after is executed. The second one then removes ‘YYYYYY_’ to store only ‘12345’ in ‘Defect’.

Accessing another level (getting into more depth)

Just take note of the structure of the XML again.

<build>

<!—test case level custom attributes -->

</testcase>

</platform>

</build>

</testplan>

</repository>

</datasource>

To access the parent tags and the corresponding xml tags within those tags we can use ‘../’. So, how can this be more useful? Consider a scenario where we need all the test cases from the file with details such as datasource_description, repository, testplan, build, platform, and testcase execution status and other details. How can we do that?

Just dive deep down to the level of a testcase and get its details. Also, get the other values using ‘../’ several times depending upon the parenthood of tags. Following code does that:

Notice the appearance of ‘../’, a number of times, to access the various fields that are not at the level of ‘testcase’.

Moving to next line (line feed)

While creating a csv file, the control needs to move to the next line for the next record. This can be accomplished by using the following construct:

Note that $Defect is the value of variable after which we need a line break.

<xsl:text>
</xsl:text> inserts a line break so that for next ‘for-each’ the entries start populating on the next row.

Handling XML with non-regular structure

Sometimes the XML on which we need to work might not be structured uniformly. By uniform we mean that, the structure does not follow a top to bottom hierarchy. The hierarchy will be present but there may be more than one path that the tags follow to reach the last tag within them. The above XML had a uniform structure, since every tag had a parent and there was proper hierarchy from datasource to testcase. ‘Proper’ hierarchy means that there are no parallel paths as is the case with below XML.

Have a look at the following XML:

(-) sign indicates that repository tag is expanded and it contains three tags – platforms, testplans, and executions in parallel. They all have a (+) sign before them indicating that they can be expanded further. On expansion the testplans tag had the following structure:

<testplan_id>

<testplan_name>

<build>

<build_id>

<testplan_platforms>

<testplan_testcases>

<testplan_testcase>

<testcase_version_id>

And executions had the structure:

<execution_testplan>

<testplan_id>

<execution_builds>

<execution_build>

<build_id>

<execution_platforms>

<execution_platform>

<execution_testcase>

<testcase_version_id>

<execution_testcase>

<defect_id>

So, as you can see the executions does not have the testplan name, just the testplan id. So, if you are preparing and csv file that must contain a defect or a execution_testcase and you need to access the testplan_name it can be done using the following XSLT code:

a) First access the testplan_id in executions tag and store it in variable TP1:

Notice the number of ‘../’ used. The code was written for defects (hitting each defect tag), so the ‘for-each’ was something like this:

Hence, to get the testplan_id we needed 7 ‘../’s. The parenthood moves like this -

Just count the number of à here. Those many ‘../’s are required.

b) Next thing was to move out of executions completely to reach the repository or its parent tag.

And then reach out to either ‘testplan_testcase’ or ‘testplan’ directly and retrieve the ‘testplan_id’ and ‘testplan_name’, if we get a match with TP1. At the ‘testplan_testcase’ level, we have to move 2 levels up to get the corresponding ‘testplan_id’. All this is happening within the ‘testplans’ tag.

Notice the use of ‘//’ to jump to ‘testplan_testcase’ and then accessing its ‘testplan_id’ and checking for its first equivalent. We just needed the first occurence here so [1] was used to get first value and continue. The complete code snippet looked as follows:

Within the internal ‘for-each’ we are taking the ‘testplan_id’ and ‘testplan_name’ from the ‘testplans’ tag. Then we come out of it to get ‘build_id’ and ‘platform_id’ for the ‘defect’. Note the use of ../ several times to get the respective values. Again, the number of ../ depends upon the arrows à (see the above explanation).

Handling a big XML file

Sometimes an XML file becomes too big to handle. In that case XML Starlet will not throw an error, but it will just hang, without doing anything. Sometimes, on the DOS prompt, it displays – ‘Out of Memory’. That is one of the problems that XSLT 2 does not tackle. Actually XML Starlet interpreter wants to load the whole XML file before parsing it, in the memory, but when the file is too big, may be because the file is accessed several times in the code, then it will hang (this might also happen when the number of ‘../’ that we give is wrong, or when we don’t comply with the hierarchy or parenthood of tags, that’s why understanding the structure of XML is extremely important). In that case there is one way we can handle such a case. By making our XML file small, or in other words, creating an intermediate file containing only the stuff that we really need, we can ease our job. This small code does the job for you:

All the tags which are contained in xsl:template tag – summary, preconditions, steps, custom_fields were all removed in the new XML file. The left over XML contained all the test cases that we needed for our use. Using this code the file size reduced from 95MB to 14MB, and we got what we most wanted – the testcase names, testcase ids and testcase version ids.

The other way is to use XSLT 3 enabled XSLT processor such as Saxon PE 9.5. XSLT 3 enables streaming of XML documents, such that the interpreter will not load the complete document in memory, but takes it as the file loads in memory.

Besides this, usage of XML Splitter is recommended to break an XML file into multiple parts so that smaller parts can be handled separately.

Extracting Data from XML files using XSL and XPath...4

Counting a tag – ‘count’ function and the tag value – ‘sum’ function

The function ‘count’ is used to count an entity in the current ‘for-each’ ‘testplan’.

So, since we are at the ‘testplan’ level, we are counting the total occurrences of the ‘testcase_id’ within that ‘testplan’. This could also be accomplished by using only

<xsl:value-of select="count(.//testcase_id)"/>

This command is equally functional, just that the former one shows the hierarchy. There is a count of ‘tc_test_design_status’. This is a conditional count, which counts all the test cases when the testcase has the custom attribute of ‘tc_test_design_status’ set as ‘Complete’. See the XML below (under a testcase tag):

Another example of conditional copying can be seen as follows, when we count the number of ‘passed’ testcases (indicated with value ‘p’ in the Example.xml)

The value for testcase to be passed must come from XML as follows:

Remember that all these counts are within ‘testplan’, so when we use ‘//’ within that, we go directly to the ‘tcexecution_status’ tag and count when its value is ‘p’.

Did you notice the XPath expression? Here it is:

‘.//build/platform/testcase/testcase_id’

On the similar lines, we might sometimes need to sum the value within the tags. Consider the following XML:

</DCDetails>

</DCDetails>

If we have to get the sum of <TTLPTCs> for both the DCDetails, then we need to use the function as:

<xsl:value-of select = “sum(DCDetails/TTLPTCs)”>

assuming that we are at the DCDetails level. The difference between count and sum is that ‘count’ is used to count the number of occurrences of a tag, while sum is used to add the contents of a tag.

We can also use conditional sum. For example, consider the following XML:

<Test_Type>DC System</Test_Type>

<BP>Purchase</BP>

<LOB>Multi - LOB</LOB>

<TTLs>

</TTLs>

</PlanDetails>

</DCBuildRecord>

</DCRecord>

<Test_Type>Integration</Test_Type>

<BP>Payments</BP>

<LOB>Multi - LOB</LOB>

<TTLs>

</TTLs>

</PlanDetails>

</DCBuildRecord>

</DCRecord>

To get the totals depending on various conditions following code was used:

<xsl:value-of select = "sum(//PlanDetails[../../DC = 'DC 8' and Test_Type = 'Integration' and BP = 'Purchase' and (Phase = '2014-R3' or Phase = 'ICP_Ph1.001.500.000')]//TTLTCs)"/>

</TTLTCs>

<xsl:text>
</xsl:text>

<xsl:value-of select = "sum(//PlanDetails[../../DC = 'DC 8' and Test_Type = 'Integration' and BP = 'Purchase' and (Phase = '2014-R3' or Phase = 'ICP_Ph1.001.500.000')]//TTLBTCs)"/>

</TTLBTCs>

<xsl:text>
</xsl:text>

<xsl:value-of select = "sum(//PlanDetails[../../DC = 'DC 8' and Test_Type = 'Integration' and BP = 'Purchase' and (Phase = '2014-R3' or Phase = 'ICP_Ph1.001.500.000')]//TTLPTCs)"/>

</TTLPTCs>

<xsl:text>
</xsl:text>

<xsl:value-of select = "sum(//PlanDetails[../../DC = 'DC 8' and Test_Type = 'Integration' and BP = 'Purchase' and (Phase = '2014-R3' or Phase = 'ICP_Ph1.001.500.000')]//TTLFTCs)"/>

</TTLFTCs>

The output of the above file could be an XML file shown below:

Carefully note the use of brackets and conditions and the use of ../ in the above example.

‘Choose:when:otherwise’ construct

Under the section – “Getting the values – conditional ‘xsl:value-of’ tag” we counted testcases that had the custom attribute of ‘tc_test_design_status’ set as ‘Complete’. For the group that had an exception, we actually had Test Design Status set at the ‘testplan’ level, so there is nothing to count. The value will be set at ‘testplan’ level. Hence for that the code was written as follows:

Here we have new construct called choose:when:otherwise, that is similar to if:then:else.

‘Choose’ is just selecting something ‘when’ a condition is true; ‘otherwise’ the other statement is executed (something else is selected).

It is evident that the condition is put at the ‘testplan’ level and if the value is set correctly ‘Complete’, we will count all testcases within that ‘testplan’.

Counting at a different level of hierarchy

The next requirement was to get the counts at the level of ‘testplans’ and also the builds. For ‘testplans’, the above points explain how to get the totals. But, now when we go down to the level of builds, then we can get totals only at the builds level, in the following way:

Ignore the conditions in between; they are similar to previously discussed code. Just take note of how we have gone one level down to builds. But now, when we will use the ‘./’ construct, we will get all the attributes (fields) at the build level only. How to get the values at the ‘testplan’ level (custom attributes, ‘testplan_name’, ‘testplan_id’, Total TCs at Test Plan level etc.)? Simple, we will use the construct ‘../’, - to go one level up from build and get the values. This will be applied to all fields that we need.

As it is evident from the above code snippet, we have just included an extra ‘../’ to get ‘datasource_description’ and ‘repository_id’ values (as compared to that when we did the same thing at the ‘testplan’ level). Also, to get the values at the ‘testplan’ level, ‘../’ has been included in place of ‘./’ since we are now looking at one level up the ‘build’. Similarly the following code will give us totals at the ‘testplan’ level even though ‘for-each’ has gone down to ‘build’ level.

Read the above statement simply as – Give total number of ‘testcase_id’s within one level up the current level.

Variables in XSLT

Use of variables in XSLT is different from the way we use them in conventional structured programming languages such as C, C++, Java etc. There is a scope limitation for each variable, out of which it cannot be used. You cannot reassign a variable within that scope and neither can you re-declare it.

This tag will put the value of ‘testplan_id’ into variable TP1. To use a variable we need to prefix it with a $ sign as follows:

Variables are immutable, meaning that when once assigned with a value, they cannot be changes. In a way, they behave like constants.