XML

Extensible Markup Language

The Extensible Markup Language (XML) is a general-purpose markup language.^[1] It is classified as an extensible language because it allows its users to define their own elements. Its primary purpose is to facilitate the sharing of structured data across different information systems, particularly via the Internet.^[2] It is used both to encode documents and serialize data. In the latter context, it is comparable with other text-based serialization languages such as JSON and YAML.^[3]

It started as a simplified subset of the Standard Generalized Markup Language (SGML), and is designed to be relatively human-legible. By adding semantic constraints, application languages can be implemented in XML. These include XHTML,^[4] RSS, MathML, GraphML, Scalable Vector Graphics, MusicXML, and thousands of others. Moreover, XML is sometimes used as the specification language for such application languages.

XML is recommended by the World Wide Web Consortium. It is a fee-free open standard. The W3C recommendation specifies both the lexical grammar, and the requirements for parsing.

Well-formed and valid XML documents

There are two levels of correctness of an XML document:

• Well-formed. A well-formed document conforms to all of XML's syntax rules. For example, if a start-tag appears without a corresponding end-tag, it is not well-formed. A document that is not well-formed is not considered to be XML; a conforming parser is not allowed to process it.
• Valid. A valid document additionally conforms to some semantic rules. These rules are either user-defined, or included as an XML schema or DTD. For example, if a document contains an undefined element, then it is not valid; a validating parser is not allowed to process it.

Well-formed documents: XML syntax

As long as only well-formedness is required, XML is a generic framework for storing any amount of text or any data whose structure can be represented as a tree. The only indispensable syntactical requirement is that the document has exactly one root element (alternatively called the document element). This means that the text must be enclosed between a root start-tag and a corresponding end-tag. The following is a "well-formed" XML document:

<book>This is a book.... </book>

The root element can be preceded by an optional XML declaration. This element states what version of XML is in use (normally 1.0); it may also contain information about character encoding and external dependencies.

<?xml version="1.0" encoding="UTF-8"?>

The specification requires that processors of XML support the pan-Unicode character encodings UTF-8 and UTF-16 (UTF-32 is not mandatory). The use of more limited encodings, such as those based on ISO/IEC 8859, is acknowledged and is widely used and supported.

Comments can be placed anywhere in the tree, including in the text if the content of the element is text or #PCDATA.

XML comments start with <!-- and end with -->. Two dashes (--) may not appear anywhere in the text of the comment.

<!-- This is a comment. -->

In any meaningful application, additional markup is used to structure the contents of the XML document. The text enclosed by the root tags may contain an arbitrary number of XML elements. The basic syntax for one element is:

<name attribute="value">content</name>

The two instances of »name« are referred to as the start-tag and end-tag, respectively. Here, »content« is some text which may again contain XML elements. So, a generic XML document contains a tree-based data structure. Here is an example of a structured XML document:

<recipe name="bread" prep_time="5 mins" cook_time="3 hours">

   <title>Basic bread</title>

   <ingredient amount="3" unit="cups">Flour</ingredient>

   <ingredient amount="0.25" unit="ounce">Yeast</ingredient>

   <ingredient amount="1.5" unit="cups" state="warm">Water</ingredient>

   <ingredient amount="1" unit="teaspoon">Salt</ingredient>

   <instructions>

     <step>Mix all ingredients together.</step>

     <step>Knead thoroughly.</step>

     <step>Cover with a cloth, and leave for one hour in warm room.</step>

     <step>Knead again.</step>

     <step>Place in a bread baking tin.</step>

     <step>Cover with a cloth, and leave for one hour in warm room.</step>

     <step>Bake in the oven at 350°F for 30 minutes.</step>

   </instructions>

 </recipe>

Attribute values must always be quoted, using single or double quotes; and each attribute name should appear only once in any element. XML requires that elements be properly nested — elements may never overlap. For example, the code below is not well-formed XML, because the em and strong elements overlap:

<!-- WRONG! NOT WELL-FORMED XML! -->

<p>Normal <em>emphasized <strong>strong emphasized</em> strong</strong></p>

<!-- Correct: Well-formed XML. -->

<p>Normal <em>emphasized <strong>strong emphasized</strong></em> <strong>strong</strong></p>

<p>Alternatively <em>emphasized</em> <strong><em>strong emphasized</em> strong</strong></p>

XML provides special syntax for representing an element with empty content. Instead of writing a start-tag followed immediately by an end-tag, a document may contain an empty-element tag. An empty-element tag resembles a start-tag but contains a slash just before the closing angle bracket. The following three examples are equivalent in XML:

<foo></foo>

<foo />

<foo/>

An empty-element may contain attributes:

<info author="John" genre="science-fiction" date="2009-Jan-01" />

Entity references

An entity in XML is a named body of data, usually text. Entities are often used to represent single characters that cannot easily be entered on the keyboard; they are also used to represent pieces of standard ("boilerplate") text that occur in many documents, especially if there is a need to allow such text to be changed in one place only.

Special characters can be represented either using entity references, or by means of numeric character references. An example of a numeric character reference is "€", which refers to the Euro symbol by means of its Unicode codepoint in hexadecimal.

An entity reference is a placeholder that represents that entity. It consists of the entity's name preceded by an ampersand ("&") and followed by a semicolon (";"). XML has five predeclared entities:

&	&	ampersand
<	<	less than
>	>	greater than
'	'	apostrophe
"	"	quotation mark

Here is an example using a predeclared XML entity to represent the ampersand in the name "AT&T":

<company_name>AT&T</company_name>

Additional entities (beyond the predefined ones) can be declared in the document's Document Type Definition (DTD). A basic example of doing so in a minimal internal DTD follows. Declared entities can describe single characters or pieces of text, and can reference each other.

<?xml version="1.0" encoding="UTF-8"?>

<!DOCTYPE example [

<!ENTITY copy "©">

<!ENTITY copyright-notice "Copyright © 2006, XYZ Enterprises">

]>

<example>

&copyright-notice;

</example>

When viewed in a suitable browser, the XML document above appears as:

<example> Copyright © 2006, XYZ Enterprises </example>

Numeric character references

Numeric character references look like entity references, but instead of a name, they contain the "#" character followed by a number. The number (in decimal or "x"-prefixed hexadecimal) represents a Unicode code point. Unlike entity references, they are neither predeclared nor do they need to be declared in the document's DTD. They have typically been used to represent characters that are not easily encodable, such as an Arabic character in a document produced on a European computer. The ampersand in the "AT&T" example could also be escaped like this (decimal 38 and hexadecimal 26 both represent the Unicode code point for the "&" character):

<company_name>AT&T</company_name>

<company_name>AT&T</company_name>

Similarly, in the previous example, notice that “&#xA9” is used to generate the “©” symbol.

See also numeric character references.

Well-formed documents

A well-formed document must conform to the following rules, among others:

• Non-empty elements are delimited by both a start-tag and an end-tag.
• Empty elements may be marked with an empty-element (self-closing) tag, such as <IAmEmpty />. This is equal to <IAmEmpty></IAmEmpty>.
• All attribute values are quoted with either single (') or double (") quotes. Single quotes close a single quote and double quotes close a double quote.
• Tags may be nested but must not overlap. Each non-root element must be completely contained in another element.
• The document complies with its declared character encoding. The encoding may be declared or implied externally, such as in "Content-Type" headers when a document is transported via HTTP, or internally, using explicit markup at the very beginning of the document. When no such declaration exists, a Unicode encoding is assumed, as defined by a Unicode Byte Order Mark before the document's first character. If the mark does not exist, UTF-8 encoding is assumed.

Element names are case-sensitive. For example, the following is a well-formed matching pair:

<Step> ... </Step>

whereas this is not

<Step> ... </step>

The careful choice of names for XML elements will convey the meaning of the data in the markup. This increases human readability while retaining the rigor needed for software parsing.

Choosing meaningful names implies the semantics of elements and attributes to a human reader without reference to external documentation. However, this can lead to verbosity, which complicates authoring and increases file size.

Automatic verification

It is relatively simple to verify that a document is well-formed or validated XML, because the rules of well-formedness and validation of XML are designed for portability of tools. The idea is that any tool designed to work with XML files will be able to work with XML files written in any XML language (or XML application). One example of using an independent tool follows:

• load it into an XML-capable browser, such as Firefox or Internet Explorer
• use a tool like xmlwf (usually bundled with expat)
• parse the document, for instance in Ruby:

irb> require "rexml/document"

irb> include REXML

irb> doc = Document.new(File.new("test.xml")).root

Valid documents: XML semantics

By leaving the names, allowable hierarchy, and meanings of the elements and attributes open and definable by a customizable schema or DTD, XML provides a syntactic foundation for the creation of purpose specific, XML-based markup languages. The general syntax of such languages is rigid — documents must adhere to the general rules of XML, ensuring that all XML-aware software can at least read and understand the relative arrangement of information within them. The schema merely supplements the syntax rules with a set of constraints. Schemas typically restrict element and attribute names and their allowable containment hierarchies, such as only allowing an element named 'birthday' to contain 1 element named 'month' and 1 element named 'day', each of which has to contain only character data. The constraints in a schema may also include data type assignments that affect how information is processed; for example, the 'month' element's character data may be defined as being a month according to a particular schema language's conventions, perhaps meaning that it must not only be formatted a certain way, but also must not be processed as if it were some other type of data.

An XML document that complies with a particular schema/DTD, in addition to being well-formed, is said to be valid.

An XML schema is a description of a type of XML document, typically expressed in terms of constraints on the structure and content of documents of that type, above and beyond the basic constraints imposed by XML itself. A number of standard and proprietary XML schema languages have emerged for the purpose of formally expressing such schemas, and some of these languages are XML-based, themselves.

Before the advent of generalised data description languages such as SGML and XML, software designers had to define special file formats or small languages to share data between programs. This required writing detailed specifications and special-purpose parsers and writers.

XML's regular structure and strict parsing rules allow software designers to leave parsing to standard tools, and since XML provides a general, data model-oriented framework for the development of application-specific languages, software designers need only concentrate on the development of rules for their data, at relatively high levels of abstraction.

Well-tested tools exist to validate an XML document "against" a schema: the tool automatically verifies whether the document conforms to constraints expressed in the schema. Some of these validation tools are included in XML parsers, and some are packaged separately.

Other usages of schemas exist: XML editors, for instance, can use schemas to support the editing process (by suggesting valid elements and attributes names, etc).

DTD

Main article: Document Type Definition

The oldest schema format for XML is the Document Type Definition (DTD), inherited from SGML. While DTD support is ubiquitous due to its inclusion in the XML 1.0 standard, it is seen as limited for the following reasons:

• It has no support for newer features of XML, most importantly namespaces.
• It lacks expressiveness. Certain formal aspects of an XML document cannot be captured in a DTD.
• It uses a custom non-XML syntax, inherited from SGML, to describe the schema.

DTD is still used in many applications because it is considered the easiest to read and write.

XML Schema

Main article: XML Schema (W3C)

A newer XML schema language, described by the W3C as the successor of DTDs, is XML Schema, or more informally referred to by the initialism for XML Schema instances, XSD (XML Schema Definition). XSDs are far more powerful than DTDs in describing XML languages. They use a rich datatyping system, allow for more detailed constraints on an XML document's logical structure, and must be processed in a more robust validation framework. XSDs also use an XML-based format which makes it possible to use ordinary XML tools to help process them, although XSD implementations require much more than just the ability to read XML.

Criticisms of XSD include the following:

• The specification is very large, which makes it difficult to understand and implement.
• The XML-based syntax leads to verbosity in schema description, which makes XSDs harder to read and write.
• Schema validation can be an expensive addition to XML parsing, especially for high volume systems.
• The modeling capabilities are very limited, with no ability to allow attributes to influence content models.
• The type derivation model is very limited, in particular that derivation by extension is rarely useful.
• Database-related data transfer has been supported with arcane ideas such as nillability but the requirements of industrial publishing are under-supported.
• The key/keyref/uniqueness mechanisms are not type aware
• The PSVI concept (Post Schema Validation Infoset) does not have a standard XML representation or Application Programming Interface, thus it works against vendor independence unless revalidation is performed.

RELAX NG

Main article: RELAX NG

Another popular schema language for XML is RELAX NG. Initially specified by OASIS, RELAX NG is now also an ISO international standard (as part of DSDL). It has two formats: an XML based syntax and a non-XML compact syntax. The compact syntax aims to increase readability and writability but, since there is a well-defined way to translate the compact syntax to the XML syntax and back again by means of James Clark's Trang conversion tool, the advantage of using standard XML tools is not lost. RELAX NG has a simpler definition and validation framework than XML Schema, making it easier to use and implement. It also has the ability to use datatype framework plug-ins; a RELAX NG schema author, for example, can require values in an XML document to conform to definitions in XML Schema Datatypes.

ISO DSDL and Other Schema Languages

The ISO DSDL (Document Schema Description Languages) standard brings together a comprehensive set of small schema languages, each targeted at specific problems. DSDL includes RELAX NG full and compact syntax, Schematron assertion language, and languages for defining datatypes, character repertoire constraints, renaming and entity expansion, and namespace-based routing of document fragments to different validators. DSDL schema languages do not have the vendor support of XML Schemas yet, and are to some extent a grassroots reaction of industrial publishers to the lack of utility of XML Schemas for publishing.

Some schema languages not only describe the structure of a particular XML format but also offer limited facilities to influence processing of individual XML files that conform to this format. DTDs and XSDs both have this ability; they can for instance provide attribute defaults. RELAX NG and Schematron intentionally do not provide these; for example the infoset augmentation facility.

International use

XML supports the direct use of almost any Unicode character (other than the ones that have special symbolic meaning in XML, itself, such as the open corner bracket, "<") in element names, attributes, comments, character data, and processing instructions. Therefore, the following is a well-formed XML document, even though it includes both Chinese and Cyrillic characters:

<?xml version="1.0" encoding="UTF-8"?>

<俄語>Данные</俄語>

Displaying XML on the web

XML documents do not carry information about how to display the data. Without using CSS or XSL, a generic XML document is rendered as raw XML text by most web browsers. Some display it with 'handles' (e.g. + and - signs in the margin) that allow parts of the structure to be expanded or collapsed with mouse-clicks.

In order to style the rendering in a browser with CSS, the XML document must include a reference to the stylesheet:

<?xml-stylesheet type="text/css" href="myStyleSheet.css"?>

Note that this is different from specifying such a stylesheet in HTML, which uses the <link> element.

Extensible Stylesheet Language (XSL) can be used to alter the format of XML data, either into HTML or other formats that are suitable for a browser to display.

To specify client-side XSL Transformation (XSLT), the following processing instruction is required in the XML:

<?xml-stylesheet type="text/xsl" href="myTransform.xslt"?>

Client-side XSLT is supported by many web browsers, but not Opera before version 9.0. Alternatively, one may use XSL to convert XML into a displayable format on the server rather than being dependent on the end-user's browser capabilities. The end-user is not aware of what has gone on 'behind the scenes'; all they see is well-formatted, displayable data.

See the XSLT article for an example of server-side XSLT in action.

XML extensions

• XPath makes it possible to refer to individual parts of an XML document. This provides random access to XML data for other technologies, including XSLT, XSL-FO, XQuery etc. XPath expressions can refer to all or part of the text, data and values in XML elements, attributes, processing instructions, comments etc. They can also access the names of elements and attributes. XPaths can be used in both valid and well-formed XML, with and without defined namespaces.
• XInclude defines the ability for XML files to include all or part of an external file. When processing is complete, the final XML infoset has no XInclude elements, but instead has copied the documents or parts thereof into the final infoset. It uses XPath to refer to a portion of the document for partial inclusions.
• XQuery is to XML what SQL and PL/SQL are to relational databases: ways to access, manipulate and return XML.
• XML Namespaces enable the same document to contain XML elements and attributes taken from different vocabularies, without any naming collisions occurring.
• XML Signature defines the syntax and processing rules for creating digital signatures on XML content.
• XML Encryption defines the syntax and processing rules for encrypting XML content.
• XPointer is a system for addressing components of XML-based internet media.

XML files may be served with a variety of Media types. RFC 3023 defines the types "application/xml" and "text/xml", which say only that the data is in XML, and nothing about its semantics. The use of "text/xml" has been criticized as a potential source of encoding problems but is now in the process of being deprecated.^[5] RFC 3023 also recommends that XML-based languages be given media types beginning in "application/" and ending in "+xml"; for example "application/atom+xml" for Atom. This page discusses further XML and MIME.

Processing XML files

Three traditional techniques for processing XML files are:

• Using a programming language and the SAX API.
• Using a programming language and the DOM API.
• Using a transformation engine and a filter

More recent and emerging techniques for processing XML files are:

• Push Parsing
• Data binding

Simple API for XML (SAX)

SAX is a lexical, event-driven interface in which a document is read serially and its contents are reported as "callbacks" to various methods on a handler object of the user's design. SAX is fast and efficient to implement, but difficult to use for extracting information at random from the XML, since it tends to burden the application author with keeping track of what part of the document is being processed. It is better suited to situations in which certain types of information are always handled the same way, no matter where they occur in the document.

DOM

DOM is an interface-oriented Application Programming Interface that allows for navigation of the entire document as if it were a tree of "Node" objects representing the document's contents. A DOM document can be created by a parser, or can be generated manually by users (with limitations). Data types in DOM Nodes are abstract; implementations provide their own programming language-specific bindings. DOM implementations tend to be memory intensive, as they generally require the entire document to be loaded into memory and constructed as a tree of objects before access is allowed. DOM is supported in Java by several packages that usually come with the standard libraries. As the DOM specification is regulated by the World Wide Web Consortium, the main interfaces (Node, Document, etc.) are in the package org.w3c.dom.*, as well as some of the events and interfaces for other capabilities like serialization (output). The package com.sun.org.apache.xml.internal.serialize.* provides the serialization (output capacities) by impementing the appropriate interfaces, while the javax.xml.parsers.* package parses data to create DOM XML documents for manipulation. [2]

Transformation engines and filters

A filter in the Extensible Stylesheet Language (XSL) family can transform an XML file for displaying or printing.

• XSL-FO is a declarative, XML-based page layout language. An XSL-FO processor can be used to convert an XSL-FO document into another non-XML format, such as PDF.
• XSLT is a declarative, XML-based document transformation language. An XSLT processor can use an XSLT stylesheet as a guide for the conversion of the data tree represented by one XML document into another tree that can then be serialized as XML, HTML, plain text, or any other format supported by the processor.
• XQuery is a W3C language for querying, constructing and transforming XML data.
• XPath is a DOM-like node tree data model and path expression language for selecting data within XML documents. XSL-FO, XSLT and XQuery all make use of XPath. XPath also includes a useful function library.

Pull Parsing

Pull parsing ^[6] treats the document as a series of items which are read in sequence using the Iterator design pattern. This allows for writing of recursive-descent parsers in which the structure of the code performing the parsing mirrors the structure of the XML being parsed, and intermediate parsed results can be used and accessed as local variables within the methods performing the parsing, or passed down (as method parameters) into lower-level methods, or returned (as method return values) to higher-level methods. Examples of pull parsers include StAX in the Java programming language, SimpleXML in PHP and System.Xml.XmlReader in .NET.

A pull parser creates an iterator that sequentially visits the various elements, attributes, and data in an XML document. Code which uses this 'iterator' can test the current item (to tell, for example, whether it is a start or end element, or text), and inspect its attributes (local name, namespace, values of XML attributes, value of text, etc.), and can also move the iterator to the 'next' item. The code can thus extract information from the document as it traverses it. The recursive-descent approach tends to lend itself to keeping data as typed local variables in the code doing the parsing, while SAX, for instance, typically requires a parser to manually maintain intermediate data within a stack of elements which are parent elements of the element being parsed. Pull-parsing code can be more straightforward to understand and maintain than SAX parsing code.

Data binding

Another form of XML Processing API is data binding, where XML data is made available as a custom, strongly typed programming language data structure, in contrast to the interface-oriented DOM. Example data binding systems include the Java Architecture for XML Binding (JAXB)^[7].

Non-extractive XML Processing API

Non-extractive XML Processing API is a new and emerging category of parsers that aim to overcome the fundamental limitations of DOM and SAX. The most representative is VTD-XML, which abolishes the object-oriented modeling of XML hierarchy and instead uses 64-bit Virtual Token Descriptors (encoding offsets, lengths, depths, and types) of XML tokens. VTD-XML's approach enables a number of interesting features/enhancements, such as high performance, low memory usage ^[8], ASIC implementation ^[9], incremental update ^[10], and native XML indexing ^{[11] [12]}.

[edit] Specific XML applications and editors

The native file format of OpenOffice.org, AbiWord, and Apple's iWork applications is XML. Some parts of Microsoft Office 2007 are also able to edit XML files with a user-supplied schema (but not a DTD), and Microsoft has released a file format compatibility kit for Office 2003 that allows previous versions of Office to save in the new XML based format. There are dozens of other XML editors available.

History

The versatility of SGML for dynamic information display was understood by early digital media publishers in the late 1980s prior to the rise of the Internet.^[13][14] By the mid-1990s some practitioners of SGML had gained experience with the then-new World Wide Web, and believed that SGML offered solutions to some of the problems the Web was likely to face as it grew. Dan Connolly added SGML to the list of W3C's activities when he joined the staff in 1995; work began in mid-1996 when Jon Bosak developed a charter and recruited collaborators. Bosak was well connected in the small community of people who had experience both in SGML and the Web. He received support in his efforts from Microsoft.

XML was compiled by a working group of eleven members,^[15] supported by an (approximately) 150-member Interest Group. Technical debate took place on the Interest Group mailing list and issues were resolved by consensus or, when that failed, majority vote of the Working Group. A record of design decisions and their rationales was compiled by Michael Sperberg-McQueen on December 4th 1997.^[16] James Clark served as Technical Lead of the Working Group, notably contributing the empty-element "<empty/>" syntax and the name "XML". Other names that had been put forward for consideration included "MAGMA" (Minimal Architecture for Generalized Markup Applications), "SLIM" (Structured Language for Internet Markup) and "MGML" (Minimal Generalized Markup Language). The co-editors of the specification were originally Tim Bray and Michael Sperberg-McQueen. Halfway through the project Bray accepted a consulting engagement with Netscape, provoking vociferous protests from Microsoft. Bray was temporarily asked to resign the editorship. This led to intense dispute in the Working Group, eventually solved by the appointment of Microsoft's Jean Paoli as a third co-editor.

The XML Working Group never met face-to-face; the design was accomplished using a combination of email and weekly teleconferences. The major design decisions were reached in twenty weeks of intense work between July and November of 1996, when the first Working Draft of an XML specification was published.^[17] Further design work continued through 1997, and XML 1.0 became a W3C Recommendation on February 10, 1998.

XML 1.0 achieved the Working Group's goals of Internet usability, general-purpose usability, SGML compatibility, facilitation of easy development of processing software, minimization of optional features, legibility, formality, conciseness, and ease of authoring. Like its antecedent SGML, XML allows for some redundant syntactic constructs and includes repetition of element identifiers. In these respects, terseness was not considered essential in its structure.

Sources

XML is a profile of an ISO standard SGML, and most of XML comes from SGML unchanged. From SGML comes the separation of logical and physical structures (elements and entities), the availability of grammar-based validation (DTDs), the separation of data and metadata (elements and attributes), mixed content, the separation of processing from representation (processing instructions), and the default angle-bracket syntax. Removed were the SGML Declaration (XML has a fixed delimiter set and adopts Unicode as the document character set).

Other sources of technology for XML were the Text Encoding Initiative (TEI), which defined a profile of SGML for use as a 'transfer syntax'; HTML, in which elements were synchronous with their resource, the separation of document character set from resource encoding, the xml:lang attribute, and the HTTP notion that metadata accompanied the resource rather than being needed at the declaration of a link; and the Extended Reference Concrete Syntax (ERCS), from which XML 1.0's naming rules were taken, and which had introduced hexadecimal numeric character references and the concept of references to make available all Unicode characters.

Ideas that developed during discussion which were novel in XML, were the algorithm for encoding detection and the encoding header, the processing instruction target, the xml:space attribute, and the new close delimiter for empty-element tags.

Versions

There are two current versions of XML. The first, XML 1.0, was initially defined in 1998. It has undergone minor revisions since then, without being given a new version number, and is currently in its fourth edition, as published on August 16, 2006. It is widely implemented and still recommended for general use. The second, XML 1.1, was initially published on February 4, 2004, the same day as XML 1.0 Third Edition, and is currently in its second edition, as published on August 16, 2006. It contains features — some contentious — that are intended to make XML easier to use in certain cases^[18] - mainly enabling the use of line-ending characters used on EBCDIC platforms, and the use of scripts and characters absent from Unicode 2.0. XML 1.1 is not very widely implemented and is recommended for use only by those who need its unique features. ^[19]

XML 1.0 and XML 1.1 differ in the requirements of characters used for element and attribute names: XML 1.0 only allows characters which are defined in Unicode 2.0, which includes most world scripts, but excludes those which were added in later Unicode versions. Among the excluded scripts are Mongolian, Cambodian, Amharic, Burmese, and others.

Almost any Unicode character can be used in the character data and attribute values of an XML 1.1 document, even if the character is not defined, aside from having a code point, in the current version of Unicode. The approach in XML 1.1 is that only certain characters are forbidden, and everything else is allowed, whereas in XML 1.0, only certain characters are explicitly allowed, thus XML 1.0 cannot accommodate the addition of characters in future versions of Unicode.

In character data and attribute values, XML 1.1 allows the use of more control characters than XML 1.0, but, for "robustness", most of the control characters introduced in XML 1.1 must be expressed as numeric character references. Among the supported control characters in XML 1.1 are two line break codes that must be treated as whitespace. Whitespace characters are the only control codes that can be written directly.

There are also discussions on an XML 2.0, although it remains to be seen^[vague] if such will ever come about. XML-SW (SW for skunk works), written by one of the original developers of XML, contains some proposals for what an XML 2.0 might look like: elimination of DTDs from syntax, integration of namespaces, XML Base and XML Information Set (infoset) into the base standard.

The World Wide Web Consortium also has an XML Binary Characterization Working Group doing preliminary research into use cases and properties for a binary encoding of the XML infoset. The working group is not chartered to produce any official standards. Since XML is by definition text-based, ITU-T and ISO are using the name Fast Infoset[3] for their own binary infoset to avoid confusion (see ITU-T Rec. X.891 | ISO/IEC 24824-1).

Patent claims

In October 2005 the small company Scientigo publicly asserted that two of its patents, U.S. Patent 5,842,213 and U.S. Patent 6,393,426 , apply to the use of XML. The patents cover the "modeling, storage and transfer [of data] in a particular non-hierarchical, non-integrated neutral form", according to their applications, which were filed in 1997 and 1999. Scientigo CEO Doyal Bryant expressed a desire to "monetize" the patents but stated that the company was "not interested in having us against the world." He said that Scientigo was discussing the patents with several large corporations.^[20]

XML users and independent experts responded to Scientigo's claims with widespread skepticism and criticism. Some derided the company as a patent troll. Tim Bray described any claims that the patents covered XML as "ridiculous on the face of it".^[21]

Because there exists a large amount of prior art relating to XML, including SGML, some legal experts believed it would be difficult for Scientigo to enforce its patents through litigation.^{[citation needed]}

Critique of XML

Commentators have offered various critiques of XML, suggesting circumstances where XML provides both advantages and potential disadvantages.^[22]

“	XML is really just data dressed up as a hooker.	”
—Dave Thomas[23]

Advantages of XML

• It is text-based.
• It supports Unicode, allowing almost any information in any written human language to be communicated.
• It can represent common computer science data structures: records, lists and trees.
• Its self-documenting format describes structure and field names as well as specific values.
• The strict syntax and parsing requirements make the necessary parsing algorithms extremely simple, efficient, and consistent.
• XML is heavily used as a format for document storage and processing, both online and offline.
• It is based on international standards.
• It can be updated incrementally.
• It allows validation using schema languages such as XSD and Schematron, which makes effective unit-testing, firewalls, acceptance testing, contractual specification and software construction easier.
• The hierarchical structure is suitable for most (but not all) types of documents.
• It manifests as plain text files, which are less restrictive than other proprietary document formats.
• It is platform-independent, thus relatively immune to changes in technology.
• Forward and backward compatibility are relatively easy to maintain despite changes in DTD or Schema.
• Its predecessor, SGML, has been in use since 1986, so there is extensive experience and software available.
• An element fragment of a well-formed XML document is also a well-formed XML document.

Disadvantages of XML

• XML syntax is redundant or large relative to binary representations of similar data.^[24]
• The redundancy may affect application efficiency through higher storage, transmission and processing costs.^[25][26]
• XML syntax is verbose, especially for human readers, relative to other alternative 'text-based' data transmission formats.^[27][28]
• The hierarchical model for representation is limited in comparison to an object oriented graph.^[29][30]
• Expressing overlapping (non-hierarchical) node relationships requires extra effort.^[31]
• XML namespaces are problematic to use and namespace support can be difficult to correctly implement in an XML parser.^[32]
• XML is commonly depicted as "self-documenting" but this depiction ignores critical ambiguities.^[33][34]
• The distinction between content and attributes in XML seems unnatural to some and makes designing XML data structures harder.^[35]
• Linking between XML documents requires the use of XLink, which is complex compared to hyperlinks^[36]
• It's hard to find an XML parser that is complete, correct, and efficient.^[37]

Introducing ADO.NET and the Typed DataSet

Introducing ADO.NET and the Typed DataSet

Introduction

Several weeks ago, I met with the CIO and development staff of a local customer. We discussed their current development strategies and project load. In the last several years they had developed several Visual Basic 6 and ASP based applications that had been extremely successful and represented the core application infrastructure for the business. These applications had become so successful that the business users had recently requested several new applications and pretty substantial enhancements to their existing applications. In reviewing these new requirements the CIO had made the decision that it was quicker and easier to rebuild the applications using .NET and design them using a Service Oriented Application approach.

In reviewing the applications it seemed everything they were doing involved database reads, writes, and manipulation. The development staff had been reviewing the articles and code samples for ADO.NET. They had spent many years refining their existing ADO based infrastructure and perfecting their use of ADO. The problem was when they starting looking at the .NET Framework, they found this new object called the DataSet. It seemed after all their research they continued to remain confused about this object when compared to the ADO recordset object they had used and loved for many years. In this article I will explain the benefits of ADO.NET and how the ADO.NET object model fits into the .NET Framework. Additionally, how the features of the DataSet object can be combined with XML to create a new type of object called the typed DataSet, which simplifies the task of data navigation.

ADO Revisited

Figure 1: The object structure for recordsets

The ADO recordset object represents a collection or records returned when a request is made to the database. This object represents the entire set of records returned from a base table or the complete results of an executed SQL command. At any time during the life of this object, it refers to a single record within the entire set of records. The object structure of the recordset collection is shown in Figure 1.

ADO contains the recordset object, which enables it to manipulate SQL data. One of the primary advantages of recordsets is the ability to loop through records stored in a recordset, accessing each of the individual records and changing, modifying, or deleting specific fields within each.

ADO also makes extensive use of cursors as the way of providing data views. Cursors enable the navigation between records in a recordset. The cursor actually sits on the current record specified in the recordset. The supported ADO cursor types are shown in Table 1.

Table 1: ADO cursor types

Cursor Type	Description
Dynamic Cursor	Enables the viewing of additions, changes, and deletions made by other users and allows forward and backward movement through a recordset object.
Keyset cursors	Acts the same as the dynamic cursor except that is prevents users from seeing records that others have added and prevents access to deleted records.
Static Cursor	Provides a static copy of a set of records that allows the location of data. Enables forward and backward movement through the recordset. Additions, changes, or deletions by other users will not be visible.
Forward-only cursor	Acts the same as the dynamic cursor except that is allows only forward scrolling. This improves performance and is the default type of recordset.

Note:
The recordset object automatically positions the current record as the first record and sets the Bottom of File (BOF) and End of File (EOF) properties to false. If there are no records, both the BOF and EOF properties are automatically set to true.

Defining a Recordset

Defining a recordset was a pretty trivial task that could be accomplished in less than 10 lines of code. For example, the following code shows how this can be done in an ASP page.

<html>

<body>

<%

set conn=Server.CreateObject("ADODB.Connection")

conn.Provider="Microsoft.Jet.OLEDB.4.0"

conn.Open(Server.Mappath("northwind.mdb"))

set rs = Server.CreateObject("ADODB.recordset")

rs.Open "Select * from Customers", conn

do until rs.EOF

    for each x in rs.Fields

       Response.Write(x.name)

       Response.Write(" = ")

       Response.Write(x.value & "<br />") 

    next

    Response.Write("<br />")

    rs.MoveNext

loop

rs.close

conn.close

%>

</body>

</html>

Enter ADO.NET

ADO.NET is not an upgrade to ADO but a complete redesign. At the core of ADO.NET is a set of libraries that are designed to communicate with a variety of back-end data stores. The ADO.NET libraries are part of the System.Data namespace and encompass a variety of additional libraries that enable data source connection, query submission, and processing results. One of the goals of the redesign of ADO.NET was to provide a hierarchical, disconnected data cache that works offline and online. The central disconnected object, called the DataSet, allows the search, filtering, navigation and storage of pending data changes. The design of ADO.NET is based on XML and enables ADO.NET to bridge the gap between the traditional database access and XML development. Developers are able to work with both XML data and database data using a common set of data interfaces as shown in Figure 2.

Figure2: The ADO.NET Object Model

ADO.NET is designed to provide consistent access to common data sources through a managed data provider. This provides a common interface for data sources like SQL Server as well as those exposed through OLE-DB and XML interfaces. Each of the managed providers is designed specifically for the type of data access it will provide. These results can either be processed directly or placed in an ADO.NET DataSet object that can then be exposed to the user, combined together, or even remoted between application tiers.

One of the strongest advantages of the DataSet object is that it can be used independently within the .NET Framework to manage locally stored data or XML files. ADO has really been the mainstay of database access for years, but there were several inherent design issues that ADO.NET was created to overcome. One of the main problems that ADO.NET is designed to solve is the lack of features needed to easily build applications that were part of .NET and the Service Oriented Architecture. For example, XML was added as an afterthought to ADO as the standard started to emerge within the industry. ADO.NET on the other hand was designed based on XML. Additionally, the ADO cursor engine makes it possible to pass disconnected ADO recordset objects between tiers in applications, but it doesn't provide a way to combine multiple recordset objects together. ADO also provides a way to submit cached changes to the database but doesn't provide programmatic control over the logic used to submit the updates. One of the major limitations for developing scalable SOA based applications that is lacking within the ADO cursor engine is the ability to submit pending changes to the database via stored procedures. Many database administrators allow users to only modify the contents of their databases through stored procedures, which is not possible through the binary object of the ADO recordset object.

.NET Data Providers

Within the .NET Framework a data provider is a collection of classes designed to allow you to communicate with a specific type of data store. The .NET Framework comes pre-built with providers for SQL Server, OLE-DB sources, Oracle, and ODBC. In addition there are several additional providers that have been released independently including DB2 and MYSQL.

The OLE DB .NET Data Provider is designed to communicate with various data stores through the OLE DB providers. The SQL Client .NET Data provider is designed to solely communicate with SQL Server databases. Each of these managed providers implements the same base classes - Connection, Command, Data Reader, Parameter and Transaction. The names and specific namespaces may change depending on the specific provider and the underlying database requirements. For example, the SQL Managed Provider uses a SQLConnection object, while the OLE DB provider uses an OLEDbConnection object. Each provider is designed to take advantage of the specific performance and limitations of the specific data source.

Regardless of which .NET data provider you use, the provider's Connection object implements the same basic set of interfaces. When opening a connection to the data store, developers create an instance of the providers connection object, set the ConnectionString and then call its Open method.

Connecting to a Data Source

The connection object represents a connection to your data source. For SQL Server you use the namespace System.Data.SQLClient.SQLConnection, and for OleDB you use System.Data.OleDb.OleDbConnection. When specifying the datasource you can provide a variety of amplifying information that describes the data source that includes its location and connection type.

The connection object is similar to the ADO Connection object in that it is used to connect and disconnect from a data source. The connection object acts as the glue through which other objects like the DataAdaptor and the Command objects communicate with the database and submit queries and retrieve results. The following code example shows how to connect to a local SQL Server database using the managed provider.

 Dim sqlConn As SqlConnection
Dim sqlCmd As SqlCommand
Dim strConstring As String
Dim intUserID As Integer

strConstring = ConfigurationSettings.AppSettings("constring")
strConstring = "Data Source=(local);Initial Catalog= _  
                Northwind;Integrated Security=SSPI"
sqlConn = New SqlConnection(strConstring)
sqlConn.Open()
 

Note:
The connection object also provides a BeginTransaction method that is used to create a Transaction object. This object is used to either commit or cancel the changes made to the database during the lifetime of the Transaction object.

DataAdapter Object

The DataAdapter is a new concept introduced to data access with the ADO.NET model. It acts as the bridge between the database and the disconnected objects in the ADO.NET object model. The DataAdaptor object's Fill method provides the mechanism to fetch the results of a query into a DataSet or a DataTable so that the data is easily accessed. The DataAdapter object is also used to submit pending changes stored in the DataSet back to the database.

The DataAdapter object is part of the Command object and exposes a variety of properties and methods. For example, the SelectCommand property contains a Command object that represents the query that is used to populate the DataSet object. Additionally, this object exposes the UpdateCommand, InsertCommand and DeleteCommand properties that correspond to the Command objects used when you submit modified, new or deleted data rows back to the database. With each of these properties you are able to call stored procedures or a SQL statement that performs the type of operation. Once completed you can call the Update method on the DataAdapter object, and ADO.NET will use the Command object created earlier to update the database as shown in the following code example.

Public Shared Function CreateCandyAdapter(ByVal conn As SqlConnection)_
As SqlDataAdapter

Dim da As SqlDataAdapter = New SqlDataAdapter
Dim cmd As SqlCommand
Dim parm As SqlParameter

' Create the SelectCommand.
cmd = New SqlCommand("sp_select", conn)
cmd.CommandType = CommandType.StoredProcedure

da.SelectCommand = cmd

' Create the UpdateCommand
cmd = New SqlCommand("sp_update", conn)
cmd.CommandType = CommandType.StoredProcedure

cmd.Parameters.Add("@CandyName", SqlDbType.VarChar, 5, "CandyName")
cmd.Parameters.Add("@CandyDescription", SqlDbType.VarChar, 40, _
"CandyPrice")

parm.SourceVersion = DataRowVersion.Original
da.UpdateCommand = cmd
Return da
  
End Function
 

In order to maintain what is happening to the database and the data, the DataAdapter object provides a variety of supporting properties that can track which tables in the database correspond to the tables in your DataSet object. The TableMappings collection is the property used to track which table in the database corresponds to which table in your DataSet object. Each table mapping has a similar property for mapping columns called a ColumnMappings collection.

Enter the DataSet

The DataSet object is central to supporting disconnected, distributed data scenarios using ADO.NET. This object is actually a memory resident representation of data that provides a consistent relational programming model regardless of the data source. The DataSet object represents a complete set of data including the related tables, constraints and relationships. Figure 3 shows the DataSet object model.

Figure 3: The Dataset object model.

The DataSet object as its name implies contains data. It provides a container for a number of DataTable objects that are stored in the DataSet's object tables collection. One of the biggest advantages of the DataSet is that you can package an entire structure into it. Instead of having to call multiple times into a middle tier object to retrieve data you can simply package the entire instance, including table references, directly into a single object that can be returned.

The data stored in the DataSet object is disconnected from the database. Any changes that are made are simply cached in a DataRow. When it's time to send the changes back the database, you can pass the entire object to the middle tier server. The DataSet provides a GetChanges method that can be used to extract only the modified rows from a DataSet. This enables specific changed rows to be extracted and passed and reduces the overhead of data across a network.

The DataSet also exposes a Merge method, which acts in conjunction with the GetChanges method. The middle tier server can submit changes to the database using the smaller DataSet returned by the Merge method and return a DataSet that contains the new data. Then using the Merge method this returned DataSet can then be merged back to the contents of the existing DataSet object as shown in the following example.

CandyDS.Merge(otherDS)
 

The DataSet also provides the ability to read and write data to it from a file or an area of memory. This means you can just save the contents of the DataSet object or the structure. ADO.NET stores the data as an XML document. It is important to remember that ADO.NET and XML are tightly coupled, which means that moving data between the two is an easy task. For example, the following code shows how to create a DataSet and then display the XML contents to the console.

Private Shared Sub DemonstrateGetXml()
   ' Create a DataSet with 1 table two columns and 10 rows.
   Dim ds As DataSet = New DataSet("myDataSet")
   Dim t As DataTable = ds.Tables.Add("Items")
   t.Columns.Add("id", Type.GetType("System.Int32"))
   t.Columns.Add("Item", Type.GetType("System.String"))

   ' Add ten rows.
   Dim r As DataRow
   Dim i As Integer
   For i = 0 To 9
       r = t.NewRow()
       r("id") = i
       r("Item")= "Item" & i
       t.Rows.Add(r)
   Next

   ' Display the DataSet contents as XML.
   Console.WriteLine( ds.GetXml() )
End Sub
 

DataSet Web Services

Within the world of the Services Oriented Architecture, the Web Service is the basic unit of work. This means that often a Web Service is constructed to only return a set of data based on a query that the front-end application sends in. This provides a perfect use for the DataSet as a way of returning data. For example, the following code section shows how data can be returned directly from a Web Service using a DataSet.

 Public Function GetRequests(ByVal RequestedStatus As
Boolean) As DataSet
       'db connection
       Dim sqlConn As SqlConnection
       Dim sqlCmd As SqlCommand
       Dim strConstring As String
       Dim intUserID As Integer

       strConstring = ConfigurationSettings.AppSettings("constring")
       sqlConn = New SqlConnection(strConstring)
       sqlConn.Open()
       sqlCmd = New SqlCommand

       With sqlCmd
           .Connection = sqlConn
           .CommandTimeout = 30
           .CommandType = CommandType.StoredProcedure
           If RequestedStatus = True Then
               .CommandText = "spGetOpenRequests"
           Else
               .CommandText = "spGetClosedRequests"
           End If

       End With

       Dim RequestDA As SqlDataAdapter = New SqlDataAdapter
       RequestDA.SelectCommand = sqlCmd
       Dim RequestDS As DataSet = New DataSet
       RequestDA.Fill(RequestDS, "RequestType")

       Return RequestDS
       sqlConn.Close()

   End Function
 

When a user accesses the data from a front-end application, he receives a stream of XML as shown in Figure 4.

Figure 4: XML returned from a DataSet object

The transportable nature of the DataSet and XML enables the client the ability to reconstitute the data on the receiving end as shown in the following example.

Dim ws As New localhost.wsSubmit
Dim xmlStatus As DataSet
xmlStatus = ws.GetRequests(RequestedStatus:=False)
 

One of the difficult parts of using the DataSet this way is that table, column, and row navigation can be difficult. The navigation is actually done through the combination of tables, columns, and rows that make up the DataSet.

Dim s As String
s = CType(dsCustomersOrders1.Tables("Customers")._
Rows(0).Item("CustomerID"), String)
 

Typed Vs. Untyped DataSets

The nature of a DataSet requires that it is either typed or untyped. By definition a typed DataSet is any DataSet that is derived from the base DataSet class that applies the information contained in the XSD to generate a typed class. Information from the schema that contains the tables, columns, and rows is generated and compiled into a new DataSet derived from the XSD and this promotes the DataSet to a first class object in the .NET Framework.

The process of inheriting the typed DataSet from the base DataSet class means that the typed class assumes all functionality of the DataSet class and can be used with methods that take an instance of the DataSet class as a parameter. This is the opposite of the untyped DataSet that has no corresponding schema and is exposed only as a collection.

Defining the Schema

The first step in defining a typed DataSet is to create the XSD that is used to control and provide structure for the DataSet. This can be done by adding a new XML schema to your project as shown in Figure 5.

Figure 5: Adding a new schema

This adds a new XSD item to your project. When defining the XSD structure, always make sure that you point to the existing Web Service namespace and provide a fully qualified name of the exposed DataSet, as shown in the following XSD.

<?xml version="1.0" encoding="utf-8" ?>

<xsd:schema id="candyTypeDataset"

targetNamespace="http://tempuri.org/CandyType.xsd"

elementFormDefault="qualified" xmlns="http://tempuri.org/CandyType.xsd"

xmlns:xsd="http://www.w3.org/2001/XMLSchema">

<xsd:element name="candyTypeDataset">

<xsd:complexType>

<xsd:sequence>

<xsd:element name="candyType" minOccurs="0"

type="xsd:string" />

</xsd:sequence>

</xsd:complexType>

</xsd:element>

</xsd:schema>

The schema provides a structure that is used to define a set of classes. The schema shown above defines a single field as a string with the name "candyType". Once the schema is created, it can be viewed using the XML view of Visual Studio. This view shows the completed XML schema as shown in Figure 6.

Figure 6: Completed XSD view.

Once completed the XSD schema can be compiled into a typed DataSet using the right click menu located on the graphical XSD display as shown in Figure 7.

Figure 7: Generating a typed DataSet.

The compilation of the typed DataSet creates two additional files as shown in Figure 8.

Figure 8: Wrapper classes created in Visual Studio.

The first file which contains the .vb extension is used to define the proxy class that is generated when the DataSet is compiled. This class contains the methods and properties that are needed to access, manipulate, and serialize the DataSet. The second file with the .xsx extension contains information about the layout of the XSD on the design surface as shown in Figure 6.

Finishing the Web Service

Once created the typed DataSet is usable within the application as a first class object. This means that when the object is created IntelliSense is now aware of the objects and methods available within the object and displays these in the object viewer as shown in Figure 9.

Figure 9: Exposed typed DataSet objects.

This enables the object to be created and using the following code to fill the typed DataSet for return to the client.

<WebMethod()> Public Function GetCandyTypes() As candyTypeDataset

'db connection

Dim sqlConn As SqlConnection

Dim sqlCmd As SqlCommand

Dim strConstring As String

Dim intUserID As Integer

strConstring = ConfigurationSettings.AppSettings("constring")

sqlConn = New SqlConnection(strConstring)

sqlConn.Open()

sqlCmd = New SqlCommand

With sqlCmd

.Connection = sqlConn

.CommandTimeout = 30

.CommandType = CommandType.StoredProcedure

.CommandText = "sp_GetCandyType"

End With

Dim CandyDA As SqlDataAdapter = New SqlDataAdapter

CandyDA.SelectCommand = sqlCmd

Dim CandyDS As New candyTypeDataset

CandyDA.Fill(CandyDS, CandyDS.Tables(0).TableName)

Return CandyDS

sqlConn.Close()

End Function

When creating the WebMethod, it is important to provide a return value that matches the name of the typed DataSet. This is how the client becomes aware of the structure and is able to use the schema defined within the return stream. For example, if we run the above WebMethod and review the WSDL generated. We can see that the Namespace we created for the schema has been added as shown in Figure 10.

Figure 10: WSDL definition showing defined namespace.

If you review further down the WSDL you can see that the new XSD has been appended as shown in Figure 11.

Figure 11: Appended XSD

Client Retrieval

Once the Web Service is completed and the reference is set, any client can retrieve the data exposed by the typed DataSet. Once created the typed DataSet becomes an object that can be retrieved using the named properties directly from the client as shown in the code example below.

Dim ws As New localhost.SweetsService

Dim ds As localhost.candyTypeDataset

ds = ws.GetCandyTypes

Dim i As Integer

For i = 0 To (ds.candyTypeDataset.Count - 1)

Dim locButton As New Button

Dim ButFace As String

locButton.Text = ds.candyTypeDataset.Item(i).candyType

Me.Controls.Add(locButton)

Next

End Sub

It is important to understand that the navigation is now by property as shown above. This means that we don't have to navigate the underlying structure using the standard navigation of the DataSet, but are able to access field by property name.

Summary

At the end of the meeting, the CIO and the development team looked closely at the code samples that we had created and had a better understanding. I spoke with them the other day and was happy to hear that they had adopted ADO.NET and the type DataSet as a way of consuming and rendering their data. Hopefully while reading this article you have come to understand the benefits of ADO.NET and the typed DataSet. These include type checking at design time which gives the advantage of statement completion. For a developer this means fewer errors during runtime from misspellings and consistent navigation that is similar to the ADO.

In this article we have covered a lot of information about the features available within ADO.NET. By no means have we covered every aspect, but only scratched the surface. Hopefully you can now understand the features and benefits of ADO.NET and how important the typed DataSet is to application development. I certainly challenge all readers as I did with the staff that I was meeting with to start exploring these as a way of better developing your data access operations.

About the Author

Thom is a Senior Technology Specialist with Microsoft Corporation in New England. He specializes in working with customers on developing and implementing .NET based applications. He is a frequent speaker and contributor to various magazines. He is also a regular writer for his personal Weblog at http://radio.weblogs.com/0131777/. If you have comments or questions about this article, please visit his Weblog and contact him.