                                    ------
                                    Apache Any23 - Developers Guide
                                    ------
                              The Apache Software Foundation
                                    ------

~~  Licensed to the Apache Software Foundation (ASF) under one or more
~~  contributor license agreements.  See the NOTICE file distributed with
~~  this work for additional information regarding copyright ownership.
~~  The ASF licenses this file to You under the Apache License, Version 2.0
~~  (the "License"); you may not use this file except in compliance with
~~  the License.  You may obtain a copy of the License at
~~
~~     http://www.apache.org/licenses/LICENSE-2.0
~~
~~  Unless required by applicable law or agreed to in writing, software
~~  distributed under the License is distributed on an "AS IS" BASIS,
~~  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
~~  See the License for the specific language governing permissions and
~~  limitations under the License.

Architectural Overview

[./images/any23-overall.png]
  
  The informal architectural diagram above shows the <<Any23>> logical modules, the main data flow and the code packages implementing such modules.

  The first module, <<Data Fetching>>, is responsible for retrieving raw data from the Web, its implementation package is <<org.apache.any23.source>>.
  The data collected by <Data Fetching> is analyzed by the <<MIMEtype Detection>> module, implemented in package <<org.apache.any23>>. 
  Such module will determine the data encoding and the content <MIME> type. 
  The identification of the MIME type is used to select a list of activable <Extractors> for the subsequent metadata extraction. 

  The next phase is performed by the <<Content Validation and Patching>> module (<<org.apache.any23.validator>>), and it is required because the most part of data
  exposed on the Web is affected by minor issues which compromise the correct working of some <Extractors>. To overcome such problems <<Any23>> 
  introduced a mechanism to detect issues and in most cases to fix them. The detection and fixing is performed using an extensible collection of <<Rules>>. 
  Currently the Validation and Patching is applied only on <DOM> based documents (<HTML>).

  The <<Metadata Extraction>> module, implemented within the <<org.apache.any23.extraction>> package, applies all the <Extractors> activated by the analysis phase 
  and generates an RDF statements stream together with an issue report. 
  The statements produced by the <Extractor>s can be filtered to remove spurious, repeated or unwanted triples using the <<Metadata Filtering>> module (<<org.apache.any23.filter>>).

  The last metadata extraction phase consists in the conversion of the filtered statements in an RDF representation format. 
  This can be done by using one of the available RDF writers provided by the <<Serialization>> module (<<org.apache.any23.writer>>).

  The other modules represented at the bottom of the diagram  add auxiliary functionalities over the core application.
  The <<Plugin Management>> module (<<org.apache.ay23.plugin>>) is responsible for the extension of the platform through the runtime detection 
  and registration of additional components included within the classpath. 
  The Plugin Manager is currently able to detect and register new Extractors, Writers and CLI tools. 
  It is foreseen the plugin support implementation for all the modules marked as (P).

  The <<CLI Tool>> module (org.apache.any23.cli) allows to run all the available CLI tools through a unified interface.

  The <<Service>> module (org.apache.any23.service) implements a REST service to use Any23 as a Web service implementing a <REST> interface.


Developers Guide

  This section introduces some <<Apache Any23>> programming fundamentals.

* {{{./dev-data-extraction.html}Data Extraction}}
  
  Explains how to extract RDF data from HTTP resources with <<Apache Any23>>.

* {{{./dev-data-conversion.html}Data Conversion}}
  
   Shows how to perform RDF data conversion with <<Apache Any23>>.

* {{{./dev-validation-fix.html}Validation and Fixing}}
  
  Demonstrates how to define validation and correction rules for HTML content with <<Apache Any23>>.
    
* {{{./dev-xpath-extractor.html}XPath Extractor}}
  
  Explains how to write custom scraping rules for extracting RDF data from any HTML content with <<Apache Any23>>.
    
* {{{./dev-microformat-extractors.html}Microformat Extractors}}
  
  Explains how to write new Microformat extractors with <<Apache Any23>> and also report interesting notes on
  microformat nesting representation.

* {{{./dev-microdata-extractor.html}Microdata Extractor}}

  Explains how it works the Microdata Extractor embedded in <<Apache Any23>>.

* {{{./dev-csv-extractor.html}CSV Extractor}}

  Explains how it works the CSV Extractor embedded in <<Apache Any23>>.
