Getting started with Apache Any23
Apache Any23 can be used:
- via CLI (command line interface) from your preferred shell environment;
- as a RESTful Webservice;
- as a library.
Apache Any23 Modules
Apache Any23 is composed of the following modules:
api/The base API definitions e.g. The Any23 API.core/The core library containing all extractor functionality.cli/A command line interface enabling easy invocation of Any23 tools.csvutils/Utility code for CSV extractions.encoding/Characterset detection and encoding.mime/Media-type detection.service/The REST service.plugins/The core additional plugins.openie/Additional extractor logic for the Open Information Extraction (Open IE) system.
Use the Apache Any23 CLI
The command-line tools support is provided by the cli module.
Once Apache Any23 has been correctly installed, if you want to use it as a command line tool, use the shell script within the cli/target/appassembler/bin/ directory. These are provided both for Unix (Linux/OSX) and Windows.
The any23 script provides analysis, documentation, testing and debugging utilities.
Simply running ./any23 without options will show the usage options.
$ cli/target/appassembler/bin/any23
A command must be specified.
Usage: any23 [options] [command] [command options]
Options:
-h, --help
Display help information.
Default: false
--plugins-dir
The Any23 plugins directory.
Default: /Users/lmcgibbn/.any23/plugins
-X, --verbose
Produce execution verbose output.
Default: false
-v, --version
Display version information.
Default: false
Commands:
extractor Utility for obtaining documentation about metadata extractors.
Usage: extractor [options] Extractor name
Options:
-a, --all
shows a report about all available extractors
Default: false
-i, --input
shows example input for the given extractor
Default: false
-l, --list
shows the names of all available extractors
Default: false
-o, --outut
shows example output for the given extractor
Default: false
microdata Commandline Tool for extracting Microdata from file/HTTP source.
Usage: microdata [options] Input document URL, {http://path/to/resource.html|file:/path/to/localFile.html}
mimes MIME Type Detector Tool.
Usage: mimes [options] Input document URL, {http://path/to/resource.html|file:///path/to/local.file|inline:// some inline content}
verify Utility for plugin management verification.
Usage: verify [options] plugins-dir
rover Any23 Command Line Tool.
Usage: rover [options] input IRIs {<url>|<file>}+
Options:
-d, --defaultns
Override the default namespace used to produce statements.
-e, --extractors
a comma-separated list of extractors, e.g. rdf-xml,rdf-turtle
Default: []
-f, --format
the output format
Default: json
-l, --log
Produce log within a file.
-n, --nesting
Disable production of nesting triples.
Default: false
-t, --notrivial
Filter trivial statements (e.g. CSS related ones).
Default: false
-o, --output
Specify Output file (defaults to standard output)
Default: java.io.PrintStream@5204062d
-p, --pedantic
Validate and fixes HTML content detecting commons issues.
Default: false
-s, --stats
Print out extraction statistics.
Default: false
vocab Prints out the RDF Schema of the vocabularies used by Any23.
Usage: vocab [options]
Options:
-f, --format
Vocabulary output format
Default: N-Quads (mimeTypes=application/n-quads, text/x-nquads, text/nquads; ext=nq)The any23 script detects a list of available utilities within the core and plugins classpath and allows to activate them.
The any23-core CLI tools are:
extractor: a utility for obtaining useful information about extractors.microdata: commandline parser to extract specific Microdata content from a web page (local or remote) and produce a JSON output compliant with the Microdata specification (http://www.w3.org/TR/microdata/).mimes: detects the MIME Type for any HTTP / file / direct input resource.verify: a utility for verifying Apache Any23 plugins.rover: the RDF extraction tool.vocab: allows to dump all the RDFSchema vocabularies declared within Apache Any23.
The Rover tool
Rover is the main extraction tool. It allows to extract metadata from local and remote (HTTP) resources, specify a custom list of extractors, specify the desired output format and other flags to suppress noise and generate advanced reports.
Extract metadata from an HTML page:
cli$ any23 rover http://yourdomain/yourfile
Extract metadata from a local resource:
cli$ any23 rover myfoaf.rdf
Specify the output format, use the option "-f" or "--format": (Default output format is TURTLE).
cli$ any23 rover -f quad myfoaf.rdf
Filtering trivial statements
By default, Apache Any23 will extract HTML/head meta information, such as links to CSS stylesheets or meta information like the author or the software used to create the html. Hence, if the user is only interested in the structured content from the HTML/body tag we offer a filter functionality, activated by the "-t" command line argument.
core$ any23 rover -t -f quad myfoaf.rdf
The ExtractorDocumentation tool
The ExtractorDocumentation returns human readable information about the registered extractors.
List all the available extractors:
cli$ any23 extractor --list
csv [org.apache.any23.extractor.csv.CSVExtractorFactory] [text/csv;q=0.1]
html-embedded-jsonld [org.apache.any23.extractor.html.EmbeddedJSONLDExtractorFactory] [text/html;q=0.02, application/xhtml+xml;q=0.02]
html-head-icbm [org.apache.any23.extractor.html.ICBMExtractorFactory] [text/html;q=0.01, application/xhtml+xml;q=0.01]
html-head-links [org.apache.any23.extractor.html.HeadLinkExtractorFactory] [text/html;q=0.05, application/xhtml+xml;q=0.05]
html-head-meta [org.apache.any23.extractor.html.HTMLMetaExtractorFactory] [text/html;q=0.02, application/xhtml+xml;q=0.02]
html-head-title [org.apache.any23.extractor.html.TitleExtractorFactory] [text/html;q=0.02, application/xhtml+xml;q=0.02]
html-mf-adr [org.apache.any23.extractor.html.AdrExtractorFactory] [text/html;q=0.1, application/xhtml+xml;q=0.1]
html-mf-geo [org.apache.any23.extractor.html.GeoExtractorFactory] [text/html;q=0.1, application/xhtml+xml;q=0.1]
html-mf-hcalendar [org.apache.any23.extractor.html.HCalendarExtractorFactory] [text/html;q=0.1, application/xhtml+xml;q=0.1]
html-mf-hcard [org.apache.any23.extractor.html.HCardExtractorFactory] [text/html;q=0.1, application/xhtml+xml;q=0.1]
html-mf-hlisting [org.apache.any23.extractor.html.HListingExtractorFactory] [text/html;q=0.1, application/xhtml+xml;q=0.1]
html-mf-hrecipe [org.apache.any23.extractor.html.HRecipeExtractorFactory] [text/html;q=0.1, application/xhtml+xml;q=0.1]
html-mf-hresume [org.apache.any23.extractor.html.HResumeExtractorFactory] [text/html;q=0.1, application/xhtml+xml;q=0.1]
html-mf-hreview [org.apache.any23.extractor.html.HReviewExtractorFactory] [text/html;q=0.1, application/xhtml+xml;q=0.1]
html-mf-hreview-aggregate [org.apache.any23.extractor.html.HReviewAggregateExtractorFactory] [text/html;q=0.1, application/xhtml+xml;q=0.1]
html-mf-license [org.apache.any23.extractor.html.LicenseExtractorFactory] [text/html;q=0.01, application/xhtml+xml;q=0.01]
html-mf-species [org.apache.any23.extractor.html.SpeciesExtractorFactory] [text/html;q=0.1, application/xhtml+xml;q=0.1]
html-mf-xfn [org.apache.any23.extractor.html.XFNExtractorFactory] [text/html;q=0.1, application/xhtml+xml;q=0.1]
html-microdata [org.apache.any23.extractor.microdata.MicrodataExtractorFactory] [text/html;q=0.1, application/xhtml+xml;q=0.1]
html-rdfa11 [org.apache.any23.extractor.rdfa.RDFa11ExtractorFactory] [application/xhtml+xml;q=0.3, application/html;q=0.3, text/html;q=0.3]
html-xpath [org.apache.any23.extractor.xpath.XPathExtractorFactory] [text/html;q=0.02, application/xhtml+xml;q=0.02]
ical [org.apache.any23.extractor.calendar.ICalExtractorFactory] [text/calendar]
jcal [org.apache.any23.extractor.calendar.JCalExtractorFactory] [application/calendar+json]
owl-functional [org.apache.any23.extractor.rdf.FunctionalSyntaxExtractorFactory] [text/owl-functional]
owl-manchester [org.apache.any23.extractor.rdf.ManchesterSyntaxExtractorFactory] [text/owl-manchester]
rdf-jsonld [org.apache.any23.extractor.rdf.JSONLDExtractorFactory] [application/ld+json;q=0.1]
rdf-nq [org.apache.any23.extractor.rdf.NQuadsExtractorFactory] [application/n-quads, text/x-nquads;q=0.1, text/rdf+nq;q=0.1, text/nq;q=0.1, text/nquads;q=0.1, text/n-quads;q=0.1]
rdf-nt [org.apache.any23.extractor.rdf.NTriplesExtractorFactory] [application/n-triples;q=0.1, text/nt;q=0.1, text/ntriples;q=0.1, text/plain;q=0.1]
rdf-trix [org.apache.any23.extractor.rdf.TriXExtractorFactory] [application/trix]
rdf-turtle [org.apache.any23.extractor.rdf.TurtleExtractorFactory] [text/turtle, text/rdf+n3, text/n3, application/n3, application/x-turtle, application/turtle]
rdf-xml [org.apache.any23.extractor.rdf.RDFXMLExtractorFactory] [application/rdf+xml, text/rdf, text/rdf+xml, application/rdf]
xcal [org.apache.any23.extractor.calendar.XCalExtractorFactory] [application/calendar+xml]
yaml [org.apache.any23.extractor.yaml.YAMLExtractorFactory] [text/x-yaml;q=0.5]The MicrodataParser tool
The MicrodataParser tool allows to apply the only MicrodataExtractor on a specific input source and returns the extracted data in the JSON format declared in the Microdata specification section JSON.
cli$ any23 microdata http://path/to/resource.html
The VocabPrinter tool
The VocabPrinter Tool prints out the RDFSchema declared by all the Apache Any23 declared vocabularies.
Just launch the command below to see all the managed vocabularies.
cli$ any23 vocab
NOTE: This tool is still in beta version.
The MimeDetector tool
The MimeDetector Tool extracts the MIME Type for a given source (http:// file:// inline://).
Examples:
cli$ any23 mimes http://www.michelemostarda.com/foaf.rdf application/rdf+xml
cli$ any23 mimes file://../src/test/resources/application/trix/test1.trx application/trix
cli$ any23 mimes 'inline://<http://s> <http://p> <http://o> .' text/n3
The PluginVerifier tool
The PluginVerifier tool allows checking installed plugin in the specified input directory
Just launch the command below to sanity-check the input plugins directory
cli$ any23 verify [/path/to/plugins/dir]
Apache Any23 CLI Plugins
The Apache Any23 ToolRunner CLI (bin/any23) supports the auto detection of Tool plugins within the classpath. For further details see Plugins section.
The default any23 CLI plugins are enlisted below.
Crawler Plugin
crawler-tool The Crawler Plugin provides basic site crawling and metadata extraction capabilities.
cli$ any23 -h
[...]
crawler Any23 Crawler Command Line Tool.
Usage: crawler [options] input IRIs {<url>|<file>}+
Options:
-d, --defaultns Override the default namespace used to
produce statements.
-e, --extractors a comma-separated list of extractors, e.g.
rdf-xml,rdf-turtle
Default: []
-f, --format the output format
Default: turtle
-l, --log Produce log within a file.
-md, --maxdepth Max allowed crawler depth.
Default: 2147483647
-mp, --maxpages Max number of pages before interrupting
crawl.
Default: 2147483647
-n, --nesting Disable production of nesting triples.
Default: false
-t, --notrivial Filter trivial statements (e.g. CSS related
ones).
Default: false
-nc, --numcrawlers Sets the number of crawlers.
Default: 10
-o, --output Specify Output file (defaults to standard
output)
Default: java.io.PrintStream@2911a3a4
-pf, --pagefilter Regex used to filter out page URLs during
crawling.
Default: .*(\.(css|js|bmp|gif|jpe?g|png|tiff?|mid|mp2|mp3|mp4|wav|wma|avi|mov|mpeg|ram|m4v|wmv|rm|smil|pdf|swf|zip|rar|gz|xml|txt))$
-p, --pedantic Validate and fixes HTML content detecting
commons issues.
Default: false
-pd, --politenessdelay Politeness delay in milliseconds.
Default: 2147483647
-s, --stats Print out extraction statistics.
Default: false
-sf, --storagefolder Folder used to store crawler temporary data.
Default: /var/folders/zz/9vvv_lbn1cs8dpwz859nmq080000gn/T/crawler-metadata-9ff4c650-10c2-41a1-9d99-ebeb3e7d21ceA usage example:
cli$ any23 crawler -s -f ntriples http://www.repubblica.it 1> out.nt 2> repubblica.log
Use Apache Any23 as a RESTful Web Service
Apache Any23 provides a Web Service that can be used to extract RDF from Web documents. Apache Any23 services can be accessed through a RESTful API.
Running the server
The server command line tool is defined within the service module. Run the any23server script
service$ ./bin/any23server
from the command line in order to start up the server, then go to to access the web interface. A live demo version of such service is running at . You can also start the server from Java by running the Apache Any23 Servlet class. Maven can be used to create a WAR file for deployment into an existing servlet container such as Apache Tomcat.
Use Apache Any23 as a Library
See our Developers guide for more details.