org.apache.any23.plugin.htmlscraper

Class HTMLScraperExtractor

java.lang.Object

org.apache.any23.plugin.htmlscraper.HTMLScraperExtractor

All Implemented Interfaces:

Extractor<InputStream>, Extractor.ContentExtractor

public class HTMLScraperExtractor
extends Object
implements Extractor.ContentExtractor

Implementation of content extractor for performing HTML scraping.

Author:

Michele Mostarda (mostarda@fbk.eu)

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.any23.extractor.Extractor

Extractor.BlindExtractor, Extractor.ContentExtractor, Extractor.TagSoupDOMExtractor

Field Summary

Fields

Modifier and Type	Field and Description
static org.eclipse.rdf4j.model.IRI	PAGE_CONTENT_AE_PROPERTY
static org.eclipse.rdf4j.model.IRI	PAGE_CONTENT_CE_PROPERTY
static org.eclipse.rdf4j.model.IRI	PAGE_CONTENT_DE_PROPERTY
static org.eclipse.rdf4j.model.IRI	PAGE_CONTENT_LCE_PROPERTY

Constructor Summary

Constructors

Constructor and Description
HTMLScraperExtractor()

Method Summary

Modifier and Type	Method and Description
void	addTextExtractor(String name, org.eclipse.rdf4j.model.IRI property, de.l3s.boilerpipe.BoilerpipeExtractor extractor)
ExtractorDescription	getDescription() Returns a ExtractorDescription of this extractor.
String[]	getTextExtractors()
void	run(ExtractionParameters extractionParameters, ExtractionContext extractionContext, InputStream inputStream, ExtractionResult extractionResult) Executes the extractor.
void	setStopAtFirstError(boolean b) If true, the extractor will stop at first parsing error, iffalse the extractor will attempt to ignore all parsing errors.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

PAGE_CONTENT_DE_PROPERTY

public static final org.eclipse.rdf4j.model.IRI PAGE_CONTENT_DE_PROPERTY

PAGE_CONTENT_AE_PROPERTY

public static final org.eclipse.rdf4j.model.IRI PAGE_CONTENT_AE_PROPERTY

PAGE_CONTENT_LCE_PROPERTY

public static final org.eclipse.rdf4j.model.IRI PAGE_CONTENT_LCE_PROPERTY

PAGE_CONTENT_CE_PROPERTY

public static final org.eclipse.rdf4j.model.IRI PAGE_CONTENT_CE_PROPERTY

Constructor Detail

HTMLScraperExtractor

public HTMLScraperExtractor()

Method Detail

addTextExtractor

public void addTextExtractor(String name,
                             org.eclipse.rdf4j.model.IRI property,
                             de.l3s.boilerpipe.BoilerpipeExtractor extractor)

getTextExtractors

public String[] getTextExtractors()

run

public void run(ExtractionParameters extractionParameters,
                ExtractionContext extractionContext,
                InputStream inputStream,
                ExtractionResult extractionResult)
         throws IOException,
                ExtractionException

Description copied from interface: Extractor

Executes the extractor. Will be invoked only once, extractors are not reusable.

Specified by:

run in interface Extractor<InputStream>

Parameters:

extractionParameters - the parameters to be applied during the extraction.

extractionContext - The document context.

inputStream - The extractor input data.

extractionResult - the collector for the extracted data.

Throws:

IOException - On error while reading from the input stream.

ExtractionException - On other error, such as parse errors.

getDescription

public ExtractorDescription getDescription()

Description copied from interface: Extractor

Returns a ExtractorDescription of this extractor.

Specified by:

getDescription in interface Extractor<InputStream>

Returns:

the object representing the extractor description.

setStopAtFirstError

public void setStopAtFirstError(boolean b)

Description copied from interface: Extractor.ContentExtractor

If true, the extractor will stop at first parsing error, iffalse the extractor will attempt to ignore all parsing errors.

Specified by:

setStopAtFirstError in interface Extractor.ContentExtractor

Parameters:

b - tolerance flag.