Content Extraction: Identifying the Main Content in HTML Documents

Except the article forming the main content most HTML documents on the WWW contain additional contents such as navigation menus, design elements or commercial banners. In the context of several applications it is necessary to draw the distinction between main and additional content automatically. Content extraction and template detection are the two approaches to solve this task. This book gives an extensive overview and detailed description of existing and newly developed algorithms from both areas. The described content extraction algorithms are evaluated under different aspects using objective performance measures. An analysis of methods to cluster web documents according to their underlying templates completes the book. In combination with a localised crawling process this clustering analysis can be used to automatically create sets of training documents for template detection. As the whole process can be automated it allows to perform template detection on a single document, thereby combining the advantages of single and multi document algorithms.