Content Extraction

When converting full web pages, you typically want the article content without navigation, ads, sidebars, cookie banners, and other boilerplate. The extraction module prunes the DOM tree before conversion, keeping only meaningful content.

Basic Usage

Tip: See extraction in action in the playground - toggle "Extract content" to compare output with and without extraction.

import { convert } from 'markdown-for-agents';

const { markdown } = convert(html, { extract: true });

With extract: true, the library strips elements using a built-in set of heuristics based on HTML tags, ARIA roles, class names, and IDs.

What Gets Stripped

By Tag Name

By Hidden State

Content that a page hides from assistive technology is decorative, collapsed, or a layout duplicate of visible text, so it is stripped the same way. Elements with aria-hidden="false" are kept.

By ARIA Role

By Class Pattern

Elements with classes matching these patterns are stripped:

• ad, ads, ad- - advertisements
• sidebar - sidebar content
• widget - widget blocks
• cookie - cookie consent banners
• popup, modal - overlays and dialogs
• breadcrumb - breadcrumb navigation
• footnote - footnotes
• share, social - social sharing widgets
• newsletter - newsletter signup forms
• comment - comment sections
• related - related content blocks

By ID Pattern

Elements with IDs matching these patterns are stripped:

• ad, ads, ad- - advertisements
• sidebar - sidebar content
• cookie - cookie consent
• popup, modal - overlays

Keeping Specific Elements

You can selectively keep elements that would otherwise be stripped:

const { markdown } = convert(html, {
    extract: {
        keepHeader: true, // Keep <header> elements
        keepFooter: true, // Keep <footer> elements
        keepNav: true, // Keep <nav> elements
        keepHidden: true // Keep hidden and aria-hidden elements
    }
});

This is useful when the page header or navigation contains important content, or when a page hides text it still wants read.

Custom Strip Rules

Add your own strip rules alongside the defaults:

const { markdown } = convert(html, {
    extract: {
        // Additional tags to strip
        stripTags: ['section', 'figure'],

        // Additional class patterns (string or RegExp)
        stripClasses: [/\bpromo\b/i, 'banner-wrapper'],

        // Additional roles to strip
        stripRoles: ['status', 'alert'],

        // Additional ID patterns (string or RegExp)
        stripIds: [/\bpopover\b/i, 'disclaimer']
    }
});

Custom patterns are additive - they extend the default patterns rather than replacing them.

Pattern Matching

• String patterns match as substrings: "banner" matches class="top-banner-wrapper"
• RegExp patterns use .test(): /\bbanner\b/i matches class="Banner" but not class="banners"

Using Extraction Directly

You can use the extraction module independently from the converter:

import { extractContent } from 'markdown-for-agents/extract';
import { parseDocument } from 'htmlparser2';

const document = parseDocument(html);
extractContent(document, { keepHeader: true });
// document is now mutated - stripped elements are removed

This is useful if you need to manipulate the DOM tree after extraction but before conversion.

Examples

Blog Post

const { markdown } = convert(blogHtml, {
    extract: true,
    baseUrl: 'https://blog.example.com'
});
// Returns just the article text with resolved image URLs

Documentation Page with Nav

const { markdown } = convert(docsHtml, {
    extract: {
        keepNav: true // Keep docs sidebar navigation
    }
});

Custom CMS with Widget Classes

const { markdown } = convert(cmsHtml, {
    extract: {
        stripClasses: [/\bcms-widget\b/, /\bcms-toolbar\b/]
    }
});