Web Crawler

The web indexer follows links to one level, meaning that it syncs the page(s) from the given URL(s) and all links in that page(s). It crawls html pages for:
– Headings
– Subheadings
– Paragraphs
– Metadata
– Pictures
– Links
– Text links