Onna’s web crawler was created to index web pages.
The web indexer follows links to one level, meaning that it syncs the page(s) from the given URL(s) and all links in that page(s). It crawls html pages for:
– Headings
– Subheadings
– Paragraphs
– Metadata
– Pictures
– Links
– Text links