Post

Web search engine or a search feature for an app?

Need to crawl pages for jobs

In that case try to use one of the available open source options like swirl or apache solr. Do not reinvent it, it’s very hard to get it right.

About 20+ years ago I built one for a customer that wanted a horse industry focused search engine. Web crawling and indexing before modern scraping tools, whee! Recently working on one for localized events. What are you getting into? #buildinpublic

Need to crawl career pages. Anything good you would recommend for me to get started?

I'll start with: do you really need to scrape career pages? What's the pain you are really trying to solve, and can you solve it easier? If you really need to, then: Python has great libraries for web scraping. Also look at scrapy.org

Larry and Sergey built one some years ago, not sure what became of it.