What is Web Scraping and is Python the best language to use for this? [closed]

What is Web Scraping and is Python the best language to use for this? If so why is python the best?



    What is Web Scraping

The process of making HTTP requests to websites and then extracting data from HTML documents (as opposed to using an official API)

    and is Python the best language to use for this?

Subjective. Argumentative. Insert favourite language (Perl Perl Perl) here.



Web scraping is a computer software technique of extracting information from websites. Usually, such software programs simulate human exploration of the Web by either implementing low-level Hypertext Transfer Protocol (HTTP), or embedding certain full-fledged Web browsers. Web scraping is closely related to Web indexing, which indexes Web content using a bot and is a universal technique adopted by most search engines. In contrast, Web scraping focuses more on the transformation of unstructured Web content, typically in HTML format, into structured data that can be stored and analyzed in a central local database or spreadsheet. Web scraping is also related to Web automation, which simulates human Web browsing using computer software. Uses of Web scraping include online price comparison, weather data monitoring, website change detection, Web research, Web content mashup and Web data integration. (Wikipedia)

Like any language, Python has certain advantages and disadvantages with it. For example, Perl is an easier language to do regular expression with. Then again, BeautifulSoup, a module for Python, makes web scraping really, really easy.




Web scraping is the process of automatically collecting Web information.

Python and Perl have vast libraries for web scraping. Even though I have a personal preference towards Perl when it comes to data extraction, because the ease at which you can use regex, technically Python is no less. However based on the active community, Python would be a better choice.

Python:

BeautifulSoap module which can extract HTML content.

Scrapy Open source framework for web scraping in Python. It is a very elegant framework.

Html5lib HTML Parser based on HTML5 specification

Perl

WWW::Mechanize -- Very powerful module for extracting webpages, and parsing, and easy to use.

Win32::IE::Mechanize -- If you are using windows and wanted to scrape Javascript based pages.

Mozilla::Mechanize -- Scrape JavaScript based pages from Linux, but you need to install GTK

If you want to use Ruby for this task, Nokogiri and Mechanize are probably right tools for the job.



If you are looking into web scraping with python, you may also want to look at lxml package. It has same ElementTree API as stdlib parser, but is much more powerfull and fast.

Plus lxml package provides brilliant lxml.cssselect module.

Depending on your needs, you may want to start with either bare lxml or some web-scraping library like Scrapy, mechanize or SmartHTTP.


Source: http://stackoverflow.com/questions/5926863/what-is-web-scraping-and-is-python-the-best-language-to-use-for-this