Commit graph

9 commits

Author SHA1 Message Date
Frédéric Guillot
1eba1730d1 Move HTTP client to its own package 2018-04-28 10:51:07 -07:00
aniran
322b265d7a Scrape parent element for iframe
Current behavior: if you have an `iframe` scraper rule, `scrapContent`
tries to return the inner HTML of the `iframe`, which turns up blank.

New behavior: like `img` elements, if an `iframe` is matched by a scraper rule,
the parent element's inner HTML (i.e. the `iframe` is returned).
2018-04-27 17:57:22 -07:00
Frédéric Guillot
3c3f397bf5 Make sure the scraper parse only HTML documents 2018-01-02 18:32:01 -08:00
Frédéric Guillot
1d8193b892 Add logger 2017-12-15 18:55:57 -08:00
Frédéric Guillot
c6d9eb3614 Improve content scraper 2017-12-13 21:30:40 -08:00
Frédéric Guillot
84d912c979 Rewrite imports 2017-12-12 21:48:13 -08:00
Frédéric Guillot
ef097f02fe Add the possibility to enable crawler for feeds 2017-12-12 19:19:36 -08:00
Frédéric Guillot
87ccad5c7f Add scraper rules 2017-12-10 20:51:04 -08:00
Frédéric Guillot
7a35c58f53 Add readability package to fetch original content 2017-12-10 19:01:38 -08:00