Add scrape and rewrite rules for quantamagazine

This is a somewhat complex React site so the rules could be a little fragile.
Text content seems to be always inside .outer--content, and most h6 elements
are fluff like "read later" or pointers to other articles. However, h6.byline
and h6.post__title__kicker are relevant to the current article.

Figure captions are sometimes inside both figure and div.outer--content
elements, sometimes only inside figure, so take both and remove the
intersection.

The figure elements sometimes contain multiple copies of images or
videos, and we just take them all. Math articles seem to use Mathjax,
which we don't add.
This commit is contained in:
Jouni K. Seppänen 2022-01-03 17:41:21 +02:00 committed by Frédéric Guillot
parent 4445cdd053
commit dcf87bd642
2 changed files with 2 additions and 0 deletions

View file

@ -26,6 +26,7 @@ var predefinedRules = map[string]string{
"oglaf.com": "add_image_title", "oglaf.com": "add_image_title",
"optipess.com": "add_image_title", "optipess.com": "add_image_title",
"peebleslab.com": "add_image_title", "peebleslab.com": "add_image_title",
"quantamagazine.org": `remove("h6:not(.byline,.post__title__kicker), #comments, .next-post__content, .footer__section, figure .outer--content")`,
"sentfromthemoon.com": "add_image_title", "sentfromthemoon.com": "add_image_title",
"thedoghousediaries.com": "add_image_title", "thedoghousediaries.com": "add_image_title",
"treelobsters.com": "add_image_title", "treelobsters.com": "add_image_title",

View file

@ -33,6 +33,7 @@ var predefinedRules = map[string]string{
"osnews.com": "div.newscontent1", "osnews.com": "div.newscontent1",
"phoronix.com": "div.content", "phoronix.com": "div.content",
"pseudo-sciences.org": "#art_main", "pseudo-sciences.org": "#art_main",
"quantamagazine.org": ".outer--content, figure",
"raywenderlich.com": "article", "raywenderlich.com": "article",
"slate.fr": ".field-items", "slate.fr": ".field-items",
"techcrunch.com": "div.article-entry", "techcrunch.com": "div.article-entry",