Project

General

Profile

Feature #3613

Extension: Personal functionality - scrape non-yt page and store in GCS bucket

Added by Ram Kordale 5 months ago. Updated 5 months ago.

Status:
Review
Priority:
High
Start date:
06/17/2024
Due date:
% Done:

90%

Estimated time:
1.20 h

Description

If config.js has "Personal" set to true and 'Scrape_Content' set to true, and it is not a Youtube page, scrape the entire page and store the txt file as yyyymmhhmmss in <env>-scraped-content, where <env> is "edutestdev", "edutestqa" or "prod" in the following format:

URL: <url>
Content: <content>

Also available in: Atom PDF