CyberCode Academy

CyberCode Academy
  • 0.0 (0)
  • קורסים
  • עדכון יומי

Welcome to CyberCode Academy — your audio classroom for Programming and Cybersecurity. 🎧 Each course is divided into a series of short, focused episodes that take you from beginner to advanced level — one lesson at a time. From Python and web development to ethical hacking and digital defense, our content transforms complex concepts into simple, engaging audio learning. Study anywhere, anytime — and level up your skills with CyberCode Academy. 🚀 Learn. Code. Secure. You can listen and download our episodes for free on more than 10 different platforms: https://linktr.ee/cybercode_academy

  1. לפני 10 שע׳

    Course 40 - Web Scraping with Python | Episode 35: Locating Dynamic Elements with Selenium and Python

    This module is basically about the core skill in Selenium automation: reliably finding the right element on a page that keeps changing.🧩 What “locating elements” really meansIn Selenium, everything you interact with is a web element, such as: buttonsinput fieldslinksimageshidden UI components used by JavaScriptModern web apps are often dynamic, meaning: IDs change on every refreshclasses are generated randomlyelements appear/disappear after AJAX callsSo the real challenge is not clicking elements — it’s finding them consistently.⚠️ The Dynamic Web ProblemUnlike static HTML pages, modern JavaScript-heavy sites: regenerate DOM elements constantlyload content asynchronouslymodify attributes at runtimeThat’s why a locator that works once may fail on the next page load.🔍 The 8 Ways to Locate ElementsSelenium gives multiple strategies. Each has a different “strength level”.1. 🆔 ID (Best option) Fastest and most reliableMust be uniqueBreaks only if developers change structure2. 🏷️ Name Works when ID is missingCommon in forms3. 🔗 Link Text Matches full hyperlink textExample: “Login”Partial Link Text Matches part of a linkMore flexible but less precise4. 🎯 CSS Selectors Very powerful and widely usedUses patterns like:classeshierarchyattributesExample idea:“div.container button.primary”5. 🧱 Tag Name Finds elements like , , Usually returns many results6. 🎨 Class NameUses CSS class attributeRisk: many elements share same class7. 🧭 XPath (Most powerful)Can navigate DOM like a treeWorks even when structure is messyKey advantage:supports relative pathscan search based on text, attributes, hierarchy⚙️ Two Core Retrieval Methods🔹 find_elementreturns single elementthrows error if not foundbest when you expect exactly one match🔹 find_elementsreturns list of elementssafe (no exception if empty)you can loop or index results🧠 Practical InsightThe real decision rule is:Use ID firstIf not available → CSS SelectorIf structure is complex → XPathIf multiple results → find_elements⚡ Key TakeawayThis module is really teaching one idea:Selenium automation fails not because of actions, but because of bad element selection strategiesSo robust scraping depends on:choosing stable attributesavoiding fragile selectorshandling dynamic DOM changes You can listen and download our episodes for free on more than 10 different platforms: https://linktr.ee/cybercode_academy

    Course 40 - Web Scraping with Python | Episode 35: Locating Dynamic Elements with Selenium and Python

  2. לפני יום אחד (1)

    Course 40 - Web Scraping with Python | Episode 34: Architecture, Setup, and Basic Web Automation

    This episode focuses on how Selenium WebDriver actually works under the hood, and then walks into the practical setup and first automation steps.🧠 Selenium WebDriver ArchitectureSelenium WebDriver is designed to control browsers as realistically as possible, which is why it uses a multi-layer architecture instead of direct code-to-browser control.🧩 1. Language BindingsThese are client libraries that let you write automation scripts in different languages: PythonJavaJavaScriptC#They translate your code into commands WebDriver can understand.🌐 2. JSON Wire Protocol (or W3C WebDriver Protocol)This is the communication layer. Your script sends HTTP requestsCommands are encoded as JSON payloadsThese requests are sent to the browser driverThink of it as:“Selenium speaking HTTP to the browser”🧭 3. Browser DriversEach browser has its own driver: Chrome → ChromeDriverFirefox → GeckoDriverTheir job is to: receive commandstranslate them into browser-native actions🖥️ 4. Real BrowserFinally, the driver controls the actual browser: opens pagesclicks elementsexecutes JavaScriptrenders content⚙️ How Execution FlowsA Selenium action follows this chain:Your Python code → Selenium library → HTTP request → Browser Driver → BrowserThis layered design is what allows cross-browser automation.🛠️ Environment Setup OverviewThe episode walks through setting up a working Selenium environment:📦 Install core libraries Selenium (automation engine)BeautifulSoup (optional parsing tool)🌐 Install browser driver Must match your browser version exactlyExample: Chrome version ↔ ChromeDriver version📓 Optional tools Jupyter Notebook for interactive testingUseful for debugging selectors step-by-step🚀 Basic WebDriver UsageOnce setup is complete, the workflow becomes:1. Start browser instance Launch Chrome/Firefox via WebDriver2. Navigate to a page Open a URL like a normal user3. Perform actions clickscrollinput textextract elements4. Close browser clean shutdown of session🧊 Headless BrowsingA key optimization introduced is headless mode.What it means: Browser runs without UINo visible window opensWhy it matters: faster executionlower memory usageideal for servers and automation pipelines🧠 Key InsightThe main idea of this episode is:Selenium is not just a scraping tool — it's a remote control system for real browsersThat’s why it can handle: JavaScript-rendered contentuser interactionsdynamic page updates You can listen and download our episodes for free on more than 10 different platforms: https://linktr.ee/cybercode_academy

    Course 40 - Web Scraping with Python | Episode 34: Architecture, Setup, and Basic Web Automation

  3. לפני יומיים (2)

    Course 40 - Web Scraping with Python | Episode 33: Foundations of Scraping Dynamic Webpages with Python and Selenium

    This episode is essentially a setup guide for moving from simple HTTP-based scraping to full browser automation using Selenium, especially for websites where content is rendered or modified by JavaScript.🌐 Web Scraping vs Dynamic Web Pages🧾 What “web scraping” means hereWeb scraping is framed as:Converting web page content into structured data for analysisBut the key challenge is that not all content is immediately visible in HTML.🧱 Static vs Dynamic Content📄 Static content Same HTML for every userCan be scraped with tools like Requests or BeautifulSoupNo JavaScript dependency⚡ Dynamic content Changes based on:user interactiontimelocationJavaScript executionOften not present in raw HTMLRequires browser simulation to access🤖 Why Selenium is NeededTraditional scrapers only download HTML.But modern websites: render content with JavaScriptload data after page loadrequire clicks/scrolling to reveal content👉 Selenium solves this by controlling a real browser.🧰 Selenium OverviewSelenium is described as an automation framework for browsers, not just a scraping tool.It allows you to: open web pagesclick buttonsscroll pagesfill formssimulate real users🧩 Core Selenium Components1. 🧪 Selenium IDE Record & playback toolUsed for quick prototypingNo coding required2. 🧬 Selenium RC (Legacy) First generation frameworkAllowed multi-language test scriptsNow largely obsolete3. 🧭 Selenium WebDriver (Main tool)This is the core engine used in real projectsIt: directly controls the browserexecutes user-like actionsinteracts with page elements👉 This is the most important part for scraping dynamic sites4. 🌐 Selenium Grid Enables parallel executionRuns tests across multiple machines/browsersUsed for scaling automation⚙️ Prerequisites for Using SeleniumBefore practical usage, you need: Python basicsHTML/CSS understandingBrowser driver setup (ChromeDriver / GeckoDriver conceptually)Ability to inspect web elements🚀 What Selenium Enables in ScrapingWith Selenium WebDriver, you can: load JavaScript-heavy pageswait for content to appearinteract with UI elementsextract final rendered DOMThis is crucial for modern websites like: dashboardssocial media pagese-commerce filtersinfinite scroll pages🧠 Key InsightThe main takeaway is:Traditional scrapers read HTML. Selenium scrapes the rendered browser state.That difference is what makes it powerful for dynamic content. You can listen and download our episodes for free on more than 10 different platforms: https://linktr.ee/cybercode_academy

    Course 40 - Web Scraping with Python | Episode 33: Foundations of Scraping Dynamic Webpages with Python and Selenium

  4. לפני 3 ימים

    Course 40 - Web Scraping with Python | Episode 32: Native Data Storage and Implementation

    This episode is about removing custom storage code from your Scrapy project and replacing it with Scrapy’s built-in Feed Export system, which turns scraping into a fully configurable data export pipeline.📤 Scrapy Feed Exporters (Automated Data Storage)🧠 Core IdeaInstead of manually writing data to files or databases, Scrapy can automatically export scraped items using:Feed Exporters = built-in serialization + storage systemThey handle: formattingwritingdestination management📊 1. Supported Output FormatsScrapy can serialize scraped data into multiple formats:🧾 File formats JSON → full structured exportJSON Lines (JSONL) → streaming-friendly formatCSV → spreadsheet-ready formatXML → hierarchical structured outputEach format is useful depending on downstream usage: JSON → APIs & appsCSV → Excel / analyticsXML → structured integrationsJSONL → big data pipelines🌍 2. Storage BackendsFeed exporters are not limited to local files.They can write directly to: 💻 Local filesystem📡 FTP servers☁️ Amazon S3 (cloud storage)This makes Scrapy suitable for:enterprise-level data pipelines without extra storage code⚙️ 3. Pipeline + Export IntegrationA key concept in this episode is the separation of concerns:🔹 Pipelines (data filtering layer)Used to: remove unwanted itemsenforce business rulesclean or block dataExample: drop books above a certain pricefilter invalid entries🔹 Feed Exporters (storage layer)Used to: take final cleaned itemsserialize themwrite them to destination🧪 4. Configuration-Driven DesignInstead of writing export logic in code, everything is moved into:🛠️ settings.pyYou define: output formatoutput destination (URI)export behaviorExample conceptually:FEEDS: output.json: format: json encoding: utf8 🔄 5. Full Data FlowSpider ↓ Item Extraction ↓ Pipelines (filter + clean) ↓ Feed Exporter (serialize) ↓ Storage (file / S3 / FTP) 🧪 6. Practical Demo InsightThe episode’s demo reinforces:✔ Filtering firstItems are removed before export via pipelines.✔ No manual savingNo open() or file handling needed.✔ Automatic export generationScrapy generates: JSON outputXML outputstructured datasets🧠 Key TakeawayThe main idea is:Scrapy becomes a configuration-driven data exporter, not just a scraper.You define: what to extract (spider)what to keep (pipelines)where to store it (feed exporters)Everything else is automated.🚀 Big PictureThis module completes the Scrapy data pipeline:StageResponsibilitySpiderExtract dataPipelineClean/filter dataFeed ExporterSerialize + store data You can listen and download our episodes for free on more than 10 different platforms: https://linktr.ee/cybercode_academy

    Course 40 - Web Scraping with Python | Episode 32: Native Data Storage and Implementation

  5. לפני 4 ימים

    Course 40 - Web Scraping with Python | Episode 31: From Item Loaders to Pipelines

    This episode is essentially about turning Scrapy from “just a scraper” into a full data processing system, where extraction, cleaning, validation, and storage are all structured and automated.🕷️ Scrapy Data Population & Processing Pipeline1. 📦 Item Loaders (Structured Data Population)Item Loaders are the layer between raw scraped HTML and structured Scrapy Items.Instead of manually assigning fields, you feed data through controlled methods:🔹 Core methods add_xpath()add_css()add_value()These methods: collect raw extracted valuespass them through processors automaticallybuild a clean final item via load_item()💡 Why this mattersInstead of: messy manual parsingscattered cleaning logicYou get:A single controlled pipeline for building structured objects🔄 Item Loader FlowResponse HTML ↓ add_xpath / add_css / add_value ↓ Input Processors (cleaning + normalization) ↓ Item Fields (structured data) ↓ load_item() ⚙️ 2. Item Pipelines (Post-Extraction Processing Layer)Item Pipelines operate after scraping, acting like a processing conveyor belt.Each pipeline class can: modify datavalidate datareject invalid itemsstore data🔹 Common Pipeline Responsibilities🧹 Data Cleaning remove unwanted charactersnormalize formatsfix inconsistent values✅ Validation check price formatsvalidate emails or URLsensure required fields exist🚫 Filtering drop invalid or unwanted itemsblock duplicatesfilter based on business rules💾 Storage save to databaseexport to JSON / CSVpush into APIs📚 3. Practical Example: Book Scraping SystemThe episode demonstrates a real workflow using a book website.🔹 Data Transformation ExampleMapCompose usageUsed to transform raw fields like: image URLs → full valid URLsbook links → normalized linkstext cleanup (whitespace, symbols)🔹 Custom Pipeline LogicExample rule:“Flag or drop books where price > threshold”So the pipeline can: mark expensive booksexclude them entirelyor route them differently🔹 Pipeline OrderingScrapy allows multiple pipelines:You define execution order in settings:Item Pipeline Order: 1. Cleaning Pipeline 2. Validation Pipeline 3. Filtering Pipeline 4. Storage Pipeline This ensures:Data always flows in a predictable transformation sequence🧠 Key Concept of the EpisodeThe main idea is:Scrapy is not a scraper — it is a data engineering pipeline frameworkYou are not just collecting data, you are: structuring it (Item Loaders)refining it (Processors)validating it (Pipelines)and storing it (Final output layer)🧩 Mental ModelLayerPurposeItem LoadersBuild structured itemsProcessorsClean + normalize fieldsPipelinesValidate + transform + storeSettingsControl execution order🚀 Big Picture InsightThis episode shows the shift from:❌ “scrape → print data”to:✅ “scrape → structure → clean → validate → store → scale” You can listen and download our episodes for free on more than 10 different platforms: https://linktr.ee/cybercode_academy

    Course 40 - Web Scraping with Python | Episode 31: From Item Loaders to Pipelines

  6. לפני 5 ימים

    Course 40 - Web Scraping with Python | Episode 30: Controlling URL Paths and Processing Scraped Data

    This episode is really about controlling Scrapy’s crawl scope and shaping data as it moves through the pipeline, so you’re not just collecting data—you’re actively engineering what gets collected and how it looks.🕷️ Scrapy Crawl Control & Data Processing Pipeline1. 🎯 URL Path Control (Allow / Deny Rules)In Scrapy, crawl behavior is tightly controlled using rule-based filtering, often inside spiders like CrawlSpider.🔹 Allow rules Define what URLs the spider is allowed to followTypically based on regex patternsUsed to target specific sections of a site (e.g., product pages)🔹 Deny rules Explicitly block unwanted pathsUseful for excluding:irrelevant categoriesadmin pagesunwanted content typesExample use cases: Allow: /products/.*Deny: /category/crime/.*, /adult/.*Key idea:You are shaping the crawler’s “attention span” using URL patterns.⚙️ 2. Data Processing Pipeline (Item Loaders)Once Scrapy extracts raw HTML data, it passes through a structured transformation system.This is where Item Loaders + Processors come in.🔄 Input vs Output Processors📥 Input Processors Run immediately after extractionClean or normalize raw scraped valuesExample: stripping whitespace, converting formats📤 Output Processors Run after all values are collectedProduce final cleaned field value🧠 3. Built-in Processor ToolsScrapy provides reusable functions to transform scraped data efficiently:🔹 MapComposeApplies functions to every item in a list.Example use: strip spacesconvert strings to integersnormalize URLs👉 Think of it as:“run this function on every extracted piece of data”🔹 JoinCombines multiple values into a single string.Example:["New", "York"] → "New York" Used when: HTML splits text into multiple nodesYou want a single clean field🔹 TakeFirstReturns: the first non-null value from a listUseful because: Scrapy often returns multiple matchesYou usually only want one final value🔗 4. Full Data Flow (Important Concept)This is the critical architecture idea in the episode:HTML Response ↓ Selectors (XPath / CSS) ↓ Item Loader ↓ Input Processors (cleaning stage 1) ↓ Output Processors (final formatting) ↓ Items ↓ Item Pipelines (storage / DB / export) 🧠 Core Insight of the EpisodeThe key idea is:Scrapy is not just scraping data — it is a data transformation pipeline systemYou don’t just extract data… You control how messy web data becomes structured business intelligence.📌 Mental ModelComponentPurposeAllow / Deny rulesControl crawl scopeInput ProcessorsClean raw extractionOutput ProcessorsFinal formattingMapComposeTransform listsJoinMerge textTakeFirstReduce noise You can listen and download our episodes for free on more than 10 different platforms: https://linktr.ee/cybercode_academy

    Course 40 - Web Scraping with Python | Episode 30: Controlling URL Paths and Processing Scraped Data

  7. לפני 6 ימים

    Course 40 - Web Scraping with Python | Episode 29: From Feed and Sitemap Spiders to CrawlSpider Demos

    This episode is really about choosing between manual control and automated crawling logic inside Scrapy, and understanding how specialized spider classes change your level of control.Here’s the structured breakdown:🕷️ Scrapy Spider Types — Practical Comparison & Feed Spiders1. Feed-Based Spiders (Structured Data Sources)These spiders are not designed for HTML pages — they target pre-structured data formats.📄 XMLFeedSpider ScrapyPurpose:Extract structured data from XML feeds.Key concept: Works by iterating through XML nodesUses itertag to define which tag to extractUses iterator mode (itnodes) for performanceBehavior:Instead of parsing a full page, it streams through XML elements one by one.📊 CSVFeedSpider ScrapyPurpose:Scrape structured CSV files directly.Key features: Custom delimiters (, ; \t)Configurable quote charactersHeader mapping → fields become item keysBehavior:Each row becomes a structured item automatically.2. SitemapSpider (Automated URL Discovery)SitemapSpider ScrapyPurpose:Crawl websites using their sitemap instead of link discovery.How it works: Reads sitemap.xmlExtracts all URLs listedFilters URLs using:regex rulescallback mapping rulesAdvantage:No need to manually discover or follow links.⚔️ 3. scrapy.Spider vs CrawlSpider (Core Comparison)🧱 A. scrapy.Spider (Manual Control)Behavior: You define:start_urlsparse() logicpagination logic manuallyWhat you control: Every requestEvery page transitionEvery extraction stepExample characteristics: CSS selectors used explicitlyMust manually follow “next page” linksFull control over flowKey idea:You are writing the crawling engine logic yourself.🤖 B. CrawlSpider (Automated Crawling)Behavior: Uses Rules + LinkExtractorsAutomatically follows linksWhat it does for you: Finds links automaticallyFilters them using regex or CSS rulesCalls callbacks automaticallyScope: Much broader by defaultCan crawl entire domains unless restrictedKey idea:You define rules — Scrapy handles navigation.🔄 4. Real Demo Insight (Quotes Scraping Example)scrapy.Spider behavior: Manually extract dataManually handle paginationPages may finish in non-sequential order (async execution)CrawlSpider behavior: Automatically follows linksLess manual parsing logicMore scalable for large websites🧠 Core Concept of the EpisodeThe real takeaway is:scrapy.Spider = precision control CrawlSpider = autonomous exploration📌 Mental ModelTypeStrengthWeaknessscrapy.SpiderFull controlMore codeCrawlSpiderAutomationLess fine-grained controlSitemapSpiderFast discoveryDepends on sitemapXML/CSV SpidersStructured feedsLimited flexibility You can listen and download our episodes for free on more than 10 different platforms: https://linktr.ee/cybercode_academy

    Course 40 - Web Scraping with Python | Episode 29: From Feed and Sitemap Spiders to CrawlSpider Demos

  8. 8 באוג׳

    Course 40 - Web Scraping with Python | Episode 28: Base and Generic Crawling Classes

    This episode is essentially about how Scrapy structures crawling logic through different spider types, and when to use each one depending on the scale and structure of the target site.Here’s the clean, structured breakdown:🕷️ Scrapy Spiders — Architecture & Types1. What a Spider Actually IsA Scrapy spider is a Python class that defines: Where to crawl (scoping)How to crawl (link following rules)What to extract (parsing logic)So every spider always answers three questions:Where do I start? → Where do I go next? → What data do I take?2. Base Class: scrapy.Spider ScrapyThis is the simplest and most flexible spider.Core structure: name → identifier for the spiderallowed_domains → restricts crawling scopestart_urls → initial entry pointsFlow: Scrapy sends requests automatically via start_requestsResponses are passed to parse()You manually extract data + generate next requestsKey idea:Full manual control over crawling logic3. CrawlSpider (Rule-Based Automation)CrawlSpiderThis is the most commonly used advanced spider.Instead of manually controlling navigation, you define rules.Core concept: Uses Link ExtractorsUses RulesAutomatically follows links that match conditionsExample behavior: “Follow all product links”“Ignore login pages”“Only crawl category pages”Why it matters:It automates link discovery instead of writing it manually.4. SitemapSpider (Structured Crawling)SitemapSpiderDesigned for websites that expose: /sitemap.xmlBehavior: Reads sitemap URLsExtracts all listed links automaticallyCrawls them without link discovery logicBest for: Large structured websitesSEO-friendly sitesE-commerce catalogs5. XMLFeedSpider & CSVFeedSpiderThese are specialized for data feeds, not HTML pages.XMLFeedSpider: Iterates over XML nodesExtracts structured fieldsCSVFeedSpider: Iterates row-by-row through CSV filesUse case:When the “website” is already a dataset feed6. CrawlSpider Rules SystemThis is the most important upgrade over base spiders.Components: Link Extractor → finds links on pagesRules → define which links to followCallback functions → process matched pagesExample logic: Follow category pagesExtract product pages onlyIgnore pagination or ads7. Parsing Mechanism (Shared Concept)Across all spiders:Parsing step always includes: Extracting structured fields (title, price, etc.)Using XPath or CSS selectorsYielding items or new requests8. Spider Selection StrategyHere’s how you choose:Spider TypeBest Use CaseSpiderCustom logic, full controlCrawlSpiderRegular websites with link patternsSitemapSpiderSEO-driven structured sitesXMLFeedSpiderXML APIs / feedsCSVFeedSpiderCSV datasets🧠 Key InsightThe real concept behind this episode is:Scrapy is not about writing scrapers — it’s about choosing the right crawling strategy. You can listen and download our episodes for free on more than 10 different platforms: https://linktr.ee/cybercode_academy

    Course 40 - Web Scraping with Python | Episode 28: Base and Generic Crawling Classes
הצגת הכול (329)

אודות

Welcome to CyberCode Academy — your audio classroom for Programming and Cybersecurity. 🎧 Each course is divided into a series of short, focused episodes that take you from beginner to advanced level — one lesson at a time. From Python and web development to ethical hacking and digital defense, our content transforms complex concepts into simple, engaging audio learning. Study anywhere, anytime — and level up your skills with CyberCode Academy. 🚀 Learn. Code. Secure. You can listen and download our episodes for free on more than 10 different platforms: https://linktr.ee/cybercode_academy

מידע

  • יוצרים
    CyberCode Academy
  • שנות פעילות
    2025 - 2026
  • פרקים
    329
  • סיווג
    תוכן הולם
  • זכויות יוצרים
    © Copyright CyberCode Academy
  • אתר התכנית
    CyberCode Academy