CyberCode Academy
  • July 23

Course 40 - Web Scraping with Python | Episode 13: Mastering Scrapy Shell, CSS, and XPath Selectors

CyberCode Academy

In this lesson, you’ll learn about: how to use Scrapy Shell for interactive crawling, how CSS selectors work for fast extraction, and how XPath enables advanced and flexible data targeting1. What is Scrapy Shell?🔹 Interactive Prototyping ToolScrapy Shell is a live testing environment where you can:

  • Test selectors before writing spiders
  • Inspect HTML responses instantly
  • Experiment with scraping logic
🔹 Key Objects Inside Shell
  • response → HTML content of the page
  • request → HTTP request details
  • spider → scraper context
👉 Key Insight
You can test everything before writing real crawling logic2. Working with Live URLs and FilesScrapy Shell supports:
  • 🌐 Live websites
  • 📄 Local HTML files
👉 This makes it ideal for debugging broken or complex pages3. CSS Selectors (Fast & Simple)🔹 Basic Extraction🔹 Common SyntaxSelectorMeaning#idSelect by ID.classSelect by classtagSelect by tag🔹 Scrapy Shell Methodsresponse.css("title").get() response.css("p").get_all() 🔹 Extract Attributesresponse.css("img::attr(src)").get() 👉 Key Insight
CSS is perfect for quick, readable extraction4. Important Behavior: Cached Responses🔹 One Hidden DetailScrapy Shell:
  • Works on cached HTML
  • Won’t reflect live changes unless restarted
👉 Key Insight
Always restart shell when debugging updated pages5. XPath Selectors (Advanced Power)🔹 Full DOM NavigationXPath lets you navigate HTML like a tree structure6. Absolute vs Relative XPath🔹 Absolute Path/html/body/div/p
  • Starts from root
  • Very strict
🔹 Relative Path//div/p
  • Searches anywhere
  • More flexible
7. Attribute Matching in XPath🔹 Using @response.xpath("//img[@id='logo']").get() 🔹 Using Wildcardsresponse.xpath("//*[@id='main']").get() 8. Using contains()🔹 Pattern Matchingresponse.xpath("//p[contains(text(), 'news')]").get() 👉 Key Insight
XPath is powerful for uncertain or messy HTML structures9. CSS vs XPathFeatureCSSXPathSimplicity✔️ Easy❌ More complexPower❌ Limited✔️ Very powerfulFlexibilityMediumVery high10. Mental ModelThink of Scrapy Shell as:
  • A laboratory
  • CSS = quick filters
  • XPath = surgical precision tools
Final TakeawayScrapy Shell bridges the gap between:👉 “guessing selectors”
and
👉 “engineered extraction logic”Once you master CSS + XPath inside the shell, you can confidently build spiders that work on even the most complex websites.

You can listen and download our episodes for free on more than 10 different platforms:
https://linktr.ee/cybercode_academy
Episode Webpage

Information

  • Show
    CyberCode Academy
  • Frequency
    Updated Daily
  • Published
    July 23, 2026 at 6:00 AM UTC
  • Length
    21 min
  • Rating
    Clean