votes up 2

Spider not found: (spider_name)

Package:
scrapy
github stars 41445
Exception Class:
KeyError

Raise code

        """
        Return the Spider class for the given spider name. If the spider
        name is not found, raise a KeyError.
        """
        try:
            return self._spiders[spider_name]
        except KeyError:
            raise KeyError(f"Spider not found: {spider_name}")

    def find_by_request(self, request):
        """
        Return the list of spider names that can handle the given request.
        """
        return [
            name for name, cls in self._spiders.items()

Ways to fix

votes up 1 votes down

Code to reproduce bug (incorrect name - spider2)

import os, shutil
from scrapy.spiderloader import SpiderLoader
from scrapy.settings import Settings

# Defining a spider inside a 'spiders' directory (normally, such file would already exist)
try:
    os.mkdir(os.path.join('.', 'spiders'))
    with open('./spiders/spider1.py', 'w') as f:
        f.write('from scrapy.spiders import Spider\nclass Spider1(Spider):\n    name = "spider1"\n    allowed_domains = ["foo.org", "bar.org"]')
except: pass

# Actual code to replicate error once files are setup (in whatever way)
try:  
    settings = Settings({ 'SPIDER_MODULES': ['spiders'] })
    spider_loader = SpiderLoader.from_settings(settings)
    spider_loader.load('spider2')

# Code to clean up test spiders
finally:
    try: shutil.rmtree('./spiders')
    except: pass

Correct:

import os, shutil
from scrapy.spiderloader import SpiderLoader
from scrapy.settings import Settings

# Defining a spider inside a 'spiders' directory (normally, such file would already exist)
try:
    os.mkdir(os.path.join('.', 'spiders'))
    with open('./spiders/spider1.py', 'w') as f:
        f.write('from scrapy.spiders import Spider\nclass Spider1(Spider):\n    name = "spider1"\n    allowed_domains = ["foo.org", "bar.org"]')
except: pass

# Actual code to replicate error once files are setup (in whatever way)
try:  
    settings = Settings({ 'SPIDER_MODULES': ['spiders'] })
    spider_loader = SpiderLoader.from_settings(settings)
    spider_loader.load('spider1')  # <-- CHANGE DONE HERE (defined spiders only include spider1)
# Code to clean up test spiders

finally:
    try: shutil.rmtree('./spiders')
    except: pass

Note: in this example, for it to be working, it is necessary to have the spiders defined in a child directory - the example code takes care of that

Jun 26, 2021 Filipino9507 answer

Add a possible fix

Please authorize to post fix