Spider not found: (spider_name)

Package:

scrapy

41445

Exception Class:

KeyError

Raise code

        """
        Return the Spider class for the given spider name. If the spider
        name is not found, raise a KeyError.
        """
        try:
            return self._spiders[spider_name]
        except KeyError:
            raise KeyError(f"Spider not found: {spider_name}")

    def find_by_request(self, request):
        """
        Return the list of spider names that can handle the given request.
        """
        return [
            name for name, cls in self._spiders.items()

Links to the raise (1)

https://github.com/scrapy/scrapy/blob/ee682af3b06d48815dbdaa27c1177b94aaf679e1/scrapy/spiderloader.py#L77

Ways to fix

Code to reproduce bug (incorrect name - spider2)

import os, shutil
from scrapy.spiderloader import SpiderLoader
from scrapy.settings import Settings

# Defining a spider inside a 'spiders' directory (normally, such file would already exist)
try:
    os.mkdir(os.path.join('.', 'spiders'))
    with open('./spiders/spider1.py', 'w') as f:
        f.write('from scrapy.spiders import Spider\nclass Spider1(Spider):\n    name = "spider1"\n    allowed_domains = ["foo.org", "bar.org"]')
except: pass

# Actual code to replicate error once files are setup (in whatever way)
try:  
    settings = Settings({ 'SPIDER_MODULES': ['spiders'] })
    spider_loader = SpiderLoader.from_settings(settings)
    spider_loader.load('spider2')

# Code to clean up test spiders
finally:
    try: shutil.rmtree('./spiders')
    except: pass

Correct:

import os, shutil
from scrapy.spiderloader import SpiderLoader
from scrapy.settings import Settings

# Defining a spider inside a 'spiders' directory (normally, such file would already exist)
try:
    os.mkdir(os.path.join('.', 'spiders'))
    with open('./spiders/spider1.py', 'w') as f:
        f.write('from scrapy.spiders import Spider\nclass Spider1(Spider):\n    name = "spider1"\n    allowed_domains = ["foo.org", "bar.org"]')
except: pass

# Actual code to replicate error once files are setup (in whatever way)
try:  
    settings = Settings({ 'SPIDER_MODULES': ['spiders'] })
    spider_loader = SpiderLoader.from_settings(settings)
    spider_loader.load('spider1')  # <-- CHANGE DONE HERE (defined spiders only include spider1)
# Code to clean up test spiders

finally:
    try: shutil.rmtree('./spiders')
    except: pass

Note: in this example, for it to be working, it is necessary to have the spiders defined in a child directory - the example code takes care of that

Jun 26, 2021

Filipino9507 3

Add a possible fix

Please authorize to post fix