votes up 2

Spider not found: (spider_name)

Package:
scrapy
github stars 41445
Exception Class:
KeyError

Raise code

        """
        Return the Spider class for the given spider name. If the spider
        name is not found, raise a KeyError.
        """
        try:
            return self._spiders[spider_name]
        except KeyError:
            raise KeyError(f"Spider not found: {spider_name}")

    def find_by_request(self, request):
        """
        Return the list of spider names that can handle the given request.
        """
        return [
            name for name, cls in self._spiders.items()
🙏 Scream for help to Ukraine
Today, 2nd July 2022, Russia continues bombing and firing Ukraine. Don't trust Russia, they are bombing us and brazenly lying in same time they are not doing this 😠, civilians and children are dying too! We are screaming and asking exactly you to help us, we want to survive, our families, children, older ones.
Please spread the information, and ask your governemnt to stop Russia by any means. We promise to work extrahard after survival to make the world safer place for all.

Ways to fix

votes up 1 votes down

Code to reproduce bug (incorrect name - spider2)

import os, shutil
from scrapy.spiderloader import SpiderLoader
from scrapy.settings import Settings

# Defining a spider inside a 'spiders' directory (normally, such file would already exist)
try:
    os.mkdir(os.path.join('.', 'spiders'))
    with open('./spiders/spider1.py', 'w') as f:
        f.write('from scrapy.spiders import Spider\nclass Spider1(Spider):\n    name = "spider1"\n    allowed_domains = ["foo.org", "bar.org"]')
except: pass

# Actual code to replicate error once files are setup (in whatever way)
try:  
    settings = Settings({ 'SPIDER_MODULES': ['spiders'] })
    spider_loader = SpiderLoader.from_settings(settings)
    spider_loader.load('spider2')

# Code to clean up test spiders
finally:
    try: shutil.rmtree('./spiders')
    except: pass

Correct:

import os, shutil
from scrapy.spiderloader import SpiderLoader
from scrapy.settings import Settings

# Defining a spider inside a 'spiders' directory (normally, such file would already exist)
try:
    os.mkdir(os.path.join('.', 'spiders'))
    with open('./spiders/spider1.py', 'w') as f:
        f.write('from scrapy.spiders import Spider\nclass Spider1(Spider):\n    name = "spider1"\n    allowed_domains = ["foo.org", "bar.org"]')
except: pass

# Actual code to replicate error once files are setup (in whatever way)
try:  
    settings = Settings({ 'SPIDER_MODULES': ['spiders'] })
    spider_loader = SpiderLoader.from_settings(settings)
    spider_loader.load('spider1')  # <-- CHANGE DONE HERE (defined spiders only include spider1)
# Code to clean up test spiders

finally:
    try: shutil.rmtree('./spiders')
    except: pass

Note: in this example, for it to be working, it is necessary to have the spiders defined in a child directory - the example code takes care of that

Jun 26, 2021 Filipino9507 answer

Add a possible fix

Please authorize to post fix