Skip to content
Back to top
View this page
Edit this page

# Engine Library

Implementations of the framework for the SearXNG engines.

- `searx.enginelib.EngineCache`

- `searx.enginelib.Engine`

- `searx.enginelib.traits`

There is a command line for developer purposes and for deeper analysis. Here is an example in which the command line is called in the development environment:

    $ ./manage dev.env
    (dev.env)$ python -m searx.enginelib --help

Hint

The long term goal is to modularize all implementations of the engine framework here in this Python package. ToDo:

- move implementations of the SearXNG’s engines loader to a new module in the `searx.enginelib` namespace.

class searx.enginelib.EngineCache(engine_name: str, expire: int | None = None)

  Persistent (SQLite) key/value cache that deletes its values again after `expire` seconds (default/max: `MAXHOLD_TIME`). This class is a wrapper around `ENGINES_CACHE` (`ExpireCacheSQLite`).

  In the git://searx/engines/demo_offline.py engine you can find an exemplary implementation of such a cache other examples are implemented in:

  - git://searx/engines/radio_browser.py

  - git://searx/engines/soundcloud.py

  - git://searx/engines/startpage.py

  For introspection of the DB, jump into developer environment and run command to show cache state:

      $ ./manage dev.env
      (dev.env)$ python -m searx.enginelib cache state

      cache tables and key/values
      ===========================
      [demo_offline        ] 2025-04-22 11:32:50 count        --> (int) 4
      [startpage           ] 2025-04-22 12:32:30 SC_CODE      --> (str) fSOBnhEMlDfE20
      [duckduckgo          ] 2025-04-22 12:32:31 4dff493e.... --> (str) 4-128634958369380006627592672385352473325
      [duckduckgo          ] 2025-04-22 12:40:06 3e2583e2.... --> (str) 4-263126175288871260472289814259666848451
      [radio_browser       ] 2025-04-23 11:33:08 servers      --> (list) ['https://de2.api.radio-browser.info',  ...]
      [soundcloud          ] 2025-04-29 11:40:06 guest_client_id --> (str) EjkRJG0BLNEZquRiPZYdNtJdyGtTuHdp
      [wolframalpha        ] 2025-04-22 12:40:06 code         --> (str) 5aa79f86205ad26188e0e26e28fb7ae7
      number of tables: 6
      number of key/value pairs: 7

  In the “cache tables and key/values” section, the table name (engine name) is at first position on the second there is the calculated expire date and on the third and fourth position the key/value is shown.

  About duckduckgo: The vqd coode of ddg depends on the query term and therefore the key is a hash value of the query term (to not to store the raw query term).

  In the “properties of ENGINES_CACHE” section all properties of the SQLiteAppl / ExpireCache and their last modification date are shown:

      properties of ENGINES_CACHE
      ===========================
      [last modified: 2025-04-22 11:32:27] DB_SCHEMA           : 1
      [last modified: 2025-04-22 11:32:27] LAST_MAINTENANCE    :
      [last modified: 2025-04-22 11:32:27] crypt_hash          : ca612e3566fdfd7cf7efe...
      [last modified: 2025-04-22 11:32:30] CACHE-TABLE--demo_offline: demo_offline
      [last modified: 2025-04-22 11:32:30] CACHE-TABLE--startpage: startpage
      [last modified: 2025-04-22 11:32:31] CACHE-TABLE--duckduckgo: duckduckgo
      [last modified: 2025-04-22 11:33:08] CACHE-TABLE--radio_browser: radio_browser
      [last modified: 2025-04-22 11:40:06] CACHE-TABLE--soundcloud: soundcloud
      [last modified: 2025-04-22 11:40:06] CACHE-TABLE--wolframalpha: wolframalpha

  These properties provide information about the state of the ExpireCache and control the behavior. For example, the maintenance intervals are controlled by the last modification date of the LAST_MAINTENANCE property and the hash value of the password can be used to detect whether the password has been changed (in this case the DB entries can no longer be decrypted and the entire cache must be discarded).

class searx.enginelib.Engine

  Class of engine instances build from YAML settings.

  Further documentation see General Engine Configuration.

  Hint

  This class is currently never initialized and only used for type hinting.

  engine_type: str

    Type of the engine (Search processors)

  paging: bool

    Engine supports multiple pages.

  max_page: int = 0

    If the engine supports paging, then this is the value for the last page that is still supported. `0` means unlimited numbers of pages.

  time_range_support: bool

    Engine supports search time range.

  safesearch: bool

    Engine supports SafeSearch

  language_support: bool

    Engine supports languages (locales) search.

  language: str

    For an engine, when there is `language: ...` in the YAML settings the engine does support only this one language:

        - name: google french
          engine: google
          language: fr

  region: str

    For an engine, when there is `region: ...` in the YAML settings the engine does support only this one region:

        .. code:: yaml

    > - name: google belgium engine: google region: fr-BE

  fetch_traits: Callable[[EngineTraits, bool], None]

    Function to to fetch engine’s traits from origin.

  traits: traits.EngineTraits

    Traits of the engine.

  categories: list[str]

    Specifies to which categories the engine should be added.

  name: str

    Name that will be used across SearXNG to define this engine. In settings, on the result page ..

  engine: str

    Name of the python file used to handle requests and responses to and from this search engine (file name from git://searx/engines without `.py`).

  enable_http: bool

    Enable HTTP (by default only HTTPS is enabled).

  shortcut: str

    Code used to execute bang requests (`!foo`)

  timeout: float

    Specific timeout for search-engine.

  display_error_messages: bool

    Display error messages on the web UI.

  proxies: dict[str, dict[str, str]]

    Set proxies for a specific engine (YAML):

        proxies :
          http:  socks5://proxy:port
          https: socks5://proxy:port

  disabled: bool

    To disable by default the engine, but not deleting it. It will allow the user to manually activate it in the settings.

  inactive: bool

    Remove the engine from the settings (disabled & removed).

  about: dict[str, dict[str, str]]

    Additional fields describing the engine.

        about:
           website: https://example.com
           wikidata_id: Q306656
           official_api_documentation: https://example.com/api-doc
           use_official_api: true
           require_api_key: true
           results: HTML

  using_tor_proxy: bool

    Using tor proxy (`true`) or not (`false`) for this engine.

  send_accept_language_header: bool

    When this option is activated (default), the language (locale) that is selected by the user is used to build and send a `Accept-Language` header in the request to the origin search engine.

  tokens: list[str]

    A list of secret tokens to make this engine private, more details see Private Engines (tokens).

  weight: int

    Weighting of the results of this engine (weight).

  setup(engine_settings: dict[str, Any]) → bool

    Dynamic setup of the engine settings.

    With this method, the engine’s setup is carried out. For example, to check or dynamically adapt the values handed over in the parameter `engine_settings`. The return value (True/False) indicates whether the setup was successful and the engine can be built or rejected.

    The method is optional and is called synchronously as part of the initialization of the service and is therefore only suitable for simple (local) exams/changes at the engine setting. The `Engine.init` method must be used for longer tasks in which values of a remote must be determined, for example.

  init(engine_settings: dict[str, Any]) → bool | None

    Initialization of the engine.

    The method is optional and asynchronous (in a thread). It is suitable, for example, for setting up a cache (for the engine) or for querying values (required by the engine) from a remote.

    Whether the initialization was successful can be indicated by the return value `True` or even `False`.

    - If no return value is given from this init method (`None`), this is equivalent to `True`.

    - If an exception is thrown as part of the initialization, this is equivalent to `False`.

  abstractmethod search(query: str, params: OfflineParamTypes) → EngineResults

    Search method of the `offline` engines

  abstractmethod request(query: str, params: OnlineParamTypes) → None

    Method to build the parameters for the request of an `online` engine.

  abstractmethod response(resp: SXNG_Response) → EngineResults

    Method to parse the response of an `online` engine.

searx.enginelib.ENGINES_CACHE: ExpireCacheSQLite = <searx.cache.ExpireCacheSQLite object>

  Global `searx.cache.ExpireCacheSQLite` instance where the cached values from all engines are stored. The MAXHOLD_TIME is 7 days and the MAINTENANCE_PERIOD is set to two hours.

## Engine traits

Engine’s traits are fetched from the origin engines and stored in a JSON file in the data folder. Most often traits are languages and region codes and their mapping from SearXNG’s representation to the representation in the origin search engine. For new traits new properties can be added to the class `EngineTraits`.

To load traits from the persistence `EngineTraitsMap.from_data` can be used.

class searx.enginelib.traits.EngineTraitsEncoder(*, skipkeys=False, ensure_ascii=True, check_circular=True, allow_nan=True, sort_keys=False, indent=None, separators=None, default=None)

  Encodes `EngineTraits` to a serializable object, see `json.JSONEncoder`.

  default(o: Any) → Any

    Return dictionary of a `EngineTraits` object.

class searx.enginelib.traits.EngineTraits(regions: dict[str, str]=<factory>, languages: dict[str, str]=<factory>, all_locale: str | None = None, data_type: Literal['traits_v1'] = 'traits_v1', custom: dict[str, ~typing.Any]=<factory>)

  The class is intended to be instantiated for each engine.

  regions: dict[str, str]

    Maps SearXNG’s internal representation of a region to the one of the engine.

    SearXNG’s internal representation can be parsed by babel and the value is send to the engine:

        regions ={
            'fr-BE' : <engine's region name>,
        }

        for key, egnine_region regions.items():
           searxng_region = babel.Locale.parse(key, sep='-')
           ...

  languages: dict[str, str]

    Maps SearXNG’s internal representation of a language to the one of the engine.

    SearXNG’s internal representation can be parsed by babel and the value is send to the engine:

        languages = {
            'ca' : <engine's language name>,
        }

        for key, egnine_lang in languages.items():
           searxng_lang = babel.Locale.parse(key)
           ...

  all_locale: str | None = None

    To which locale value SearXNG’s `all` language is mapped (shown a “Default language”).

  data_type: Literal['traits_v1'] = 'traits_v1'

    Data type, default is ‘traits_v1’.

  custom: dict[str, Any]

    A place to store engine’s custom traits, not related to the SearXNG core.

  get_language(searxng_locale: str, default: str | None = None) → str | None

    Return engine’s language string that best fits to SearXNG’s locale.

    Parameters:

      - searxng_locale – SearXNG’s internal representation of locale selected by the user.

      - default – engine’s default language

    The best fits rules are implemented in `searx.locales.get_engine_locale`. Except for the special value `all` which is determined from `EngineTraits.all_locale`.

  get_region(searxng_locale: str, default: str | None = None) → str | None

    Return engine’s region string that best fits to SearXNG’s locale.

    Parameters:

      - searxng_locale – SearXNG’s internal representation of locale selected by the user.

      - default – engine’s default region

    The best fits rules are implemented in `searx.locales.get_engine_locale`. Except for the special value `all` which is determined from `EngineTraits.all_locale`.

  copy()

    Create a copy of the dataclass object.

  classmethod fetch_traits(engine: Engine | types.ModuleType) → EngineTraits | None

    Call a function `fetch_traits(engine_traits)` from engines namespace to fetch and set properties from the origin engine in the object `engine_traits`. If function does not exists, `None` is returned.

  set_traits(engine: Engine | types.ModuleType) → None

    Set traits from self object in a `Engine` namespace.

    Parameters:

      engine – engine instance build by `searx.engines.load_engine()`

class searx.enginelib.traits.EngineTraitsMap

  A python dictionary to map `EngineTraits` by engine name.

  ENGINE_TRAITS_FILE: Path = PosixPath('/home/runner/work/searxng/searxng/searx/data/engine_traits.json')

    File with persistence of the `EngineTraitsMap`.

  save_data()

    Store EngineTraitsMap in in file `self.ENGINE_TRAITS_FILE`

  classmethod from_data() → EngineTraitsMap

    Instantiate `EngineTraitsMap` object from `ENGINE_TRAITS`

  set_traits(engine: Engine | types.ModuleType)

    Set traits in a `Engine` namespace.

    Parameters:

      engine – engine instance build by `searx.engines.load_engine()`