Skip to content
Back to top
View this page
Edit this page

# Utility functions for the engines

Utility functions for the engines

searx.utils.XPathSpecType: TypeAlias = str | lxml.etree.XPath

  Type alias used by `searx.utils.get_xpath`, `searx.utils.eval_xpath` and other XPath selectors.

searx.utils.searxng_useragent() → str

  Return the SearXNG User Agent

searx.utils.gen_useragent(os_string: str | None = None) → str

  Return a random browser User Agent

  See searx/data/useragents.json

searx.utils.gen_gsa_useragent() → str

  Return a random “Google Go App” User Agent suitable for Google

  See searx/data/gsa_useragents.txt

class searx.utils.HTMLTextExtractor

  Internal class to extract text from HTML

searx.utils.html_to_text(html_str: str) → str

  Extract text from a HTML string

  Args:

    - html_str (str): string HTML

  Returns:

    - str: extracted text

  Examples:

        >>> html_to_text('Example <span id="42">#2</span>')
        'Example #2'

        >>> html_to_text('<style>.span { color: red; }</style><span>Example</span>')
        'Example'

        >>> html_to_text(r'regexp: (?&lt;![a-zA-Z]')
        'regexp: (?<![a-zA-Z]'

        >>> html_to_text(r'<p><b>Lorem ipsum </i>dolor sit amet</p>')
        'Lorem ipsum </i>dolor sit amet</p>'

        >>> html_to_text(r'&#x3e &#x3c &#97')
        '> < a'

searx.utils.markdown_to_text(markdown_str: str) → str

  Extract text from a Markdown string

  Args:

    - markdown_str (str): string Markdown

  Returns:

    - str: extracted text

  Examples:

        >>> markdown_to_text('[example](https://example.com)')
        'example'

        >>> markdown_to_text('## Headline')
        'Headline'

searx.utils.extract_text(xpath_results: list[ElementBase | _Element] | ElementBase | _Element | str | Number | bool | None, allow_none: bool = False) → str | None

  Extract text from a lxml result

  - If `xpath_results` is a list of `ElementType` objects, extract the text from each result and concatenate the list in a string.

  - If `xpath_results` is a `ElementType` object, extract all the text node from it ( `lxml.html.tostring`, `method="text"` )

  - If `xpath_results` is of type `str` or `Number`, `bool` the string value is returned.

  - If `xpath_results` is of type `None` a `ValueError` is raised, except `allow_none` is `True` where `None` is returned.

searx.utils.normalize_url(url: str, base_url: str) → str

  Normalize URL: add protocol, join URL with base_url, add trailing slash if there is no path

  Args:

    - url (str): Relative URL

    - base_url (str): Base URL, it must be an absolute URL.

  Example:

        >>> normalize_url('https://example.com', 'http://example.com/')
        'https://example.com/'
        >>> normalize_url('//example.com', 'http://example.com/')
        'http://example.com/'
        >>> normalize_url('//example.com', 'https://example.com/')
        'https://example.com/'
        >>> normalize_url('/path?a=1', 'https://example.com')
        'https://example.com/path?a=1'
        >>> normalize_url('', 'https://example.com')
        'https://example.com/'
        >>> normalize_url('/test', '/path')
        raise ValueError

  Raises:

    - lxml.etree.ParserError

  Returns:

    - str: normalized URL

searx.utils.extract_url(xpath_results: list[ElementBase | _Element] | ElementBase | _Element | str | Number | bool | None, base_url: str) → str

  Extract and normalize URL from lxml Element

  Example:

        >>> def f(s, search_url):
        >>>    return searx.utils.extract_url(html.fromstring(s), search_url)
        >>> f('<span id="42">https://example.com</span>', 'http://example.com/')
        'https://example.com/'
        >>> f('https://example.com', 'http://example.com/')
        'https://example.com/'
        >>> f('//example.com', 'http://example.com/')
        'http://example.com/'
        >>> f('//example.com', 'https://example.com/')
        'https://example.com/'
        >>> f('/path?a=1', 'https://example.com')
        'https://example.com/path?a=1'
        >>> f('', 'https://example.com')
        raise lxml.etree.ParserError
        >>> searx.utils.extract_url([], 'https://example.com')
        raise ValueError

  Raises:

    - ValueError

    - lxml.etree.ParserError

  Returns:

    - str: normalized URL

searx.utils.dict_subset(dictionary: MutableMapping[Any, Any], properties: set[str]) → MutableMapping[str, Any]

  Extract a subset of a dict

  Examples:

        >>> dict_subset({'A': 'a', 'B': 'b', 'C': 'c'}, ['A', 'C'])
        {'A': 'a', 'C': 'c'}
        >>> >> dict_subset({'A': 'a', 'B': 'b', 'C': 'c'}, ['A', 'D'])
        {'A': 'a'}

searx.utils.humanize_bytes(size: int | float, precision: int = 2)

  Determine the human readable value of bytes on 1024 base (1KB=1024B).

searx.utils.humanize_number(size: int | float, precision: int = 0)

  Determine the human readable value of a decimal number.

searx.utils.convert_str_to_int(number_str: str) → int

  Convert number_str to int or 0 if number_str is not a number.

searx.utils.extr(txt: str, begin: str, end: str, default: str = '') → str

  Extract the string between `begin` and `end` from `txt`

  Parameters:

    - txt – String to search in

    - begin – First string to be searched for

    - end – Second string to be searched for after `begin`

    - default – Default value if one of `begin` or `end` is not found. Defaults to an empty string.

  Returns:

    The string between the two search-strings `begin` and `end`. If at least one of `begin` or `end` is not found, the value of `default` is returned.

  Examples:

        >>> extr("abcde", "a", "e")
        "bcd"
        >>> extr("abcde", "a", "z", deafult="nothing")
        "nothing"

searx.utils.int_or_zero(num: list[str] | str) → int

  Convert num to int or 0. num can be either a str or a list. If num is a list, the first element is converted to int (or return 0 if the list is empty). If num is a str, see convert_str_to_int

searx.utils.to_string(obj: Any) → str

  Convert obj to its string representation.

searx.utils.ecma_unescape(string: str) → str

  Python implementation of the unescape javascript function

  https://www.ecma-international.org/ecma-262/6.0/#sec-unescape-string https://developer.mozilla.org/fr/docs/Web/JavaScript/Reference/Objets_globaux/unescape

  Examples:

        >>> ecma_unescape('%u5409')
        '吉'
        >>> ecma_unescape('%20')
        ' '
        >>> ecma_unescape('%F3')
        'ó'

searx.utils.remove_pua_from_str(string: str)

  Removes unicode’s “PRIVATE USE CHARACTER”s (PUA) from a string.

searx.utils.get_engine_from_settings(name: str) → dict[str, dict[str, str]]

  Return engine configuration from settings.yml of a given engine name

searx.utils.get_xpath(xpath_spec: str | XPath) → XPath

  Return cached compiled `lxml.etree.XPath` object.

  `TypeError`:

    Raised when `xpath_spec` is neither a `str` nor a `lxml.etree.XPath`.

  `SearxXPathSyntaxException`:

    Raised when there is a syntax error in the XPath selector (`str`).

searx.utils.eval_xpath(element: ElementBase | _Element, xpath_spec: str | XPath) → Any

  Equivalent of `element.xpath(xpath_str)` but compile `xpath_str` into a `lxml.etree.XPath` object once for all. The return value of `xpath(..)` is complex, read XPath return values for more details.

  `TypeError`:

    Raised when `xpath_spec` is neither a `str` nor a `lxml.etree.XPath`.

  `SearxXPathSyntaxException`:

    Raised when there is a syntax error in the XPath selector (`str`).

  `SearxEngineXPathException:`

    Raised when the XPath can’t be evaluated (masked `lxml.etree..XPathError`).

searx.utils.eval_xpath_list(element: ElementBase | _Element, xpath_spec: str | XPath, min_len: int | None = None) → list[Any]

  Same as `searx.utils.eval_xpath`, but additionally ensures the return value is a `list`. The minimum length of the list is also checked (if `min_len` is set).

searx.utils.eval_xpath_getindex(element: ElementBase | _Element, xpath_spec: str | XPath, index: int, default: Any = <searx.utils._NotSetClass object>) → Any

  Same as `searx.utils.eval_xpath_list`, but returns item on position `index` from the list (index starts with `0`).

  The exceptions known from `searx.utils.eval_xpath` are thrown. If a default is specified, this is returned if an element at position `index` could not be determined.

searx.utils.get_embeded_stream_url(url: str)

  Converts a standard video URL into its embed format. Supported services include Youtube, Facebook, Instagram, TikTok, Dailymotion, and Bilibili.

searx.utils.js_obj_str_to_python(js_obj_str: str) → Any

  Convert a javascript variable into JSON and then load the value

  It does not deal with all cases, but it is good enough for now. chompjs has a better implementation.

searx.utils.parse_duration_string(duration_str: str) → timedelta | None

  Parse a time string in format MM:SS or HH:MM:SS and convert it to a timedelta object.

  Returns None if the provided string doesn’t match any of the formats.