Notes:
- Safesearch doesn't seem to work properly?
- In theory multiple languages are supported, but even in the web UI, they don't work properly
- Possibly, we could cache the request hashes (h query parameter), I'm not sure if it ever changes
The AOL engines deliver no results or too many incorrect results; in #5972 it is
reported that the AOL-images do not work, and on AOL-web the language selection
seems to depend more on the IP than on the selected language / Locally, the
language selection works for me, on the public server only English results are
displayed, which significantly worsens the result list.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
As a side effect, Cloudscraper is no longer needed.
It probably only ever worked by setting the correct request headers,
so we don't really need it since we can just set the right request
headers and ciphersuites ourselves.
Karmasearch seem to crash when searching with long queries >= 100 characters.
The returned JSON is exactly this `["",[]]`, which will crash when trying to
access `resp.json()["results"]`
Close: https://github.com/searxng/searxng/issues/5911
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Unsplash started using [Anubis](https://anubis.techaro.lol/)
for blocking crawlers. Therefore, requests using common
user agents (e.g. Firefox, Chrome) must pass a JavaScript
challenge.
However, other user agents seem unaffected for now, hence
settings the UA to something different does still work.
URLs, name and colors are automatically rendered into manifest.json.
Furthermore user preference of theme (light, dark, black) and theme colors are
respected. Theme colors can be set in settings.yml
Submitting an issue on GitHub isn't a end user task .. most issue reports coming
from this form are not filled out, since end users are not familiar with the
needs of a developer community.
Closes: https://github.com/searxng/searxng/issues/5820
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
I found a bypass using the Android Google App this time. However, unlike the iPhone GSA method, this one does have rate limits. Although it took a couple of hundred consecutive requests to trigger them.
* [enh] engines: rework bing engine
Only Bing-Web has been reworked.
Some features now require JavaScript (paging and time-range results).
Cookies no longer work, parameters such as `cc`, `ui`, ... alter the results.
The engine only appears to use the locale from `Accept-Language` header properly.
The rest of Bing's child engines (Bing-Image, Bing-Video, ...) seem to benefit
from using `mkt` param in conjunction with the `Accept-Language` header
override, although Bing-Web does not (?)
* [enh] explicit mkt
* [fix] engines: bing_videos.py
https://github.com/searxng/searxng/pull/5793#pullrequestreview-3881883250
Remove |safe filter from 6 template locations where data from external
search engine APIs was rendered as raw HTML without sanitization. Jinja2
autoescape now properly escapes these fields.
The |safe filter was originally added in commit 213041adc (March 2021)
by copying the pattern from result.title|safe and result.content|safe.
However, title and content are pre-escaped via escape() in webapp.py
lines 704-706 before highlight_content() adds trusted <span> tags for
search term highlighting. The metadata, info.value, link.url_label,
repository, and filename fields never go through any escaping and flow
directly from external API responses to the template.
Affected templates and their untrusted data sources:
- macros.html: result.metadata from DuckDuckGo, Reuters, Presearch,
Podcast Index, Fyyd, bpb, moviepilot, mediawiki, and others
- paper.html: result.metadata from academic search engines
- map.html: info.value and link.url_label from OpenStreetMap
user-contributed extratags
- code.html: result.repository and result.filename from GitHub API
Example exploit: a search engine API returning
metadata='<img src=x onerror=alert(document.cookie)>' would execute
arbitrary JavaScript in every user's browser viewing that result.
Since about a month, the website just says "temporarily unavailable", so it's safe to assume that it's just no longer working
Related:
- https://github.com/searxng/searxng/pull/3798
Google recently changed the DOM structure for mobile-centric responses, causing the `google_videos` engine to return zero results and the main `google` engine to drop the majority of its results (due to missing snippets or failed URL parsing). These changes restore the functionality and improve the result count for both engines.
This patch updates the parsing logic for both the `google` and `google_videos` engines to handle the modern HTML structure returned by Google when using GSA (Google Search App) User-Agents.
**Specific changes include:**
* **Google Videos (`gov`)**:
* Updated title XPath to support `role="heading"`.
* Improved URL extraction to correctly decode Google redirectors (`/url?q=...`) using `unquote`.
* Added support for the `WRu9Cd` class to capture publication metadata (author/date).
* Broadened thumbnail search and added a fallback to YouTube's `hqdefault.jpg`.
* **Google Web**:
* Relaxed the strict snippet (`content`) requirement. Valid results are no longer discarded if a snippet is missing in the mobile UI.
* Hardened URL extraction to handle both direct and redirected URLs safely.
* Improved thumbnail extraction by searching the entire result block.
Removes the `fasttext-predict` dependency and the language detection code.
If a user now selects `auto` for the search language, the detected language now
falls back directly to the `Accept-Language` header sent by the browser (which was already the fallback when fasttext returned no result).
- fasttext's [language detection is unreliable](https://github.com/searxng/searxng/issues/4195) for some languages, especially short search queries, and in particular for queries containing proper names which is a common case.
- `fasttext-predict` consumes [significant memory](https://github.com/searxng/searxng/pull/1969#issuecomment-1345366676) without offering users much real value.
- the upstream fasttext project was archived by Meta in 2024
- users already have two better alternatives: the `Accept-Language` header and the search-syntax language prefix (e.g. `:fr` or `:de`).
Related: https://github.com/searxng/searxng/issues/4195
Closes: https://github.com/searxng/searxng/issues/5790
Bnyro
