Commit Graph

1127 Commits

Author SHA1 Message Date
Matthieu Gautier 1514661c26 Protect search from multi threading race condition.
libzim's search is not thread safe (mainly because xapian is not).
So we must protect our search objects from multi thread calls.

The best way to do this is to associate a mutex to the `zim::Searcher`
and lock the searcher each time we access object derivated from the
searcher (search, results, iterator, ...)
2022-06-02 12:37:52 +02:00
Matthieu Gautier e5ea210d2c Add a template specialization for ConcurrentCache storing shared_ptr
When ConcurrentCache store a shared_ptr we may have shared_ptr in used
while the ConcurrentCache has drop it.
When we "recreate" a value to put in the cache, we don't want to recreate
it, but copying the shared_ptr in use.

To do so we use a (unlimited) store of weak_ptr (aka `WeakStore`)
Every created shared_ptr added to the cache has a weak_ptr ref also stored
in the WeakStore, and we check the WeakStore before creating the value.
2022-06-02 12:37:52 +02:00
Matthieu Gautier 2b38d2cf1b Copy the lrucache test from libzim.
- Adapt lrucache.cpp for rigth include path
  and use `kiwix::lru_cache` instead of `zim::lru_cache`.
- Add missing `#include <set>` in lrucache.h
2022-06-02 12:37:52 +02:00
Matthieu Gautier 0081b4d8e7 Make the limit of zim files per search configurable.
The default value is 0, which means no limit.
2022-06-02 12:37:52 +02:00
Matthieu Gautier b74910b2af Limit the number of zim in multizim fulltext search.
We are currently limiting to 5 but it will be changed in next commit.
2022-06-02 12:37:50 +02:00
Matthieu Gautier cf30233358 Prefix env variable name with `KIWIX_` 2022-06-02 12:23:43 +02:00
Matthieu Gautier f0065fdd6f Introduce Error exception to do i18n 2022-06-02 12:23:42 +02:00
Matthieu Gautier c72132054d Move i18n helper functions 2022-06-02 12:22:28 +02:00
Matthieu Gautier 077ceac5a5 Make the search_rendered handle multizim search.
This introduce a intermediate mustache object to store information
about the request made by the user.
2022-06-02 12:22:28 +02:00
Matthieu Gautier 39d0a56be8 Use selectBooks in handle_search 2022-06-02 12:22:28 +02:00
Matthieu Gautier 76d5fafb72 Introduce `selectBooks`
`selectBooks` allow us to parse a query in a "standard" way to get
the book(s) on which the user want to work.
2022-06-02 12:22:28 +02:00
Matthieu Gautier 4438106c2f Add a prefix in get_search_filter
The prefix will be used to parse a "query to select book" in different context.
For now we have only one context : selecting books for the catalog search.
But we will want to select books to do fulltext search on them
(will be done in later commit)
2022-06-02 12:22:28 +02:00
Matthieu Gautier 76ebfd7ea4 Move get_search_filter and subrange. 2022-06-02 12:22:27 +02:00
Matthieu Gautier 22996e4a6b Allow user to select multiple books when doing search. 2022-06-02 12:22:27 +02:00
Matthieu Gautier 98c54b2279 Handle multiple arguments in RequestContext. 2022-06-02 12:22:27 +02:00
Matthieu Gautier 854623618c Use the newly introduced searcherCache for multizim searcher. 2022-06-02 12:22:25 +02:00
Matthieu Gautier fd0edbba80 Use a set of id as key for a the searcher Cache.
It will allow use to cache seacher for multiple zim files.
2022-05-24 14:55:48 +02:00
Matthieu Gautier f5af0633ec Move the searcher cache into the Library 2022-05-24 14:55:48 +02:00
Matthieu Gautier 740581c55c Link the cache size to the book count.
Unless explicitly set via user env variable.
2022-05-24 14:55:48 +02:00
Matthieu Gautier 582e3ec46d Use a concurrent cache to store Archive cache. 2022-05-24 14:55:48 +02:00
Matthieu Gautier 28fb76bbc2 Remove m_readers in `Library::impl`
It is a deprecated interface and it is a simple wrapper on Archive.
2022-05-24 14:55:48 +02:00
Matthieu Gautier 7c688a4acc Move `getCacheLength` to a generic helper function `getEnvVar` 2022-05-24 14:55:48 +02:00
Matthieu Gautier 66b2449800 Remove unnecessary catch
Catch of std::exception is already made in `handle_request`
2022-05-23 19:17:28 +02:00
Matthieu Gautier aad95e3413 Introduce a results intermediate object in the template rendering.
Url in href must not be html encoded. As we already url encode the path, it
is ok to have `'` in the url.
2022-05-23 19:16:14 +02:00
Matthieu Gautier f0dd34b6db Introduce buildQueryData helper in SearchRenderer 2022-05-23 19:13:25 +02:00
Matthieu Gautier bbdde93f49 Introduce a pagination object to render search result. 2022-05-23 19:12:17 +02:00
Matthieu Gautier cb62da65c3 Raise a exception if something went wrong in the template rendering. 2022-05-23 10:56:39 +02:00
Matthieu Gautier 288b4ae7df Fix count of remote books in `Library::Impl::getBookCount` 2022-05-23 10:56:39 +02:00
Matthieu Gautier 52c12b0c2f Introduce `Library::Impl::getBookCount`
We simply introduce a `getBookCount` which is not protected by a lock.
2022-05-23 10:56:39 +02:00
Matthieu Gautier 4695f47dd2 Introduce operator+= to simplify response creation. 2022-05-23 10:56:39 +02:00
Matthieu Gautier f42f6a60df Use extractFromString to parse request argument.
On top of reusing code, it throw a exception if we cannot convert given
argument in the type we want.
2022-05-23 10:56:39 +02:00
Matthieu Gautier 717c39f2ef Better ExtractFromString
- Throw a exception if we cannot extract from string.
  (We throw the same exception as `std::sto*`)
- Add a specialization to extract string from string
- Add some unit test
2022-05-23 10:56:39 +02:00
Matthieu Gautier aa1f73472d Remove unecessary BookDB helper class.
It was needed to not expose Xapian in public header.
Now we can remove it and directly use a Xapian db.
2022-05-23 10:56:39 +02:00
Matthieu Gautier 090c2fd31a Move LibraryBase out of public API.
We use composition instead of inheritance to implement Library.
2022-05-23 10:56:39 +02:00
Veloman Yunkan 84c68d4d7b Search results pagination bugfix
Search results pagination is disabled for a single page outcome too.
2022-05-18 12:45:47 +04:00
Veloman Yunkan 3b9f28b2b5 Applied cache-id to search_results.css
The story of search_results.css

static/skin/search_results.css was extracted from
static/templates/no_search_result.html before the latter was dropped.

static/templates/no_search_result.html in turn seems to be a copied and
edited version of static/templates/search_result.html.

In the context of exploratory work on the internationalization of
kiwix-serve (PR #679) I noticed duplication of inline CSS across those
two templates and intended to eliminated it. That goal was not fully
accomplished (static/templates/search_result.html remained untouched)
because by that time PR #679 grew too big and the efforts were diverted
into splitting it into smaller ones. Thus search_results.css slipped
into one of those small PRs, without making much sense because nothing
really justifies preserving custom CSS in the "Fulltext search unavailable"
error page.

At the same time, it served as the only case where a link to a cacheable
resource is generated in C++ code (rather than found in a template).
This poses certain problems to the handling of cache-ids. A workaround
is to expel the URL into a template so that it is processed by
`kiwix-resources`. This commit merely demonstrates that solution. But
whether it should be preserved (or rather the "Fulltext search
unavailable" page should be deprived of CSS) is questionable.
2022-05-02 20:37:22 +04:00
Matthieu Gautier fba0f09f4f Do not compress content smaller than 1400 Bytes 2022-04-27 18:23:39 +02:00
Matthieu Gautier 0d294c50a5 [SERVER] Support gzip encoding instead of deflate.
The `compress` function is copied from httplib
2022-04-27 18:23:38 +02:00
Veloman Yunkan 927c12574a Preliminary support for Accept-Language: header
In the absence of the "userlang" query parameter in the URL, the value
of the "Accept-Language" header is used. However, it is assumed that
"Accept-Language" specifies a single language (rather than a comma
separated list of languages possibly weighted with quality values).

Example:

Accept-Language: fr
// should work

Accept-Language: fr-CH, fr;q=0.9, en;q=0.8, de;q=0.7, *;q=0.5
// The requested language will be considered to be
// "fr-CH, fr;q=0.9, en;q=0.8, de;q=0.7, *;q=0.5".
// The i18n code will fail to find resources for such a language
// and will use the default "en" instead.
2022-04-13 16:40:20 +02:00
Veloman Yunkan 9987fbd488 Fixed CI build failure under android_arm* 2022-04-13 16:40:20 +02:00
Veloman Yunkan a0d9a824e1 Internationalized searchbox tooltip 2022-04-13 16:40:20 +02:00
Veloman Yunkan 11be821c46 Internationalized "Go to a randomly selected page"
At this point a potential issue has been revealed. Now we produce
the final HTML via 2-level template expansion

1. Render parameterized messages
2. Render the HTML template

In which templates we should use double mustache "{{}}" (HTML-escaping)
tags and where we may use triple mustache "{{{}}}" (non-escaping) tags?
2022-04-13 16:40:20 +02:00
Veloman Yunkan 3da81a3d0f Internationalized "Go to the main page" button 2022-04-13 16:40:20 +02:00
Veloman Yunkan f73be3cde7 Initializing mustache data via initializer list 2022-04-13 16:40:20 +02:00
Veloman Yunkan c2bfeb4030 "Go to welcome page" is internationalized 2022-04-13 16:40:20 +02:00
Veloman Yunkan 6f3db20078 Internationalized "Fulltext search unavailable" page 2022-04-13 16:40:20 +02:00
Veloman Yunkan fbd23a8329 Fully internationalized 400, 404 & 500 error pages 2022-04-13 16:40:20 +02:00
Veloman Yunkan d2c864b010 Internationalized raw-entry-not-found message 2022-04-13 16:40:20 +02:00
Veloman Yunkan 779382642b Internationalized bad raw access datatype message 2022-04-13 16:40:20 +02:00
Veloman Yunkan ca7e0fb4a0 Internationalized random article failure message 2022-04-13 16:40:20 +02:00