Commit Graph

1339 Commits

Author SHA1 Message Date
Matthieu Gautier ead1474ead Make `SearchRendered` taking a const pointer. 2023-10-05 16:37:22 +02:00
Matthieu Gautier 1316dec37c Make the `Server` keep a `shared_ptr` instead of a raw NameMapper pointer.
Same as for `Library`, we want to be sure that the `NameMapper`
actually exists when the server is using it.
2023-10-05 16:37:22 +02:00
Matthieu Gautier a5557eeb25 Make the `Server` keep a `shared_ptr` instead of a raw Library pointer.
We want to be sure that `Library` actually exists when we use it.
While it is not a silver bullet (user can still create a shared_ptr on
a raw pointer), making the `Server` keep `shared_ptr` on the library
help us a lot here.
2023-10-05 16:36:18 +02:00
Matthieu Gautier efcbf6ef1e Make the `UpdatableNameMapper` keep a `shared_ptr`.
Same as `Manager`, we want to be sure that `Library` actually exists
when we use it.
2023-09-25 16:31:55 +02:00
Matthieu Gautier 139b561253 Make the `Manager` keep a `shared_ptr` instead of a raw Library reference.
We want to be sure that `Library` actually exists when we modify it.
While it is not a silver bullet (user can still create a shared_ptr on
a raw pointer), making the `Manager` keep `shared_ptr` on the library
help us a lot here.
2023-09-25 16:30:56 +02:00
Matthieu Gautier 49e99e7c22 Remove dumpers from the public API.
All those dumper were not used by any of our other projects.
They are only used internally, either by `Library::writeToFile` or the
server.
2023-09-19 16:46:58 +02:00
Veloman Yunkan f8c3a1fd2e Added default user language to viewer_settings.js
The default user language determined from the value of "Accept-Language"
header is communicated to the client via the /viewer_settings.js
endpoint.
2023-09-09 19:37:49 +04:00
Veloman Yunkan b5b98e7a61 RIP userlang cookie
This commit drops the usage of the userlang cookie in the backend but
not in the frontend. UI language control should be broken at this point
and will be fixed in the next few commits.
2023-09-09 19:37:49 +04:00
Nikhil Tanwar c938101c70 Allow multiple category support
Created a generic function multipleQuery which takes:
1. string (representing a comma separated list)
2. param (the value to query on using Xapian).

Category and language query will use this function.
2023-07-26 18:15:45 +05:30
Nikhil Tanwar 385931f229 Move getLanguageSelfName to tools.h
This is a general utility which other ports can get use of.
Added tests
2023-07-26 16:02:32 +05:30
Nikhil Tanwar 94d6bef402 Introduce readCategoriesFromFeed()
Added a function to load categories stored in an OPDS stream
2023-07-26 16:02:32 +05:30
Nikhil Tanwar a28c2973e9 Introduce readLanguagesFromFeed()
Added a new function to read languages stored in an OPDS feed
2023-07-26 16:02:32 +05:30
Nikhil Tanwar 5c3a997de4 make beautifyFileSize public
This general function will be useful in other kiwix apps
2023-07-21 22:19:45 +05:30
Veloman Yunkan e9d9d85427 Added Gungbe language code 2023-07-10 23:28:04 +04:00
Veloman Yunkan f41007989b Fool-proof checking of book illustration presence
Now (in a library.xml flow) a book is considered to contain an
illustration only if both "faviconMimeType" and "favicon" attributes
are set to non-empty values.
2023-06-24 20:10:41 +04:00
Anthony J. Bentley df164aefe5 Include netinet/in.h everywhere except Windows
According to POSIX, sockaddr_in is declared in netinet/in.h.
Some POSIX systems (notably OpenBSD and FreeBSD) declare it in
only this header, so including it is required. Others, like Linux,
are are more lax in exposing symbols to the namespace, providing
sockaddr_in via additional headers, but it does no harm to include
the standard header on such systems.
2023-05-27 12:42:29 +03:00
Veloman Yunkan dc58e278c7 git mv src/server/internalServer_catalog{_v2,}.cpp 2023-04-25 12:48:49 +04:00
Veloman Yunkan 9994302312 Explicit charset in OPDS response MIME types 2023-04-25 12:48:29 +04:00
Veloman Yunkan 8c190cf34f Moved InternalServer::handle_catalog() 2023-04-25 12:48:10 +04:00
Veloman Yunkan 1273570e01 Deduplication of OPDS MIME type strings 2023-04-25 12:47:02 +04:00
Veloman Yunkan 6f0e55d603 A slight simplification of the mustache template
Got rid of the partial vs full entries logic in the mustache template -
now it is entirely contained in `OPDSDumper::dumpOPDSFeedV2()`.
2023-04-18 14:45:51 +04:00
Veloman Yunkan 9fd7f7da34 Really enable Chromium to display PDFs in the viewer iframe
The previous "fix" (merged PR #924) was buggy. It not only didn't work
in Chromium v90, but in more recent versions too.

I verified that this fix works in Firefox (v111) and Chromium (v90):

- Attempts by the ZIM content to break out of the viewer iframe are
blocked.
- PDFs are displayed in the viewer.
2023-04-10 16:42:21 +04:00
Veloman Yunkan a6659cbe96 Enable Chromium to display PDFs in the viewer iframe
This fix requires Chromium version above 90.
2023-04-05 15:08:29 +02:00
Nikhil Tanwar 951e15c665 No results display in /nojs
Shows a link to reset filter if there are no books.
2023-03-29 19:03:30 +05:30
Nikhil Tanwar cc35fe503f Translations for /nojs endpoint
Uses the string from #846 for translations.
A couple new translations are also added for <title> tag.
2023-03-29 19:03:29 +05:30
Nikhil Tanwar 37aadb86fb language/category filtering in /nojs endpoint
Adds language and category filter in /nojs.
Unlike the main page, the filtering is only done after user submits the form.
2023-03-29 19:02:58 +05:30
Nikhil Tanwar f843ea48f0 Add Results label
Shows "x results" label where x = number of books based on filters
2023-03-29 19:02:58 +05:30
Nikhil Tanwar a48e2e6f06 Add search form for /nojs endpoint
Adds an html form to search books by the q= parameter
2023-03-29 19:02:58 +05:30
Nikhil Tanwar 0f7e11bd86 Add download-links to tiles in /nojs
The download-link links to /nojs/download/<bookname> for all 4 types of downloads.
2023-03-29 19:02:56 +05:30
Nikhil Tanwar dbded6eee2 Add links to content for tiles in /nojs
If the tiles are now clicked, they redirect to main page of book.
2023-03-28 21:50:47 +05:30
Nikhil Tanwar c1d7cc37fd Add tags in tiles for /nojs endpoint
Adds span elements for tags
2023-03-28 21:49:31 +05:30
Nikhil Tanwar 6071b98fb7 Import book tiles
Tries to copy the same design of tiles as main page with javascript enabled
2023-03-28 21:49:31 +05:30
Nikhil Tanwar dca47d35f7 Introduce /nojs endpoint
Adds /nojs endpoint for fallback.
Currently, it serves an HTML with book names in library
2023-03-28 20:25:44 +05:30
Nikhil Tanwar d8656ec149 Introduce HTMLDumper
HTMLDumper class will be used to dump library in HTML format. It inherits from LibraryDumper
2023-03-28 20:25:44 +05:30
Nikhil Tanwar f1873876b2 Extract LibraryDumper from OPDSDumper
This change creates a new common class for dumping the library into various formats: LibraryDumper
2023-03-28 20:25:44 +05:30
Veloman Yunkan eb002ae306 Deprecated Book::getLanguage()
Introduced `Book::getCommaSeparatedLanguages()` instead.
2023-03-08 15:24:53 +01:00
Veloman Yunkan 2550306052 One more usage of Book::getLanguages()
`Book::getLanguages()` is used instead of `Book::getLanguage()` when
determining the set of languages for a collection of books.
2023-03-08 15:24:53 +01:00
Veloman Yunkan 51fcb90dc0 Library::updateBookDB() uses Book::getLanguages() 2023-03-08 15:24:53 +01:00
Veloman Yunkan b1ad319d52 Enter Book::getLanguages() 2023-03-08 15:24:53 +01:00
Veloman Yunkan 5bda7fd45c Support for multilang ZIMs 2023-03-08 15:24:53 +01:00
Veloman Yunkan ac742e9da2 Redirection of slashless root URL
With non-empty root location, the canonic form of the root URL for a
kiwix server is now required to end with a slash (to match the situation
for an empty root location). This requirement enables usage of relative
URLs on the welcome page and resources/scripts loaded through that page.

A slashless root URL is redirected to the slashful version.
2023-02-22 17:54:20 +04:00
Veloman Yunkan 2e0124710a `?count=0` OPDS catalog queries return 0 results
... which is a useful way of finding out the total number of results
with the least consumption of resources.
2023-02-10 19:15:29 +01:00
Veloman Yunkan e8c8a297b5 Registered MIME-types for .ico and .json
As a result, favicon.ico stopped being considered a compressible resource.
2023-02-10 15:07:00 +01:00
Nikhil Tanwar 12f0614350 Add Serbo-croate language name
Adds "srpskohrvatski" as name for "hbs" language tag.
2023-02-10 09:20:23 +05:30
Veloman Yunkan c2fffacbbd Renamed a data member 2023-02-09 10:40:23 +01:00
Veloman Yunkan 02f631fdb6 Got rid of RequestContext::full_url 2023-02-09 10:40:23 +01:00
Veloman Yunkan 05a66ead6e URI-encoding of the root location part
Now the root location is URI-encoded too.

In order to properly test this change the root location in the tests was
changed from "/ROOT" to "/ROOT#?" (or "/ROOT%23%3F" in URI-encoded form),
which is why this commit is so big.
2023-02-09 10:40:07 +01:00
Veloman Yunkan 97f0314fe6 Saving a few CPU cycles
This silly optimization in fact helps to avoid a somewhat more serious
waste of CPU cycles that would otherwise result in the next commit.
2023-02-08 22:16:27 +01:00
Veloman Yunkan a7fe4193e3 Preparing to save a few CPU cycles 2023-02-08 22:16:27 +01:00
Veloman Yunkan 2c5e84b6b3 Simpler fullURL2LocalURL() 2023-02-08 22:16:27 +01:00
Veloman Yunkan 71a66e0528 Passing of unrooted URL into RequestContext()
This change doesn't make much sense on its own - the real goal is to
prepare some ground for easier implementation of URI-encoding of the root
location.
2023-02-08 22:16:27 +01:00
Veloman Yunkan a807ce27f1 URI-encoding when redirecting legacy URLs to /content
Testing of this functionality revealed that the query part containing +
symbols (as replacement for spaces in the parameter values) isn't
forwarded properly as the + symbols are URI-encoded (this is a bug on
the part of the `RequestContext::get_query()` the result of which
already contains URI-encoded +'s).
2023-02-08 22:16:27 +01:00
Veloman Yunkan 2e9bec95b0 Proper URI-encoding in InternalServer::build_redirect()
- Before this change `InternalServer::build_redirect()` only URI-encoded the
  article path, ignoring the book name and/or the root location components of
  the URL.

- In order to be able to test this fix, corner_cases.zim was renamed to
  contain a couple of special URL symbols in its filename. The
  `create_corner_cases_zim_file` script was updated accordingly.
2023-02-08 22:16:09 +01:00
Matthieu Gautier 1ba588272c Get `Waiting` downloads before `Active` ones.
`Waiting` can become `Active` while we are getting the downloads.
We may have rare case where we miss a download if we get `Active` before
`Waiting`.
2023-02-08 15:42:17 +01:00
Matthieu Gautier 2c3b7409aa Remove the default value of follow parameter in `updateStatus`.
`false` is a pretty bad default value as most user want to track
the real download.

By removing the default value, we force user to make a choice.
We could have change the default value to true but it would have been
a silent API change and we don't want that.
2023-02-08 15:42:17 +01:00
Matthieu Gautier 18b7b5f277 Mark constant methods as const. 2023-02-08 15:42:17 +01:00
Matthieu Gautier 0e612de4d1 Make `Downloader` return shared_ptr instead of raw pointer.
This is dangerous by nature to return raw pointer on internal data.
2023-02-08 15:42:17 +01:00
Matthieu Gautier 52ae5c3a5f Make Downloader thread safe. 2023-02-08 15:42:17 +01:00
Matthieu Gautier d1fe1b89ae Do not automatically update the status of existing Download.
User may already have a pointer to the `Download` and it is not protected
against concurrent access.

We could update the status of new created `Download` as by definition,
no one have a pointer on it.
But it better to not do it neither :
- For consistency
- Because the first call on update status may be long on windows (because
  of file preallocation). It is better to not block the downloader for that.
2023-02-08 15:42:17 +01:00
Matthieu Gautier 1aa8521e15 Remove the lock.
As we now build a new request handle for every request, we don't need
a lock.

libcurl itself is thread safe as long as we don't share a handle.
2023-02-08 15:42:17 +01:00
Matthieu Gautier 95ebb6a492 Build a new curl "handle" at everyrequest instead of reusing the same one. 2023-02-08 15:42:17 +01:00
Veloman Yunkan ca079a72cc Some clean-up 2023-01-25 19:15:12 +04:00
Veloman Yunkan 471c5b89f4 Dropped the 2nd param of urlEncode()
`urlEncode(str)` is now equivalent to the previous `urlEncode(str, true)`.
2023-01-25 19:15:12 +04:00
Veloman Yunkan 3bf8211b70 Made 2nd param of urlEncode() mandatory
This is a precautionary step before dropping the said parameter.
2023-01-25 19:15:12 +04:00
Veloman Yunkan ec81d5904d Proper URI-encoding in kiwix::getSearchUrl() 2023-01-25 19:15:12 +04:00
Veloman Yunkan 63e0d5c7c2 RequestContext::get_query() is fully URI-encoded 2023-01-25 19:15:12 +04:00
Veloman Yunkan 772243e832 Category name is fully URI-encoded 2023-01-25 19:15:12 +04:00
Veloman Yunkan bad13d76b4 Removed unused code 2023-01-25 19:15:12 +04:00
Veloman Yunkan 0bde4d9412 Properly URI-encoded links in search results
Special URI symbols occurring in the item path part of the search result
link were NOT encoded, because that would also encode the path separator (/)
symbol. Now that `urlEncode()` never encodes the / symbol, it is safe to
encode all other URI-special symbols in the path.
2023-01-25 19:15:12 +04:00
Veloman Yunkan 239b108fa7 / is no longer a reserved char for urlEncode()
This change is a quick hack solving known issues with URI-encoding in
libkiwix.

This change removes the slash character from the list of URL separator
symbols in URL encoding/decoding utilities, and makes it a symbol that
is safe to leave unencoded.

Effects:

- `urlEncode()` never encodes the '/' symbol (even when it is requested
  to encode the URL separator symbols too).

- `urlDecode(str)`/`urlDecode(..., false)` will now decode %2F to '/';
  other encoded URL separator symbols are NOT decoded when the second
  argument of `urlDecode()` is set to false (which is the default).
2023-01-25 19:15:12 +04:00
Veloman Yunkan c5ccbd37e2 Extracted isHarmlessUriChar() 2023-01-25 19:15:12 +04:00
Veloman Yunkan aa2e443eb8 Fixed indentation
Replaced tabs with spaces.
2023-01-25 19:15:11 +04:00
Veloman Yunkan 82d477009d '#' is a URI delimiter symbol 2023-01-25 19:15:11 +04:00
Veloman Yunkan e49081da80 Fixed urlEncode() for chars below 0x10 2023-01-25 19:15:11 +04:00
Veloman Yunkan e35e7585e0 Server sets userlang cookie as global and permanent
Without specifying the "Path" attribute of the cookie in the "Set-Cookie" header
we end up with multiple instances of the cookie for different URLs. We
want a single "global" cookie for kiwix-serve. Besides we want it to be
"permanent" rather than a session cookie, hence the large (1-year-long)
TTL value for the "Max-Age" attribute.
2023-01-24 19:01:32 +01:00
Veloman Yunkan fcb97c3c06 Sparing use of "Set-Cookie: userlang=..." header
Server adds the "Set-Cookie: userlang=..." header to the response only
if the "userlang" cookie is not already present with the same value.
2023-01-24 19:01:32 +01:00
Veloman Yunkan 8eb527389e URI-encoding of redirections to URLs with special symbols 2023-01-10 17:41:59 +04:00
Veloman Yunkan 28e9fb48b6 Properly implemented parseUserLanguagePreferences() 2022-12-14 15:34:46 +01:00
Veloman Yunkan 634f3fcf14 Properly implemented selectMostSuitableLanguage() 2022-12-14 15:34:46 +01:00
Veloman Yunkan 88597e1834 Enter selectMostSuitableLanguage() 2022-12-14 15:34:46 +01:00
Veloman Yunkan 69b3e1f8a7 Moved user language preferences into i18n.{h,cpp} 2022-12-14 15:34:46 +01:00
Veloman Yunkan 669d8898ac Enter UserLangPreferences 2022-12-14 15:34:46 +01:00
Veloman Yunkan 14f0f79061 User language control via userlang cookie 2022-12-14 15:34:46 +01:00
Veloman Yunkan 1d74b5e311 Server sets the userlang cookie on every response 2022-12-14 15:34:46 +01:00
Emmanuel Engelhart 2d42d6dc60
Gzip compress HTTP response for Web fonts 2022-12-07 19:21:27 +01:00
Veloman Yunkan 4966f4155d Fixed handling of backslashes in suggestions 2022-11-17 11:51:53 +04:00
Veloman Yunkan 0f0ae1cfed A small refactoring 2022-11-17 11:51:53 +04:00
Veloman Yunkan da78aae62b kiwix::Suggestions gives up its temporary pedigree 2022-11-17 11:51:53 +04:00
Veloman Yunkan abcd4ade99 kiwix::Suggestions::getJSON() 2022-11-17 11:51:53 +04:00
Veloman Yunkan 7a9780eb90 kiwix::Suggestions::addFTSearchSuggestion() 2022-11-17 11:51:53 +04:00
Veloman Yunkan 51bd881211 kiwix::Suggestions::add() 2022-11-17 11:51:53 +04:00
Veloman Yunkan f36f1661d5 Got rid of result count tracker variable 2022-11-17 11:51:53 +04:00
Veloman Yunkan 18f4a58237 Conception of kiwix::Suggestions 2022-11-17 11:51:53 +04:00
Matthieu Gautier 8cec014691 Use new `zim::Archive::getMediaCount` from libzim.
As libzim also changed the behavior of `zim::Archive::getArticleCount`,
we don't need the hack, and we don't need the code to parse `M/Counter`.
2022-11-02 13:15:47 +01:00
Veloman Yunkan 7d69ece27d OPDS can be filtered using more than one language
From now on, the `lang` parameter of the /catalog/search,
/catalog/v2/entries, and /catalog/v2/partial_entries endpoints is
interpreted as a comma-separated list of languages.
2022-11-01 19:16:30 +01:00
Veloman Yunkan c87add1419 Removed an unused variable 2022-11-01 19:16:30 +01:00
Veloman Yunkan cb02dbd92a RequestContext preserves the exact query string
Before this change RequestContext::get_query() returned a reordered
query string (alphabetically sorted by the parameter names).

This fix facilitiates testing of responses where the request URL appears
in the response.
2022-10-31 13:28:21 +04:00
Veloman Yunkan 9409e8bd91 Preventing confusion of tongues in multizim search
Multizim search requires that all selected books be in the same
language.

No new URL query parameter was introduced for specifying the intended
search language - `books.filter.lang` can be used for that purpose.

The server_search unit-test was updated to use a slightly cheating
library xml file where the language of example.zim was tweaked from "en"
to "eng" in order to match that of zimfile.zim. Note that this change
drops from the tested server two other goofy ZIM files corner_cases.zim
and poor.zim that have been/are included in ServerTest.
2022-10-31 13:27:57 +04:00
Veloman Yunkan cd62b5dd91 Some clean-up 2022-10-31 13:22:15 +04:00
Veloman Yunkan 414d7ae4fe Fixed indentation 2022-10-31 13:22:15 +04:00