smallsolar
/
libkiwix
mirror of https://github.com/kiwix/libkiwix.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
1,509 Commits 23 Branches 58 Tags 131 MiB
Commit Graph

869 Commits

Author SHA1 Message Date
Veloman Yunkan e028bcbb04 Android's java.io.FileDescriptor is different 2021-02-10 14:19:47 +01:00
Veloman Yunkan 9cdf7a44c0 JNIKiwixReader can open an embedded ZIM archive 2021-02-10 14:19:47 +01:00
Veloman Yunkan 4d23e44de7 JNIKiwixReader ctor taking a file descriptor 
... and a corresponding unit test
 2021-02-10 14:19:47 +01:00
Veloman Yunkan 98d69ef59b Added testReader unit-test for the java wrapper 2021-02-10 14:19:47 +01:00
Veloman Yunkan e40827fbac Renamed the java wrapper unit test runner script 2021-02-10 14:19:47 +01:00
Veloman Yunkan a798e0c0a1 Made the java wrapper unit test run & pass 
The kiwixlib java wrapper unit test can be run manually via the
src/wrapper/java/org/kiwix/testing/compile_test.sh script.

The test ZIM files in src/wrapper/java/org/kiwix/testing were created
using the create_test_zimfiles. They must be updated/re-generated and
committed in git whenever their source data or the create_test_zimfiles
script changes. Note: small.zim.embedded is not used at this point, it
was created for testing the enhancement coming in a few commits.
 2021-02-10 14:19:47 +01:00
Matthieu Gautier 24b2e6e585 Remove unnecessary include. 2021-01-26 17:53:25 +01:00
Matthieu Gautier 3fd1310008 Use c++11 std::thread instead of pthread. 2021-01-26 17:53:25 +01:00
Matthieu Gautier 4749656828 Do not crash if zim file has no `Counter` metadata. 2021-01-26 15:15:27 +01:00
Emmanuel Engelhart 84895c4036 Better </head> detection regex 2021-01-18 13:16:56 +01:00
Emmanuel Engelhart a8bf9dd5b4 Better Kiwix Serve Taskbar insertion (after charset definition) 2021-01-18 11:18:53 +01:00
Emmanuel Engelhart a61c94ef10 Add GPLv3 header 2021-01-18 10:54:33 +01:00
Emmanuel Engelhart 8c43fd8d36 Fix taskbar insertion in case of '<head>' attributes 2021-01-11 14:37:19 +01:00
Emmanuel Engelhart 3e2810dff4 Support 'video/*' * 'audio/*' mimetypes in getMediaCount() 2021-01-07 12:32:32 +01:00
Emmanuel Engelhart 44c4aa931a Better use kiwix::startsWith() 2021-01-03 15:17:03 +01:00
Emmanuel Engelhart 95b32b168d More robust getMediaCount() 2021-01-01 17:05:32 +01:00
Matthieu Gautier 1002c15e0d Remove unnecessary checks. 
`Reader` cannot be created with a null `zimArchive`.
We don't have to check for zimArchive being not null.
 2020-12-09 14:25:02 +01:00
Matthieu Gautier d51000c4a9 Use new libzim method `hasFulltextIndex` to check for fulltext index. 2020-12-09 14:25:02 +01:00
Matthieu Gautier ba302bed33 Use new libzim method `getFaviconEntry` to get the favicon. 2020-12-09 14:25:02 +01:00
Steve Wills 6900b4e506 fix build on FreeBSD 
With this header, sockaddr_in and INADDR_ANY are not defined
 2020-12-07 09:38:46 -05:00
Matthieu Gautier 1a5a2e7a8e Adapt kiwix-lib to the new libzim api. 2020-12-02 12:16:48 +01:00
Matthieu Gautier d87079ec13 Remove deprecated method in the reader. 2020-11-24 19:00:52 +01:00
Veloman Yunkan 0f8fe1f63f Alternative implementation of parseMimetypeCounter() 2020-10-29 14:11:27 +04:00
Matthieu Gautier 08464f23bc Better parsing of `M/Counter` 
Mimetype may contain a parameters.
Then, the mimetype would be something like "text/html;foo=bar;foz=baz"

It will contains a `;` and `=` and it conflicts with the same operators
we use to separate the items in our list.

We have to use a more advanced algorithm which takes the context into
account.

Fix #416
 2020-10-28 16:03:18 +01:00
Matthieu Gautier ef42abea4b Add some tests of `parseMimetypeCounter` 2020-10-28 14:44:23 +01:00
Matthieu Gautier 4407dd12bd Move mimetypeCounter parsing in its own function. 2020-10-28 14:08:06 +01:00
Matthieu Gautier 632583ede2 Add missing include 2020-10-07 18:43:57 +02:00
Matthieu Gautier 61f9d4ab3a Stop the internal server only if it exists. 2020-10-07 14:36:45 +02:00
Matthieu Gautier 470bfc3f1f Better variable name for outStream. 2020-08-28 15:27:03 +02:00
Matthieu Gautier ea3180cb8c Better error printing. 2020-08-28 15:27:03 +02:00
Matthieu Gautier 72d3f8f8e2 Fix segmentation fault with curl requests. 
Use a heap allocated buffer (with lifetime of Aria2 class) instead of
a stack allocated one.

Original fix made by @ZaWertun. Kudos to him.

Fix #kiwix/kiwix-desktop#123, kiwix/kiwix-desktop#513
and kiwix/kiwix-desktop#423
 2020-08-26 12:42:16 +02:00
Matthieu Gautier af9e03904c Use std::mutex and std::unique_lock instead of pthread mutex/lock. 
It simplify a bit the code and ensure that mutex is correctly unlock
even in case of exception.
 2020-08-26 12:30:56 +02:00
Matthieu Gautier 39611cbd60 Wait for waitingThread to exit before destroying the subprocess memory. 
WaitingThread read some shared memory with the SubProcess
(`mutex`, `m_running`).
When we destroy the SubProcess, we must be sure that WaitingThread has
correctly finished else we may have invalid read/write on freed memory.
 2020-08-26 12:26:04 +02:00
Matthieu Gautier 6f0d3003ac Remove `m_compress` member. 2020-08-13 11:16:41 +02:00
Matthieu Gautier ee17b0739a Fix compilation on CI native dyn. 
On the CI, the native_dyn docker image is setup with a packaged version
on libmicrohttpd for which `MHD_HTTP_RANGE_NOT_SATISFIABLE` is not
defined.

When the CI will be fixed, we can revert this commit.
 2020-08-13 11:16:41 +02:00
Matthieu Gautier 47436f7bdd Move some header setting in response's constructors. 
It make easier to understand what is somehow constant and what depends
of the context.
 2020-08-13 11:16:41 +02:00
Matthieu Gautier 3352c95314 Remove the `RedirectResponse` and use a basic `Response` with header. 2020-08-13 11:16:41 +02:00
Matthieu Gautier 77123ac74c Move the adding of 304 headers in 304 factory. 
This avoid us to create a ContentResponse just to have some correct
headers.
 2020-08-13 11:16:41 +02:00
Matthieu Gautier 9078f0ac6e Remove `ResponseMode`. 2020-08-13 11:16:41 +02:00
Matthieu Gautier 8d6567d067 Create a utility builder for 416 response. 
Also add a map in the response to store specific headers.
 2020-08-13 11:16:41 +02:00
Matthieu Gautier 6d5cddca12 Fix android compilation 
Android clang complains about the fact it cannot move the
`std::unique_ptr<ContentResponse>` into a `std::unique_ptr<Response>&&`
(for the implicit `std::unique_ptr<Response>` constructor).
Let's help him a bit.
 2020-08-13 11:16:41 +02:00
Matthieu Gautier a3939e9a05 Move all the content code in the ContentResponse. 2020-08-13 11:16:41 +02:00
Matthieu Gautier eee621d15b Move small utilities method to create response in Response class. 2020-08-13 11:16:41 +02:00
Matthieu Gautier 7b2ee37437 Move the entry response to its own class. 2020-08-13 11:16:41 +02:00
Matthieu Gautier f014fb2895 Introduce a ContentResponse. 
This is only an "interface" for now as other type of response (entry) may
be "transformed" to a ContentResponse.
We cannot move all the code in the class.
 2020-08-13 11:16:41 +02:00
Matthieu Gautier 1011d1ff0b Move the redirection response in its own class. 
The redirection is the easiest to move, let's start with this one.
 2020-08-13 11:16:41 +02:00
Matthieu Gautier 9e351b279e Remove `get_default_response` in favor of a static Response method. 
We want to build different kind of response depending of the context.
 2020-08-13 11:16:41 +02:00
Matthieu Gautier a0bdc0821c Move internalServer code into its own source files. 2020-08-13 11:16:41 +02:00
Matthieu Gautier a819d9e3e0 Make the server handle pointer to response instead of plain response. 
This is a preparatory work.
We will specialize the response and so we need a pointer to response
instead of plain response.
 2020-08-13 11:16:41 +02:00
manan jethwani c74b935a9b added pageLength for search_pagination 2020-08-12 02:08:02 +05:30
Matthieu Gautier a55d504017 Fix getArticleCount. 
With #403, the article mimetype may be different than "text/html".
It can also be "text/html; raw=true".
(And in fact it already could have any kind of optional argument).
 2020-08-11 18:27:54 +02:00
Matthieu Gautier 87b5adcaf4 Make the response responsible to detect if we must introduce taskbar. 
The response detect if taskbar must be added depending of the mimetype.

Now, `set_taskbar` can be call unconditionally
(no need to check for the mimetype)

And we don't need to call set_taskbar if we have no information to set.
 2020-08-11 18:27:54 +02:00
Veloman Yunkan c4e6313c90 x in a --> a.contains(x) in meson.build files 2020-08-11 18:17:18 +02:00
renaud gaudin 3f25a3d005 Fixed #391: prevent taskbar and blocker at article level 
Some HTML articles are meant to be displayed through a viewer. In this case,
we know we don't want the server to inject the taskbar nor the link blocker
because the content is not a user-ready web page but a partial element of it.

Such articles still need to be `text/html` to be parsed properly by browsers.

This changes the way we decide to display the tasbar or not.
Previously, we were adding it to every article with a MIME __starting with__ `text/html`.
Now, we're additionally preventing it on `text/html` MIME if there is a `;raw=true` string inside.

This leaves articles with MIME `text/html;raw=true` (warc2zim convention) outside
of the taskbar target.

For similar reasons, the external-link blocker is set to apply to the same set of articles.
Previously, it was applied to all articles which was an (unoticable) mistake.
 2020-08-07 09:26:24 +02:00
MananJethwani 599aaa4c1b added code for status code 204 for empty return of search. 2020-08-01 01:45:42 +05:30
Veloman Yunkan 3d425f44de Request header case is ignored 
Originally reported against case sensitivity of the Range header
(see issue #387), this fix applies to all request headers (since
according to RFC 7230 all header fields are case-insensitive, see
https://tools.ietf.org/html/rfc7230#section-3.2). However, a
corresponding unit-test was added only for the Range header.
 2020-07-30 16:01:51 +02:00
Matthieu Gautier 7ece383004 Add support for samba path on windows. 
Fix kiwix/kiwix-desktop#429
 2020-07-15 11:40:02 +02:00
Kelson cf8e8b94eb Fix compilation with libmicrohttpd v0.97.1 2020-07-08 14:42:46 +02:00
Matthieu Gautier 4d307e18eb Add new thread safe suggestion API. 
Previous API were using an internal vector to store the suggestions search
results.

The new API takes a vector as out argument. So user can call the functions
without having to protect the search.

We should change the android API to reflect the change but it is a bit
more complex to do at JNI level. As android do not call it multithreaded
we are safe for now. And we need the new API asap for kiwix-desktop.

So we keep the same API on android for now, the new api will be made
in next version.
 2020-07-01 17:16:13 +02:00
Kunal Mehta fb79cde729 Pass -latomic for architectures that need it 
Some architectures, specifically armel, mipsel, m68k & powerpc in
Debian, need to explicitly link to atomic.

Use meson to see if the target's CPU family is one of those, and if so,
pass -latomic to the linker.

Tested on armel and mipsel machines to verify passing -latomic works, and
on armhf and amd64 to ensure normal builds aren't broken.

Fixes #371.
 2020-06-29 00:18:13 -07:00
Matthieu Gautier ff605873ed Include missing `algorithm` header. 
`min` and `max` functions are defined here.
 2020-06-10 15:27:51 +02:00
Veloman Yunkan 05ef5d5f51
Assertion in ByteRange allows 0-sized content 
The assertion in the ByteRange constructor was written under the assumption that the content must have non-zero size. Now it allows that corner case.
 2020-06-02 21:53:47 +04:00
Veloman Yunkan f52b220d01 Dropped RequestContext::has_range() 2020-05-26 14:10:26 +04:00
Veloman Yunkan 50a850f3a9 Fixed a comment 2020-05-26 14:04:18 +04:00
Veloman Yunkan 886ae17274 Fixed a CodeFactor issue 2020-05-26 13:59:47 +04:00
Veloman Yunkan 85d6daabac Rolled back minor unneeded changes 2020-05-26 13:10:50 +04:00
Veloman Yunkan 5f1918d005 Split a long line 2020-05-26 13:04:03 +04:00
Veloman Yunkan 16bd79fa1b Final clean-up of byte_range.{h,cpp} 2020-05-26 12:50:08 +04:00
Veloman Yunkan c2ebdefe8d Handling of unsatisfiable ranges 2020-05-26 02:11:26 +04:00
Veloman Yunkan 37032892a4 Fixed compilation error under win32_* 
ERROR is a macro under Windows
 2020-05-26 01:58:17 +04:00
Veloman Yunkan 6b43438b74 Fixed compilation error under native_dyn 
MHD_HTTP_RANGE_NOT_SATISFIABLE is not defined in the older version of
libmicrohttpd (that is used under CI/Linux native_dyn).
 2020-05-26 01:54:36 +04:00
Veloman Yunkan 7301bf89bb Some refactoring of byte-range parsing 2020-05-26 01:50:29 +04:00
Veloman Yunkan ff23b28e7c Removed unnecessary qualifier 2020-05-26 01:41:37 +04:00
Veloman Yunkan 931e95f391 Invalid byte ranges result in 416 responses 2020-05-26 01:40:07 +04:00
Veloman Yunkan f7571b5b69 Content-Range header is set only for partial content 2020-05-25 17:42:18 +04:00
Veloman Yunkan 801ad18a89 ByteRange::resolve() 2020-05-25 17:27:35 +04:00
Veloman Yunkan 67a347c0c4 Moved byte-range parsing to byte_range.cpp 2020-05-25 17:21:10 +04:00
Veloman Yunkan 693905eb68 Default constructed ByteRange is a full range 2020-05-25 17:17:56 +04:00
Veloman Yunkan f3e79c6b4c Introduced src/server/byte_range.cpp 2020-05-25 16:43:44 +04:00
Veloman Yunkan 52f207eaa6 Support for single-ended byte ranges 2020-05-25 16:37:01 +04:00
Veloman Yunkan 67294217a8 ByteRange::Kind 2020-05-25 16:23:44 +04:00
Veloman Yunkan d111a40ce8 Response::m_byteRange 2020-05-23 20:35:22 +04:00
Veloman Yunkan 0c5bb3fcfe Moved ByteRange to a header file of its own 2020-05-23 20:08:53 +04:00
Veloman Yunkan 3fba8c20a0 Converted RequestContext::ByteRange to a class 
Also renamed the `range_pair` data member of `RequestContext` to `byteRange_`
 2020-05-23 19:59:47 +04:00
Veloman Yunkan 54db6049b7 Byte-range parsing not exposed in the header file 2020-05-23 18:58:19 +04:00
Veloman Yunkan 81c38d6b2b parse_byte_range() without side-effects 2020-05-23 18:53:16 +04:00
Veloman Yunkan e6a86c02ae Got rid of RequestContext::accept_range 2020-05-23 17:15:42 +04:00
Veloman Yunkan a0f7f32570 Re-ordered function definitions 2020-05-23 17:11:26 +04:00
Veloman Yunkan c39fce8839 RequestContext::parse_byte_range() 2020-05-23 17:09:51 +04:00
Veloman Yunkan de37489c53 Range header starts with a unit spec 
After this commit valid ranges of the form "bytes=firstbyte-lastbyte" should
be handled correctly.
 2020-05-22 17:17:31 +04:00
Veloman Yunkan 2a35a86de6 Fixed the size value used creating a response 
In case of a partial response the size of the response is different
from the served entry size.
 2020-05-22 16:49:35 +04:00
Veloman Yunkan 0a30a77c08 Handling of out of bound byte ranges 2020-05-22 16:46:38 +04:00
Veloman Yunkan 1a99bacfe3 Byte ranges are inclusive 
The second component of a byte range, if present, designates the
index of the last byte to be included in the partial response.
 2020-05-22 16:30:43 +04:00
Kelson 94c2ab4395 Add two OPDS related mime-types to compress for HTTP 2020-05-18 08:19:51 +02:00
Kelson 0f07cab920 Small HTTP header beautification 2020-05-17 20:19:19 +02:00
Veloman Yunkan 5f0a9d0b08 Added a comment clarifying a non-obvious case 2020-05-15 15:17:04 +04:00
Veloman Yunkan 54f5dbbd35 Handling of If-None-Match conditional requests 2020-05-14 17:01:22 +04:00
Veloman Yunkan 95a5cde359 ETags are set in the response as needed 
Also added server-unit tests related to ETags in the response.
 2020-05-14 17:01:22 +04:00
Veloman Yunkan 3d08ef43f2 HEAD request is not rejected 
libmicrohttpd handles HEAD requests by dropping the body of the response
(if any). Hence letting a HEAD request through into the code that
processes GET requests is safe.

Also added server unit-tests related to the handling of HEAD requests.
 2020-05-14 17:01:22 +04:00
Veloman Yunkan bfa51c2d87 Refactoring: got rid of duplicate get_mime_type() 2020-04-29 18:33:25 +04:00