Commit Graph

582 Commits

Author SHA1 Message Date
Matthieu Gautier 29e554b47b Include pthread 2018-10-29 14:30:35 +01:00
Matthieu Gautier 68dc4d40b5 Include windows.h before synchapi.h 2018-10-29 12:20:00 +01:00
Matthieu Gautier 2682fa8f9c Remove unecessary variable or output. 2018-10-26 14:19:10 +02:00
Matthieu Gautier a22f962722 Correctly store the size of the book in the library.
`reader.getFileSize()` return ko.
2018-10-26 14:18:40 +02:00
Matthieu Gautier a1876e3b27 Add a method converta2toa3 to convert language code alpha2 to alpha3.
Qt give use alpha2 language code but we use alpha3.
2018-10-26 14:18:06 +02:00
Matthieu Gautier ad654ead08 Do not force the download port to be 80.
We may want to use url with port != 80.
2018-10-24 11:56:38 +02:00
Matthieu Gautier c6206edfb4 Do not always download the favicon of a book. Download as needed.
When parsing a opds feed, the favicon is a url, not a dataurl.
If we download the favicon all the times, it may take a lot of time to
parse the feed.

We store the url and download the favicon only when needed (when displayed)
2018-10-24 11:56:05 +02:00
Matthieu Gautier c20ae18bff An opds feed can also be the openSearch result.
We must be able to set the correct entry in the feed for a searchResult.
2018-10-24 11:51:38 +02:00
Matthieu Gautier b1508c0b98 Better listBooksIds supported mode.
Only have REMOTE or LOCAL is a bit restrictive. By using flags a user
can specify for complex request.
2018-10-24 11:50:11 +02:00
Matthieu Gautier 34021994cd Fix for Android
- No std::to_string. We have to implement it with a ostringstream
- No pthread_cancel. So we use pthread_kill to send a signal to the thread.
2018-10-24 10:48:53 +02:00
Matthieu Gautier 910ce5f10d Fix for Windows
- "winsock2.h" needs to be included before "windows.h". But if a
  compilation unit include "windows.h" and after "networkTools.h", we
  fails and it is complicated to handle. The include must not be in the
  header but in the cpp
- windows define some ERROR macro. It is a pitty but we cannot use `ERROR`
  in our enum.
- If build statically using mingw we need to define `CURL_STATICLIB`
2018-10-24 10:47:12 +02:00
Matthieu Gautier c66c7e9c20 Store the size of the book in OPDSFeed. 2018-10-24 10:47:12 +02:00
Matthieu Gautier ad69fdd8c0 Move the download method from the downloader to networkTools.
The download method is a simple method to download content.
It use curl to download the content instead of aria.
2018-10-24 10:47:12 +02:00
Matthieu Gautier a73ef23f6e Keep the book size in byte in memory (instead of in kb)
We keep the size in kb in library.xml for compatibility.
2018-10-24 10:47:12 +02:00
Matthieu Gautier fe6d5fa93e Store the downloadId in the book (and in the library). 2018-10-24 10:47:12 +02:00
Matthieu Gautier 43ff8565d1 Add a Download class to encapsulate a aria2 download. 2018-10-24 10:47:12 +02:00
Matthieu Gautier f718c4c472 Add a LibraryManipulator.
Library client (kiwix-desktop) need to know when a book is added to
library by the manager. By using a LibraryManipulator, we can do
dependency injection.
2018-10-24 10:47:12 +02:00
Matthieu Gautier 8176a6eded Be more resilient to potential aria2 error. 2018-10-24 10:47:12 +02:00
Matthieu Gautier bb1f777078 Store the aria2 session and recover from it. 2018-10-24 10:47:12 +02:00
Matthieu Gautier 829c34dd69 Store in the book instance if the given path is valid.
The path may exist and not be valid if the zim file is not truncated
(ie, interrupted download)
2018-10-24 10:47:12 +02:00
Matthieu Gautier 9c0f9696ed Better beautifyInteger and beautifyFileSize. 2018-10-24 10:47:12 +02:00
Matthieu Gautier be6dc01b4f Add few helper methods to xmlrpc objects. 2018-10-24 10:47:12 +02:00
Matthieu Gautier 18fc5cb4df Correctly set the aria2 secret rpc. 2018-10-24 10:47:12 +02:00
Matthieu Gautier 996829e4d7 Allow a OPDSDumper to dump only a subset of the library. 2018-10-24 10:47:12 +02:00
Matthieu Gautier 5128861136 Remove default value for book pointer of `readBookFromPath`.
This is a nonsense to accept NULL pointer here.
2018-10-24 10:47:12 +02:00
Matthieu Gautier 7804bf2276 Reimplement listBooksIds.
No real improvement.
2018-10-24 10:47:12 +02:00
Matthieu Gautier 99e313f915 Clean includes of manager.h 2018-10-24 10:47:12 +02:00
Matthieu Gautier 839320d5e7 Move the `Book` class in its own source file. 2018-10-24 10:47:12 +02:00
Matthieu Gautier 1e8f85eaff Rename methods `title()` into `getTitle()`.
Same for all attributes.
2018-10-24 10:47:12 +02:00
Matthieu Gautier e0704b3b21 Move the initialization code of a book from xml|opds into Book. 2018-10-24 10:47:12 +02:00
Matthieu Gautier 57fbb98bca Do not store the favicon base64 encoded in the book.
The fact that the favicon is base64 encoded in a storage detail.
2018-10-24 10:47:12 +02:00
Matthieu Gautier c7f9218350 base64_encode takes a string instead of a char* 2018-10-24 10:47:12 +02:00
Matthieu Gautier 66a9a69480 Move the code updating a book from a reader in the Book class. 2018-09-06 18:30:37 +02:00
Matthieu Gautier 04b05dd68b Remove `removeBookById` from the Manager.
Use the same method of the `Library`.
2018-09-06 18:30:37 +02:00
Matthieu Gautier aa6772b345 Remove the "last" book functionnality.
- This is not used by any application.
- This is application specific and should not be stored in the library
  (who is a list of book).
2018-09-06 18:30:37 +02:00
Matthieu Gautier efae3e0d2f Do not make the `Manager` responsible to create the `Library`.
The `Manager` manage a library already existing.
This avoid the Library clone stuff.
2018-09-06 18:30:37 +02:00
Matthieu Gautier bba3c252e4 Make the member of the book protected.
It is up to the book to manage its attribute.

Also remove the `absolutePath` (and `indexAbsolutePath`). The `Book::path` is always stored
absolute.
The fact that the path can be stored absolute or relative in the
`library.xml` is not relevant for the book.
2018-09-06 18:30:37 +02:00
Matthieu Gautier 57ac6f0305 Use a map to store the Library's books.
Having the books sorted is useless.
We handle books by id not by index.
2018-09-06 18:30:37 +02:00
Matthieu Gautier 541fb0cfd1 Remove the "current" book functionnality.
- This is not used by any application.
- This is application specific and should not be stored in the library
  (who is a list of book).
2018-09-06 18:30:37 +02:00
Matthieu Gautier c9eac04050 Make the Library`s book vector private.
Move a lot of methods from Manager to Library. Because books is private
and thoses methods are better in Library.
2018-09-06 18:30:37 +02:00
Matthieu Gautier 741c67786a Add `update` method to `Book`. 2018-09-06 18:30:37 +02:00
Matthieu Gautier db9000f706 Make the downloader use the aria2c wrapper instead of the aria2 library. 2018-09-06 18:30:34 +02:00
Matthieu Gautier 0a93cb0872 Add aria2 downloader using subprocess aria2c. 2018-09-06 18:29:49 +02:00
Matthieu Gautier f4846c1ac8 Add a tool's function to get the data directory.
The data directory is where kiwix application should store data.
2018-08-29 15:28:52 +02:00
Matthieu Gautier 9b516ac35d Add a small wrapper around pugixml do handle xmprpc. 2018-08-29 15:28:52 +02:00
Matthieu Gautier 79b780b75b Move the function to convert from xml_node to string in otherTools.
This can be usefull elsewhere than in opds_dumper
2018-08-29 15:28:52 +02:00
Matthieu Gautier f3dd83907d Add backend to launch subprocess.
The windows backend is not tested.
2018-08-29 15:28:52 +02:00
Matthieu Gautier 7c634738dd Update jni build script to java jdk 8+.
With jdk8, `javac` has an option `-h` to generate the header files of
native classes.
So there is no need to run `javah` several times.

As there is now only one command to run (`javac`), there is no need for
the wrapper script `gen_kiwix.sh`.

Fix #167
2018-08-20 12:17:29 +02:00
Matthieu Gautier fd9b6569af Include unistd.h only on unix platform. 2018-08-03 19:22:39 +02:00
Matthieu Gautier 3cf58b5f5b Make libaria2 an optional dependency.
We don't compile libaria2 on Windows.
2018-08-03 19:22:39 +02:00
Matthieu Gautier dbcc9140b9 [Android] Better error message when failing to read in zim file.
Let's print the exception's message to allow us to better understand
what went wrong.
2018-07-31 11:49:44 +02:00
mhutti1 d61580f599 Set JNI values to NULL on error 2018-07-27 10:11:24 +02:00
mhutti1 3227b29c90 Follow redirects in favicons 2018-07-27 10:11:24 +02:00
mhutti1 709baae934 Convert all JNI cerrs to Android log messages 2018-06-29 16:35:11 +02:00
mhutti1 ea8cd9f1a9 Correctly pass 0 through JNI if ZIM file is corrupted 2018-06-29 15:30:13 +02:00
Matthieu Gautier 5025ee4963 Fix icudt version.
We have move to icu version 58 a while ago.
2018-06-11 14:38:32 +02:00
Matthieu Gautier 9aaf82a36d Explicitly use icu namespace.
Fix #145
2018-06-11 14:36:34 +02:00
mhutti1 fa99cce68d Decode reserved characters in URLs 2018-05-20 18:41:39 +01:00
Matthieu Gautier 622d2fc23d Remove unused static resources.
Stop words are not use anymore since a long time now that indexing has
been moved to libzim. No need to embedded them in kiwix-lib.
2018-05-15 11:30:30 +02:00
Matthieu Gautier c0b1c6013e Fix parsing of url
Fix kiwix/kiwix-tools#193
2018-05-14 17:41:05 +02:00
Matthieu Gautier 1dd828e79c Fix pathExists and check for correct path for xapian index.
The correct path for xapian database should be "X/fulltext/xapian",
not "Z//fulltextIndex/xapian".

So lets check for the right path and fallback to the wrong one (but
used in old zims).

The double '/' in the path is a bug of zimwriterfs and is specific
to the xapian database.
We must handle this correctly in `hasFulltextIndex` and not (buggly) in
`pathExists`.
(Hopefully, it seems that pathExists were used only by hasFulltextIndex)
2018-04-19 18:04:07 +02:00
Matthieu Gautier 135028c16a Introduce better API to manipulate entries in a zim file.
The previous API suffer different problems:
- It was difficult to handle articles redirecting to other article.
- It was not possible to get few information (title) without getting
  the whole content.

The new API introduce the new class `Entry` that act as a proxy to an
article in the zim file.

Methods of `Reader` now return an `Entry` and the user has to call
`Entry`'s methods to get useful information.
No redirection is made explicitly.
If an entry is not found, an exception is raised instead of returning
an invalid `Entry`.

The common pattern to get the content of an entry become :

```
std::string content;
try {
  auto entry = reader.getEntryFromPath(path);
  entry = entry.getFinalEntry();
  content = entry.getContent();
} catch (NoEntry& e) {
  ...
}
```

Older methods are keep (with the same behavior) but are marked as
deprecated.
2018-04-19 18:04:07 +02:00
Matthieu Gautier 47ce044e3e Add method to `Manager` to populate the library from a opds stream.
The library's books are created in the metadata in the opds.
As the opds stream is by definition a distant "library", there is no
zim to read to complete missing information.

This can lead to incomplete `library.xml`.
2018-04-19 17:53:08 +02:00
Matthieu Gautier 1f091da3f4 Add a downloader tools to download files.
The downloader is using libaria2.

For now, only one download can be run a the time.
A download will start only if (and as soon as) no download is running.
2018-04-19 17:53:08 +02:00
Matthieu Gautier d4fefd1a57 Add a function to create a temporary directory. 2018-04-19 17:53:05 +02:00
Matthieu Gautier 9f86b59d1d Add a function to get the content of a file. 2018-04-19 17:53:02 +02:00
Matthieu Gautier 2164faba44 Add a potential search description link in the opds stream. 2018-04-19 17:08:01 +02:00
Matthieu Gautier ad92af928b Be able to filter a library.
This generate a new library only with the corresponding books.
2018-04-19 17:08:01 +02:00
Matthieu Gautier ee51c470b4 Allow the manager to dump the opds feed of the whole library. 2018-04-19 17:08:01 +02:00
Matthieu Gautier ec097ab267 Try to compile kiwix-lib without warning. 2018-04-18 16:57:27 +02:00
Matthieu Gautier d686de7ec3 Set the RPATH of kiwix-lib.
As we cannot change (DY)LD_LIBRARY_PATH on macos, we have to use rpath.
2018-04-17 16:27:31 +02:00
Matthieu Gautier 313f6731b0 Correctly pre-increment loopCounter.
If we check the later the `loopCounter` with 42, we must pre-increment the
content. Else, in case of infinite loop, the `loopCounter` will be 43.

Related to kiwix/kiwix-tools#168
2018-03-25 17:21:40 +02:00
Matthieu Gautier ee6831d665 Check `internal->_search` before using it.
If a search has been set and a user try to get the nextResult or
restart the search, `internal->_search` will be NULL.
2018-03-12 17:45:18 +01:00
Matthieu Gautier f8a2e4c503 Only add a reader to the searcher if the reader as fulltext index.
`libzim` will not search in zim file without embedded fulltext index.
If we don't want to mess up with result index, we must not store "wrong"
reader.

Fix #111
2018-03-12 17:34:45 +01:00
Matthieu Gautier 57a197d38d Make getCurrentBookId const. 2018-03-12 17:34:45 +01:00
Matthieu Gautier b6ba10af2a Remove unnecessary currentArticleOffset.
This protected member is never used.
2018-03-12 17:34:45 +01:00
Matthieu Gautier f93f50087b Remove unnecessary setBookIndex.
We can use default argument instead of creating a new method.
2018-03-12 17:34:45 +01:00
Matthieu Gautier 6adf95c329 Make JNIKiwixReader throw an exception if something goes wrong at creation.
If the `nativeHandle` is null, the JNIKiwixReader is invalid and we must
not use it.

Throwing an exception for the caller code to handle this properly.
And previously, user code has no way to detect something went wrong :/
2018-02-01 17:18:54 +01:00
mhutti1 97bcf57d53 Allow JNI to access search snippets 2017-12-15 16:02:49 +00:00
mhutti1 f303c7502d Fix JNI to work with kiwix-android 2017-12-14 17:32:03 +00:00
Matthieu Gautier 16bd34e6a6 Add a method in the JNI API to get direct access information.
For binary content (not compressed), it could be interesting to
directly read the content in the zim file instead of using `kiwix-lib`.

This method returns the needed information to do so (if possible).
2017-12-13 17:22:26 +01:00
Matthieu Gautier 5a953f191b Remove a small warning. 2017-12-13 17:11:10 +01:00
Matthieu Gautier 41c92cfc3c Better calculate the start of the last search page.
The increment between pages should always be a multiple of
`resultCountPerPage`.
2017-11-27 12:39:04 +00:00
Matthieu Gautier 64dc5131c0 Be able to specify the global contentHumanReadableId without a index.
Even if we use the add_reader method to search into embedded full text
index, we need to specify the global `contentHumanReadableId` as it will
be used to generate "page links".
2017-11-27 12:37:13 +00:00
kelson42 28b0588df4 Better URL encoding 2017-11-23 19:26:41 +01:00
Matthieu Gautier 4e5d9f0360 Add a API to get only a part of a article content.
Add the jni method `getContentPart` to get only a part of the artcicle
content.

The method can be used to get a part of the content or to know the size
of the full content.
2017-11-22 19:06:54 +00:00
mhutti1 520c1edf31 Fix JNI android integration 2017-11-22 14:54:03 +00:00
mhutti1 d2f7503cfa Fix JNI for android integration 2017-11-22 14:54:03 +00:00
Matthieu Gautier 7a59779b77 Change JNI API of kiwix-lib (mainly to support multi-zims search)
This is a major API break. User code will have to be rewritten.

Before this commit, API was a unique object wrapping the library and
handle a global state with one `Reader` and one `Writer` at the time.

Now, the API is axed around three main objects :
 - The `JNIKiwixReader`, a wrapper around a `kiwix::Reader` (who allow to
   read one zim)
 - The `JNIKiwixSearcher`, a wrapper around a `kiwix::Searcher` (who allow
   to search through one or more reader(s))
 - The `JNIKiwixSearcher.Result` a result of a search. Allowing to get all
   information about a result (title, url, content, snippet, ...)
2017-11-22 14:54:03 +00:00
Matthieu Gautier 766b64dddc Update gen_kiwix.sh to not be dependent of the number of arguments. 2017-11-22 14:46:01 +00:00
Matthieu Gautier b9ac7084ac Add small API to do geo query.
This is a small quick and dirty API to do geo query.

It is not possible with this API to do a query search and a geo search.
It's either one or the other.

We should think about a better global API to do searching and provide
both of them in the same time (libzim does it).
2017-11-14 17:32:06 +01:00
Matthieu Gautier 0e8c8f68c5 Extend search limits to 140.
70 is a too small limit for the number of results.
Users need at least 100.

As the html rendering will fails with more than 144 results,
explicitly limits the number of search to 140.

Fixes kiwix/kiwix-tools#92
2017-11-06 12:23:13 +01:00
Matthieu Gautier 382655d83c Explicitly set ctpp2 iIMaxSteps to extends search beyond 68 results.
Ctpp2 templates have a limit step number. If the template to render is
too big, the rendering fails, throwing an exception.

From our tests, it seems that, with the template we have, the default
step limit allow us to render 68 results only.

By doubling the limit, we can render up to 144 results.
2017-11-06 12:10:48 +01:00
Matthieu Gautier 5c2280e7c7 Claims that multi part zim has no embedded full text index.
We cannot search into an embedded fulltext index if the zim is multipart.
Instead of crashing, let's pretend we have no fulltext index.
2017-08-15 14:19:53 -04:00
Chris Li cf93c8719f Allow kiwix-lib to compile without ctpp2c.
ctpp2c is used to pre-compile the template resource.
However, on OSX, ctpp2c seems to be difficult to compile, as we don't need
ctpp2 at all on OSX/iOS, lets just stop to force the use of ctpp2c.
2017-08-14 10:42:16 -04:00
Matthieu Gautier 1ff1bf6168 Always set the humanReadableName with the readable in kiwix-search.
We always need a humanReadableName associated with a content to search in.
Do not separate the two values (human readable name and zim) in two
different functions.

This way, we avoid miss-use of the Searcher who could lead to segfault.
2017-08-10 09:20:11 -04:00
Matthieu Gautier d3acae1fd2 getContent* methods also allow to get the title.
Add a `title` write argument to `getContent*` methods.
This argument is filled with the title of the content get.

Also update the JNI accordingly.

Related to kiwix/kiwix-android#214
2017-07-26 11:03:32 +02:00
kelson42 1d1dfbf4da Fix JNI licensing #71 2017-07-22 09:29:28 +02:00
Matthieu Gautier bc5f4f5de4 Use right contentId to generate the article url in search template.
As we do multisearch, we must use the associated contentID of the result
to generate the url.
2017-07-18 10:04:40 +02:00
Matthieu Gautier 9cc329dbd2 Support multi-zims search in kiwix-lib.
All the code was already in zimlib.
It is mainly a update of the code using zimlib.

No JNI change for now to not break the API.
2017-07-18 10:04:40 +02:00