The correct path for xapian database should be "X/fulltext/xapian",
not "Z//fulltextIndex/xapian".
So lets check for the right path and fallback to the wrong one (but
used in old zims).
The double '/' in the path is a bug of zimwriterfs and is specific
to the xapian database.
We must handle this correctly in `hasFulltextIndex` and not (buggly) in
`pathExists`.
(Hopefully, it seems that pathExists were used only by hasFulltextIndex)
The previous API suffer different problems:
- It was difficult to handle articles redirecting to other article.
- It was not possible to get few information (title) without getting
the whole content.
The new API introduce the new class `Entry` that act as a proxy to an
article in the zim file.
Methods of `Reader` now return an `Entry` and the user has to call
`Entry`'s methods to get useful information.
No redirection is made explicitly.
If an entry is not found, an exception is raised instead of returning
an invalid `Entry`.
The common pattern to get the content of an entry become :
```
std::string content;
try {
auto entry = reader.getEntryFromPath(path);
entry = entry.getFinalEntry();
content = entry.getContent();
} catch (NoEntry& e) {
...
}
```
Older methods are keep (with the same behavior) but are marked as
deprecated.
The library's books are created in the metadata in the opds.
As the opds stream is by definition a distant "library", there is no
zim to read to complete missing information.
This can lead to incomplete `library.xml`.
The downloader is using libaria2.
For now, only one download can be run a the time.
A download will start only if (and as soon as) no download is running.
If we check the later the `loopCounter` with 42, we must pre-increment the
content. Else, in case of infinite loop, the `loopCounter` will be 43.
Related to kiwix/kiwix-tools#168
`libzim` will not search in zim file without embedded fulltext index.
If we don't want to mess up with result index, we must not store "wrong"
reader.
Fix#111
If the `nativeHandle` is null, the JNIKiwixReader is invalid and we must
not use it.
Throwing an exception for the caller code to handle this properly.
And previously, user code has no way to detect something went wrong :/
For binary content (not compressed), it could be interesting to
directly read the content in the zim file instead of using `kiwix-lib`.
This method returns the needed information to do so (if possible).
Even if we use the add_reader method to search into embedded full text
index, we need to specify the global `contentHumanReadableId` as it will
be used to generate "page links".
Add the jni method `getContentPart` to get only a part of the artcicle
content.
The method can be used to get a part of the content or to know the size
of the full content.
This is a major API break. User code will have to be rewritten.
Before this commit, API was a unique object wrapping the library and
handle a global state with one `Reader` and one `Writer` at the time.
Now, the API is axed around three main objects :
- The `JNIKiwixReader`, a wrapper around a `kiwix::Reader` (who allow to
read one zim)
- The `JNIKiwixSearcher`, a wrapper around a `kiwix::Searcher` (who allow
to search through one or more reader(s))
- The `JNIKiwixSearcher.Result` a result of a search. Allowing to get all
information about a result (title, url, content, snippet, ...)
This is a small quick and dirty API to do geo query.
It is not possible with this API to do a query search and a geo search.
It's either one or the other.
We should think about a better global API to do searching and provide
both of them in the same time (libzim does it).
70 is a too small limit for the number of results.
Users need at least 100.
As the html rendering will fails with more than 144 results,
explicitly limits the number of search to 140.
Fixeskiwix/kiwix-tools#92
Ctpp2 templates have a limit step number. If the template to render is
too big, the rendering fails, throwing an exception.
From our tests, it seems that, with the template we have, the default
step limit allow us to render 68 results only.
By doubling the limit, we can render up to 144 results.
ctpp2c is used to pre-compile the template resource.
However, on OSX, ctpp2c seems to be difficult to compile, as we don't need
ctpp2 at all on OSX/iOS, lets just stop to force the use of ctpp2c.
We always need a humanReadableName associated with a content to search in.
Do not separate the two values (human readable name and zim) in two
different functions.
This way, we avoid miss-use of the Searcher who could lead to segfault.
Add a `title` write argument to `getContent*` methods.
This argument is filled with the title of the content get.
Also update the JNI accordingly.
Related to kiwix/kiwix-android#214
This was not necessary when searching in only one zim file as `url` was
enough to get the article (and so the content).
If we want to search in several zim in the same time, we need a way to get
the content directly.
If a embedded fulltext database is present, suggestion will search in it :
- insensitive case search.
- search for terms in the middle of the title.
- xapian will try to complete the last word of the query (as if a '*'
were added at the end)
Compilation fails on clang 3.4.1 (and presumably later, tho I haven't tested) with
```
src/reader.cpp:131:59: error: no viable conversion from 'iterator' (aka '__map_iterator<typename __base::iterator>') to 'std::map<std::string, unsigned int>::const_iterator' (aka '__map_const_iterator<typename __base::const_iterator>')
std::map<std::string, unsigned int>::const_iterator it = counterMap.find("text/html");
^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/include/c++/v1/map:713:29: note: candidate constructor (the implicit copy constructor) not viable: no known conversion from 'iterator' (aka '__map_iterator<typename __base::iterator>') to 'const std::__1::__map_const_iterator<std::__1::__tree_const_iterator<std::__1::__value_type<std::__1::basic_string<char>, unsigned int>, std::__1::__tree_node<std::__1::__value_type<std::__1::basic_string<char>, unsigned int>, void *> *, long> > &' for 1st argument
class _LIBCPP_TYPE_VIS_ONLY __map_const_iterator
^
```
because we are not using the right type for the map iterator. As we are using
C++11, let's use `auto` and make compiler set the right type for us.
Android will look in specific repository to find native libs.
We need to use the `android_abi` name defined in the cross_compilation
file instead of `cpu_family`.
libzim only know how to read embedded full text index in a zim file.
This is nice as we want to embedded the full text index in zim file and
not have separated full text index.
However, we still have some zim+separated index we have to read.
So we have to support the search in separated index for a while.
`kiwix-android` is using `kiwix-lib` as an external java application now.
So we need `kiwix-lib` build system to also install application files
(manifest, resources, ..).
In C++11 `linux` is not a reserved word, so compilators do not define it.
A correct way to check if we are compiling for linux is to check for
`__linux__`.
Unicode headers ends by defining the DONE symbol in a enum.
It can clash with other includes.
(For instance the httpd.h from apache who use `#define DONE -2`).
Both project should not declare such common symbols publicly but we have
to do with them anyway.
Commit b8d950c removes this symbol.
The indexer is not used anymore and will be soon removed.
So for now, just remove the call to readStopWords until we totally
remove the indexer code.
The commit 83d2725 adapt the jni wrapper to the new search API but try to
use the `Result` class from the `kiwix` namespace but `Result` is not in
the namespace.
A correct fix would be to move `Result` in `kiwix` but it also change the
API for other tools (kiwix-tools). As we will move the search
functionality in `zimlib` it is better to just do this silly fix and
update the API latter when moving the search functionality.
The `for( auto elem: elems)` syntax is a C++11 syntax.
We are not using C++11 (even if it would be good idea).
This works on recent compiler (on Fedora 25) but fails on older one
(on Travis).
We don't need to generate a vector of result when we do a search.
We better to just keep the handle to the current MSetIterator and
generate the wanted values when needed.
- No more dependency to reswrap binary (everything is done in python)
- Resource strings can be directly accessed.
As side effect, it add a check at compilation if the resource is
declared and compiled in the binary.
- The resource content can be overwritten at runtime with a env variable.
There is also few clean in the static as some files shoul be in the tools
directory.
The compile_resource script is install to let other project use it.
There is no more integrated build of dependencies in the build system.
Dependencies are discovered using pkg-config except for ctpp2 where there
is no pkg-config file.
Binary content do not need to be modified, so we don't need to copy it.
We can directly serve it from the internal zim (cluster) buffer.
The handle_content function now getArticleObjectByDecodedUrl instead of
getContentByDecodedUrl.
This is to get the mimetype of the article and copy the content only when
needed (getContentByDecodedUrl always copy the content).
Thus, handle_content is a bit more complex as it need to do some
manipulation previously made in getContentByDecodedUrl.
The main change is that if the content is binary, we serve the content
with a callback response who will get the content chunks directly from
the blob buffer.
The fulltext index in the zim is at the url /Z/fulltextIndex/xapian.
We do not need to specifie the Z in the url as this is automatically add with the namespace.
The XapianSearcher creation must fail with a exception if we cannot open
the xapian database.
So, if we try to open a zim and there is no embedded index, we must fail.
We raise the custom exception NoXapianIndexInZim in this case.
- kiwix-search will try the embedded ft-index if no indexPath is specified
in the library.xml or command argument.
- kiwix-search also uses xapianSearcher and xapianSearcher needs zimlib,
so add zimlib in compilation flag for searcher.
- kiwix-serve already depends on zimlib.
mhutti1
Matthieu Gautier
kelson42
Chris Li
Steve Wills
Matthieu Gautier
kelson42
Chris Li
dattaz