Commit Graph

253 Commits

Author SHA1 Message Date
Alexander Sashnov da31dd428e remove 'zimwriterfs' project from build and CI configuration
'zimwriterfs' is now a part of zim-tools (openzim/zim-tools/#92)
2020-07-08 16:54:14 +02:00
Kelson 3de886f805 Upgrade libmicrohttpd to 0.9.71 2020-07-08 14:57:38 +02:00
MiguelRocha e72589a850 Added a flag to force kiwix build to not do fast_clone
- Docopt cannot do fast_clone cause it is getting a
    specific commit id. This will be changed when a new
    release is available.
2020-05-25 10:57:22 +02:00
MiguelRocha 69c33da0bc Now kiwix-build retrives docopt from a commit that has pkgconfig support. 2020-05-25 10:57:04 +02:00
MiguelRocha 3e2df9a8e8 Added docopt cpp dependency to zimtools. It will be needed to the new version of zimdump
Do not build docopt on android and iOS.
2020-04-27 13:39:40 +02:00
Kelson 7f6a8856b7 Remove deprecated Mustache patch 2020-04-21 09:31:21 +02:00
Kelson 5d1f5bda1f Update to Mustache v4.1 2020-04-21 09:31:21 +02:00
Matthieu Gautier 99226e0c6a Build zstd in flatpak.
Revert 10c767e8ce
2020-04-20 15:14:31 +02:00
Matthieu Gautier ffdee634f4 Do not force a reconfigure of meson projects.
If meson.build changes, the ninja command will rebuild it.
And its avoid us to delete the build directory.
2020-04-10 00:38:47 +02:00
Veloman Yunkan 25d9e18e3d Temporary fix for facebook/zstd#2073 2020-04-07 14:12:17 +04:00
Veloman Yunkan ece4df3721 Added zstd dependency 2020-04-06 20:17:09 +02:00
Kelson 9be35a4dc0
Fix qt-everywhere-src-5.10.1.tar.xz URL (#431) 2020-04-06 20:16:01 +02:00
Matthieu Gautier fb07b58812 Fix recompilation of the CI of meson project.
We were assuming that meson project correspond to our projects and so we
were always building them, even if they were already compiled.
(This way, a simple `kiwix-build` is enough to recompile the WIP code of
our project).

However, on the CI, we do not archive the source code/build directory in
the base deps archive. So when we try to compile, the compile step of
meson projects fails because the source are not here.
We have a small workaround for pugixml but as zstd is also meson, it is
time to do something correct.

By default, all projects now try to skip if a build is already present.
Our projects are marked as `force_build` and so, they do not try to skip.
2020-04-06 19:15:40 +02:00
Matthieu Gautier 4fa5f34125 Build all make based dependency using -O3 option.
This will build all Make, CMake, QMake dependency being build with -O3.
So all our dependencies will be build with -O3.

Our project (meson) are already build in release when needed.
kiwix-desktop should be build optimized in case of release but we need
to handle it correctly. At least not build in O3 per default as it may
break debug.
2020-03-24 16:50:38 +01:00
Matthieu Gautier 71c46d984a Build lzma with assembler optimization. 2020-03-18 21:03:48 +01:00
Matthieu Gautier 40b5013279 Add some colors in the terminal. 2020-02-25 15:48:20 +01:00
Matthieu Gautier f504cf54e1 Do not reset the source directory if there is modification.
We simply do a `git merge --ff-only` instead of a `git reset`. This way:

- No change on the local repository
    => update to `remote/master`
- If you switch branch and local branch is behind `remote/master`
    => fast forward to `remote/master`
- If you switch branch and branches diverge
    => No update of the directory
- On branch master (or a branch behind `origin/master`) with compatible
  change in the working tree.
    =>  update of the directory
- On branch master (or a branch behind `origin/master`) but incompatible
  change in the working tree.
    =>  No update.

This way, sources are keep up-to-date as far as possible.
2020-02-25 15:46:09 +01:00
Matthieu Gautier 94c98261c7 Fix compilation on different platform in the same time.
If we modify the dependency's configure_env, we may change
the dictionary with value of the first platform. Then, when
we use it for the second platform, the previous values are used.

Do not modify the dep_conf_env dictionary and then we are good.
2020-02-24 18:24:07 +01:00
Matthieu Gautier 32e3b5b5c4 Fix compilation of kiwix-lib-app.
- We should not try to set the cross compilation flags when we use
  gradle.
- Fix anyway the setting of tho cross compilation flags for android sdk
  platform (the default implementation use the `static` attributes that
  doesn't exist for android)
2020-02-24 15:30:31 +01:00
Matthieu Gautier 7b6c79482a Make the dependency responsible to set the compilation env.
Instead of having the run_command function setting the
env from the buildEnv, this is the dependency that create the
env and then pass it to the run_command function.

This way, each dependency will be able to set a specific env.
2020-02-21 16:29:23 +01:00
Matthieu Gautier 7d742f807f Fix setting of environment using env from dependency.
If `self.configure_env` is a (computed) property, the previous code
never format the value of the env.
2020-02-21 15:12:17 +01:00
renaud gaudin 3569c277a9 switch to older version of libmicrohttpd
libmicrohttpd 0.9.69 is not working properly on macOS (catalina)
2020-02-12 11:09:46 +01:00
renaud gaudin 5af0a8bfe7 fixed crash in xapian-core configure
use a copy of configure_env to prevent edits on what's being iterated
2020-02-11 13:10:21 +00:00
Matthieu Gautier 0a37c8a573 Change args given to kiwix-lib meson for android. 2020-01-28 15:49:52 +01:00
Kelson b92a111fcc
Upgrade libmicrohttpd to 0.9.69 (#394) 2019-12-16 16:57:59 +01:00
Kelson 8420dcb474 Bump-up Xapian to 1.4.14 (#382) 2019-12-09 01:12:48 +01:00
Matthieu Gautier a82d6430dc Remove support of kiwix-android and kiwix-custom-app. 2019-12-05 12:30:10 +01:00
Matthieu Gautier b6bbf9b098 Do not install gradle. Use gradlew script. 2019-12-05 12:30:10 +01:00
Kelson e602897e6b
Newest version of libcurl (#387) 2019-12-05 06:29:00 -05:00
Kelson 0085a5c565 Revert "Bump-up Xapian to 1.4.14 (#382)"
This reverts commit f4f9f6f833.
2019-11-25 03:34:41 +01:00
Kelson f4f9f6f833
Bump-up Xapian to 1.4.14 (#382) 2019-11-24 07:37:39 -05:00
Kelson 6b43f5ad48
Update libmicrohttpd to 0.9.66 (#365)
* Update libmicrohttpd to 0.9.66

* Proper Bash syntax highlighting
2019-09-06 15:54:44 +02:00
Matthieu Gautier f2369c6819 Build project in debug by default instead of debugoptimized. 2019-09-04 17:10:44 +02:00
Matthieu Gautier 2e347b2b33 Do not generate the pom file in kiwix-build.
Now we generate the pom file in kiwix-lib side, we don't need to generate
it ourselves.
2019-08-21 14:27:24 +02:00
Matthieu Gautier f80afdecbc Move to last version of libmicrohttpd.
Again.. 0dcaa28 move back to a previous version (probably a wrong rebase)
2019-08-19 18:42:01 +02:00
Matthieu Gautier bd43b9cb10 Do not download gtest in flatpak. 2019-08-19 17:02:44 +02:00
Matthieu Gautier 0dcaa28e35 libmicrohttpd is now a dependency of kiwix-lib, not kiwix-tools.
And we have to install libmicrohttpd on windows.
2019-08-10 19:44:10 +02:00
Matthieu Gautier 07f8150eb6 Use the name "kiwix-lib" for the source of "kiwix-lib-app". 2019-08-01 14:57:06 +02:00
Matthieu Gautier 7c2527c38f Do not try to download subproject for libzim in flatpak. 2019-07-31 16:36:31 +02:00
Matthieu Gautier 7d659baa37 Package kiwix-serve within kiwix-desktop.
Now that kiwix-desktop use kiwix-serve to serve the content, we need to
compile and package kiwix-serve (kiwix-tools).
2019-07-24 11:17:39 +02:00
Matthieu Gautier 9bbc3930ae Print the duration of each steps. 2019-07-24 10:25:52 +02:00
Matthieu Gautier 3a1cd6ddb2 Build the kiwix-lib android "application" in kiwix-build.
This is not a real application as it is dependency for kiwix-android.
But it a app in the sens that it is packaged for android build system.
2019-07-24 10:25:52 +02:00
Kelson 4c342c5b9d
Update libmicrohttpd to 0.9.65 (#347) 2019-07-23 14:55:34 +02:00
Matthieu Gautier ef9755eff5 Remove android mips platforms
Also correctly rename Android classes.
2019-06-14 14:13:18 +02:00
Isaac Hutt 06da96f85f Update gradle to 5.2 2019-06-03 06:26:28 +02:00
Kelson 456537e188 Disable Xapian remote backend 2019-06-02 17:15:26 +02:00
Matthieu Gautier f9d5e0aa21 Use an already made archive for the armhf toolchains instead of git clone.
First, we don't need to clone all the git repository, we only need a
sub-directory.
Secondly, as we the base_deps archives only archive the sub-directory,
subsequent run of kiwix-build in the ci will try to do a git update on
a non git repository.

By making ourselves a tar archive, we simplify the process.

The archive on download.kiwix.org has been made with :
`tar czf raspberrypy-tools.tar.gz raspberrypi-tools/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian-x64`
`raspberrypi-tools` being a clone of `https://github.com/raspberrypi/tools`
commit sha is 5caa7046982f0539cf5380f94da04b31129ed521
2019-05-02 17:43:11 +02:00
Matthieu Gautier 318c524f4c Use our own server to store libmagic. 2019-04-30 15:28:21 +02:00
Matthieu Gautier 2970a2d445 Update to new version of lzma and get it from kiwix.org server.
It seems that the downloading of the archive stored on sourceforge
fails sometime.

Use the archive stored in our server (download.kiwix.org) instead.
And also move to the last version of lzma.
2019-03-06 10:24:09 +01:00
Matthieu Gautier 1341c8b401 Do not strip libraries in release_mode.
Meson use gcc's `-s` option to create the archive.
This option should make gcc meemic `ar` tool but in some version of gcc,
it seems there is a bug and the `-s` option is not taken into account.

See https://github.com/mesonbuild/meson/issues/4138

So, do not try to strip libraries. We will strip the final binaries anyway.
2019-03-04 13:53:26 +01:00
Matthieu Gautier 505961be4c Build in release mode.
- Dependency are installed "striped".
- Our project are build "debugoptimized" by default and "release" when
  building release instead of "debug"

We need to update the `base_deps_meta_version` as we are changing how
dependencies are compiled.
2019-02-19 17:23:14 +01:00
Matthieu Gautier f2c3a8a4f6 Add a patch to allow compilation of xapian on android. 2019-02-18 17:02:13 +01:00
Matthieu Gautier 6a1fdb8620 Use xapian 1.4.10 insteal of 1.4.7
Fix #264
2019-02-18 14:32:45 +01:00
Matthieu Gautier 8d13ef673a Update the list of android packages to install. 2019-02-05 16:22:09 +01:00
Isaac Hutt f73187f7c9 Update gradle dependency 2019-02-05 16:22:09 +01:00
Matthieu Gautier 4eb95fd4cc Pass correct option `-Dstatic_linkage` to libzim in native_mixed platform.
In native_mixed, we want to build a shared lib but we need to compile
with the static lib of the dependencies.
2019-02-04 17:01:28 +01:00
Matthieu Gautier 466dc8b64a Fix `-fPIC` CXXFLAGS. 2019-02-04 17:01:28 +01:00
Matthieu Gautier 50167460dc Move the version of org.kde in the versions.py file. 2019-01-23 16:04:02 +01:00
Matthieu Gautier b2d2e489bf [iOS] Use the correct option `b_bitcode` for libzim and kiwix-lib. 2019-01-17 16:50:49 +01:00
Matthieu Gautier c4a3da03c3 Correctly install mustache in flatpak.
We need a simple buildsystem to "build" mustache" as we simply need to
copy the header.
2019-01-15 14:44:02 +01:00
Matthieu Gautier 961076bb87 [ANDROID] Build only the apk, not everything.
It seems that build everything on android side is a bit too expensive.
Let's do as is done on android CI, build only the apk.
2019-01-15 11:26:11 +01:00
Matthieu Gautier 1a0e0ad717 Update to last version of libmagic (5.35) 2019-01-09 11:17:41 +01:00
Matthieu Gautier 8005aff6c0 Remove ctpp2 dependency.
kiwix-lib doesn't use ctpp2 anymore but mustache.
So remove ctpp2 dependency and add mustache.
2019-01-09 11:17:41 +01:00
Matthieu Gautier 17a786092a Do not set unused configure options in flatpak. 2018-12-12 11:19:07 +01:00
Matthieu Gautier f96ae19c88 Disable websocket support in aria2.
We don't need it. It could speed up a bit the compilation.
2018-11-30 11:37:23 +01:00
Matthieu Gautier 833c5844a1 Use the right option to build kiwix_desktop and pugixml on flatpak. 2018-11-27 17:22:47 +01:00
Matthieu Gautier ed63ffe24c Add aria2 dependency.
kiwix-desktop needs the aria2 binary to download content.
kiwix-desktop will use aria2 throw lib-kiwix so it is somehow a kiwix-lib
dependency, but it is used only by kiwix-desktop.

And we know to install it only on platforms where we are supporting
kiwix-destkop.
2018-11-27 17:22:47 +01:00
birros 8d58d8d7cb First version of a flatpak builder.
This commit add a first version code to build a flatpak of kiwix-desktop.
It is mainly based on the PR #254 of @birros (hence he is the author of this commit)

However there is some differences :
- I (@mgautier) create a new builder to run the flatpak build instead of using a new
  dependency.
- Use the flatpak platform to install org.kde.Platform and org.kde.Sdk

This code version doesn't correctly work but I wanted to commit the birros' code
without too many modification (even if there is a lot).
2018-11-27 09:43:52 +01:00
Matthieu Gautier 0600be96ac Force the build of libcurl instead of using system one.
On ubuntu, the libcurl package is compiled with libkrb (kerberos).
However, no libkrb5.a is provided and so, we cannot link statically.

As we are not using kerberos, we can (and must) compile ourselves libcurl.
2018-10-31 15:33:12 +01:00
Matthieu Gautier 1e67a6b50f Add libcurl dependency.
This is needed for new kiwix-lib version.
We also remove libaria as kiwix-lib use aria2 binary directly.
2018-10-23 11:57:58 +02:00
Kelson e9ae0da048 Move to Xapian 1.4.7 2018-09-20 18:27:30 +02:00
Matthieu Gautier 1569068332 Do not set version for main project if we are not making release. 2018-09-06 16:10:43 +02:00
Matthieu Gautier ecc6927de8 Use git archive to create dist source archive.
qmake seems to generate wrong "dist" rules for "make".
It try to copy the translation files as if it was in current directory
instead of looking the source dir.

dist archive is usefull with autotools where we want to run autogen before.
Or with meson when meson run tests before making the archive.

With qmake, it seems useless, let's simply archive what is git.
2018-09-05 19:38:58 +02:00
Matthieu Gautier df0b684869 Correctly find the version of a dependency.
We have separated the version in two separated dicts since a long time :/
2018-09-05 19:31:38 +02:00
Matthieu Gautier e22ceab92a Do not use buffered header on android.
Android devices are low memory devices, use last libzim compilation option
to avoid copy/mmap index header in memory.
2018-08-01 18:07:13 +02:00
Matthieu Gautier 5c83ba9d10 Fix compilation of kiwix-custom-app.
This is forgotten dependency. Not updated when we moved to multi-arch
build.
Now, it is fixed.
2018-07-19 16:19:55 +02:00
Emmanuel Engelhart 79a08e001b Use faster svn export instead of svn checkout 2018-07-16 10:15:28 +02:00
Matthieu Gautier 0f389f7cc9 Get icu sources from an archive instead of svn.
SVN checkout can be pretty long to do (especially if icu serveur is pretty
slow).

By downloading an archive on our own server, we avoid a spending too many
time downloading.

The archive is a simple targz of the SVN checkout.
2018-07-11 11:00:55 +02:00
Emmanuel Engelhart e90ff48ec9 Remove support of old Xapian chert backend 2018-06-24 20:51:31 +02:00
Matthieu Gautier d076c6b8d7 Do not set zimwriterfs version directly in the dependency.
The version is tracked in `kiwixbuild/versions.py`, so we must no set
it in `kiwixbuild/dependencies/zimwriterfs.py`.
2018-06-18 17:56:13 +02:00
Matthieu Gautier 1cbfcef9ce Fix kiwix-desktop dependencies. 2018-06-14 18:18:45 +02:00
Matthieu Gautier 8506cdeffd Correctly pass CMAKE_* to qmake.
qmake doesn't use the env variables, we must pass the variables in the
command line.
2018-06-14 18:18:45 +02:00
Matthieu Gautier 4047ec20dd Add xapian patch to correctly include sys types. 2018-06-14 18:18:45 +02:00
Matthieu Gautier 398f77f9f3 Add kiwix-desktop dependency. 2018-06-12 16:12:31 +02:00
Matthieu Gautier cf8f303362 Add qt(webengine) dependency.
Add qt dependency and QMakeBuilder.
2018-06-12 16:12:15 +02:00
Matthieu Gautier 1109b6b155 Better compilation of icu4c.
- Disable icuio and layoutex.
- Hardcode UTF8 as charset (we always use utf8 and hardcoding him improve
  performances)
- Do not include default utf headers
- Do not use `using namespace icu`

See http://source.icu-project.org/repos/icu/trunk/icu4c/readme.html#RecBuild
for more information about options.

Increment base dependencies version as we are compiling icu differently.
2018-06-11 18:05:10 +02:00
Matthieu Gautier 4dea44d305 Fix travis CI 2018-06-08 10:29:26 +02:00
Matthieu Gautier daefadecc9 Better gradle option.
Enable `--no-daemon` : Using the daemon make gradle keep "configuration"
(like plugin path). If gradle installation path change, it will break
kiwix-android build.

Enable `--build-cache` : This is not the case by default :/
This should greatly improve compilation speed.
2018-06-07 10:50:30 +02:00
Matthieu Gautier 369b805e59 Add ios_multi_arch fatlib.
Fix #113
2018-06-06 17:41:12 +02:00
Matthieu Gautier 5f5fbfe082 Rename `targets` option to `target`. 2018-06-06 17:41:12 +02:00
Matthieu Gautier db82455f03 Be able to build multi-arch android apk.
By building kiwix-android on the `android` platform, we can now build
`kiwix-lib` on all `android_<arch>` platforms, and so have all
architectures in the same apk.

Fix #108
2018-06-06 17:41:12 +02:00
Matthieu Gautier 9b85791705 Gradle is a dependency of the android platform, not of other dependencies. 2018-06-06 17:41:11 +02:00
Matthieu Gautier c8c0192675 Simplify cross-compilation
Now we can make reference to build step in another target platform, we can
simplify cross-compilation fo libmagic and icu4c.
2018-05-31 11:30:26 +02:00
Matthieu Gautier 4d6b6994bc Merge toolchain and dependencies.
Toolchain now became "simple" dependencies.
They are referenced by Platform.

Platform are now created at demande and a neutralPlatform now exists.
2018-05-31 11:30:25 +02:00
Matthieu Gautier daad1c98c6 Make the options global. 2018-05-31 11:25:59 +02:00
Matthieu Gautier 041826d0e8 Move from target logic to steps logic.
This is the big change !!!!

Instead of handling target as primary object and prepare/build targets,
we are handling build steps.

A build step may be a source (preparation) or a build (of the source).
Actualy, a step is a tuple (context, Builder or Source).

The context define the context of the step. It can be :
- 'source', for a Source step
- 'neutral' or the name of a platform for Build step.

Target becomes a "Class only" class.
2018-05-31 11:25:59 +02:00
Matthieu Gautier 754df8df1d Make the platform responsible to create the buildenv.
It is now the platform who is responsible to:
- correctly add the toolchains as needed
- setup the cross environment
- create the cross files.
2018-05-31 11:25:54 +02:00
Matthieu Gautier 115fbfa147 Move dependencies declaration of a target into the builder.
This is the builder that depends on other target, not the target itself.
2018-05-31 11:14:31 +02:00
Matthieu Gautier 7e0b403ccc Move `run_command` method out of buildEnv class. 2018-05-31 11:14:31 +02:00
Matthieu Gautier 816e06a512 Make the neutralEnv global.
No need to pass the neutralEnv from instance to instance.
2018-05-31 10:42:37 +02:00
Matthieu Gautier b950feb893 Move Builder and BuildEnv in specific module.
Introduce also a "NeutralEnv", a build environment independent of the
targeted platform. All `Source` now build using the neutralEnv.

Most of toolchains are also using neutralEnv except android_ndk who is
specific to a platform.

As toolchain are neutral, platform specific environment variables are now
set by the platformInfo directly instead of the toolchain.
2018-05-22 16:58:39 +02:00
Matthieu Gautier f1d1fc654c Better MakeBuilder configure system.
`all_configure_option` should be ALL configure option.
2018-05-22 16:43:37 +02:00
Matthieu Gautier 53585b838a Split the dependencies.py file into several parts. 2018-05-22 16:43:37 +02:00