Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Build Caffe on CentOS 6 with dependencies included #77

Merged
merged 1 commit into from
Sep 22, 2015

Conversation

cypof
Copy link
Contributor

@cypof cypof commented Aug 8, 2015

First draft of how we could get Caffe to run on every Linux distrib. cppbuild.sh now compiles all dependencies from source, and keeps the libs in cppbuild/lib for inclusion in the jar file. I only worked on x86-64.

If we run this on CentOS 6, which has the oldest glibc (2.12), it should run anywhere. There is only one additional dependency, but it is only needed at compile time for OpenBLAS:
sudo yum install gcc-gfortran

An easy way to try on a brand new CentOS is with docker:
docker run -it centos:6 bash

A strange thing is that one of the libraries seems to #define FLOAT, which breaks the compilation of the OpenCV headers in Caffe, so it is necessary to modify caffe/examples/cpp_classification/classification.cpp to add #undef FLOAT before the opencv headers (line 2).

@saudet
Copy link
Member

saudet commented Aug 8, 2015

Looking good! Thanks a lot! Though, if you look under the ffmpeg module, I keep the archive download and unarchiving separate from the build phase, because we can basically reuse the same source for all platforms. I'll probably check myself linux-x86 and macosx eventually if I have to, but organizing the script file in advance would help. Anyway, let me know when you have this tested and I'll merge what you have! Thanks

And to fix any compile errors we can also include patch files, but it's better if we figure out a way, for example, to not compile the examples. :)

@bhack
Copy link

bhack commented Aug 14, 2015

How do you think we will handle dependencies modularity after BVLC/caffe#2523 and BVLC/caffe#2619 (android)?

@cypof
Copy link
Contributor Author

cypof commented Aug 14, 2015

@bhack Not sure, but if it is easy to embed them all in the jar that should do it for now.

@bhack
Copy link

bhack commented Aug 14, 2015

Minimizing dependencies would be a great thing for apk size and forward pass only use in android. So the problem it is about how to compile preset with the minimal dependencies we want.

@cypof
Copy link
Contributor Author

cypof commented Aug 15, 2015

Ah OK, we should import the Makefile.config options from cppbuild.sh, and only build and package the corresponding libraries.

@bhack
Copy link

bhack commented Aug 15, 2015

Yes or pilot the relative cmake flags. One positive thing of Cmake it is that support external git dependencies. We could add that in caffe directly.

@cypof
Copy link
Contributor Author

cypof commented Aug 15, 2015

Interesting, I will look at this.

@cypof
Copy link
Contributor Author

cypof commented Aug 15, 2015

@saudet I refactored things to be closer to ffmpeg, and added Mac support. It's not finished but most of it should be there.

@bhack
Copy link

bhack commented Aug 15, 2015

I really hope that we can remove boost dependencies with BVLC/caffe#2537

@saudet
Copy link
Member

saudet commented Aug 16, 2015

Great, thanks @cypof! A couple of questions though. I understand that Caffe's build doesn't have a proper "make install", but for the other libraries, if we configure the builds with a prefix to, say, our local cppbuild directory, and run make install, that might neatly place all header files under include and the shared libraries in lib. Have you tested that and found it did not work properly?

And, just for reference, what are the issues with building HDF5 from source on Linux?

@cypof
Copy link
Contributor Author

cypof commented Aug 16, 2015

I tried to avoid make install because for many libraries it also installs the static one (.a), and symbolic links to the library version (like .so.8.0) which might not be handled correctly when unzipping the jar. Also some file systems have problems with links, like when virtualbox mounts a host folder in a VM. That's useful e.g. for when running docker.

@saudet
Copy link
Member

saudet commented Aug 17, 2015

The Builder of JavaCPP already takes care of not bundling symlinks, and the shared librairies are used by default on Linux, so there shouldn't be any problems using standard 'make install'...

@saudet
Copy link
Member

saudet commented Sep 19, 2015

So, would you have any more recent changes to the build script? Or should I go, merge this, and polish it on my end?

@cypof
Copy link
Contributor Author

cypof commented Sep 19, 2015

I will try to go back to that this weekend, or maybe next week. I have something a bit better.

@saudet
Copy link
Member

saudet commented Sep 22, 2015

libcaffe.so doesn't link with the static libraries and the dynamic ones don't get included in the JAR file, but otherwise seems to be working fine on my machine so I'll merge and finish the work on my end :) Thanks!!

saudet added a commit that referenced this pull request Sep 22, 2015
Build Caffe with dependencies included
@saudet saudet merged commit e5fb064 into bytedeco:master Sep 22, 2015
@cypof
Copy link
Contributor Author

cypof commented Sep 22, 2015

Thanks. After playing a bit with it I feel it would be even better to separate the dependencies and caffe itself in different jars. This way we could use maven to download the dependencies and work on our own version of caffe without to worry about which version of library gets loaded, between the maven one and the locally compiled. It would also be great with the include-files-in-a-jar stuff I am working on. Over time the dependencies library might turn into full fledged presets.

@saudet
Copy link
Member

saudet commented Sep 23, 2015

Yes, lots of stuff to be done :)

saudet added a commit that referenced this pull request Sep 26, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants