Status Quo
As Haskellers we live in a world where our tools just keep getting better. See for example the GHC release notes that contains gems like this: Eq and Show are no longer required for Num. GHC releases break backwards compatibility so that they can bring us really valuable updates and improvements.It's not just GHC that is a moving target. Everything in the Haskell community is improving and moving. Hackage gets tons of new packages and updated packages each week. Updates to libraries on Hackage tend to follow in the precedent set by GHC. Breaking API changes are common, and always well intentioned.
The package versioning policy (PVP) helps by allowing us to express our package dependencies in terms of what we expect to be a breaking change and what is meant to be a non-breaking change. Even so, accidental API breakages creep in and some authors fail to follow the PVP due to ignorance or choice.
Problem
The downside to all the wonderful flux is that it's hard to keep your code building in the varied environments that it ends up in. Fixing a deprecation warning on one version of GHC puts you at risk of not building on a previous, but important, release of GHC. For example, in 7.4.1, mkTyCon is now deprecated in favor of the recently added mkTyCon3.
Build with -Werror -Wall and you're even more likely to run into problems. Actually, I highly recommend disabling -Werror before posting to hackage. Feel free to use it for dev builds.
How can each of us be sure that when we release to hackage that our source builds on the most machines possible? How can we as a community make sure that our software is buildable with minimum hassle?
Proposal
I think the only real solution for this problem is testing. I need to try my software in the different environments that I want to support. At a high level, that's Windows, OSX, and Linux. At a more detailed level that includes testing on multiple versions of ghc and trying out different versions of the dependencies from hackage and comparing against my testsuite. That's exactly how we do it at work and it works well.
I think it's time for a community visible build service. Apache offers this for projects under their care. Details about their service are found here.
My current thinking is to combine Hudson or Jenkins, possibly with some form of sandboxing such as sydbox. Do sandboxing tools exist for Windows and OSX? Another approach to sandboxing is to clone a virtual machine and discard it after the build finishes or a timeout occurs.
A completely different approach is to trust the community to not abuse the servers and let anyone apply for an account, with new user vetting. I think this is my preferred stance. If we make it so build machines are disposable and easy to redeploy from scratch, resource usage is monitored or capped per-request, and community members are moderating use, then I believe it should be possible to use the honor system.
Each build machine would have multiple versions of GHC installed. Additional extra-libraries could be petitioned for so that lots of C bindings would be buildable.
Users need to be able to submit projects, and their dependencies, for build testing before pushing the result to hackage.
What else does it need to do?
Call for Volunteers
I could build a private set of build machines for my own use and maintain them, but I don't think I'm alone in wanting this. Who is with me?
What about creating a package on Hackage that can post build results? Then anyone can install this locally to report problems, right?
ReplyDelete@shapr, If I understand your suggestion correctly, it more about reporting when people can and cannot build something that is on hackage? I think that is quite useful but solves a different (related) problem. I want people to have confidence *before* they post to hackage. And I want people to be able to get that confidence during development. So for your package to be useful without the public build infrastructure I think it would require people to build the development version with said tool.
ReplyDeleteDoes that make sense?
That would be awesome indeed! But how feasible is this?
ReplyDeleteAnd if we extend this, we could use the "benchmark" section for cabal files that Johan Tibell has been working on (IIRC) to run benchmarks against different versions of the dependencies and of ghc, letting us spot potential performance bottlenecks, regressions, etc.
This is cool idea, but sounds hard.
ReplyDeleteEspecially building for every possible combination of dependencies.
For example, to be sure about package conduit-0.2.0 for one platform, there must be performed 71280 builds.
Python folks do it with buildbot:
ReplyDeletehttp://www.python.org/dev/buildbot/all/waterfall
Having just been involved in setting up a Jenkins infrastructure at work, this idea definitely crossed my mind too.
ReplyDeleteI suspect it would be best to 'just do it', to see what benefits would arise from the simplest application of the concept, and then iterate from there.
A logical starting point would be the current Haskell Platform, targeting one Linux, Mac and Windows build environment.
The obvious question is what infrastructure is available, and/or how it would be funded.