(irc log from june 2nd 2008)

<eMBee> good evening
<eMBee> is anyone here aware of this project: http://vcs-pkg.org/ ?
<Tybstar> eMBee: someone should just tell them about Conary!
<eMBee> they are potentially interested in using conary
<Tybstar> nice
<eMBee> someone from conary should join and answer their questions
<eMBee> nobody there currently appears to know anything about it
<eMBee> the main issue appears to be that they need to retain the ability to make debian or rpm packages
<eMBee> i think this should be possible
<eMBee> since this is a cross-distribution effort it would be great if some conary experts could join and contribute
<Tybstar> hmm
<Tybstar> we did talk about this a long time ago in some form. i think that we could generate debs…
<Tybstar> at least produce a ccs2deb converter
<Tybstar> that would be a lot of work
<eMBee> why?
<eMBee> don’t you just need to unpack the files and repack them?
<Tybstar> not really…there are a lot of other things (file tags, scripts to run, etc) that would have to be handled
<Tybstar> it would actually be easier to convert deb to ccs
<Tybstar> as i understand it
<Tybstar> i’m not terribly familiar wtih the inner workings of deb
<Tybstar> sorry, gotta run
<Tybstar> i’ll bookmark that page
<eMBee> join the list
<eMBee> it would be great if rpath would support this effort
<eMBee> if all the major distributions use conary as a backend then we could access all their packages in our conary based systems
<eMBee> and while it would not unify the package managers it would at least unify all the package sources, and that would be a huge step ahead in the whole linux distribution world
<gxti> adapting conary as a backend for arbitrary package managers is a bit of a long shot… it’s quite possible to convert other distributions’ packages to conary format, and probably the other way around, but that requires lots of special casing and hard work
<eMBee> what about using conary as a source repository for other package formats?
<gxti> that’s what i mean
<gxti> you can’t store arbitrary things; it already has to be in conary’s format
<eMBee> what do you mean with that?
<eMBee> i am not talking abouit storing a debian source package in conary, but telling debian to pull the source from conary before it builds the binary
<eMBee> that should not be hard
<eMBee> not harder than telling deb to pull from git
<gxti> you could use conary to do that, but it seems like a very odd thing to do
<eMBee> why would it be odd?
<eMBee> it would not be any more odd than using git for the same reason
<gxti> git is a much more powerful source manager than conary is
<gxti> and if you’re only going to use it for sources, i don’t see why you would pick conary
<eMBee> ok
<eMBee> could conary instead use git as a source backend?
<gxti> no
<eMBee> why not?
<gxti> conary’s source control works because the binaries are tightly bound to the sources - you can tell where the source came from just by looking at the version
<eMBee> ahh
<eMBee> now i see where this is getting at
<gxti> using conary for source control for another distro that isn’t using conary just means that you’re using it like a weak VCS
<eMBee> yes
<gxti> it’d be possible, but very silly
<eMBee> ok
<gxti> now i imagine that with factories, you actually could conjure up something where you manage recipes in git
<gxti> that would be neat to try
<eMBee> factories?
<gxti> a relatively new feature where you can programatically create recipes
<gxti> especially handy for “building” RPMs, JARs, WARs, etc into a conary repo
<gxti> but you can use it for regular building too
<eMBee> ok, back to your ealier comment, you said the source control works because the binaries are closely tied to the source
<eMBee> can you elaborate that please?
<gxti> conary’s “thing” is that when you build a trove, you build it from a particular source version, and the binaries you produce have a version derived from the version of those sources
<gxti> so you can always trace back from the binaries to the sources they were built from
<gxti> so when you build the sources with this version:
<gxti> /conary.rpath.com@rpl:devel//2/2.9.10-2
<gxti> you end up with binaries with this version:
<gxti> /conary.rpath.com@rpl:devel//2/2.9.10-2-0.1
<gxti> note the -0.1 on the end
<gxti> so given any binary, you can find its source just by stripping off that binary count
<eMBee> ok, how does that prevent conary from storing the source in git?
<eMBee> you could just use that source version as a tag
<gxti> the source might live in git, but ultimately it has to end up in the conary repository or conary cannot build it
<gxti> so it’s quite possible that you could write a factory that loads a recipe out of git
<eMBee> how does ‘ending up in the conary repository’ look like?
<gxti> well, your typical recipe is a recipe file that references some tarball on a remote server via addArchive, right?
<gxti> when you do ‘cvc commit’, cvc fetches that tarball, and commits the tarball and the recipe file to the conary repo
<eMBee> yes
<gxti> which means that it creates a foobar:source trove with those two files in it
<gxti> if you use addGitSnapshot, the same thing happens, only conary has to make the tarball first
<gxti> so it does a git clone/pull/whatever-git-does-i-cant-remember, makes a tarball of the latest revision, and commits that, and uses it exactly like addArchive from then on
<gxti> so the actual build has nothing to do with git; it’s just unpacking a tarball like it would with anything else
<eMBee> that’s how it would work today, but what if conary replaced its source storage with git? then it would not need to copy the source. and if a recipe used addArchive it would instead push the sources into git
<gxti> but it can’t just replace its “source storage” - that’s half the appeal of conary in the first place, that the binary can be directly traced to a source trove with a similar version
<gxti> the idea is to gather as many things as possible into the repository such that you can see exactly what was used to build it
<gxti> you can go back and look at the tarball created by addGitSnapshot if there is any doubt as to what was built
<eMBee> that’s the part i don’t get, how does having the source in git prevent you to trace it?
<gxti> and while git certainly has similar concepts (as far as identifying revisions), it’s not bound to the binaries at all
<eMBee> what means bind? other than a version string? git revisions could be tagged with that string
<eMBee> and conary could store the git revision string too, just to be on the safe side
<eMBee> rpath would still host this git repository
<eMBee> the user would not see the difference
<gxti> it would be nearly impossible to replace the source backend of conary… technically infeasible, that is
<gxti> more practical is something like the factory approach i suggested, where the recipes are sourced from git, but ultimately end up in the repo
<gxti> conary just cannnot separate the sources and binaries like you are suggesting
<eMBee> well, i am looking at this from the aspect of integrating the package managment of various distributions
<eMBee> and i don’t dispute your answer, i am just trying to learn a bit more about it so i can actually understand it
<gxti> the factory approach would be sufficient for managing a recipe in git; i think replacing the backend is a mountain’s worth of work
<gxti> for not much purpose
<eMBee> i think having a common source storage for managing packages and patches accross all linux distributions would be a lot of gain
<eMBee> but of course on the conary side this could still be done withaddGitSnapshot
<gxti> well, it would be done at a higher level than that
<gxti> addGitSnapshot is for getting the package’s sources from git
<gxti> you couldn’t get the recipe that way, because you need to put addGitSnapshot in the recipe… a bit of a bootstrapping problem
<eMBee> ah
<gxti> i’m not sure what the project’s goals are as far as unifying distros
<gxti> like if they want to do crazy things like using the same ‘spec file’ for all the targets
<eMBee> as far as i understand the main goal is to unify things like patch mangment since many patches are the same for most distributions (i too copied patches from a debian package to build a conary package)
<eMBee> i guess the goals are still under discussion, which is another reason why it would be good if conary people would participate
<gxti> if you just wanted to keep patches in source control and leave the recipes/specs to each distro, then addGitSnapshot would be sufficient
<Mark__T_mib> eMBee: you don’t have to unify patches
<Mark__T_mib> it’s more important to bring them upstream
<gxti> Mark__T_mib: ideally, yes
<eMBee> sometimes this is not possible
<gxti> Mark__T_mib: realistically, it’s not going to happen
<Mark__T_mib> too often it just doesn’t happen, because upstream developers don’t know of the patches
<eMBee> i think in a case where most distributions do use the same patch, it would be great if that were obvious, so that if someone improves a patch, everyone can benefit
<eMBee> having a common patch managment would help pushing pathces upstream since then there is only one place for upstream to find patches
<eMBee> it will be easier to see which paches are applied in which distribution
<eMBee> everyone would still be free to branch off and do whatever they want
<Mark__T_mib> I think having one place for patches is almost as impossible as having all patches applied upstream, though
<eMBee> why do you think that?
<gxti> it’s a noble goal; it certainly makes it more obvious to upstream what they are lacking
<jtate> a good branching VCS could make it relatively easy to be a patch dumping ground
* dugan: thinks it would be cool if cooking could generate different package types.
<Mark__T_mib> oops need to go…
<Mark__T_mib> cya
* LanceHaig: agrees with dugan
* eMBee nods. lets go back to that topic: what would be involved to build other packages from conary?
<gxti> dugan: yeah, and that sounds much more plausible than switching out the backend
<gxti> although there’s certainly no lack of challenges
<LanceHaig> it would be cool to have foresight in your chosen package style
<eMBee> challenges?
<gxti> yeah, like translating tag handlers
<eMBee> oh, you mean there are channelges to generate other package formats, sure
<gxti> tag handlers are typically provided by different packages but i imagine that most other package managers don’t work that way, so you’d have to copy them
<eMBee> what are tag handlers?
<gxti> scripts that run after files are installed or modified
<gxti> for example, when you install a library, ldconfig is run automatically
<gxti> or when you install a font, the font listing in that directory is regenerated
<eMBee> so there is a seperate package to provides the handler? and in the actual installed package there is just a tag that triggers the handler?
<gxti> yes, although iirc you can provide a handler in the same package that uses it
<eMBee> well, we are talking about building other package formats so this would need to handle the case where the handler is not present
<gxti> i imagine that most of the time you would not want to use the taghandlers conary is aware of anyway
<gxti> for example, our initscript handler is useless on systems not using redhat initscripts
<eMBee> well, it would not be useless on fedora/redhat
<eMBee> also i don’t think conary should be tied to redhat init scripts
<so_solid_moo> aren’t redhat moving to upstart scripts?
<gxti> so_solid_moo: i don’t keep up with redhat, but rpl3 is going to have upstart, and since we loosely follow redhat’s initscripts, it’s quite possible
<eMBee> back to the other topic: how are binaries stored in the conary repository?
<gxti> eMBee: when you commit a changeset, everything except the raw file contents is stored into the database, and the files go into disk in a content store
<eMBee> everything covers? the recipe? scripts? versions? dependencies?
<gxti> yes
<gxti> well, the recipe is a file, but the :souce component goes into the DB too, when you did ‘cvc commit’
<gxti> and the :source component references the recipe
<gxti> (the recipe ultimately goes into the content store)
<eMBee> the source tar file should go into the content store too, right?
<gxti> yes, as part of the :source component
<eMBee> ok, so, for most package formats the source tarball, and the binary files would be identical (assuming the same build options are used)
<eMBee> the difference would only be the rest
<gxti> well, the sources and binaries will be different depending on how the package builds
<eMBee> right
<gxti> even for something like a pure python library where the files just get copied, the source will have something like foobar.tar.bz2 while the binary will have lots of individual files in /usr/lib/python2.4/site-packages/…
<eMBee> i mean if the package builds identical then the result would be too
<gxti> yes; if the source is unchanged, then a rebuild will generally produce the same binaries
<gxti> it can be affected by what is on the build system too (e.g. a new python)
<gxti> that’s another goal of conary; since you can trace the binaries back to the sources, then you can always rebuild the sources and get similar binaries
<eMBee> i am now extending that to different package formats. with rthe same source and the same build instructions (and the same versions of build dependencies) the binaries should still be the same
<gxti> yes
<eMBee> the differences would be that debian, redhat and conary have different scripts
<eMBee> so all it would take is to store those scripts?
<eMBee> and of course unify the build process
<gxti> you could theoretically write a rather simple program that builds a recipe into a changeset, then converts the changeset into a deb or whatever
<gxti> the trick is all the distro-specific stuff, like scripts
<gxti> but i’m not really convinced that that’s what the project needs
<eMBee> well, i am tryung to explore the feasability of using conary as a backend for deb and rpm packages
<gxti> and i’m still not sure it’s a great idea :P
<eMBee> i am not saying it is one
<gxti> certainly an interesting one
<eMBee> but if it is not a great idea, then i am trying to find out why
<gxti> it might very well be worth it to use conary’s build tools, but i don’t know how well the other distribution’s communities will appreciate that
* dugan: only thinks its interesting because I think the build system is pretty cool and flexible. If it could be broken out and available as a format for others to use, maybe other package developers would consider it as their format if that’s not the part they’re trying in innovate on.
<LanceHaig> suse have the open build system (or something like that)
<LanceHaig> and it builds for a host of distro’s
<LanceHaig> nothing like conary though
<dugan> yes, but rpm specs suck
<LanceHaig> they do
<LanceHaig> it builds for deb as well:-)
<eMBee> well, one goal of the vcs-pkg project is to get an answer to that. there are people from different distributions participating, and i definetly thing conary should participate as well. whether the actual distributions will accept the result is only a part of the quest
* gxti: needs to look into other distro’s build systems to see what competition conary has :P
<gxti> i’ve used portage (gentoo) but that isn’t a traditional build system
<gxti> it makes no attempt at isolation since its purpose is of course to run directly on the target system
<eMBee> well, one thing that will be hard to compete with for conary is that debian builds on a dozend architectures. at least until conary is ported
<gxti> conary is aware of other architectures, it just hasn’t been used there yet
<gxti> there’s little benefit to the company to maintain rPath Linux, for example, on sparc :P
<eMBee> of course
<gxti> i am, however, working (an overstatement, i haven’t touched it in weeks) on a lightweight embedded distro that might be easier to port
<gxti> but until i get it working on i586, i’m not going to start buying hardware :P
<eMBee> if debian were to use conary as a backend, asll the architectures would need to work
<gxti> conary should have no issues working on other arches. its cross-compiling support is actually very good.
<eMBee> debian would take care of the porting though (assuming someone there was convinced that conary is a god fit)
<eMBee> good even
<gxti> and you don’t even need to worry about running conary on those architectures, because we’re only talking about building with conary, not managing
<eMBee> right
<eMBee> but some parts of conary need to run to manage the build
<gxti> if you want a pure build, yes
<gxti> you could always cross-compile everything :P
* gxti: plans on making his entire distro cross-compileable
* gxti: is utterly insane
<eMBee> i think for some architectures cross compilation may not work
<devnet> eMBee: the worst part about what you’re looking at is that dpkg supports interaction with the user
<devnet> eMBee: conary doesn’t
<devnet> debs almost always have the stupid idea that a user cares what font they use for a simple package
<eMBee> at buildtime?
<devnet> then prompt for user input
<devnet> not at build time, at install time
<eMBee> but that should be irrelevant to using conary for building only
<devnet> should it?
<devnet> I’m not so sure
<eMBee> how does asking the user at install time affect the build?
<devnet> I’m thinking that with things like triggers…how do you effectively employ those when building with conary?
<gxti> conary wouldn’t be responsible for adding the user interaction
<gxti> that has to happen afterwards anyway, because conary doesn’t know how to make .debs
* eMBee nods
<eMBee> the interaction comes from scripts. the debian version of the conary build tool would add those scripts to the repository, and the debian packaging tool would get them back from the repo (asuming that these two parts are seperated when conary is used)
<gxti> apt-get would have no interaction with conary whatsoever
* eMBee nods
* devnet: has no idea…I just know that when talking with the conary guru’s last week about this very subject
<devnet> some of them expressed distaste for even attempting
<gxti> well yes, it’s completely evil
<devnet> mainly due to things like triggers
<gxti> but that’s their thing
<gxti> we just won’t let people do it with conary :P
<devnet> lol
<devnet> :P good plah
<devnet> s/plah/plan
<gxti> plah!
<eMBee> devnet: can you elaborate on that discussion? (was it on irc? do you have a log?)
<devnet> eMBee: I wish I could…but it dealt with some non-public things here at rPath
<eMBee> oh, ok
<eMBee> are there any objections to publishing the recent discussion on deb, rpm, conary integration on the web?
<gxti> feel free
<eMBee> thanks, i plan to worward it to the vcs-pkg list or possibly put it in their wiki