These are usually stripped out later by the dependency analyzer but this
can't help the correctness of the code (e.g. with option handling, which
depends on there being only one right Module to apply options to).
By moving this code and associated utility methods into Application we
nearly complete the long-in-progress refactoring of the monolithic
kdesrc-build script into proper classes/objects.
The kdesrc-build script itself is nearly bare, but there's still a bit
more to do.
At least it should be easier to make individual test scripts to test
different bugs instead of having a monstrosity of a kdesrc-build-test
script that has to maintain a common state.
I've tested this in each way that I'm able (run needing to make a new
source dir, build dir, new install and log dir, run updating existing
checkouts, bzr/svn/git/kde-projects all tested, etc.). But this was a
large move so let me know if I've missed anything.
This is to support being able to improve and expand the test suite by
(eventually) having a simple function call to setup different modules to
be tested.
This is a long-overdue change that allows for overloading the
stringification operator to produce better error messages.
BuildException was previously not an actual module since kdesrc-build
itself was just a single script file. Now that we allow modules on disk
there is no reason to leave the exception class as a figment of Perl's
imagination.
Before this we would try to apply branch-group to a module, it would
fail (since plain git modules don't *have* any kde-projects metadata),
and we would invariably select master instead of choosing based on the
plain git module's branch option.
Note that this still doesn't solve the problem of modules that are from
kde-projects but *don't* have branch-group metadata; if both module
branch-group and global branch are active for such a module, the
module-specific branch-group will be tried and resolve to 'master'
independent of the global branch setting.
Whether this is a feature or not is not yet certain. :)
Now the priority order for all of the various "I want to checkout this
commit" options are centralized in one spot, and both cloning and
updating obey the same options now (they didn't before).
The spec allows for (or hints at, I forget) entries beginning with a "_"
in the layers and groups keys of the JSON object, so let's make sure we
filter those out just in case someone introduces one.
We need a better name for this feature.
But basically, this allows for creating kde-project repository-wide
"branch groups". For instance, you could list the git branch to use for
the "stable Qt/KDE 4" release/maintenance branch, the "new development
Qt/KDE 4" development branch (typically master, but that has been
changing), and the "upcoming next major release" branch (for us, this
means 'frameworks' sometimes, means 'master' sometimes, other modules
may have different ideas).
By using the 'branch-group' option (which remains to be further
documented) you can say that you want to defer to the metadata about the
repository branches stored in
kde-build-metadata/logical-module-structure. kdesrc-build should even
track branches for you as they are updated by the Release Team, if you
simply want to hang out on stable but compile your own stuff.
Note that this support requires Perl's JSON module (which should come
with Perl 5.14 if I understand it right, but you probably want to
install JSON::XS from CPAN). JSON support is only required for this
feature, it is not a new overall dependency.
See also: http://community.kde.org/Infrastructure/Project_Metadata
The ->isa() checks still work for superclasses, and knowing that a given
module originally came from a kde-project module set is useful in the
code. In fact this should be considered a regression from the module-set
parse refactoring patch.
This is unfortunately a giant change, as all of the functionality that
is encompassed into module-sets currently had to migrate over to
multiple separate classes, including the new ksb::ModuleSet class and
subclasses.
This was a long-overdue change, however, and should allow for accurately
tracking a source module-set for a given module.
On the other hand this migration of logic has made it easier to
understand each of the individual pieces where they stand (e.g. there is
no longer a separate expandXMLModules and expandModuleSets).
In addition we can properly handle ignore-modules with wildcards just as
we do with use-modules (they even use the same matching logic) which
means that it is safe to integrate this into master (assuming no extra
boogs get added, of course).
This will also help with fixing some of the extant module-selection bugs
(321883, 299415).
BUG:321275
CCBUG:321667
Kind of a mouthful, but remove a filter that was added to avoid
automatically including an actual git repository's children that were
also git repositories. E.g. think of something like a SuperBuild git
module that also had its component git modules as logical children
within projects.kde.org.
This filter is unnecessary now that kdesrc-build supports both kde.org
metadata (kde-build-metadata/build-script-ignore) and user-configurable
module filtering (ignore-modules option).
More importantly this filter precludes many other desirable types of
group syntax (e.g. including all of kdebindings just by asking for
kdebindings).
So we remove the filter. The trouble I've seen so far is that it is
now rather difficult to build *only* kdelibs, since kde/kdelibs is a
logical parent of nepomuk-core, nepomuk-widgets, and kactivities. This
can be worked-around by using a normal single module declaration. E.g.
module kdelibs
repository kde:kdelibs
end module
On the other hand, the recently-added config file option
"ignore-modules" should now remove any kde-projects modules that contain
a path component(s) matching the ignored module. You may have to be
specific with a given ignore atom for this reason (e.g. ignore
playground/libs, not 'libs' otherwise you'll also ignore
kdegraphics/libs at this point... though that was also true before this
commit).
BUG:321667
The previous version worked on modules that had *no* user (a setup most
of mine had as the user was set in the ssh config).
For modules with a user set the trailing '@' was included in the uid
which make the check fail even when the module was correct. (Though I
had thought I had tested this case with other modules).
In the past some KDE SVN modules used the "svn external" feature to pull
in other SVN repositories. These svn-external links were on the *server*
side so the link URL had to use a single scheme.
This scheme was https, which caused quite some problems for anonsvn
users just trying to update from SVN being an unattended kdesrc-build
run, as the SSL cert for svn.kde.org was not signed by a well-known CA.
As a result kdesrc-build updated svn automatically with the expected KDE
SVN SSL signature to avoid blocking on the interactive warning.
This is no longer required. The modules that used svn externals have
pretty much moved onto git now. The new svn server does not even have an
https interface, so this is dead code now, and removed accordingly.
Once I figure out a "preferred" solution for this use case I'll probably
convert to that, until then no reason to pollute with warnings.
I've run a search for other uses of given/when and didn't see any.
As opposed to reading /all/ output for a given module and only
displaying it immediately before the build phase starts. This makes it
very disconcerting to track git-clone progress...
Though this only works for Git modules, it should at least fix the
annoying "OMG YOUR REPO MOVED" warnings that never get fixed when in
async mode (which is the default).
Previously you simply never saw the warning due to the now-fixed output
squelching, now we need to fix this for good as well.
In synchronous mode it was impossible to get the build subroutine to
attempt to build a module which had no source update, as the updater
code would filter out its build phase *in addition* to marking it as
up-to-date.
This never broke the async code as filtering out the build phase was
always unimplemented (which is what led to a specific IPC type for "code
up-to-date" in the first place).
The build thread owns the TTY in the current setup. To keep the update
and build processess from fighting over the TTY when in async mode,
we've been simply squelching the update output up to this point, except
for errors.
This is not really the best idea as it can leave out kind of important
details. With this change we configure the logging subsystem to forward
its messages that it would ordinarily print out to instead go over an
IPC channel.
The IPC channel knows which module is currently being operated on, tags
the message, and forwards it over to destination proc's IPC handler,
which extracts it and holds onto the message until that proc reaches the
same module.
Once the receiver proc is processing the module, those messages are all
output at once and then normal processing resumes.
In case some messages are never read out this way (e.g. a module is
updated but never built) we have a message scavenger right after the
build is done to give those messages as well.
It's kind of annoying to be unable to build all modules in a certain
kde-projects module grouping, *except* for some certain chosen ones.
Now you can: You can add the ignore-modules option in your module-sets
using kde-projects repository.
It accepts a space-separated list of module names (either kdefoo or
kde/module/kdefoo style) to ignore.
Although the intention is to only filter out matching modules in *that*
module-set, it would actually perform the filtering even if you
accidentally put it in the wrong module set due to a fluke of the
implementation (I recommend not relying on this, however).
The documentation has also been updated, including an example of the
usage.
For whatever dumb reason I coded the "ignored kde-projects modules"
handling (kde-build-metadata/build-script-ignore) by having the code
that reads in the list of ignored modules go and immediately update the
build context.
For obvious reasons this only works when the update is actually
performed, which means ignore information wasn't used in pretend mode.
We fix this by moving the build context update to the rest of the
metadata handling.
This commit removes the 'megaclass' code that IPC::Pipe has become to
support two different dataflow paths in a single class.
Instead, we use up to 2 different IPC objects in async mode to give the
dataflow needed.
Splitting this up required moving the message boundary detection code
back to IPC::Pipe. It is now expected that a given IPC subclass is able
to pass messages with message boundaries intact, and to block while
waiting for messages if concurrency is supported. IPC::Null already
maintains message boundaries and doesn't support concurrency.
Previously IPC and IPC::Pipe cooperated to perform message boundary
detection, by inserting and removing '\n' characters. This was always
kind of a hack. Now we simply pass the message length across the pipe
followed by the message, and ignore the idea completely in IPC.
The impetus for all of this had been to improve performance, but
unfortunately that hasn't happened here, although it's still a positive
change from the code simplicity front.
The most noticeable effect is probably that it should be damn near
impossible not to have automoc build first now, at least in comparison
to other kde-projects modules. It also should take care of the vast
majority of the code needed for bug 312324. Still needed is handling for
Qt/Qt5.
BUG:312324
FIXED-IN:1.16
We now read in and should be able to handle branch-specific dependencies
and module dependency exclusions.
For full support we still need to find a way to properly handle Qt vs.
Qt5 modules (e.g. for phonon Five and automoc).
The last commit's first attempt had an error where the kde/foo/* syntax
would be stripped to '*', which the KDEXMLReader actually understands
perfectly. Unfortunately a bare '*' reads in all modules, including
websites and other unpleasant things.
I only caught this by chance, so add a sanity check to make this an
error in case I miss it in the future.
This should fix the big ugly error messages for this particular
scenario. The code in question is trying to determine a good branch name
that it can use, which means it need to verify that the requested branch
name *doesn't* exist. This would correspond to an exit code of 1 for the
given git command.
In pretend mode all safe_system() calls give an exit code of 0, so it
appears that there are no safe unused branch names.
Since even in pretend mode the source directory should exist in this
code path I switched to use a system() call (with the correct full
branch name to boot).
I'm not sure the other part of the bug is fixed (i.e. checking out a
local branch immediately after cloning or whatever dfaure was seeing) so
I'll leave open for now until I can do further testing.
CCBUG:311758
Michael Pyne