I've done quite a bit of talking about how taskbot could be flexible but a lot of it has been rather abstract and a more concrete example would be helpful.
The Task
To step through this, I'm going to use one of the simpler tests from the existing PUATP suite that is currently running in AutoQA: rpmlint.
As described by the upstream project, "rpmlint is a tool for checking common errors in rpm packages.". It is rather straight-forward in terms of execution and scheduling:
- rpmlint is run for every build done in koji
- it is run on only one package at a time and is independent of other updates/packages
- rpmlint's results are the lines printed to stdout
There is a bit of complexity in which rpmlint warnings/errors constitute a failure but for the sake of simplicity, I'm assuming the following for the sake of discussion:
- PASS - no errors or warnings from rpmlint
- WARN - one or more warnings in the rpmlint output but no errors
- FAIL - one or more errors exist within the output from rpmlint
AutoQA Execution Process
AutoQA's execution process is currently as follows:
Scheduling
A script is run through cron on the autoqa master server - this script looks for any new builds in koji or updates in bodhi since the last time it was run. If there are new builds in koji which are queued for testing, the watcher script creates an autotest job to run rpmlint on the rpms contained in that build.
Delegation
Once the rpmlint job has been scheduled in autotest, it will eventually be queued for execution on one of the client machines (a mix of bare metal and VMs).
Execution
The following is all executed and done on the test client but is coordinated by the master.
Initialization
The client pulls in files from a specific place on the master - these files contain
- any tests not distributed with upstream autotest
- any libraries or executables that are needed for testing
Once the workspace has been initialized, the arguments are read and the rpmlint test (not the executable) is started.
Preparation
In order to run rpmlint, we need an rpm to run it against, so we download the rpm from koji that corresponds to the envr passed in as an argument. The code to do this is in the AutoQA library.
Execution
The test executes rpmlint via subprocess, capturing the output and exit code from the execution.
Analysis and Reporting
In the case of rpmlint, AutoQA doesn't do any reporting unless the package maintainer has opted into receiving results via email. However, html logs are generated for every test run.
As this doesn't provide a great point for comparison, the options for reporting in AutoQA are
- Leave a comment on the relevant bodhi update (when scheduled against an update instead of a koji build)
- Update or create an entry in ResultsDB via XMLRPC
Both of these operations use the AutoQA library downloaded from the master.
Cleanup
The logs are shipped off to the autotest master before the test is complete and the autotest master is notified that this client is ready for another test.
Taskbot's Execution Process
Taskbot is similar to AutoQA in how it approaches the test but there are some important differences in a few of the details.
Scheduling
When koji emits a message that a build has been completed, the rules engine would submit a new task to the execution engine via its' restful interface with the following contents:
- TASKNAME : rpmlint
- ARGUMENTS: build-to-test-1.2.3-4.fc99
Delegation
Once scheduled, the task execution engine would delegate the task to a buildslave which does the actual work of the test.
Execution
Once we get into execution, it becomes a little more difficult to describe exactly what would happen in taskbot in general terms. By design, this part is very flexible, allowing for multiple test runners or tasks which wouldn't traditionally be defined as tests.
What I have here is how I've started to implement rpmlint but it code isn't finished and I suspect that things will change over time.
The slave starts execution of the task, following a set of steps as defined by the master:
Initialization
Update the test libraries (equivalent to the AutoQA library) using yum or pip.
- The libraries needed for testing would be installed/updated at this time but the process of changing these libraries would be independent of the system as a whole.
Clone the 'rpmlint' task repository from a known central git server; similar to how package building works
- Since we have control over this repository, we can safely assume that it obeys a certain format with regard to the names and purposes of executable files (python, bash, perl etc.)
Run the 'INIT' executable in the cloned git repo to do other updating and pre-test preparation.
- In this case, we don't need to do anything as the rpmlint test is pretty simple
Preparation
Running the 'PREPARE' executable in the cloned git repository, the envr passed in as an argument to this test would be used to lookup and download the matching rpm from koji to ./rpms
Execution
Running the 'RUNTEST' executable, rpmlint would be run against all of the rpms in ./rpms, keeping track of which output goes with which rpm. Output would be saved in a known location, probably to disk as a semi-formatted text file
Analysis
Running the 'ANALYSIS' executable, the output from execution would be read and the result would be determined.
Reporting
Using the 'REPORT' executable, a comment would be made on the update and the test result would be submitted to a ResultsDB-like system using a RESTful interface and json.
Cleanup
Using the 'CLEANUP' executable, any downloaded rpms and intermediate files would be purged from the system.
Differences
To summarize what I see as the important differences between the two executions as described above, we have a nifty table:
| Item | AutoQA | Taskbot |
|---|---|---|
| Core Library Updates | Updated on the master with a system update | Updated via yum prior to test execution, completely independent of the system as a whole |
| Test Library Updates | Updated on the master with a system update | Updated during initialization using any number of methods, independent of the system as a whole |
| Test updates | Updated on the master with a system update | Git head is pulled in at the start of test execution, git repo updates are 100% independent of the rest of the system |
| Non-Python Executables | Can be done as a subprocess, currently no non-python support in AutoQA library | Executables can be in any supported language as long as their results are compatible with taskbot |
Note that initial library support for Taskbot is likely to be python only. While a main goal is to support the usage of many languages during tasks, limiting initial library support is a way to keep the scope in check.
Conclusion
I hope that a more concrete example helps to illustrate more of why I'm proposing taskbot instead of just trying to incrementally improve AutoQA.
Yes, I did gloss over some of the complexities and details and I know that there are a few conceptual holes in what I'm proposing but keep in mind - it's not done yet and it isn't final; the description is a proof of concept so that we can talk about more than abstract, un-implemented ideas.
As usual, please send suggestions in the comments here or start/continue a thread on qa-devel.