I had a build chain that ended late recently. Looking into it the two root causes were the build chain starting late, and another set of jobs starting late.
Build Chain time line.
3:30 Chron job triggers the root job.
3:32:10-3:32:18 (from tc ui)*. Composite job runs
3:35:39 - 3:36:09 Job1, First non-composite job runs.
1. Why did Job1 start so late? 5:40 after start/3:20 after composite job is really slow.
Job1 could have run on four separate hosts. Agent logs show nothing going on for the other three hosts from 3:32 to about 3:36. I don't know why it started late at 3:35:39.
*Note. I don't completely understand timings on the webui and agent logs. They aren't the same.
Timeline of jobs on agent.
3:34:58. Job2 starts. TC Web UI > Overview
3:34:58. Job2 starts. Starting Build in Agent log.
3:35:38. Job2 ends. TC Web UI > Overview.
3:35:38. Job2 ends. Build finished in Agent log.
3:35:39. Job1 starts. TC Web UI > Overview
3:35:39. Job1 starts. Starting Build in Agent log.
3:36:09. Job1 ends. TC Web UI > Overview
3:36:09. Job1 ends. Build finished in Agent log.
3:36:48. Job2 Build finished. TC Web UI > Build Log
3:37:43. Job1 Build finished. TC Web UI > Build Log
2. I'm not sure why the end times in TC Web UI > Build Log are different than TC Web UI > Overview and the agent log. What's going on here?
3. I've got a set of upload jobs that started late. Why did this upload job start 2-3min late?
Each individual upload client job has a dependency on a specific run client job, the run client jobs are really quick. They also have a shared dependence on a run server job.
Run Server: 03:40 - 04:00. TC Web UI > Overview
Run Server: 04:02:57 : Build finished. TC Web UI > Build log.
Upload Client: 04:02:58 - 04:03:06 TC Web UI > Overview
This makes more sense if Upload Client was dependent on the 4:02:57 Build finished time from the web UI, but I saw Job1 start before Job2 logged that on the web ui. I use reserved agents for Run Server and Run/Upload Client job. Agents should be able to run jobs whenever dependencies are finished.