Teamcity for scientific computing
Hi All,
I have recently started looking at ways to bring ideas and principles from the world of industry software development, and especially DevOps, into the world of scientific computing. Among other things, I am looking for ways to harness the principles of CI/CD for the benefit of computing-based research loops, and I wonder if anyone has already gone in this direction.
The idea in principle is to have researches run small models on their own local machines, perhaps while using an IDE like PyeCharm, etc., and then, when they are ready to submit computing jobs to larger facilities (HPC, HTC, Grid networks, etc.), do so but actually committing their code to an appropriate GIT repository. I guess you can see how something like Teamcity may be useful for implementing this mode of working.
So if anyone can point me to some useful materials that will save me from re-inventing the wheel, I’d appreciate that very much.
Best,
Oren Shani
Please sign in to leave a comment.
Hi,
this is a very interesting scenario. I have to admit I'm not aware of any products that do this task specifically, although if the computing services offer APIs for pushing jobs, it should be relatively easy to automate it. TeamCity would already allow you to automate most of that process by picking up the commits to your jobs, running any tests you might want to run in a build agent (such as for example ensuring there is no major issues with it before sending it), then after validation, you could send it to the grid via their API using a command line runner. In a way this is what TeamCity does for the majority of users, just changing the endpoint of where to push the code/artifacts.
While I'm not aware of any plugin that connects directly to any of this services, another, maybe better option to replace the command line runner would be to create your own runner plugin that picks up the jobs from the working folder and pushes them to the servers.