How to build python packages with twine and wheel

Answered

I'm trying to build A python package and then push to AWS Code Artifact. I do not understand how to run the commands within Teamcity to accomplish this. To run on a normal command line I would perform the following steps (assuming python is already installed)

pip install wheel twine
python setup.py sdist bdist_wheel
twine upload --repository codeartifact dist/*

I guess the first 2 steps can be done through the runner type python, but I have not figured out how to work with installing packages or using virtualenv inside Teamcity yet. The third step I have no idea how to run because it's not a python runner but a runner type of command line. Python is not installed on that command line. So how do I run twine inside Teamcity?

10 comments
Comment actions Permalink

If you can run all of your commands from a command-line, you should be able to run all of the commands as a command-line build step in TeamCity. However, it would make sense to use virtual environments in most cases, so you may want to add the commands to create or switch to a venv you wish to use. I would suggest something like:

python -m venv /mybuildenv/
source /mybuildenv/bin/activate
pip install wheel twine
python setup.py sdist bdist_wheel
twine upload --repository codeartifact dist/*
deactivate

The Python Runner is a new recent addition to TeamCity. I am not aware that it provides support for wheel or twine, but I could be wrong. From what I understand, it is mostly useful for running python scripts and some testing tools. For the details on the Python Runner, please refer to https://www.jetbrains.com/help/teamcity/python.html.

0
Comment actions Permalink

When I run python --version via the command line runner type I get an error of python not found. So I don't see how this solution works.

0
Comment actions Permalink

It sounds like python isn't installed on your build agent. In order to run python commands, you will need to have python installed on your build agent. The command line build step will be executed on the build agent, so any software or build tools used in the build will need to be installed on the build agent.

0
Comment actions Permalink

Python is installed on the host machine but I have TeamCity running inside docker containers. I can try to figure out how to get TeamCity able to access python on the host machine.

0
Comment actions Permalink

Do you mean you're running your build agent within a docker container? You can add python to the docker image you're using in order to use it in your builds.

0
Comment actions Permalink

I edited my earlier comment. I wrote that terribly, sorry. Yes, my build agents are in docker containers. I'm running the TeamCity Server and build agents within a docker swarm.

0
Comment actions Permalink

If you're using the jetbrains/teamcity-agent image, you can customize it by installing Python. 

Customization
You can customize the image via the usual Docker procedure:

  • Run the image
docker run -it -e SERVER_URL="<url to TeamCity server>"  \ 
 -v <path to agent config folder>:/data/teamcity_agent/conf  \
 --name="my-customized-agent"  \
 jetbrains/teamcity-minimal-agent  \

  • Enter the container
docker exec -it my-customized-agent bash

  • Change whatever you need (Install Python)
  • Exit and create a new image from the container:
docker commit my-customized-agent <the registry where you what to store the image>

https://hub.docker.com/r/jetbrains/teamcity-agent

1
Comment actions Permalink

I just installed python on the docker container, thank you for the information! Will I have to do this every time I update to the latest team city docker version?

For those who read this in the future to install python you need to log into the docker container as root and run apt-get update before trying to install python because that will install the build-essential package.

I will need to access the setup.py from my repository and then upload the the generated files. What location are the repo files pulled into on the agent? I have the repo connected via Version Control Settings already.

0
Comment actions Permalink

Will I have to do this every time I update to the latest team city docker version?

Well, you would need to do it anytime you wish to update the base image or make any changes to the environment. It may be easier to use a dockerfile to make your image so you don't have to rebuild it by hand every time. Then you could add an extra build configuration to automate it using the Docker Runner to build the image with the dockerfile.

If you go this far down the rabbit hole, you may also want to explore using the Docker Wrapper to run your commands within a specific docker image while the agent runs on the docker host or in another container with 'docker-in-docker'. This would save you from having to rebuild and administer the agent every time. For example, you could have a build configuration or build step to produce a docker image with python installed, then use the Docker Wrapper with this image in your command line step to run your python script. 

I will need to access the setup.py from my repository and then upload the the generated files. What location are the repo files pulled into on the agent? I have the repo connected via Version Control Settings already.

The files are pulled into the build checkout directory. The checkout directory is configured in the Checkout Settings section on the Version Control Settings page; the default Auto (recommended) value is strongly advised, but it is also possible to configure a custom checkout directory.

1
Comment actions Permalink

Yea, I think the best thing to do is create a custom Dockerfile for it.  And thanks for the info on running docker wrapper. I might have to use that soon even if not for python. My next goal is to run Terraform from Teamcity. I will definitely at least experiment with that docker wrapper here.

0

Please sign in to leave a comment.