cp1252 encoding when running python via Teamcity
Hi :)
We have some weird behaviour when running python via teamcity vs on a console (on the agent machine):
When running a python-build step via Teamcity the encoding is always cp1252, running the same command in a terminal shows utf-8.
Simply executing a python step with the script content:
import sys
print(sys.stdout.encoding)
print(sys.stdin.encoding)
Will result in the output of “utf-8”, “utf-8” in a console but show “cp1252”, “cp1252” when executed via a build agent.
Running the script on the agent (but not through teamcity) through a java ProcessBuilder also results in utf-8.
So far we've tried setting teamcity.runner.commandline.stdstreams.encoding to UTF-8, and launching the teamcity agent with -Dfile.encoding=utf-8 -Dconsole.encoding=utf-8 without success.
I'm running out of ideas where I could still look/investigate further, any help would be appreciated.
Thanks!
Please sign in to leave a comment.
Using the
```
sys.stdout.reconfigure(encoding='utf-8')
sys.stdin.reconfigure(encoding='utf-8')
```
will make sure to use utf-8 in your script no matter the environment.
Please let me know if you have additional questions.
Best regards,
Anton
Hey Anton,
I appreciate the response, though the approach using reconfigure is suboptimal (not supported in all python versions, having to modify the scripts themselves).
I found a way to deal with this issue by configuring the environment parameter: PYTHONIOENCODING to “utf-8:surrogateescape”, I'd still be interested in how teamcity runs the subprocesses, which causes the difference in encoding when running scripts via teamcity vs. running them in a command line. So if you have any documentation on that I'd be thankful.
Thanks for the help!
Quick update: While python no longer raises exceptions and sys.stdout.encoding is now set to utf-8, the teamcity logs now show ? instead of non-ascii characters.
I've tried solutions that you can find on the internet, so far without success (the ones found here: https://stackoverflow.com/questions/9213241/teamcity-build-log)
I know that the solution I proposed is suboptimal, but I tested it, and it works for non-ASCII characters output. Will it work for you?
Best regards,
Anton
Hi Anton,
yes it works when reconfiguring stdin/stdout. Unfortunately that means that we would have to modify potentially many scripts but it's definitely better than not having a working solution :)
Also setting the following parameters either via env. param or as regular system environment variable has the same effect as reconfiguring stdout/stdin (so our issue is solved):
param("env.PYTHONIOENCODING","UTF8")
param("env.PYTHONLEGACYWINDOWSSTDIO","True")
Thanks for the support!
Good to know. Thank you!
Best regards,
Anton