Running in Docker Swarm can not connect to Remote Database

When running in docker-compose everything works as expected but when running the same file as a swarm teamcity keeps throwing UnknownHostException. I'm having it connect to a remote postgres database hosted by aws (RDS). I've repeated this issue locally and when running on an EC2 instance to verify the issue. Anyone have an idea what is going on here? For reference this is the docker compose file:


version: "3"

services:
 teamcity-server:
 privileged: true
 image: jetbrains/teamcity-server:2020.2.2
 hostname: teamcity-server
 ports:
  - "8111:8111"
  - "5432:5432"
volumes:
 - ./data_dir:/data/teamcity_server/datadir
 - ./log_dir:/opt/teamcity/logs

teamcity-agent:
 privileged: true
 image: jetbrains/teamcity-agent:2020.2.2
 environment:
  - SERVER_URL=http://teamcity-server:8111
  - AGENT_NAME=regular_agent
  - DOCKER_IN_DOCKER=start
volumes:
  - "/var/run/docker.sock:/var/run/docker.sock"

 

teamcity-minimal-agent:
 privileged: true
 image: jetbrains/teamcity-minimal-agent:2020.2.2
 environment:
  - SERVER_URL=http://teamcity-server:8111
  - AGENT_NAME=minimal_agent
  - DOCKER_IN_DOCKER=start
 volumes:
  - "/var/run/docker.sock:/var/run/docker.sock"

8 comments
Comment actions Permalink

You're getting the UnknownHostException when TeamCity server attempts to connect to the database? Are you able to ping the database hostname from within the TeamCity server container?

0
Comment actions Permalink

I did not ping it. I tried to install that capability into the container but  I was having trouble. There is some lock on the apt files to install new packages and there is no suder in the container either. I did get the exact IP address of the RDS database and used that instead of the URL. Which won't work RDS because blocks it and does the DNS resolution itself. It gave me a different error message which I believe gives us more information to the solution.

Could not connect to PostgreSQL server.
the connection attempt failed.: org.postgresql.util.PSQLException: The connection attempt failed. Caused by: java.net.NoRouteToHostException: No route to host (Host unreachable)

I believe this s a response from the RDS's firewall. I might be wrong though. I think this different error suggest that there is something going on with the DNS resolution of the proper URL.

0
Comment actions Permalink

You should be able to install inetutils-ping, but you'll have to connect as root. Try:

docker exec -it -u root < your teamciy server container id > bash

root@teamcity-server:/# apt update
root@teamcity-server:/# apt install inetutils-ping

I'm not sure about your 'No route to host' error. It seems to me that this error is probably the result of the container not resolving the hostname of your database. If you can connect to the same database when running via docker-compose, I would be surprised if the firewall changes behavior when running from a swarm.

0
Comment actions Permalink

The command was a little off but I fixed it.

docker exec -it -u root  id_of_container /bin/bash

I was able to install everything and could not ping the database. The network in swarm is slightly different then in docker-compose. docker-compose uses bridge networks, docker-swarm uses overlay networks. They are very similar. I tried using bridge networks, but that did not work. I also tried running running on the host by adding this to the docker compose file:

network_mode: host

 

0
Comment actions Permalink

When you attempt to ping the database, does it resolve the hostname to an IP address or does it respond with "Unknown Host"?

0
Comment actions Permalink

I tried this all on my local machine and on my ec2 server.

Locally: it will resolve the IP correctly but can't connect because RDS blocks access if not on a vpn. When I connect to the vpn and ping rds again through the docker container it does not resolve the ip with a message of unknown host.

EC2: I can't reach the internet to install inetutils-ping. This might be an issue with the ec2 server itself, but I'm really not sure.

I'm not sure what to make of all this.

0
Comment actions Permalink

If it cannot resolve the hostname to an IP address, it won't be able to reach it. Maybe try adding a specific DNS server entry to the teamcity-server section of docker-compose.yml. There is a really good explanation here: https://stackoverflow.com/questions/44724497/what-is-overlay-network-and-how-does-dns-resolution-work

 teamcity-server:
 privileged: true
 image: jetbrains/teamcity-server:2020.2.2
 hostname: teamcity-server
 ports:
  - "8111:8111"
  - "5432:5432"
volumes:
 - ./data_dir:/data/teamcity_server/datadir
 - ./log_dir:/opt/teamcity/logs
dns: <your dns server>
0
Comment actions Permalink

On my ec2 server I put the dns equal to the VPC dns server and it work!! I set two settings in my docker compose file:

dns: 169.254.169.253
network_mode: host

Here's some info on VPC DNS in case some else read this post: https://docs.aws.amazon.com/vpc/latest/userguide/vpc-dns.html#vpc-dns-support

Do you know why this was an issue at all? I've connected other containers in a docker swarm to RDS without any problems. Why was it issue now?

0

Please sign in to leave a comment.