TeamCity > MySQL replication breaks on large downloaded_artifacts table size

The link to the bug on bugs.mysql.com isn't working.  Here's the full write-up for the issue below. 

Please note, the write-up below was sanitized for security and compliance reasons.  The database being referred to is a teamcity db and the table is the downloaded artifacts table.   

Over the past few days we've attempted to dump/restore/configure
slave replication from our current primary master-db-01 (running
MySQL 5.6) to a new slave: slave-db-01 (running MySQL 5.7).

It is our belief that MIXED style replication from a MySQL 5.6
master to a MySQL 5.7 slave is broken and is silently failing.

It is also our belief that the slave is incorrectly reporting
the health of the SQL thread.

Attmpt 1)

- Restore completed at 19:02
- Seconds behind master reached zero within 40mins
- Seconds behind master started increasing at 23:32

Attempt 2)
- Restore completed at 21:18
- Seconds behind master reached zero within 20mins
- Seconds behind master started increasing at 00:21 (3hrs later)

Both times where the slave replication stopped. We confirmed that the execution
pointer is stalled at the DELETE FROM db1.table1 query:

Here's the command I'm running to view that execution state in
the binary logs (executed on Slave slave-db-01):

```
mysqlbinlog \
--read-from-remote-server \
-u replicator \
-p \
-h master-db-01 \
--base64-output=DECODE-ROWS \
--verbose \
--start-position=${EXEC_MASTER_LOG_POS} \
${RELAY_MASTER_LOG_FILE}
```

Here's an example of one of the DELETE queries we found in the binary logs where the execution pointer was stopped:

```
DELETE FROM `db1`.`table1`
WHERE
@1=6555242
@2=6549166
@3=1524006825925
@4="foo.pom"
```

Here are some thoughts on why it may be taking a long time to delete from the db1.table1 table:

1. Table size is big (38G).
2. Table does not have a PRIMARY KEY

$ mysql --execute="\W; SHOW CREATE TABLE db1.table1\G"
*************************** 1. row ***************************
Table: table1
Create Table: CREATE TABLE `table1` (
`field3` bigint(20) DEFAULT NULL,
`field2` bigint(20) DEFAULT NULL,
`field3` bigint(20) DEFAULT NULL,
`field4` text COLLATE utf8_bin,
KEY `table1_field2` (`field2`),
KEY `table1_field3_field2` (`field3`,`field2`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin

3. The query may not be optimized

mysql> \W; EXPLAIN SELECT field3 FROM db1.table1 WHERE field3=6555242 AND field2=6549166 AND field3=1524006825925 AND field4="foo"\G;
Show warnings enabled.
ERROR:
No query specified

*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: table1
partitions: NULL
type: ref
possible_keys: table1_field2_id,table1_field3_ts_id
key: table1_field2_id
key_len: 9
ref: const
rows: 1
filtered: 5.00
Extra: Using where
1 row in set, 1 warning (0.00 sec)

Note (Code 1003): /* select#1 */ select `db1`.`table1`.`field3`
AS `field3` from `db1`.`table1` where
((`db1`.`table1`.`field4` = 'monkey')
and (`db1`.`table1`.`field3` = 1524006825925)
and (`db1`.`table1`.`field2` = 6549166)
and (`db1`.`table1`.`field3` = 6555242))
ERROR:
No query specified
```

Here are a few similar stories where I found people experiencing a similar issue

https://mariadb.com/kb/en/library/replication-stops-after-one-hour-or-so/
https://dba.stackexchange.com/q/36909
https://stackoverflow.com/q/8547827/86263

Here is the master configuration:

[mysqld]
binlog_format=MIXED
datadir=/var/lib/mysql
expire_logs_days=9
innodb_buffer_pool_size=20G
log_bin
server_id=1
skip-name-resolve
socket=/var/lib/mysql/mysql.sock
sql_mode=NO_ENGINE_SUBSTITUTION,STRICT_TRANS_TABLES
symbolic-links=0
tmpdir=/var/lib/mysql/mysql_tmpdir

[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

Here's the slave configuration:

[mysqld]
bind-address=0.0.0.0
binlog-ignore-db="mysql"
datadir=/var/lib/mysql
expire_logs_days=9
general_log=1
general_log_file=/var/log/mysql/general.log
innodb_buffer_pool_size=20G
log-bin=mysql-bin.log
log_error=/var/log/mysql/err.log
log_queries_not_using_indexes=ON
log_slave_updates=TRUE
log_slow_slave_statements=ON
long_query_time=2
max_allowed_packet=32M
max_connections=2000
replicate_wild_do_table=db1.%
server_id=2
socket=/var/lib/mysql/mysql.sock
sql_mode=NO_ENGINE_SUBSTITUTION,STRICT_TRANS_TABLES
symbolic-links=0
sync_binlog=0

[mysqld_safe]
numa_interleave=1
pid-file=/var/run/mysqld/mysqld.pid

Here's the current slave status (broken state):

Slave_IO_State: Waiting for master to send event
Master_Host: master-db-01
Master_User: <redacted>
Master_Port: <redacted>
Connect_Retry: 60
Master_Log_File: mysqld-bin.001041
Read_Master_Log_Pos: 186867120
Relay_Log_File: slave-db-01-relay-bin.000007
Relay_Log_Pos: 91856648
Relay_Master_Log_File: mysqld-bin.001039
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table: db1.%
Replicate_Wild_Ignore_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 91856450
Relay_Log_Space: 2334354113
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: 36583
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
Replicate_Ignore_Server_Ids:
Master_Server_Id: 1
Master_UUID: b6bf3f5f-3a30-11e5-88f9-025254918c34
Master_Info_File: /var/lib/mysql/master.info
SQL_Delay: 0
SQL_Remaining_Delay: NULL
Slave_SQL_Running_State: Reading event from the relay log
Master_Retry_Count: 86400
Master_Bind:
Last_IO_Error_Timestamp:
Last_SQL_Error_Timestamp:
Master_SSL_Crl:
Master_SSL_Crlpath:
Retrieved_Gtid_Set:
Executed_Gtid_Set:
Auto_Position: 0
Replicate_Rewrite_DB:
Channel_Name:
Master_TLS_Version:

Here is the SHOW GLOBAL VARIABLES output:

Variable_name Value
auto_increment_increment 1
auto_increment_offset 1
autocommit ON
automatic_sp_privileges ON
avoid_temporal_upgrade OFF
back_log 450
basedir /usr/
big_tables OFF
bind_address 0.0.0.0
binlog_cache_size 32768
binlog_checksum CRC32
binlog_direct_non_transactional_updates OFF
binlog_error_action ABORT_SERVER
binlog_format ROW
binlog_group_commit_sync_delay 0
binlog_group_commit_sync_no_delay_count 0
binlog_gtid_simple_recovery ON
binlog_max_flush_queue_time 0
binlog_order_commits ON
binlog_row_image FULL
binlog_rows_query_log_events OFF
binlog_stmt_cache_size 32768
bulk_insert_buffer_size 8388608
character_set_client latin1
character_set_connection latin1
character_set_database latin1
character_set_filesystem binary
character_set_results latin1
character_set_server latin1
character_set_system utf8
character_sets_dir /usr/share/mysql/charsets/
check_proxy_users OFF
collation_connection latin1_swedish_ci
collation_database latin1_swedish_ci
collation_server latin1_swedish_ci
completion_type NO_CHAIN
concurrent_insert AUTO
connect_timeout 10
core_file OFF
datadir /var/lib/mysql/
date_format %Y-%m-%d
datetime_format %Y-%m-%d %H:%i:%s
default_authentication_plugin mysql_native_password
default_password_lifetime 0
default_storage_engine InnoDB
default_tmp_storage_engine InnoDB
default_week_format 0
delay_key_write ON
delayed_insert_limit 100
delayed_insert_timeout 300
delayed_queue_size 1000
disabled_storage_engines
disconnect_on_expired_password ON
div_precision_increment 4
end_markers_in_json OFF
enforce_gtid_consistency OFF
eq_range_index_dive_limit 200
event_scheduler OFF
expire_logs_days 9
explicit_defaults_for_timestamp OFF
flush OFF
flush_time 0
foreign_key_checks ON
ft_boolean_syntax + -><()~*:""&|
ft_max_word_len 84
ft_min_word_len 4
ft_query_expansion_limit 20
ft_stopword_file (built-in)
general_log ON
general_log_file /var/log/mysql/general.log
group_concat_max_len 1024
gtid_executed
gtid_executed_compression_period 1000
gtid_mode OFF
gtid_owned
gtid_purged
have_compress YES
have_dynamic_loading YES
have_geometry YES
have_profiling YES
have_query_cache YES
have_rtree_keys YES
have_statement_timeout YES
have_symlink DISABLED
host_cache_size 703
hostname slave-db-01
ignore_builtin_innodb OFF
ignore_db_dirs
init_connect
init_file
init_slave
innodb_adaptive_flushing ON
innodb_adaptive_flushing_lwm 10
innodb_adaptive_hash_index ON
innodb_adaptive_hash_index_parts 8
innodb_adaptive_max_sleep_delay 150000
innodb_api_bk_commit_interval 5
innodb_api_disable_rowlock OFF
innodb_api_enable_binlog OFF
innodb_api_enable_mdl OFF
innodb_api_trx_level 0
innodb_autoextend_increment 64
innodb_autoinc_lock_mode 1
innodb_buffer_pool_chunk_size 134217728
innodb_buffer_pool_dump_at_shutdown ON
innodb_buffer_pool_dump_now OFF
innodb_buffer_pool_dump_pct 25
innodb_buffer_pool_filename ib_buffer_pool
innodb_buffer_pool_instances 8
innodb_buffer_pool_load_abort OFF
innodb_buffer_pool_load_at_startup ON
innodb_buffer_pool_load_now OFF
innodb_buffer_pool_size 21474836480
innodb_change_buffer_max_size 25
innodb_change_buffering all
innodb_checksum_algorithm crc32
innodb_checksums ON
innodb_cmp_per_index_enabled OFF
innodb_commit_concurrency 0
innodb_compression_failure_threshold_pct 5
innodb_compression_level 6
innodb_compression_pad_pct_max 50
innodb_concurrency_tickets 5000
innodb_data_file_path ibdata1:12M:autoextend
innodb_data_home_dir
innodb_deadlock_detect ON
innodb_default_row_format dynamic
innodb_disable_sort_file_cache OFF
innodb_doublewrite ON
innodb_fast_shutdown 1
innodb_file_format Barracuda
innodb_file_format_check ON
innodb_file_format_max Barracuda
innodb_file_per_table ON
innodb_fill_factor 100
innodb_flush_log_at_timeout 1
innodb_flush_log_at_trx_commit 2
innodb_flush_method
innodb_flush_neighbors 1
innodb_flush_sync ON
innodb_flushing_avg_loops 30
innodb_force_load_corrupted OFF
innodb_force_recovery 0
innodb_ft_aux_table
innodb_ft_cache_size 8000000
innodb_ft_enable_diag_print OFF
innodb_ft_enable_stopword ON
innodb_ft_max_token_size 84
innodb_ft_min_token_size 3
innodb_ft_num_word_optimize 2000
innodb_ft_result_cache_limit 2000000000
innodb_ft_server_stopword_table
innodb_ft_sort_pll_degree 2
innodb_ft_total_cache_size 640000000
innodb_ft_user_stopword_table
innodb_io_capacity 500
innodb_io_capacity_max 2000
innodb_large_prefix ON
innodb_lock_wait_timeout 50
innodb_locks_unsafe_for_binlog OFF
innodb_log_buffer_size 16777216
innodb_log_checksums ON
innodb_log_compressed_pages ON
innodb_log_file_size 50331648
innodb_log_files_in_group 2
innodb_log_group_home_dir ./
innodb_log_write_ahead_size 8192
innodb_lru_scan_depth 1024
innodb_max_dirty_pages_pct 75.000000
innodb_max_dirty_pages_pct_lwm 0.000000
innodb_max_purge_lag 0
innodb_max_purge_lag_delay 0
innodb_max_undo_log_size 1073741824
innodb_monitor_disable
innodb_monitor_enable
innodb_monitor_reset
innodb_monitor_reset_all
innodb_numa_interleave OFF
innodb_old_blocks_pct 37
innodb_old_blocks_time 1000
innodb_online_alter_log_max_size 134217728
innodb_open_files 1495
innodb_optimize_fulltext_only OFF
innodb_page_cleaners 4
innodb_page_size 16384
innodb_print_all_deadlocks OFF
innodb_purge_batch_size 300
innodb_purge_rseg_truncate_frequency 128
innodb_purge_threads 4
innodb_random_read_ahead OFF
innodb_read_ahead_threshold 56
innodb_read_io_threads 4
innodb_read_only OFF
innodb_replication_delay 0
innodb_rollback_on_timeout OFF
innodb_rollback_segments 128
innodb_sort_buffer_size 1048576
innodb_spin_wait_delay 6
innodb_stats_auto_recalc ON
innodb_stats_include_delete_marked OFF
innodb_stats_method nulls_equal
innodb_stats_on_metadata OFF
innodb_stats_persistent ON
innodb_stats_persistent_sample_pages 20
innodb_stats_sample_pages 8
innodb_stats_transient_sample_pages 8
innodb_status_output OFF
innodb_status_output_locks OFF
innodb_strict_mode ON
innodb_support_xa ON
innodb_sync_array_size 1
innodb_sync_spin_loops 30
innodb_table_locks ON
innodb_temp_data_file_path ibtmp1:12M:autoextend
innodb_thread_concurrency 0
innodb_thread_sleep_delay 10000
innodb_tmpdir
innodb_undo_directory ./
innodb_undo_log_truncate OFF
innodb_undo_logs 128
innodb_undo_tablespaces 0
innodb_use_native_aio ON
innodb_version 5.7.21
innodb_write_io_threads 4
interactive_timeout 28800
internal_tmp_disk_storage_engine InnoDB
join_buffer_size 262144
keep_files_on_create OFF
key_buffer_size 8388608
key_cache_age_threshold 300
key_cache_block_size 1024
key_cache_division_limit 100
keyring_operations ON
large_files_support ON
large_page_size 0
large_pages OFF
lc_messages en_US
lc_messages_dir /usr/share/mysql/
lc_time_names en_US
license GPL
local_infile ON
lock_wait_timeout 31536000
locked_in_memory OFF
log_bin ON
log_bin_basename /var/lib/mysql/mysql-bin
log_bin_index /var/lib/mysql/mysql-bin.index
log_bin_trust_function_creators OFF
log_bin_use_v1_row_events OFF
log_builtin_as_identified_by_password OFF
log_error /var/log/mysql/err.log
log_error_verbosity 3
log_output FILE
log_queries_not_using_indexes ON
log_slave_updates ON
log_slow_admin_statements OFF
log_slow_slave_statements ON
log_statements_unsafe_for_binlog ON
log_syslog OFF
log_syslog_facility daemon
log_syslog_include_pid ON
log_syslog_tag
log_throttle_queries_not_using_indexes 0
log_timestamps UTC
log_warnings 2
long_query_time 2.000000
low_priority_updates OFF
lower_case_file_system OFF
lower_case_table_names 0
master_info_repository FILE
master_verify_checksum OFF
max_allowed_packet 33554432
max_binlog_cache_size 18446744073709547520
max_binlog_size 1073741824
max_binlog_stmt_cache_size 18446744073709547520
max_connect_errors 100
max_connections 2000
max_delayed_threads 20
max_digest_length 1024
max_error_count 64
max_execution_time 0
max_heap_table_size 16777216
max_insert_delayed_threads 20
max_join_size 18446744073709551615
max_length_for_sort_data 1024
max_points_in_geometry 65536
max_prepared_stmt_count 16382
max_relay_log_size 0
max_seeks_for_key 18446744073709551615
max_sort_length 1024
max_sp_recursion_depth 0
max_tmp_tables 32
max_user_connections 0
max_write_lock_count 18446744073709551615
metadata_locks_cache_size 1024
metadata_locks_hash_instances 8
min_examined_row_limit 0
multi_range_count 256
myisam_data_pointer_size 6
myisam_max_sort_file_size 9223372036853727232
myisam_mmap_size 18446744073709551615
myisam_recover_options OFF
myisam_repair_threads 1
myisam_sort_buffer_size 8388608
myisam_stats_method nulls_unequal
myisam_use_mmap OFF
mysql_native_password_proxy_users OFF
net_buffer_length 16384
net_read_timeout 30
net_retry_count 10
net_write_timeout 60
new OFF
ngram_token_size 2
offline_mode OFF
old OFF
old_alter_table OFF
old_passwords 0
open_files_limit 5000
optimizer_prune_level 1
optimizer_search_depth 62
optimizer_switch index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,engine_condition_pushdown=on,index_condition_pushdown=on,mrr=on,mrr_cost_based=on,block_nested_loop=on,batched_key_access=off,materialization=on,semijoin=on,loosescan=on,firstmatch=on,duplicateweedout=on,subquery_materialization_cost_based=on,use_index_extensions=on,condition_fanout_filter=on,derived_merge=on
optimizer_trace enabled=off,one_line=off
optimizer_trace_features greedy_search=on,range_optimizer=on,dynamic_range=on,repeated_subselect=on
optimizer_trace_limit 1
optimizer_trace_max_mem_size 16384
optimizer_trace_offset -1
parser_max_mem_size 18446744073709551615
performance_schema ON
performance_schema_accounts_size -1
performance_schema_digests_size 10000
performance_schema_events_stages_history_long_size 10000
performance_schema_events_stages_history_size 10
performance_schema_events_statements_history_long_size 10000
performance_schema_events_statements_history_size 10
performance_schema_events_transactions_history_long_size 10000
performance_schema_events_transactions_history_size 10
performance_schema_events_waits_history_long_size 10000
performance_schema_events_waits_history_size 10
performance_schema_hosts_size -1
performance_schema_max_cond_classes 80
performance_schema_max_cond_instances -1
performance_schema_max_digest_length 1024
performance_schema_max_file_classes 80
performance_schema_max_file_handles 32768
performance_schema_max_file_instances -1
performance_schema_max_index_stat -1
performance_schema_max_memory_classes 320
performance_schema_max_metadata_locks -1
performance_schema_max_mutex_classes 210
performance_schema_max_mutex_instances -1
performance_schema_max_prepared_statements_instances -1
performance_schema_max_program_instances -1
performance_schema_max_rwlock_classes 40
performance_schema_max_rwlock_instances -1
performance_schema_max_socket_classes 10
performance_schema_max_socket_instances -1
performance_schema_max_sql_text_length 1024
performance_schema_max_stage_classes 150
performance_schema_max_statement_classes 193
performance_schema_max_statement_stack 10
performance_schema_max_table_handles -1
performance_schema_max_table_instances -1
performance_schema_max_table_lock_stat -1
performance_schema_max_thread_classes 50
performance_schema_max_thread_instances -1
performance_schema_session_connect_attrs_size 512
performance_schema_setup_actors_size -1
performance_schema_setup_objects_size -1
performance_schema_users_size -1
pid_file /var/run/mysqld/mysqld.pid
plugin_dir /usr/lib64/mysql/plugin/
port 3306
preload_buffer_size 32768
profiling OFF
profiling_history_size 15
protocol_version 10
query_alloc_block_size 8192
query_cache_limit 1048576
query_cache_min_res_unit 4096
query_cache_size 1048576
query_cache_type OFF
query_cache_wlock_invalidate OFF
query_prealloc_size 8192
range_alloc_block_size 4096
range_optimizer_max_mem_size 8388608
rbr_exec_mode STRICT
read_buffer_size 131072
read_only OFF
read_rnd_buffer_size 262144
relay_log
relay_log_basename /var/lib/mysql/slave-db-01-relay-bin
relay_log_index /var/lib/mysql/slave-db-01-relay-bin.index
relay_log_info_file relay-log.info
relay_log_info_repository FILE
relay_log_purge ON
relay_log_recovery OFF
relay_log_space_limit 0
report_host
report_password
report_port 3306
report_user
rpl_stop_slave_timeout 31536000
server_id 2
server_id_bits 32
server_uuid 73375dd4-1815-11e8-ba91-0252547bd7b4
session_track_gtids OFF
session_track_schema ON
session_track_state_change OFF
session_track_system_variables time_zone,autocommit,character_set_client,character_set_results,character_set_connection
session_track_transaction_info OFF
sha256_password_proxy_users OFF
show_compatibility_56 OFF
show_old_temporals OFF
skip_external_locking ON
skip_name_resolve OFF
skip_networking OFF
skip_show_database OFF
slave_allow_batching OFF
slave_checkpoint_group 512
slave_checkpoint_period 300
slave_compressed_protocol OFF
slave_exec_mode STRICT
slave_load_tmpdir /tmp
slave_max_allowed_packet 1073741824
slave_net_timeout 60
slave_parallel_type DATABASE
slave_parallel_workers 0
slave_pending_jobs_size_max 16777216
slave_preserve_commit_order OFF
slave_rows_search_algorithms TABLE_SCAN,INDEX_SCAN
slave_skip_errors OFF
slave_sql_verify_checksum ON
slave_transaction_retries 10
slave_type_conversions
slow_launch_time 2
slow_query_log OFF
slow_query_log_file /var/lib/mysql/slave-db-01-slow.log
socket /var/lib/mysql/mysql.sock
sort_buffer_size 262144
sql_auto_is_null OFF
sql_big_selects ON
sql_buffer_result OFF
sql_log_off OFF
sql_mode STRICT_TRANS_TABLES,NO_ENGINE_SUBSTITUTION
sql_notes ON
sql_quote_show_create ON
sql_safe_updates OFF
sql_select_limit 18446744073709551615
sql_slave_skip_counter 0
sql_warnings OFF
stored_program_cache 256
super_read_only OFF
sync_binlog 0
sync_frm ON
sync_master_info 10000
sync_relay_log 10000
sync_relay_log_info 10000
system_time_zone UTC
table_definition_cache 1147
table_open_cache 1495
table_open_cache_instances 16
thread_cache_size 28
thread_handling one-thread-per-connection
thread_stack 262144
time_format %H:%i:%s
time_zone SYSTEM
tls_version TLSv1,TLSv1.1
tmp_table_size 16777216
tmpdir /tmp
transaction_alloc_block_size 8192
transaction_isolation REPEATABLE-READ
transaction_prealloc_size 4096
transaction_read_only OFF
transaction_write_set_extraction OFF
tx_isolation REPEATABLE-READ
tx_read_only OFF
unique_checks ON
updatable_views_with_limit YES
version 5.7.21-log
version_comment MySQL Community Server (GPL)
version_compile_machine x86_64
version_compile_os Linux
wait_timeout 28800

Hi Sean, sorry for the necro. Did you ever figure this out or find a way to work around it?

Just coming here via google regarding TeamCity/Primary Keys as I plan to use Galera MariaDB and I have read that a primary key (and InnoDB instead of MyISAM) is necessary especially on deletes. Looking at the above logs sounds like it is a primary key issue.

30 Tables do not have a primary key in TeamCity, so I plan to manually add them myself and see how it goes.