v27

The themes for this release are cluster stability, security and operability. - cf-mysql v27 should include MariaDB 10.0.24 and Galera 25.3.14 [#114625673]

Deprecation Warning: This will be the last cf-mysql-release to use MariaDB version 10.0. Subsequent releases will begin to use MariaDB 10.1, which became generally available late last year. Be aware that there may be compatibility differences between 10.0 and 10.1, so take care to test out changes before upgrading past cf-mysql-release v27. - Note: cf-mysql now requires stemcells 3125 or more recent. As a security precaution, we recommend you update stemcells frequently. - Note: If you’ve enabled syslogging, please see the security notice below regarding root credentials.

Pre-Start

Last year, we wrote a fussy story: - As an Operator, I’d like configure an overall timeout which will break down into individual timeouts for each of the MySQL job’s startup phases, so that large SST’s can succeed [#100598304]

This condensed update.canary_watch_time, update.update_watch_time and DatabaseStartupTimeout into simply mysql_startup_timeout, and tried to intelligently split that up into sane defaults for each of the stages of database startup.

V27 moves much of this into the BOSH pre-start phase of the job lifecycle. This eliminates many timeout-related failures, although the system may still perform lengthy SSTs during monit start under certain circumstances. - Leverage BOSH pre-start process [#100444318], [#114259781], [#117744313], [#117583221], [#117549509]

Defensive Enhancements

When restarting, a node re-syncs to the cluster using one of two methods, IST or SST. In the case of SST, it throws away its local data, and adopts the dataset of the cluster. We’ve found rare circumstances when that’s not desirable. This release includes an optional Interruptor, which prevents a node from SST when doing so would delete its own database. We’ve also provided a rejoin-unsafe errand that automates the process of forcing a node to re-join the cluster after an Operator has determined that it’s safe to do so. - As an Operator, I do not want a possibly-sick node to re-join the cluster if it means there could be data loss. [#118416629], [#118416751], [#118793289]

Galera comes with a few known limitations. In an effort to try to protect users, and enhance stability, we’ve made the following changes. - Ensure MyISAM experimental replication is disabled [#120629227] - As an Operator, I don’t want a Galera-based cluster to accept XA transactions [#121461877] - As an Operator, I’d like my replication sums to be checked [#121008813]

The Operator, or automation, may to want to block all access to the MariaDB cluster, to enable backups, diagnosis, etc. - As an Operator, I can enable and disable all traffic passing through the proxy [#127033173]

Security & Logging

More users are beginning to deploy cf-mysql into production. We’ve prioritized enabling logs and creating diagnostics that are useful to both Operators and Database Administrators to troubleshoot and audit an active cluster. We’ve also made changes to run more securely, in the event that an unauthorized user gains access to the virtual machines. - Mariadb < 10.0.22 expose credentials in mysql.err.log when performing sst [#122147897]

Note: cf-mysql-release v25 and earlier expose the credentials of the mysql root account. If you’ve enabled syslog, the root password will have been transmitted outside of the cf-mysql deployment. You must change the root password of your root account after upgrading to this version of cf-mysql-release. - We’ve changed default syslog priority level from local1.error to the CAPI recommended standard of user.error. - As an Operator, I’d like to be able to see the stdout/err of mariadb_ctl control script [#118612267] - As a mysql operator, I would like to feel safe knowing that my mariadb process is not running as a privileged user (mysqld now runs as user vcap) [#115623381] - As an operator, I do not want to see my mysql passwords in ps aux [#120031195] - As an Operator, I’d like to be able to learn more about slow queries [#113421765] - As an Operator, I’d like MySQL binlogs enabled on each Master node [#121089515] - As an Operator, I’d like to enable additional debug logging to understand replication behavior. [#120996959] - Log diagnostic cluster data to a CSV file [#118415807], [#121713351], [#127034351] - As an Operator, I don’t want users to be able to access databases whose names start with ‘test’ on a production deployment so I have confidence that my DB is secure by default [#96828790], [#117195649] - Bug: cloudfoundry/cf-mysql-release #116: Enabling syslog doesn’t appear to send mysql.err.log to syslog [#127031107] - Bug: Log SST script output to mysql.err.log [#127664015]

Audit Logs

• cloudfoundry/cf-mysql-release #89: Enable audit_plugin [#113966825]
  • As an Operator, I’d like BOSH to preserve/rotate audit_logs [#118504463]
  • As an Operator, I do not want cluster_health_logger queries in the server audit log [#119423815]
  • Audit logs should be in persistent disk [#127034681], [#122210877]

Quota Enforcer

• As an Operator, I’d like the Quota Enforcer to connect as a non-root super user so that we can exclude the QE’s activity from the audit logs [#118502911]
• quota-enforcer user should not need ‘GRANT ALL PRIVILEGES’ [#119029525]
• As an operator, I would like to provide a list of users that the quota enforcer will ignore [#119045503]
• As an Operator, I’d like the Quota Enforcer to run less frequently so that it doesn’t fill up debugging log files. [#127033101]

Enhancements and Bug Fixes

• Update cf-mysql-release to Golang 1.7 [#128205943]
• As an operator, I want to configure the number of open files available to mysqld via max_open_files [#117110409]
• As an Operator, I’d like to have a property to change the name of the MySQL service in the Marketplace [#115805005]
• Enable innodb_large_prefix [#116578809]
• Length of time GRA files are kept around should be increased [#118606579]
• Bug: As an Operator, I want cf-mysql acceptance tests to work with recent versions of cf-release [#117671103]
• Bug: As an Operator, I’d like to change my database pre-seeds without running the bootstrap procedure [#122647541], [#127938125]
• Bug: Fix an out-of control CPU issue on the service broker VMs.
  • As an operator, I do not want monit to try and start mariadb_ctrl multiple times at the same time [#119498227]

Community Contributed Enhancements

• cloudfoundry/cf-mysql-release #87: Resolve package name, job name, and port conflicts [#113088085]
• cloudfoundry/cf-mysql-release #100: Add ability to override number of compilation workers [#117224143]
• Change the name of the deployment as it appears in bosh deployments: cloudfoundry/cf-mysql-release #101: Add ability to override deployment name [#117225385]
• cloudfoundry/cf-mysql-release #97: Trim release size [#116155371]
• cloudfoundry/cf-mysql-release #110: ./scripts/generate-deployment-manifest now takes a real cf manifest [#121524303]

Documentation

• Fix arbitrator docs to get rid of 2-step deploy instructions [#115301809]
• As an Operator, I’d like an example for the api_url property [#115819685]
• Seeded database documentation and spec file example [#115451413]
• Simplified the spec files for the service broker so that service plans are only specified where necessary. The broker errands do not need to know about the entire services hash [#116383627], [#116334151]
• README.md should reference specific spiff version [#123626451]
• As an Operator, I’d like to discover documentation that describes how to recover when a node fails due to SST outside of pre-start. [#128550817]
• Bug: README.md still refers to sample_aws_stub.yml file [#121088767]
• Bug: cf_mysql.mysql.galera_healthcheck.db_password does not exist in example stubs file [#128622041]

Interestings

• As a [service which embeds cf-mysql], I would like to enable internal clients to communicate consistently with a single switchboard proxy in an HA deployment so that they incur fewer deadlock errors [#118507365]
• Acceptance test errand should have a configurable test password [#127359061]
• We’ve updated the version of the cf cli for use in smoke tests: Operator can see that the smoke test passes when deploying MySQL v.xyz in an environment where cf-release v.abc is running [#123165451]

Manifest Changes

Note: Since cf-mysql v23, manifest generation requires spiff version 1.0.7.

MySQL Job Spec Changes

• Added cf_mysql.mysql.max_open_files to allow you to configure the maximum number of open files. This defaults to 65536.
• Added cf_mysql.mysql.galera_port to allow the port for Galera cluster communication to be customized.
• Added cf_mysql.mysql.disable_auto_sst. When this is true (the default) then nodes will be prevented from performing a SST.
• Removed network_name.

Audit log support

• Added cf_mysql.mysql.server_audit_events to configure the types of events that will be logged. If unspecified then logging is disabled.
• Added cf_mysql.mysql.server_audit_file_rotations to configure the number of audit file rotations to keep. The default is 30.
• Added cf_mysql.mysql.server_audit_file_rotate_size_in_mb which controls the size of each audit log file (default `100).
• Added cf_mysql.mysql.server_audit_excluded_users to exclude database users from the audit log. The following users will always be excluded: cluster-health-logger, galera-healthcheck, quota-enforcer.

Cluster health logger

• Added cf_mysql.mysql.cluster_health.password to set the cluster health logger database password. This must be specified.
• Added cf_mysql.mysql.cluster_health.log_interval to control the frequency in seconds with which cluster information is logged. This defaults to every 30 seconds.

Galera Healthcheck

• Moved cf_mysql.mysql.healthcheck_port to cf_mysql.mysql.galera_healthcheck.port
• Moved cf_mysql.mysql.bootstrap_endpoint.username to cf_mysql.mysql.galera_healthcheck.endpoint_username
• Moved cf_mysql.mysql.bootstrap_endpoint.password to cf_mysql.mysql.galera_healthcheck.endpoint_password
• Added cf_mysql.mysql.galera_healthcheck.db_password to set the database password for the galera-healthcheck user. This must be specified.

Binlogs

• Added cf_mysql.mysql.binlog_enabled which enables the binary log if true (the default).
• Added cf_mysql.mysql.binlog_expire_days to set the number of days to retain binlogs for. This defaults to 60 days.

Debug logging

Configuring additional debug logging with these properties may result in user data being logged to disk. - Added cf_mysql.mysql.wsrep_debug. When set to 1 or 2, the database server logs additional debug events for Galera replication. The default is 0. - Added cf_mysql.mysql.log_conflicts. This defaults to true.

Proxy Job Spec Changes

• The default port for the Proxy API has changed from 80 to 8080.
• Added cf_mysql.proxy.consul_enabled to enable experimental support for registering the proxy nodes as a service in Consul.
• Added cf_mysql.proxy.consul_service_name to set the name of the published service in Consul. Defaults to mysql.
• Removed network_name.

CF MySQL Broker Job Spec Changes

• Added cf_mysql.broker.port to configure the port the broker will listen on. This defaults to 8081 (previously hard-coded to port 80).
• Added cf_mysql.broker.quota_enforcer.password to set the database password for the quota-enforcer user. This must be specified.
• Added cf_mysql.broker.quota_enforcer.ignored_users to specify an array of database users that the quota enforcer will not enforce quotas for.
• Added cf_mysql.broker.quota_enforcer.pause to configure how frequently quotas should be enforced in seconds (defaults to 1). This was previously hard-coded to 1 second.

Arbitrator Job Spec Changes

• Moved cf_mysql.mysql.healthcheck_port to cf_mysql.mysql.galera_healthcheck.port
• Moved cf_mysql.mysql.bootstrap_endpoint.username to cf_mysql.mysql.galera_healthcheck.endpoint_username
• Moved cf_mysql.mysql.bootstrap_endpoint.password to cf_mysql.mysql.galera_healthcheck.endpoint_password

Acceptance Tests Errand Spec Changes

• Added cf_mysql.acceptance_tests.password to make the smoke test password configurable. A regression in v26 meant that a configured password was not used when specified.

Bootstrap Errand Spec Changes

• Removed cf_mysql.mysql.database_startup_timeout. The bootstrap time is now unbounded to allow for possibly lengthy SST.
• Moved cf_mysql.mysql.bootstrap_endpoint.username to cf_mysql.mysql.galera_healthcheck.endpoint_username
• Moved cf_mysql.mysql.bootstrap_endpoint.password to cf_mysql.mysql.galera_healthcheck.endpoint_password

- name: "cf-mysql"
  version: "27"
  url: "https://bosh.io/d/github.com/cloudfoundry/cf-mysql-release?v=27"
  sha1: "40d210801b41fcf805cde3c47d6c762636fc36e0"

bosh upload-release --sha1 40d210801b41fcf805cde3c47d6c762636fc36e0 \
  "https://bosh.io/d/github.com/cloudfoundry/cf-mysql-release?v=27"