[Pdns-users] Using AXFR vs. Mysql Replication

[Max Clark]

Hi all,

We have been using the mysql backend with native master->slave
replication for some time now. Due to growth, distributed nameservers,
and the desire to make our DNS infrastructure more stand alone we are
interested in switching to an AXFR based zone replication to the
public pdns servers which will be feed by masters running against the
mysql backend.

With mysql native replication zero configuration is required on the
slave instances to configure new zones - I understand there is a
configuration option for a supermaster in pdns so that the slaves will
autogenerate zones based on the master but I am unclear on the proper
use for deployment.

Can anyone point me to a howto or give configuration advise for what
we want to accomplish?

Thanks,
Max

[Lorens Kockum]

Post by Max Clark
Hi all,
We have been using the mysql backend with native master->slave
replication for some time now. Due to growth, distributed nameservers,
and the desire to make our DNS infrastructure more stand alone we are
interested in switching to an AXFR based zone replication to the
public pdns servers which will be feed by masters running against the
mysql backend.

I would not recommend AXFRs, especially as you have replication
running.

I have two problems with using AXFRs:

The pdns server does not remember AXFRs or SOA checks so
that it can re-NOTIFY the slaves who have not AXFRd a given
zone within some reasonable time. This means that if for
some reason an UDP NOTIFY packet is lost, the zone wil not
be updated, and if the NOTIFY is the initial one, the zone
will not be configured on the slaves until the zone is
changed and a new NOTIFY is sent.

When a zone is removed from the master, the slave does not
recognize that. I would like the slave to remove the zone
from the database when it expires.

Post by Max Clark
With mysql native replication zero configuration is required on the
slave instances to configure new zones - I understand there is a
configuration option for a supermaster in pdns so that the slaves will
autogenerate zones based on the master but I am unclear on the proper
use for deployment.

The slaves need to have the master's IP(s) configured in pdns.conf as
their supermasters. The zones in the master database should not be type
NATIVE any more, but type MASTER. I think that's about it, apart from
the different allow-axfr access lists.

HTH.

[Duane]

Post by Lorens Kockum
The pdns server does not remember AXFRs or SOA checks so
that it can re-NOTIFY the slaves who have not AXFRd a given
zone within some reasonable time. This means that if for
some reason an UDP NOTIFY packet is lost, the zone wil not
be updated, and if the NOTIFY is the initial one, the zone
will not be configured on the slaves until the zone is
changed and a new NOTIFY is sent.

No but slaves track it and update themselves periodically.

Post by Lorens Kockum
When a zone is removed from the master, the slave does not
recognize that. I would like the slave to remove the zone
from the database when it expires.

This can be fixed using a shell script, there are other problems with
the AXFR mechanism, but these are mostly non-issues.
--
Best regards,
Duane

http://www.cacert.org - Free Security Certificates
http://www.nodedb.com - Think globally, network locally
http://www.sydneywireless.com - Telecommunications Freedom
http://e164.org - Because e164.arpa is a tax on VoIP

"In the long run the pessimist may be proved right,
but the optimist has a better time on the trip."

[Lorens Kockum]

Post by Duane

Post by Lorens Kockum
The pdns server does not remember AXFRs or SOA checks so
that it can re-NOTIFY the slaves who have not AXFRd a given
zone within some reasonable time. This means that if for
some reason an UDP NOTIFY packet is lost, the zone wil not
be updated, and if the NOTIFY is the initial one, the zone
will not be configured on the slaves until the zone is
changed and a new NOTIFY is sent.

No but slaves track it and update themselves periodically.

Unless it's the very first packet, in which case the slave doesn't know
that it's supposed to track the domain.

Also, you have a trade-off between the load imposed by slaves
polling server for each and every domain "periodically",
and the length of the "periodically", which should be short
enough to make clients happy. I don't think "your domain isn't
working because our slave failed to update from the master" is
a good argument, so the period between checks in the absence
of notifies would need to be, what, five minutes? Ten? Say ten
minutes, for ten thousand domains that's two thousand packets a
minute just for checking that nothing's changed, which IMHO is a
bit steep.

Of course there *are* notifies, but I seem to lose them
regularly :-)

Post by Duane

Post by Lorens Kockum
When a zone is removed from the master, the slave does not
recognize that. I would like the slave to remove the zone
from the database when it expires.

This can be fixed using a shell script

Yes.

Post by Duane
there are other problems with
the AXFR mechanism, but these are mostly non-issues.

Agreed.

[Max Clark]

So the general consensus seems to be AXFR replication is not a good
idea. We've lost name resolution a couple of times now because of
issues with Mysql replication (and once because mysql went down
causing the nameservers to go down, which caused name resolution for
client access to fail when we tried to bring the nameservers back up -
really stupid problem).

Outside of pulling the zones into tinydns format and replicating that
- is there another option that I should be looking at?

Thanks,
Max

Post by Lorens Kockum

Post by Duane

Post by Lorens Kockum
The pdns server does not remember AXFRs or SOA checks so
that it can re-NOTIFY the slaves who have not AXFRd a given
zone within some reasonable time. This means that if for
some reason an UDP NOTIFY packet is lost, the zone wil not
be updated, and if the NOTIFY is the initial one, the zone
will not be configured on the slaves until the zone is
changed and a new NOTIFY is sent.

No but slaves track it and update themselves periodically.

Unless it's the very first packet, in which case the slave doesn't know
that it's supposed to track the domain.
Also, you have a trade-off between the load imposed by slaves
polling server for each and every domain "periodically",
and the length of the "periodically", which should be short
enough to make clients happy. I don't think "your domain isn't
working because our slave failed to update from the master" is
a good argument, so the period between checks in the absence
of notifies would need to be, what, five minutes? Ten? Say ten
minutes, for ten thousand domains that's two thousand packets a
minute just for checking that nothing's changed, which IMHO is a
bit steep.
Of course there *are* notifies, but I seem to lose them
regularly :-)

Post by Duane

Post by Lorens Kockum
When a zone is removed from the master, the slave does not
recognize that. I would like the slave to remove the zone
from the database when it expires.

This can be fixed using a shell script

Yes.

Post by Duane
there are other problems with
the AXFR mechanism, but these are mostly non-issues.

Agreed.
_______________________________________________
Pdns-users mailing list
http://mailman.powerdns.com/mailman/listinfo/pdns-users

[Lukas Beeler]

Post by Max Clark
issues with Mysql replication (and once because mysql went down
causing the nameservers to go down, which caused name resolution for
client access to fail when we tried to bring the nameservers back up -
really stupid problem).

MySQL has had a few issues with replication (i've got
an open bug report with the MySQL people), and we've started to
monitor replication much more closely, and also started
seperating MySQL Instances for various services.

Are you using Nagios or a similar tool? MySQL replication can
break silently if you don't monitor it, which can cause major
headaches.

If you don't want to rollout nagios or something similar, i'd
recommend you to use nagios check scripts, and run them in a
cronjob with a small wrapper around them which sends SMS.

Personally, all the problems we had with MySQL happened because
we don't use a testbed environment for all changes.

If you need any help with nagios, let me know, i'd be happy to
share my configuration and scripts (don't have time to publish
them all properly).

[Max Clark]

Lukas,

I would be very interested in hearing more about your nagios
configuration to monitor mysql and mysql replication. Anything you can
provide would be a great help.

Thanks,
Max

Post by Lukas Beeler

Post by Max Clark
issues with Mysql replication (and once because mysql went down
causing the nameservers to go down, which caused name resolution for
client access to fail when we tried to bring the nameservers back up -
really stupid problem).

MySQL has had a few issues with replication (i've got
an open bug report with the MySQL people), and we've started to
monitor replication much more closely, and also started
seperating MySQL Instances for various services.
Are you using Nagios or a similar tool? MySQL replication can
break silently if you don't monitor it, which can cause major
headaches.
If you don't want to rollout nagios or something similar, i'd
recommend you to use nagios check scripts, and run them in a
cronjob with a small wrapper around them which sends SMS.
Personally, all the problems we had with MySQL happened because
we don't use a testbed environment for all changes.
If you need any help with nagios, let me know, i'd be happy to
share my configuration and scripts (don't have time to publish
them all properly).
_______________________________________________
Pdns-users mailing list
http://mailman.powerdns.com/mailman/listinfo/pdns-users

[Lorens Kockum]

Post by Max Clark
So the general consensus seems to be AXFR replication is not a good
idea. We've lost name resolution a couple of times now because of
issues with Mysql replication (and once because mysql went down
causing the nameservers to go down, which caused name resolution for
client access to fail when we tried to bring the nameservers back up -
really stupid problem).
Outside of pulling the zones into tinydns format and replicating that
- is there another option that I should be looking at?

Weeeell. My opinon is that of someone doing AXFR replication,
and seeing the bad side of that, and thinking that on the other
side of the fence, the grass is greener.

If you're having problems with mysql replication, then look at
the problems I have with AXFR replication, and maybe you'll
decide that AXFR replication is better in your situation.

I know that AXFR replication can get me a call saying "how come
my domain is not up to date on the secondary", but I've never
ever come close to even thinking that something might ever
impact *all* my domains.

HTH.

[Duane]

Post by Max Clark
Can anyone point me to a howto or give configuration advise for what
we want to accomplish?

I have nothing against PDNS and we happily use it as a primary master,
however we ran into some pretty big problems with zone transfers and we
aren't able to run mysql replication due to non-technical reasons.

We are sort of doing something similar, however there is a better option
then even what we're doing at present and are planning to get round to
upgrading at some point.

I'll explain it rather then confusing things by including our current
configuration.

Firstly grab the tinydns tools, you do this to dump out all zones on the
local machine to a flat file. Then using rsync have the remote machines
check for updates periodically.

You will need to use some kind of shell script, I tend to use PHP, to
compare the current SOA of your master(s) against the remote
installations. Any changes you simply trigger rsync to update, we've
tried a number of secondary options and so far tinydns has been the best
option for us since it uses flat files and is able to cope with large
amounts of requests per second. Something neither bind nor PDNS/MySQL
can cope with.

The above method can be locked to specific IPs etc and can be compressed
and run over SSL so the zone can't be sniffed over the wire during a
transfer, and like mysql replication only changes are transferred.
--
Best regards,
Duane

http://www.cacert.org - Free Security Certificates
http://www.nodedb.com - Think globally, network locally
http://www.sydneywireless.com - Telecommunications Freedom
http://e164.org - Because e164.arpa is a tax on VoIP

"In the long run the pessimist may be proved right,
but the optimist has a better time on the trip."

[Tony Adams]

I too am interested in a method for monitoring the status of mysql
replication. I've been very fortunate so far in that it seems to just
work, but I'd like to have a better system in place for the eventuality
that it won't.

Thanks

Tony Adams
Systems Engineer
E Solutions Corporation

-----Original Message-----
Sent: Saturday, June 24, 2006 2:43 PM
Subject: Re: [Pdns-users] Using AXFR vs. Mysql Replication
Lukas,
I would be very interested in hearing more about your nagios
configuration to monitor mysql and mysql replication. Anything you can
provide would be a great help.
Thanks,
Max

Post by Lukas Beeler

Post by Max Clark
issues with Mysql replication (and once because mysql went down
causing the nameservers to go down, which caused name resolution

for

Post by Lukas Beeler

Post by Max Clark
client access to fail when we tried to bring the nameservers back

up -

Post by Lukas Beeler

Post by Max Clark
really stupid problem).

MySQL has had a few issues with replication (i've got
an open bug report with the MySQL people), and we've started to
monitor replication much more closely, and also started
seperating MySQL Instances for various services.
Are you using Nagios or a similar tool? MySQL replication can
break silently if you don't monitor it, which can cause major
headaches.
If you don't want to rollout nagios or something similar, i'd
recommend you to use nagios check scripts, and run them in a
cronjob with a small wrapper around them which sends SMS.
Personally, all the problems we had with MySQL happened because
we don't use a testbed environment for all changes.
If you need any help with nagios, let me know, i'd be happy to
share my configuration and scripts (don't have time to publish
them all properly).
_______________________________________________
Pdns-users mailing list
http://mailman.powerdns.com/mailman/listinfo/pdns-users

--
Max Clark
http://www.clarksys.com
_______________________________________________
Pdns-users mailing list
http://mailman.powerdns.com/mailman/listinfo/pdns-users

[Lukas Beeler]

Post by Tony Adams
I too am interested in a method for monitoring the status of mysql
replication. I've been very fortunate so far in that it seems to just
work, but I'd like to have a better system in place for the eventuality
that it won't.

[ resend of the message i originally wrote to Max ]
[ I've stripped to attachment, and put it here:
http://projectdream.org/~lb/check_mysqlrepl ]

Here's a public site of our Nagios setup:

http://status.sylon.net/

Attached is the check_mysqlrepl script i use to monitor my MySQL
Replication setup. This script needs to be run once per slave.

I've modified this script to check MySQL Replication using MySQL
5 correctly. If you're using a previous version of MySQL, you
will need to adjust some variables in this scripts, in Line 97.

Use SHOW MASTER STATUS\G and SHOW SLAVE STATUS\G to find out
which values you need. Pay attention that you monitor the correct
variables, e.G.
There's Relay_Log_Pos, which shows to which position the log
files haben been TRANSFERRED. There's also Exec_Master_Log_Pos,
which shows to which position the log files have been
SUCCESSFULLY EXECUTED. You need to monitor the latter value, the
first is worthless and will only be triggered in case master and
slave lose connectivity.

You could also check for Slave_SQL_Running, but i didn't do this
yet (saw no need). Running the replication check script is rather
easy:

e.G.

/usr/share/nagios/check_mysqlrepl --crit 10000 --warn 5000
--master michelle --slave janna --dbuser repltest --dbpass
foo --mport 3308 --sport 3308

michelle is the hostname of the master, janna the hostname of the
slave. There's also a check_mysql C Program, which is part of
nagios-plugins. You can get them at nagios.org.

If you want, i could also send you my complete nagios config, but
i would need to edit out all passwords first. So i didn't do this
proactively yet ;)

If you want to monitor several Key values of your MySQL database,
i can also recommend you to use cacti and this Host template:

http://forums.cacti.net/about11010-0-asc-0.html

This will give you lots of nice graphs about the usage of your
database, which can be very helpful when looking for performance
troubles.

We're also using http://www.ecall.ch to send SMS with Nagios,
which is very cool (you get notified of any problem immediately).

A proper implementation of nagios WILL take some effort, but it
will be worth it.

Hope this helps,

Lukas