[Pdns-users] Acheiving 40,000 QPS

[Lee Standen]

Hi All,

We're looking to trial PowerDNS as an option to replace our Nominum CNS servers in the next few months, and we're impressed by the performance claimed by PowerDNS.

I've deployed a recursive server and have attempted to replay some captured traffic, but based on my initial testing, I'm only able to pull about 2500 requests per second.

Does any documentation exist which describes which settings must be changed to achieve the claimed 40,000 requests per second?

The server is currently deployed on an IBM HS-21XM blade, which is an 8 core Xeon system.

Thanks

[bert hubert]

Hi Lee,

In the past few months, quite some CNS servers have been replaced by the
PowerDNS Recursor, so you are not alone!

The 40,000 number has been reproduced by a number of separate parties,
sample output is below.

First, please read http://doc.powerdns.com/recursor-performance.html
especially the bit about firewalls. PowerDNS does aggressive source port
randomization, and may stress out intervening firewalls.

Second, try: dnsperf -q 1000 -l 100 < queryfile-example-3million

I just did, and the output is:
Parse input file: multiple times
Run time limit: 100 seconds
Ran through file: 0 times

Queries sent: 2995336 queries
Queries completed: 2991690 queries
Queries lost: 3646 queries

Avg request size: 41 bytes
Avg response size: 80 bytes

Percentage completed: 99.88%
Percentage lost: 0.12%

Started at: Mon May 17 08:23:12 2010
Finished at: Mon May 17 08:24:57 2010
Ran for: 104.828020 seconds

Queries per second: 28539.029927 qps

This is on a server with a firewall in place, and the first run. Second run
with firewall state tracking disabled as per the instructions on
doc.powerdns.com:

Statistics:

Parse input file: multiple times
Run time limit: 100 seconds
Ran through file: 1 times

Queries sent: 5700362 queries
Queries completed: 5694333 queries
Queries lost: 6029 queries

Avg request size: 41 bytes
Avg response size: 80 bytes

Percentage completed: 99.89%
Percentage lost: 0.11%

Started at: Mon May 17 08:26:33 2010
Finished at: Mon May 17 08:28:18 2010
Ran for: 104.711111 seconds

Queries per second: 54381.363598 qps

I hope this helps answer your question. The main trick is to have enough
queries operating in parallel, which incidentally is a close match to 'real
life' heavy use operation.

The operating system of the numbers above is Ubuntu Karmic Koala.

Let me know if you have further questions.

Kind regards,

Bert Hubert

Post by Lee Standen
Hi All,
We're looking to trial PowerDNS as an option to replace our Nominum CNS servers in the next few months, and we're impressed by the performance claimed by PowerDNS.
I've deployed a recursive server and have attempted to replay some captured traffic, but based on my initial testing, I'm only able to pull about 2500 requests per second.
Does any documentation exist which describes which settings must be changed to achieve the claimed 40,000 requests per second?
The server is currently deployed on an IBM HS-21XM blade, which is an 8 core Xeon system.
Thanks
_______________________________________________
Pdns-users mailing list
http://mailman.powerdns.com/mailman/listinfo/pdns-users

[Lee Standen]

Thanks for that information, Bert.

I had seen the documentation before... perhaps I should provide some more information.

Here are the options I'm using at the moment (I've tried several values):

-------------------------------------------------------
disable-packetcache=no
local-address=0.0.0.0
local-port=53
max-cache-entries=4000000
max-packetcache-entries=4000000
single-socket=off
threads=8
-------------------------------------------------------


Here's the results of the second run using the same command line you had below... the data file I found using a quick google search for the filename you used:

./queryperf -q 1000 -l 100 < queryfile-example-3million

-------------------------------------------------------
Statistics:

Parse input file: multiple times
Run time limit: 100 seconds
Ran through file: 0 times

Queries sent: 1285008 queries
Queries completed: 1284175 queries
Queries lost: 833 queries
Queries delayed(?): 0 queries

RTT max: 5.000198 sec
RTT min: 0.000002 sec
RTT average: 0.074771 sec
RTT std deviation: 0.265890 sec
RTT out of range: 1 queries

Percentage completed: 99.94%
Percentage lost: 0.06%

Started at: Mon May 17 07:15:32 2010
Finished at: Mon May 17 07:17:17 2010
Ran for: 104.875333 seconds

Queries per second: 12244.776377 qps
-------------------------------------------------------


I just double-checked the hardware and I've actually got 4 cores available, not 8 as I initially thought:

-------------------------------------------------------
processor : 3
vendor_id : GenuineIntel
cpu family : 6
model : 15
model name : Intel(R) Xeon(R) CPU 5150 @ 2.66GHz
stepping : 6
cpu MHz : 2667.294
cache size : 4096 KB
-------------------------------------------------------



Lastly, I'm using the amd64 deb from the powerdns web site on Ubuntu 10.04 rather than building a new binary. It was assumed that those binaries would be optimized, but if they're not, I'm happy to try building it myself.

Thanks in advance for any suggestions you might have :)






-----Original Message-----
From: bert hubert [mailto:***@netherlabs.nl]
Sent: Monday, 17 May 2010 2:31 PM
To: Lee Standen
Cc: pdns-***@mailman.powerdns.com
Subject: Re: [Pdns-users] Acheiving 40,000 QPS

Hi Lee,

In the past few months, quite some CNS servers have been replaced by the
PowerDNS Recursor, so you are not alone!

The 40,000 number has been reproduced by a number of separate parties,
sample output is below.

First, please read http://doc.powerdns.com/recursor-performance.html
especially the bit about firewalls. PowerDNS does aggressive source port
randomization, and may stress out intervening firewalls.

Second, try: dnsperf -q 1000 -l 100 < queryfile-example-3million

I just did, and the output is:
Parse input file: multiple times
Run time limit: 100 seconds
Ran through file: 0 times

Queries sent: 2995336 queries
Queries completed: 2991690 queries
Queries lost: 3646 queries

Avg request size: 41 bytes
Avg response size: 80 bytes

Percentage completed: 99.88%
Percentage lost: 0.12%

Started at: Mon May 17 08:23:12 2010
Finished at: Mon May 17 08:24:57 2010
Ran for: 104.828020 seconds

Queries per second: 28539.029927 qps

This is on a server with a firewall in place, and the first run. Second run
with firewall state tracking disabled as per the instructions on
doc.powerdns.com:

Statistics:

Parse input file: multiple times
Run time limit: 100 seconds
Ran through file: 1 times

Queries sent: 5700362 queries
Queries completed: 5694333 queries
Queries lost: 6029 queries

Avg request size: 41 bytes
Avg response size: 80 bytes

Percentage completed: 99.89%
Percentage lost: 0.11%

Started at: Mon May 17 08:26:33 2010
Finished at: Mon May 17 08:28:18 2010
Ran for: 104.711111 seconds

Queries per second: 54381.363598 qps

I hope this helps answer your question. The main trick is to have enough
queries operating in parallel, which incidentally is a close match to 'real
life' heavy use operation.

The operating system of the numbers above is Ubuntu Karmic Koala.

Let me know if you have further questions.

Kind regards,

Bert Hubert

Post by Lee Standen
Hi All,
We're looking to trial PowerDNS as an option to replace our Nominum CNS servers in the next few months, and we're impressed by the performance claimed by PowerDNS.
I've deployed a recursive server and have attempted to replay some captured traffic, but based on my initial testing, I'm only able to pull about 2500 requests per second.
Does any documentation exist which describes which settings must be changed to achieve the claimed 40,000 requests per second?
The server is currently deployed on an IBM HS-21XM blade, which is an 8 core Xeon system.
Thanks
_______________________________________________
Pdns-users mailing list
http://mailman.powerdns.com/mailman/listinfo/pdns-users

[Lee Standen]

Looks like I've narrowed down the issue from this setup.

The issue was one of concurrency... by boosting it up to 4000 outstanding, I was able to break over the 30,000 per second mark. It's quite possible that some combination of settings means we need to run more concurrent queries to reach

./queryperf -q 4000 -l 60 < queryfile-example-3million

Given the hardware is below what I thought we were running on, I'm quite happy with the performance level -- further, the fact it only dropped 1.26% of queries when running at this performance level, I'm even more impressed.

-----------------------------------

Statistics:

Parse input file: multiple times
Run time limit: 60 seconds
Ran through file: 0 times

Queries sent: 1982466 queries
Queries completed: 1957446 queries
Queries lost: 25020 queries
Queries delayed(?): 0 queries

RTT max: 5.005290 sec
RTT min: 0.000003 sec
RTT average: 0.061326 sec
RTT std deviation: 0.296004 sec
RTT out of range: 10 queries

Percentage completed: 98.74%
Percentage lost: 1.26%

Started at: Mon May 17 07:48:36 2010
Finished at: Mon May 17 07:49:41 2010
Ran for: 65.000016 seconds

Queries per second: 30114.546433 qps





-----Original Message-----
From: pdns-users-***@mailman.powerdns.com [mailto:pdns-users-***@mailman.powerdns.com] On Behalf Of Lee Standen
Sent: Monday, 17 May 2010 3:23 PM
To: bert hubert
Cc: pdns-***@mailman.powerdns.com
Subject: Re: [Pdns-users] Acheiving 40,000 QPS

Thanks for that information, Bert.

I had seen the documentation before... perhaps I should provide some more information.

Here are the options I'm using at the moment (I've tried several values):

-------------------------------------------------------
disable-packetcache=no
local-address=0.0.0.0
local-port=53
max-cache-entries=4000000
max-packetcache-entries=4000000
single-socket=off
threads=8
-------------------------------------------------------


Here's the results of the second run using the same command line you had below... the data file I found using a quick google search for the filename you used:

./queryperf -q 1000 -l 100 < queryfile-example-3million

-------------------------------------------------------
Statistics:

Parse input file: multiple times
Run time limit: 100 seconds
Ran through file: 0 times

Queries sent: 1285008 queries
Queries completed: 1284175 queries
Queries lost: 833 queries
Queries delayed(?): 0 queries

RTT max: 5.000198 sec
RTT min: 0.000002 sec
RTT average: 0.074771 sec
RTT std deviation: 0.265890 sec
RTT out of range: 1 queries

Percentage completed: 99.94%
Percentage lost: 0.06%

Started at: Mon May 17 07:15:32 2010
Finished at: Mon May 17 07:17:17 2010
Ran for: 104.875333 seconds

Queries per second: 12244.776377 qps
-------------------------------------------------------


I just double-checked the hardware and I've actually got 4 cores available, not 8 as I initially thought:

-------------------------------------------------------
processor : 3
vendor_id : GenuineIntel
cpu family : 6
model : 15
model name : Intel(R) Xeon(R) CPU 5150 @ 2.66GHz
stepping : 6
cpu MHz : 2667.294
cache size : 4096 KB
-------------------------------------------------------



Lastly, I'm using the amd64 deb from the powerdns web site on Ubuntu 10.04 rather than building a new binary. It was assumed that those binaries would be optimized, but if they're not, I'm happy to try building it myself.

Thanks in advance for any suggestions you might have :)






-----Original Message-----
From: bert hubert [mailto:***@netherlabs.nl]
Sent: Monday, 17 May 2010 2:31 PM
To: Lee Standen
Cc: pdns-***@mailman.powerdns.com
Subject: Re: [Pdns-users] Acheiving 40,000 QPS

Hi Lee,

In the past few months, quite some CNS servers have been replaced by the
PowerDNS Recursor, so you are not alone!

The 40,000 number has been reproduced by a number of separate parties,
sample output is below.

First, please read http://doc.powerdns.com/recursor-performance.html
especially the bit about firewalls. PowerDNS does aggressive source port
randomization, and may stress out intervening firewalls.

Second, try: dnsperf -q 1000 -l 100 < queryfile-example-3million

I just did, and the output is:
Parse input file: multiple times
Run time limit: 100 seconds
Ran through file: 0 times

Queries sent: 2995336 queries
Queries completed: 2991690 queries
Queries lost: 3646 queries

Avg request size: 41 bytes
Avg response size: 80 bytes

Percentage completed: 99.88%
Percentage lost: 0.12%

Started at: Mon May 17 08:23:12 2010
Finished at: Mon May 17 08:24:57 2010
Ran for: 104.828020 seconds

Queries per second: 28539.029927 qps

This is on a server with a firewall in place, and the first run. Second run
with firewall state tracking disabled as per the instructions on
doc.powerdns.com:

Statistics:

Parse input file: multiple times
Run time limit: 100 seconds
Ran through file: 1 times

Queries sent: 5700362 queries
Queries completed: 5694333 queries
Queries lost: 6029 queries

Avg request size: 41 bytes
Avg response size: 80 bytes

Percentage completed: 99.89%
Percentage lost: 0.11%

Started at: Mon May 17 08:26:33 2010
Finished at: Mon May 17 08:28:18 2010
Ran for: 104.711111 seconds

Queries per second: 54381.363598 qps

I hope this helps answer your question. The main trick is to have enough
queries operating in parallel, which incidentally is a close match to 'real
life' heavy use operation.

The operating system of the numbers above is Ubuntu Karmic Koala.

Let me know if you have further questions.

Kind regards,

Bert Hubert

Post by Lee Standen
Hi All,
We're looking to trial PowerDNS as an option to replace our Nominum CNS servers in the next few months, and we're impressed by the performance claimed by PowerDNS.
I've deployed a recursive server and have attempted to replay some captured traffic, but based on my initial testing, I'm only able to pull about 2500 requests per second.
Does any documentation exist which describes which settings must be changed to achieve the claimed 40,000 requests per second?
The server is currently deployed on an IBM HS-21XM blade, which is an 8 core Xeon system.
Thanks
_______________________________________________
Pdns-users mailing list
http://mailman.powerdns.com/mailman/listinfo/pdns-users

[Brad Dameron]

Perf testing of 2/4/6/8 threads still shows optimal performance at 4
threads. Cache size of 3M, using test file of 3M addresses.

Tested on HP blades with quad-core X5570 CPU and 48GB RAM, 146GB 15k SAS
drives running on CentOS 5.x with a custom RPM.

2 threads:
new.2.1: Maximum throughput: 40316.000000 qps
new.2.2: Maximum throughput: 45260.000000 qps
new.2.3: Maximum throughput: 51250.000000 qps
new.2.4: Maximum throughput: 58752.000000 qps
new.2.5: Maximum throughput: 63724.000000 qps

4 threads:
new.4.1: Maximum throughput: 68176.000000 qps
new.4.2: Maximum throughput: 85418.000000 qps
new.4.3: Maximum throughput: 98502.000000 qps
new.4.4: Maximum throughput: 97442.000000 qps
new.4.5: Maximum throughput: 98750.000000 qps

6 threads:
new.6.1: Maximum throughput: 85418.000000 qps
new.6.2: Maximum throughput: 98660.000000 qps
new.6.3: Maximum throughput: 96212.000000 qps
new.6.4: Maximum throughput: 95466.000000 qps
new.6.5: Maximum throughput: 96830.000000 qps

8 threads:
new.8.1: Maximum throughput: 89584.000000 qps
new.8.2: Maximum throughput: 96630.000000 qps
new.8.3: Maximum throughput: 92082.000000 qps
new.8.4: Maximum throughput: 93842.000000 qps
new.8.5: Maximum throughput: 86078.000000 qps


FYI here is full output from the 'resperf' tool:

DNS Resolution Performance Testing Tool
Nominum Version 1.0.1.0

[Status] Sending
[Status] Waiting for more responses
[Status] Testing complete

Statistics:

Queries sent: 2999999
Queries completed: 2989899
Queries lost: 10100
Ran for: 100.000000 seconds
Maximum throughput: 97442.000000 qps
Lost at that point: 1.32%


Brad


-----Original Message-----
From: pdns-users-***@mailman.powerdns.com
[mailto:pdns-users-***@mailman.powerdns.com] On Behalf Of Lee
Standen
Sent: Monday, May 17, 2010 12:57 AM
To: pdns-***@mailman.powerdns.com
Subject: Re: [Pdns-users] Acheiving 40,000 QPS

Looks like I've narrowed down the issue from this setup.

The issue was one of concurrency... by boosting it up to 4000
outstanding, I was able to break over the 30,000 per second mark. It's
quite possible that some combination of settings means we need to run
more concurrent queries to reach

./queryperf -q 4000 -l 60 < queryfile-example-3million

Given the hardware is below what I thought we were running on, I'm quite
happy with the performance level -- further, the fact it only dropped
1.26% of queries when running at this performance level, I'm even more
impressed.

-----------------------------------

Statistics:

Parse input file: multiple times
Run time limit: 60 seconds
Ran through file: 0 times

Queries sent: 1982466 queries
Queries completed: 1957446 queries
Queries lost: 25020 queries
Queries delayed(?): 0 queries

RTT max: 5.005290 sec
RTT min: 0.000003 sec
RTT average: 0.061326 sec
RTT std deviation: 0.296004 sec
RTT out of range: 10 queries

Percentage completed: 98.74%
Percentage lost: 1.26%

Started at: Mon May 17 07:48:36 2010
Finished at: Mon May 17 07:49:41 2010
Ran for: 65.000016 seconds

Queries per second: 30114.546433 qps





-----Original Message-----
From: pdns-users-***@mailman.powerdns.com
[mailto:pdns-users-***@mailman.powerdns.com] On Behalf Of Lee
Standen
Sent: Monday, 17 May 2010 3:23 PM
To: bert hubert
Cc: pdns-***@mailman.powerdns.com
Subject: Re: [Pdns-users] Acheiving 40,000 QPS

Thanks for that information, Bert.

I had seen the documentation before... perhaps I should provide some
more information.

Here are the options I'm using at the moment (I've tried several
values):

-------------------------------------------------------
disable-packetcache=no
local-address=0.0.0.0
local-port=53
max-cache-entries=4000000
max-packetcache-entries=4000000
single-socket=off
threads=8
-------------------------------------------------------


Here's the results of the second run using the same command line you had
below... the data file I found using a quick google search for the
filename you used:

./queryperf -q 1000 -l 100 < queryfile-example-3million

-------------------------------------------------------
Statistics:

Parse input file: multiple times
Run time limit: 100 seconds
Ran through file: 0 times

Queries sent: 1285008 queries
Queries completed: 1284175 queries
Queries lost: 833 queries
Queries delayed(?): 0 queries

RTT max: 5.000198 sec
RTT min: 0.000002 sec
RTT average: 0.074771 sec
RTT std deviation: 0.265890 sec
RTT out of range: 1 queries

Percentage completed: 99.94%
Percentage lost: 0.06%

Started at: Mon May 17 07:15:32 2010
Finished at: Mon May 17 07:17:17 2010
Ran for: 104.875333 seconds

Queries per second: 12244.776377 qps
-------------------------------------------------------


I just double-checked the hardware and I've actually got 4 cores
available, not 8 as I initially thought:

-------------------------------------------------------
processor : 3
vendor_id : GenuineIntel
cpu family : 6
model : 15
model name : Intel(R) Xeon(R) CPU 5150 @ 2.66GHz
stepping : 6
cpu MHz : 2667.294
cache size : 4096 KB
-------------------------------------------------------



Lastly, I'm using the amd64 deb from the powerdns web site on Ubuntu
10.04 rather than building a new binary. It was assumed that those
binaries would be optimized, but if they're not, I'm happy to try
building it myself.

Thanks in advance for any suggestions you might have :)






-----Original Message-----
From: bert hubert [mailto:***@netherlabs.nl]
Sent: Monday, 17 May 2010 2:31 PM
To: Lee Standen
Cc: pdns-***@mailman.powerdns.com
Subject: Re: [Pdns-users] Acheiving 40,000 QPS

Hi Lee,

In the past few months, quite some CNS servers have been replaced by the
PowerDNS Recursor, so you are not alone!

The 40,000 number has been reproduced by a number of separate parties,
sample output is below.

First, please read http://doc.powerdns.com/recursor-performance.html
especially the bit about firewalls. PowerDNS does aggressive source port
randomization, and may stress out intervening firewalls.

Second, try: dnsperf -q 1000 -l 100 < queryfile-example-3million

I just did, and the output is:
Parse input file: multiple times
Run time limit: 100 seconds
Ran through file: 0 times

Queries sent: 2995336 queries
Queries completed: 2991690 queries
Queries lost: 3646 queries

Avg request size: 41 bytes
Avg response size: 80 bytes

Percentage completed: 99.88%
Percentage lost: 0.12%

Started at: Mon May 17 08:23:12 2010
Finished at: Mon May 17 08:24:57 2010
Ran for: 104.828020 seconds

Queries per second: 28539.029927 qps

This is on a server with a firewall in place, and the first run. Second
run with firewall state tracking disabled as per the instructions on
doc.powerdns.com:

Statistics:

Parse input file: multiple times
Run time limit: 100 seconds
Ran through file: 1 times

Queries sent: 5700362 queries
Queries completed: 5694333 queries
Queries lost: 6029 queries

Avg request size: 41 bytes
Avg response size: 80 bytes

Percentage completed: 99.89%
Percentage lost: 0.11%

Started at: Mon May 17 08:26:33 2010
Finished at: Mon May 17 08:28:18 2010
Ran for: 104.711111 seconds

Queries per second: 54381.363598 qps

I hope this helps answer your question. The main trick is to have enough
queries operating in parallel, which incidentally is a close match to
'real life' heavy use operation.

The operating system of the numbers above is Ubuntu Karmic Koala.

Let me know if you have further questions.

Kind regards,

Bert Hubert

Post by Lee Standen
Hi All,
We're looking to trial PowerDNS as an option to replace our Nominum

CNS servers in the next few months, and we're impressed by the
performance claimed by PowerDNS.

Post by Lee Standen
I've deployed a recursive server and have attempted to replay some

captured traffic, but based on my initial testing, I'm only able to pull
about 2500 requests per second.

Post by Lee Standen
Does any documentation exist which describes which settings must be

changed to achieve the claimed 40,000 requests per second?

Post by Lee Standen
The server is currently deployed on an IBM HS-21XM blade, which is an 8 core Xeon system.
Thanks
_______________________________________________
Pdns-users mailing list
http://mailman.powerdns.com/mailman/listinfo/pdns-users

This email may contain confidential and privileged material for the sole use of the intended recipient. Any review, use, distribution or disclosure by others is strictly prohibited. If you are not the intended recipient (or authorized to receive for the recipient), please contact the sender by reply email and delete all copies of this message.

[bert hubert]

Post by Lee Standen
Thanks for that information, Bert.
I had seen the documentation before... perhaps I should provide some more information.
-------------------------------------------------------
disable-packetcache=no
local-address=0.0.0.0

It might be wise to set this to the actual address you are using, but this
will not impact performance.

Post by Lee Standen
local-port=53
max-cache-entries=4000000
max-packetcache-entries=4000000
single-socket=off

I'd remove this one.

Post by Lee Standen
threads=8

4 might be better given your hardware. At 8, some scheduling infelicities
might pop up.

Post by Lee Standen
Percentage completed: 99.94%
Percentage lost: 0.06%
Queries per second: 12244.776377 qps

This is decidedly disappointing. Can you retry with threads=2 and threads=4?
Can you try a 'second run' with the same data? This can rule out certain
scnarios.
Is this real hardware or virtualized?

Post by Lee Standen
Lastly, I'm using the amd64 deb from the powerdns web site on Ubuntu 10.04
rather than building a new binary. It was assumed that those binaries
would be optimized, but if they're not, I'm happy to try building it
myself.

They are pretty ok - the numbers I posted earlier are also from those debs.

Bert

[Imre Gergely]

First things first: use threads=4 or threads=8 and packet-cache.

This is how I did it:

Run a couple of dnsperf's like this:

./dnsperf -d queryfile-example-3million -s <IP> -p <port> -t 2 -l
$((3600*10)) -q 2000 -Q 2000 > logfile.log 2>&1

This will run a dnsperf from a queryfile for 10 hours (so you will see
something on the graphs), with 2000qps. Now, run a couple of these
simultaneously and see what happens.

Probably not a scientific method for precise testing :), but you should
see the qps on the graphs (make sure you create some graphs). You should
run the dnsperfs from another machine IMHO (or maybe 2-3 other machines).

Here's a bigger queryfile to replay if you need it (real life customer
queries from two months ago):

http://hosting.astral.ro/~gimre/queryfile-24mil.txt.bz2

Let us know how it goes.

Disclaimer: I'm no (power)DNS expert or anything, so the above could be
worthless :)

Post by Lee Standen
Hi All,
We’re looking to trial PowerDNS as an option to replace our Nominum CNS
servers in the next few months, and we’re impressed by the performance
claimed by PowerDNS.
I’ve deployed a recursive server and have attempted to replay some
captured traffic, but based on my initial testing, I’m only able to pull
about 2500 requests per second.
Does any documentation exist which describes which settings must be
changed to achieve the claimed 40,000 requests per second?
The server is currently deployed on an IBM HS-21XM blade, which is an 8 core Xeon system.
Thanks
_______________________________________________
Pdns-users mailing list
http://mailman.powerdns.com/mailman/listinfo/pdns-users