[Pdns-users] pdns-recursor: Only two threads used when threads=4

[Grace Wang]

Sorry for not including a subject in the original post.

Hi There,
I installed pdns-recursor 4.0.3 on a CentOS 6.5 box, that has 20 physical
cores and 128G RAM. When I set the threads in /etc/pdns-recursor/
recursor.conf to be 4 or 12, then issued tons of queries (one query sent
multiple times using multiple dnsperf instances simultaneously) at it, I
noticed only 2 of the threads were working actively, each using almost 100%
CPU and ~90% CPU, respectively. The other threads were using 0% CPU.
With a previous version pdns-recursor 3.6.0 and the same test case, I
could get 1 million qps, but with this new version that's supposed to have
better performance, I am only getting ~200k qps.
The way I installed it was by executing the following command (copied from
yum install epel-release yum-plugin-priorities && curl -o
/etc/yum.repos.d/powerdns-rec-40.repo https://repo.powerdns.com/repo
-files/centos-rec-40.repo && yum install pdns-recursor
The only thing I changed was local-address, allow-from, and threads (in
the /etc/pdns-recursor/recursor.conf file).
Could you give me some insights on why this is happening, and what I
should change in order to get all the threads to work?
Thanks a lot,
Grace
_______________________________________________
Pdns-users mailing list
https://mailman.powerdns.com/mailman/listinfo/pdns-users

[Øystein Viggen]

I installed pdns-recursor 4.0.3 on a CentOS 6.5 box, that has 20 physical
cores and 128G RAM. When I set the threads in /etc/pdns-recursor/
recursor.conf to be 4 or 12, then issued tons of queries (one query sent
multiple times using multiple dnsperf instances simultaneously) at it, I
noticed only 2 of the threads were working actively, each using almost 100%
CPU and ~90% CPU, respectively. The other threads were using 0% CPU.

Could it be that pdns-recursor isn't optimized very well for serving the
same same query at a very high rate?

With about 2000 queries/second of real-world DNS traffic from servers
and users, I'm seeing a good spread of the load over 24 threads on
pdns-recursor 4.0.3. (Ubuntu 16.04, if that matters).

Øystein

[abang]

Hi Grace,

Since 3.7.0, Recursor distributes queries to the threads itself. See
https://doc.powerdns.com/md/recursor/settings/#pdns-distributes-queries

Additional it hashes the queries with the goal to send the same query to
the same thread. This improves the cache hit rate and thus the average
latency and the overall performance.

I guess your query generator sends only 2 different queries (or more
queries which lead to only 2 different hash results).

Winfried

Post by Øystein Viggen

I installed pdns-recursor 4.0.3 on a CentOS 6.5 box, that has 20 physical
cores and 128G RAM. When I set the threads in /etc/pdns-recursor/
recursor.conf to be 4 or 12, then issued tons of queries (one query sent
multiple times using multiple dnsperf instances simultaneously) at it, I
noticed only 2 of the threads were working actively, each using almost 100%
CPU and ~90% CPU, respectively. The other threads were using 0% CPU.

Could it be that pdns-recursor isn't optimized very well for serving the
same same query at a very high rate?
With about 2000 queries/second of real-world DNS traffic from servers
and users, I'm seeing a good spread of the load over 24 threads on
pdns-recursor 4.0.3. (Ubuntu 16.04, if that matters).
Øystein
_______________________________________________
Pdns-users mailing list
https://mailman.powerdns.com/mailman/listinfo/pdns-users

[Grace Wang]

Thanks to Winfried and others for the response!

It was indeed the pdns-distributes-queries config that did the trick. My
query file was only 1 query, so all the queries were sent to one thread. I
tried it with a query file with 500 queries, the load was distributed to
all 12 threads evenly.

The latency of version 4.0.3 is much smaller than that of version 3.6.0.

Thanks again,
Grace

Post by abang
Hi Grace,
Since 3.7.0, Recursor distributes queries to the threads itself. See
https://doc.powerdns.com/md/recursor/settings/#pdns-distributes-queries
Additional it hashes the queries with the goal to send the same query to
the same thread. This improves the cache hit rate and thus the average
latency and the overall performance.
I guess your query generator sends only 2 different queries (or more
queries which lead to only 2 different hash results).
Winfried

I installed pdns-recursor 4.0.3 on a CentOS 6.5 box, that has 20 physical

Post by Øystein Viggen

cores and 128G RAM. When I set the threads in /etc/pdns-recursor/
recursor.conf to be 4 or 12, then issued tons of queries (one query sent
multiple times using multiple dnsperf instances simultaneously) at it, I
noticed only 2 of the threads were working actively, each using almost 100%
CPU and ~90% CPU, respectively. The other threads were using 0% CPU.

Could it be that pdns-recursor isn't optimized very well for serving the

same same query at a very high rate?
With about 2000 queries/second of real-world DNS traffic from servers
and users, I'm seeing a good spread of the load over 24 threads on
pdns-recursor 4.0.3. (Ubuntu 16.04, if that matters).
Øystein
_______________________________________________
Pdns-users mailing list
https://mailman.powerdns.com/mailman/listinfo/pdns-users

_______________________________________________
Pdns-users mailing list
https://mailman.powerdns.com/mailman/listinfo/pdns-users