Rancher K3S “nameserver limits exceeded”

You’ve created a Rancher K3S cluster and for some reason your ingress URL are not working with the flood of errors “Nameserver limits exceeded”.

Below errors are logged in syslog 

pi-wrkr01 k3s[354088]: I0612 12:04:55.339233  354088 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="a19e2669d074f2faa869ec29f9ced0656c3dbd80cb65f0ae6ed4dafb2f60f9fb"
Jun 12 11:43:23 k3s-pi-wrkr01 k3s[34016]: E0612 11:43:23.391005   34016 dns.go:136] "Nameserver limits exceeded" err="Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 192.168.15.1 4.2.2.2 8.8.8.8"
Jun 12 11:43:55 k3s-pi-wrkr01 k3s[34016]: E0612 11:43:55.391080   34016 dns.go:136] "Nameserver limits exceeded" err="Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 192.168.15.1 4.2.2.2 8.8.8.8"
Jun 12 11:44:47 k3s-pi-wrkr01 k3s[34016]: E0612 11:44:47.390679   34016 dns.go:136] "Nameserver limits exceeded" err="Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 192.168.15.1 4.2.2.2 8.8.8.8"
Jun 12 11:44:58 k3s-pi-wrkr01 k3s[34016]: E0612 11:44:58.391809   34016 dns.go:136] "Nameserver limits exceeded" err="Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 192.168.15.1 4.2.2.2 8.8.8.8"
Jun 12 11:45:19 k3s-pi-wrkr01 k3s[34016]: W0612 11:45:19.016239   34016 sysinfo.go:203] Nodes topology is not available, providing CPU topology
Jun 12 11:45:53 k3s-pi-wrkr01 k3s[34016]: E0612 11:45:53.390227   34016 dns.go:136] "Nameserver limits exceeded" err="Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 192.168.15.1 4.2.2.2 8.8.8.8"
Jun 12 11:46:02 k3s-pi-wrkr01 k3s[34016]: E0612 11:46:02.391550   34016 dns.go:136] "Nameserver limits exceeded" err="Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 192.168.15.1 4.2.2.2 8.8.8.8"
Jun 12 11:47:03 k3s-pi-wrkr01 k3s[34016]: E0612 11:47:03.391764   34016 dns.go:136] "Nameserver limits exceeded" err="Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 192.168.15.1 4.2.2.2 8.8.8.8"
Jun 12 11:47:22 k3s-pi-wrkr01 k3s[34016]: E0612 11:47:22.391179   34016 dns.go:136] "Nameserver limits exceeded" err="Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 192.168.15.1 4.2.2.2 8.8.8.8"

Fix:

Do not have DNS nameservers more than 3 lines. This is due to the Kubernetes CoreDNS restrictions

A sample working DNS configuration file showing entries for 3 nameservers

# This file is managed by man:systemd-resolved(8). Do not edit.
#
# This is a dynamic resolv.conf file for connecting local clients directly to
# all known uplink DNS servers. This file lists all configured search domains.
#
# Third party programs should typically not access this file directly, but only
# through the symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a
# different way, replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.

nameserver 192.168.15.1
nameserver 4.2.2.2
nameserver 8.8.8.8
search .

Please refer the code for more information

https://github.com/kubernetes/kubernetes/blob/c970a46bc1bcc100bbbfabd5c12bd4c5d87f8aea/pkg/apis/core/validation/validation.go#L2944-L2953

Solarwinds IPAM CRUD/Update using Orion/SDK

Note to myself for future use. Did this while troubleshooting Solarwinds IPAM IP reservation automation.

import orionsdk
import requests
from requests.adapters import HTTPAdapter
from requests.packages.urllib3.util.retry import Retry
from pprint import pprint

def retry_session(retries=3,
                  backoff_factor=0.3,
                  status_forcelist=(500, 502, 504)):
    session = requests.Session()
    retry = Retry(
        total=retries,
        read=retries,
        connect=retries,
        backoff_factor=backoff_factor,
        status_forcelist=status_forcelist)
    adapter = HTTPAdapter(max_retries=retry)
    session.mount('http://', adapter)
    session.mount('https://', adapter)
    return session
class VPNSolarWinds():
    def __init__(self,**kwargs):
        try:
            self.swis = orionsdk.SwisClient(kwargs["host"],
                                        kwargs["user"],
                                        kwargs["password"],
                                        session=retry_session())   #verify="server.pem",
        except Exception as e:
            print("connectionError: {}".format(e))
            
obj = VPNSolarWinds(user="admin",password="password@123",host="10.10.10.10") #sample lab solarwinds ipam tool


query = """ SELECT 
                ipn.subnetid,
                ipn.IPAddress, 
                ipn.Status, 
                ipn.Alias, 
                ipn.MAC, 
                ipn.DnsBackward, 
                ipn.DhcpClientName, 
                ipn.SysName, 
                ipn.Description, 
                ipn.Contact, 
                ipn.Location, 
                ipn.SysObjectID, 
                ipn.Vendor, 
                ipn.VendorIcon, 
                ipn.MachineType, 
                ipn.Comments, 
                ipn.ResponseTime, 
                ipn.LastBoot, 
                ipn.LastSync, 
                ipn.LastCredential, 
                ipn.AllocPolicy, 
                ipn.SkipScan, 
                ipn.LeaseExpires, 
                ipn.DnsBy, 
                ipn.MacBy, 
                ipn.StatusBy, 
                ipn.SystemDataBy,
                ipn.Uri 
            FROM IPAM.IPNode ipn JOIN IPAM.Subnet sbn 
            ON ipn.subnetid=sbn.subnetid 
            WHERE sbn.DisplayName='{subnet_name}' AND ipn.IPAddress='{ip}'""".format(subnet_name="some-subnet",ip="10.20.20.20")
# Added "Uri" to the above column names            


output = obj.swis.query(query)
uri = (output['results'][0]['Uri'])
# obj.swis.update(uri, Status='Reserved')
obj.swis.update(uri, Status='Used')
output_after = obj.swis.query(query)
status = (output['results'][0]['Status'])
print("Status:", status)
    
# pprint(results)

Reference: https://github.com/solarwinds/OrionSDK/wiki/IPAM-4.5.x-API#crud-operations-for-ip-address

#IPAM IP RESERVATION STATUS.
#Value    Name 
#0        Unknown 
#1        Used 
#2        Available 
#4        Reserved
#8        Transient
#16       Blocked