Access Logging

Nova Nodes can stream HTTP or TCP "access logs" to a syslog server on your network. This may often be a graylog cluster, etc.


TCP vs HTTP access logs

Nova ADCs log in two ways, TCP and HTTP. All ADCs will automatically use TCP logging when access logging is enabled, except for SSL Termination and HTTP ADCs. These will use HTTP logging.

HTTP logging has additional information, and can have additional custom log format flags included. Naturally, it it's only available on HTTP/S traffic.


Performance Considerations

Logging can often be seen as the enemy of high performance systems. Nova Nodes can generate hundreds of thousands of log lines per second (one line per request) which can slow the performance of the Node slightly, and also can be very difficult for you (as the client) to consume.

For this reason we also support log sampling - sending only a customizable percentage of the logs. For example a sample value of 5 will send 5 in 100 (or 1 in 20) log lines to your server, selected randomly. This is good for performance analytics, but bad for security and debugging of course.


Log Formats

By default Nova uses two formats, once for TCP and one for HTTP. You can also customize these to include whatever information is needed.

Default TCP Format

>>> Feb  6 12:12:56 localhost \
      nova[14387]: 10.0.1.2:33313 [06/Feb/2009:12:12:51.443] fnt \
      bck/srv1 0/0/5007 212 -- 0/0/0/0/3 0/0

  Field   Format                                Extract from the example above
      1   process_name '[' pid ']:'                               nova[14387]:
      2   client_ip ':' client_port                             10.0.1.2:33313
      3   '[' accept_date ']'                       [06/Feb/2019:12:12:51.443]
      4   frontend_name                                                    fnt
      5   backend_name '/' server_name                                bck/srv1
      6   Tw '/' Tc '/' Tt*                                           0/0/5007
      7   bytes_read*                                                      212
      8   termination_state                                                 --
      9   actconn '/' feconn '/' beconn '/' srv_conn '/' retries*    0/0/0/0/3
     10   srv_queue '/' backend_queue                                      0/0

Default HTTP Format

>>> Feb  6 12:14:14 localhost \
      nova[14389]: 10.0.1.2:33317 [06/Feb/2009:12:14:14.655] http-in \
      static/srv1 10/0/30/69/109 200 2750 - - ---- 1/1/1/1/0 0/0 {nova-adc.com} \
      {} "GET /index.html HTTP/1.1"

  Field   Format                                Extract from the example above
      1   process_name '[' pid ']:'                               nova[14389]:
      2   client_ip ':' client_port                             10.0.1.2:33317
      3   '[' request_date ']'                      [06/Feb/2009:12:14:14.655]
      4   frontend_name                                                http-in
      5   backend_name '/' server_name                             static/srv1
      6   TR '/' Tw '/' Tc '/' Tr '/' Ta*                       10/0/30/69/109
      7   status_code                                                      200
      8   bytes_read*                                                     2750
      9   captured_request_cookie                                            -
     10   captured_response_cookie                                           -
     11   termination_state                                               ----
     12   actconn '/' feconn '/' beconn '/' srv_conn '/' retries*    1/1/1/1/0
     13   srv_queue '/' backend_queue                                      0/0
     14   '{' captured_request_headers* '}'                     {nova-adc.com}
     15   '{' captured_response_headers* '}'                                {}
     16   '"' http_request '"'                      "GET /index.html HTTP/1.1"

Custom Formats

Nova ADCs support customizing what is logged (vs the defaults above). We do this using variables that you can use in your log format string.

% precedes log format variables. Variables can take arguments using braces ('{}'), and multiple arguments are separated by commas within the braces. Flags may be added or removed by prefixing them with a '+' or '-' sign.

Special variable "%o" may be used to propagate its flags to all other variables on the same format string. This is particularly handy with quoted ("Q") and escaped ("E") string formats.

Examples

Nova Default TCP Format
%ci:%cp [%t] %ft %b/%s %Tw/%Tc/%Tt %B %ts %ac/%fc/%bc/%sc/%rc %sq/%bq

Nova Default HTTP Format
%ci:%cp [%tr] %ft %b/%s %TR/%Tw/%Tc/%Tr/%Ta %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r 

Variables

Below are the available variables.

  +---+------+-----------------------------------------------+-------------+
  | R | var  | field name (8.2.2 and 8.2.3 for description)  | type        |
  +---+------+-----------------------------------------------+-------------+
  |   | %o   | special variable, apply flags on all next var |             |
  +---+------+-----------------------------------------------+-------------+
  |   | %B   | bytes_read           (from server to client)  | numeric     |
  | H | %CC  | captured_request_cookie                       | string      |
  | H | %CS  | captured_response_cookie                      | string      |
  |   | %H   | hostname                                      | string      |
  | H | %HM  | HTTP method (ex: POST)                        | string      |
  | H | %HP  | HTTP request URI without query string (path)  | string      |
  | H | %HQ  | HTTP request URI query string (ex: ?bar=baz)  | string      |
  | H | %HU  | HTTP request URI (ex: /foo?bar=baz)           | string      |
  | H | %HV  | HTTP version (ex: HTTP/1.0)                   | string      |
  |   | %ID  | unique-id                                     | string      |
  |   | %ST  | status_code                                   | numeric     |
  |   | %T   | gmt_date_time                                 | date        |
  |   | %Ta  | Active time of the request (from TR to end)   | numeric     |
  |   | %Tc  | Tc                                            | numeric     |
  |   | %Td  | Td = Tt - (Tq + Tw + Tc + Tr)                 | numeric     |
  |   | %Tl  | local_date_time                               | date        |
  |   | %Th  | connection handshake time (SSL, PROXY proto)  | numeric     |
  | H | %Ti  | idle time before the HTTP request             | numeric     |
  | H | %Tq  | Th + Ti + TR                                  | numeric     |
  | H | %TR  | time to receive the full request from 1st byte| numeric     |
  | H | %Tr  | Tr (response time)                            | numeric     |
  |   | %Ts  | timestamp                                     | numeric     |
  |   | %Tt  | Tt                                            | numeric     |
  |   | %Tw  | Tw                                            | numeric     |
  |   | %U   | bytes_uploaded       (from client to server)  | numeric     |
  |   | %ac  | actconn                                       | numeric     |
  |   | %b   | backend_name                                  | string      |
  |   | %bc  | beconn      (backend concurrent connections)  | numeric     |
  |   | %bi  | backend_source_ip       (connecting address)  | IP          |
  |   | %bp  | backend_source_port     (connecting address)  | numeric     |
  |   | %bq  | backend_queue                                 | numeric     |
  |   | %ci  | client_ip                 (accepted address)  | IP          |
  |   | %cp  | client_port               (accepted address)  | numeric     |
  |   | %f   | frontend_name                                 | string      |
  |   | %fc  | feconn     (frontend concurrent connections)  | numeric     |
  |   | %fi  | frontend_ip              (accepting address)  | IP          |
  |   | %fp  | frontend_port            (accepting address)  | numeric     |
  |   | %ft  | frontend_name_transport ('~' suffix for SSL)  | string      |
  |   | %lc  | frontend_log_counter                          | numeric     |
  |   | %hr  | captured_request_headers default style        | string      |
  |   | %hrl | captured_request_headers CLF style            | string list |
  |   | %hs  | captured_response_headers default style       | string      |
  |   | %hsl | captured_response_headers CLF style           | string list |
  |   | %ms  | accept date milliseconds (left-padded with 0) | numeric     |
  |   | %pid | PID                                           | numeric     |
  | H | %r   | http_request                                  | string      |
  |   | %rc  | retries                                       | numeric     |
  |   | %rt  | request_counter (HTTP req or TCP session)     | numeric     |
  |   | %s   | server_name                                   | string      |
  |   | %sc  | srv_conn     (server concurrent connections)  | numeric     |
  |   | %si  | server_IP                   (target address)  | IP          |
  |   | %sp  | server_port                 (target address)  | numeric     |
  |   | %sq  | srv_queue                                     | numeric     |
  | S | %sslc| ssl_ciphers (ex: AES-SHA)                     | string      |
  | S | %sslv| ssl_version (ex: TLSv1)                       | string      |
  |   | %t   | date_time      (with millisecond resolution)  | date        |
  | H | %tr  | date_time of HTTP request                     | date        |
  | H | %trg | gmt_date_time of start of HTTP request        | date        |
  | H | %trl | local_date_time of start of HTTP request      | date        |
  |   | %ts  | termination_state                             | string      |
  | H | %tsc | termination_state with cookie status          | string      |
  +---+------+-----------------------------------------------+-------------+

    R = Restrictions : H = mode http only ; S = SSL only