14.5.2.8. memcached Logs

If you enable verbose mode, using the -v, -vv, or -vvv options, then the information output by memcached includes details of the operations being performed.

Without the verbose options, memcached normally produces no output during normal operating.

  • Output when using -v

    The lowest verbosity level shows you:

    • Errors and warnings

    • Transient errors

    • Protocol and socket errors, including exhausting available connections

    • Each registered client connection, including the socket descriptor number and the protocol used.

      For example:

      32: Client using the ascii protocol
      33: Client using the ascii protocol
      

      Note that the socket descriptor is only valid while the client remains connected. Non-persitant connections may not be effectively represented.

    Examples of the error messages output at this level include:

    <%d send buffer was %d, now %d
    Can't listen for events on fd %d
    Can't read from libevent pipe
    Catastrophic: event fd doesn't match conn fd!
    Couldn't build response
    Couldn't realloc input buffer
    Couldn't update event
    Failed to build UDP headers
    Failed to read, and not due to blocking
    Too many open connections
    Unexpected state %d
    
  • Output when using -vv

    When using the second level of verbosity, you get more detailed information about protocol operations, keys updated, chunk and network operatings and details.

    During the initial start-up of memcached with this level of verbosity, you are shown the sizes of the individual slab classes, the chunk sizes, and the number of entries per slab. These do not show the allocation of the slabs, just the slabs that would be created when data is added. You are also given information about the listen queues and buffers used to send information. A sample of the output generated for a TCP/IP based system with the default memory and growth factors is given below:

    shell> memcached -vv
    slab class   1: chunk size     80 perslab 13107
    slab class   2: chunk size    104 perslab 10082
    slab class   3: chunk size    136 perslab  7710
    slab class   4: chunk size    176 perslab  5957
    slab class   5: chunk size    224 perslab  4681
    slab class   6: chunk size    280 perslab  3744
    slab class   7: chunk size    352 perslab  2978
    slab class   8: chunk size    440 perslab  2383
    slab class   9: chunk size    552 perslab  1899
    slab class  10: chunk size    696 perslab  1506
    slab class  11: chunk size    872 perslab  1202
    slab class  12: chunk size   1096 perslab   956
    slab class  13: chunk size   1376 perslab   762
    slab class  14: chunk size   1720 perslab   609
    slab class  15: chunk size   2152 perslab   487
    slab class  16: chunk size   2696 perslab   388
    slab class  17: chunk size   3376 perslab   310
    slab class  18: chunk size   4224 perslab   248
    slab class  19: chunk size   5280 perslab   198
    slab class  20: chunk size   6600 perslab   158
    slab class  21: chunk size   8256 perslab   127
    slab class  22: chunk size  10320 perslab   101
    slab class  23: chunk size  12904 perslab    81
    slab class  24: chunk size  16136 perslab    64
    slab class  25: chunk size  20176 perslab    51
    slab class  26: chunk size  25224 perslab    41
    slab class  27: chunk size  31536 perslab    33
    slab class  28: chunk size  39424 perslab    26
    slab class  29: chunk size  49280 perslab    21
    slab class  30: chunk size  61600 perslab    17
    slab class  31: chunk size  77000 perslab    13
    slab class  32: chunk size  96256 perslab    10
    slab class  33: chunk size 120320 perslab     8
    slab class  34: chunk size 150400 perslab     6
    slab class  35: chunk size 188000 perslab     5
    slab class  36: chunk size 235000 perslab     4
    slab class  37: chunk size 293752 perslab     3
    slab class  38: chunk size 367192 perslab     2
    slab class  39: chunk size 458992 perslab     2
    <26 server listening (auto-negotiate)
    <29 server listening (auto-negotiate)
    <30 send buffer was 57344, now 2097152
    <31 send buffer was 57344, now 2097152
    <30 server listening (udp)
    <30 server listening (udp)
    <31 server listening (udp)
    <30 server listening (udp)
    <30 server listening (udp)
    <31 server listening (udp)
    <31 server listening (udp)
    <31 server listening (udp)
    

    Using this verbosity level can be a useful way to check the effects of the growth factor used on slabs with different memory allocations, which in turn can be used to better tune the growth factor to suit the data you are storing in the cache. For example, if you set the growth factor to 4 (quadrupling the size of each slab):

    shell> memcached -f 4 -m 1g -vv
    slab class   1: chunk size     80 perslab 13107
    slab class   2: chunk size    320 perslab  3276
    slab class   3: chunk size   1280 perslab   819
    slab class   4: chunk size   5120 perslab   204
    slab class   5: chunk size  20480 perslab    51
    slab class   6: chunk size  81920 perslab    12
    slab class   7: chunk size 327680 perslab     3
    ...
    

    During use of the cache, this verbosity level also prints out detailed information on the storage and recovery of keys and other information. An example of the output during a typical set/get and increment/decrement operation is shown below.

    32: Client using the ascii protocol
    <32 set my_key 0 0 10
    >32 STORED
    <32 set object_key 1 0 36
    >32 STORED
    <32 get my_key 
    >32 sending key my_key
    >32 END
    <32 get object_key 
    >32 sending key object_key
    >32 END
    <32 set key 0 0 6
    >32 STORED
    <32 incr key 1
    >32 789544
    <32 decr key 1
    >32 789543
    <32 incr key 2
    >32 789545
    <32 set my_key 0 0 10
    >32 STORED
    <32 set object_key 1 0 36
    >32 STORED
    <32 get my_key 
    >32 sending key my_key
    >32 END
    <32 get object_key 
    >32 sending key object_key1 1 36
    
    >32 END
    <32 set key 0 0 6
    >32 STORED
    <32 incr key 1
    >32 789544
    <32 decr key 1
    >32 789543
    <32 incr key 2
    >32 789545
    

    During client communication, for each line, the initial character shows the direction of flow of the information. The < for communication from the client to the memcached server and > for communication back to the client. The number is the numeric socket descriptor for the connection.

  • Output when using -vvv

    This level of verbosity includes the transitions of connections between different states in the event library while reading and writing content to/from the clients. It should be used to diagnose and identify issues in client communication. For example, you can use this information to determine if memcached is taking a long time to return information to the client, during the read of the client operation or before returning and completing the operation. An example of the typical sequence for a set operation is provided below:

    <32 new auto-negotiating client connection
    32: going from conn_new_cmd to conn_waiting
    32: going from conn_waiting to conn_read
    32: going from conn_read to conn_parse_cmd
    32: Client using the ascii protocol
    <32 set my_key 0 0 10
    32: going from conn_parse_cmd to conn_nread
    > NOT FOUND my_key
    >32 STORED
    32: going from conn_nread to conn_write
    32: going from conn_write to conn_new_cmd
    32: going from conn_new_cmd to conn_waiting
    32: going from conn_waiting to conn_read
    32: going from conn_read to conn_closing
    <32 connection closed.
    

All of the verbosity levels in memcached are designed to be used during debugging or examination of issues. The quantity of information generated, particularly when using -vvv, is significant, particularly on a busy server. Also be aware that writing the error information out, especially to disk, may negate some of the performance gains you achieve by using memcached. Therefore, use in production or deployment environments is not recommended.

Copyright © 2010-2024 Platon Technologies, s.r.o.           Home | Man pages | tLDP | Documents | Utilities | About
Design by styleshout