4.8 Memory management

According to the documentation for curl_easy_setopt, the type of the third argument when option is CURLOPT_ERRORBUFFER is char*. Above, we've defined set-curl-option-errorbuffer to accept a :pointer as the new option value. However, there is a CFFI type :string, which translates Lisp strings to C strings when passed as arguments to foreign function calls. Why not, then, use :string as the CFFI type of the third argument? There are two reasons, both related to the necessity of breaking abstraction described in Breaking the abstraction.

The first reason also applies to CURLOPT_URL, which we will use to illustrate the point. Assuming we have changed the type of the third argument underlying set-curl-option-url to :string, look at these two equivalent forms.

  (set-curl-option-url *easy-handle* "http://www.cliki.net/CFFI")
   
  == (with-foreign-string (url "http://www.cliki.net/CFFI")
       (foreign-funcall "curl_easy_setopt" easy-handle *easy-handle*
                        curl-option :url :pointer url curl-code))

The latter, in fact, is mostly equivalent to what a foreign function call's macroexpansion actually does. As you can see, the Lisp string "http://www.cliki.net/CFFI" is copied into a char array and null-terminated; the pointer to beginning of this array, now a C string, is passed as a CFFI :pointer to the foreign function.

Unfortunately, the C abstraction has failed us, and we must break it. While :string works well for many char* arguments, it does not for cases like this. As the curl_easy_setopt documentation explains, “The string must remain present until curl no longer needs it, as it doesn't copy the string.” The C string created by with-foreign-string, however, only has dynamic extent: it is “deallocated” when the body (above containing the foreign-funcall form) exits.

If we are supposed to keep the C string around, but it goes away, what happens when some libcurl function tries to access the URL string? We have reentered the dreaded world of C “undefined behavior”. In some Lisps, it will probably get a chunk of the Lisp/C stack. You may segfault. You may get some random piece of other data from the heap. Maybe, in a world where “dynamic extent” is defined to be “infinite extent”, everything will turn out fine. Regardless, results are likely to be almost universally unpleasant.¹

Returning to the current set-curl-option-url interface, here is what we must do:

  (let (easy-handle)
    (unwind-protect
      (with-foreign-string (url "http://www.cliki.net/CFFI")
        (setf easy-handle (curl-easy-init))
        (set-curl-option-url easy-handle url)
        #|do more with the easy-handle, like actually get the URL|#)
      (when easy-handle
        (curl-easy-cleanup easy-handle))))

That is fine for the single string defined here, but for every string option we want to pass, we have to surround the body of with-foreign-string with another with-foreign-string wrapper, or else do some extremely error-prone pointer manipulation and size calculation in advance. We could alleviate some of the pain with a recursively expanding macro, but this would not remove the need to modify the block every time we want to add an option, anathema as it is to a modular interface.

Before modifying the code to account for this case, consider the other reason we can't simply use :string as the foreign type. In C, a char * is a char *, not necessarily a string. The option CURLOPT_ERRORBUFFER accepts a char *, but does not expect anything about the data there. However, it does expect that some libcurl function we call later can write a C string of up to 255 characters there. We, the callers of the function, are expected to read the C string at a later time, exactly the opposite of what :string implies.

With the semantics for an input string in mind — namely, that the string should be kept around until we curl_easy_cleanup the easy handle — we are ready to extend the Lisp interface:

  (defvar *easy-handle-cstrings* (make-hash-table)
    "Hashtable of easy handles to lists of C strings that may be
  safely freed after the handle is freed.")
   
  (defun make-easy-handle ()
    "Answer a new CURL easy interface handle, to which the lifetime
  of C strings may be tied.  See `add-curl-handle-cstring'."
    (let ((easy-handle (curl-easy-init)))
      (setf (gethash easy-handle *easy-handle-cstrings*) '())
      easy-handle))
   
  (defun free-easy-handle (handle)
    "Free CURL easy interface HANDLE and any C strings created to
  be its options."
    (curl-easy-cleanup handle)
    (mapc #'foreign-string-free
          (gethash handle *easy-handle-cstrings*))
    (remhash handle *easy-handle-cstrings*))
   
  (defun add-curl-handle-cstring (handle cstring)
    "Add CSTRING to be freed when HANDLE is, answering CSTRING."
    (car (push cstring (gethash handle *easy-handle-cstrings*))))

Here we have redefined the interface to create and free handles, to associate a list of allocated C strings with each handle while it exists. The strategy of using different function names to wrap around simple foreign functions is more common than the solution implemented earlier with curry-curl-option-setter, which was to modify the function name's function slot.²

Incidentally, the next step is to redefine curry-curl-option-setter to allocate C strings for the appropriate length of time, given a Lisp string as the new-value argument:

  (defun curry-curl-option-setter (function-name option-keyword)
    "Wrap the function named by FUNCTION-NAME with a version that
  curries the second argument as OPTION-KEYWORD.
   
  This function is intended for use in DEFINE-CURL-OPTION-SETTER."
    (setf (symbol-function function-name)
            (let ((c-function (symbol-function function-name)))
              (lambda (easy-handle new-value)
                (funcall c-function easy-handle option-keyword
                         (if (stringp new-value)
                           (add-curl-handle-cstring
                            easy-handle
                            (foreign-string-alloc new-value))
                           new-value))))))

A quick analysis of the code shows that you need only reevaluate the curl-option enumeration definition to take advantage of these new semantics. Now, for good measure, let's reallocate the handle with the new functions we just defined, and set its URL:

  cffi-user> (curl-easy-cleanup *easy-handle*)
  => NIL
  cffi-user> (setf *easy-handle* (make-easy-handle))
  => #<FOREIGN-ADDRESS #x09844EE0>
  cffi-user> (set-curl-option-nosignal *easy-handle* 1)
  => 0
  cffi-user> (set-curl-option-url *easy-handle*
                                  "http://www.cliki.net/CFFI")
  => 0

For fun, let's inspect the Lisp value of the C string that was created to hold "http://www.cliki.net/CFFI". By virtue of the implementation of add-curl-handle-cstring, it should be accessible through the hash table defined:

  cffi-user> (foreign-string-to-lisp
              (car (gethash *easy-handle* *easy-handle-cstrings*)))
  => "http://www.cliki.net/CFFI"

Looks like that worked, and libcurl now knows what URL we want to retrieve.

Finally, we turn back to the :errorbuffer option mentioned at the beginning of this section. Whereas the abstraction added to support string inputs works fine for cases like CURLOPT_URL, it hides the detail of keeping the C string; for :errorbuffer, however, we need that C string.

In a moment, we'll define something slightly cleaner, but for now, remember that you can always hack around anything. We're modifying handle creation, so make sure you free the old handle before redefining free-easy-handle.

  (defvar *easy-handle-errorbuffers* (make-hash-table)
    "Hashtable of easy handles to C strings serving as error
  writeback buffers.")
   
  ;;; An extra byte is very little to pay for peace of mind.
  (defparameter *curl-error-size* 257
    "Minimum char[] size used by cURL to report errors.")
   
  (defun make-easy-handle ()
    "Answer a new CURL easy interface handle, to which the lifetime
  of C strings may be tied.  See `add-curl-handle-cstring'."
    (let ((easy-handle (curl-easy-init)))
      (setf (gethash easy-handle *easy-handle-cstrings*) '())
      (setf (gethash easy-handle *easy-handle-errorbuffers*)
              (foreign-alloc :char :count *curl-error-size*
                             :initial-element 0))
      easy-handle))
   
  (defun free-easy-handle (handle)
    "Free CURL easy interface HANDLE and any C strings created to
  be its options."
    (curl-easy-cleanup handle)
    (foreign-free (gethash handle *easy-handle-errorbuffers*))
    (remhash handle *easy-handle-errorbuffers*)
    (mapc #'foreign-string-free
          (gethash handle *easy-handle-cstrings*))
    (remhash handle *easy-handle-cstrings*))
   
  (defun get-easy-handle-error (handle)
    "Answer a string containing HANDLE's current error message."
    (foreign-string-to-lisp
     (gethash handle *easy-handle-errorbuffers*)))

Be sure to once again set the options we've set thus far. You may wish to define yet another wrapper function to do this.