возвращаемое значение != код ошибки. В случае ошибки возвращается всегда -1, а список возможных кодов ошибок есть в мане:
RETURN VALUE
       Usually, on success zero is returned.  A few ioctl() requests  use  the
       return  value  as an output parameter and return a nonnegative value on
       success.  On error, -1 is returned, and errno is set appropriately.

ERRORS
       EBADF  d is not a valid descriptor.

       EFAULT argp references an inaccessible memory area.

       EINVAL Request or argp is not valid.

       ENOTTY d is not associated with a character special device.

       ENOTTY The specified request does not apply to the kind of object  that
              the descriptor d references.