When you call pthread_mutex_trylock() you'll attempt to lock the mutex. If the mutex is currently unlocked you'll get the lock, and this function will return zero. However, if the mutex is locked this call won't block. Rather, it will return a non-zero EBUSY error value.
Threads in C
pthreads library and threaded programming in C