memset( pxNewTCB->pxStack, tskSTACK_FILL_BYTE, usStackDepth * sizeof( portSTACK_TYPE ) ); I am using a simulator, is it possible that the memory in this simulator is not big enough so program goes into deadlock at this line?

A single line like this cannot deadlock. I think your definition of deadlock is not correct, unless you have seen something in the implementation of memset() (in its source code) that could. People using simulators waste a lot of time on this forum. In addition you do not state which processor you are using, which compiler you are using, which simulator you are using, or what the problem you are experiencing is. Don’t expect to get a solution.

sorry.. I’m using simulator WSim, msp430-gcc, freeRTOS on wsn430 platform.
I want to setup a peer communicating via MAC address on simulators WSim and WSNet. In fact there is a example given by http://senstools.gforge.inria.fr/doku.php?id=os:freertos:examples, 05_network_device and 06_coordinator, I tested it and failed.
Now, I don’t know where the problems come from, simulator, driver or system…

First, “Deadlock” is a fairly specific term dealing with synchronization and is not applicable here. If by “Deadlock” you mean the program appears stuck here, note that memset can be a slow operation for a simulator, as it has in it a loop with a count equal to the number of entries to set.